Kubernetes v1.2.2 api-server dosen't start - docker

Question

I attempt to deploy Pachyderm (a docker bigdata platform) on kubernetes. Limited by Pachyderm, I have to install kubernetes v1.2.2, an old version. I follow the guide here http://kubernetes.io/docs/getting-started-guides/docker/ to deploy Kubernetes on local server via docker. The guide can work with the kubernetes >=1.3.0, but when I use it to deploy kubernetes 1.2.2, I met some problems.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec38ae951f09 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube apiserver" 8 seconds ago Exited (255) 7 seconds ago k8s_apiserver.78ec1de_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_d26fc24e
55c1b13bb610 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" 8 seconds ago Up 8 seconds k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_1cb4c220
b9f0e5b3a7a9 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube scheduler" 9 seconds ago Up 8 seconds k8s_scheduler.fc12fcbe_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_e5065506
9cd613d272bc gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube apiserver" 9 seconds ago Exited (255) 8 seconds ago k8s_apiserver.78ec1de_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_c04426af
49fe2c409386 gcr.io/google_containers/etcd:2.2.1 "/usr/local/bin/etcd " 10 seconds ago Up 9 seconds k8s_etcd.7e452b0b_k8s-etcd-127.0.0.1_default_1df6a8b4d6e129d5ed8840e370203c11_a6f11fdb
5b208be18c71 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube controlle" 10 seconds ago Up 9 seconds k8s_controller-manager.70414b65_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_c377c5e9
df194f3cf663 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube proxy --m" 10 seconds ago Up 9 seconds k8s_kube-proxy.9a9f4853_k8s-proxy-127.0.0.1_default_5e5303a9d49035e9fad52bfc4c88edc8_63ec0b04
58b53ec28fbe gcr.io/google_containers/pause:2.0 "/pause" 10 seconds ago Up 9 seconds k8s_POD.6059dfa2_k8s-etcd-127.0.0.1_default_1df6a8b4d6e129d5ed8840e370203c11_21034b2e
df48fe4cdf0a gcr.io/google_containers/pause:2.0 "/pause" 10 seconds ago Up 9 seconds k8s_POD.6059dfa2_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_4867dbbc
fe6b74c2a881 gcr.io/google_containers/pause:2.0 "/pause" 10 seconds ago Up 9 seconds k8s_POD.6059dfa2_k8s-proxy-127.0.0.1_default_5e5303a9d49035e9fad52bfc4c88edc8_fad2c558
4c00ad498916 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube kubelet -" 25 seconds ago Up 24 seconds kubelet
From the docker container table, it can be observed that my apiserver is down when deploying kubernetes1.2.2. The restart interval of kubernetes apiserver obeys expontional backoff algorithm. But never work.
Then,
sv: batch/v1
mv: extensions/__internal
I0727 06:06:27.593708 1 genericapiserver.go:82] Adding storage destination for group batch
W0727 06:06:27.593745 1 server.go:383] No RSA key provided, service account token authentication disabled
F0727 06:06:27.593767 1 server.go:410] Invalid Authentication Config: open /srv/kubernetes/basic_auth.csv: no such file or directory
Please see docker logs of kubernetes apiserver here. Note that some authentication error occurred seems that the Kubernetes does not have required key to be permitted.Also see the controller manager log here. The controller manager wait for the apiserver, however the apiserver hasn't ran ever. The controller manager is also dump.
E0727 06:07:10.604801 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:11.604832 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:12.604752 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:13.604803 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:14.604332 1 nodecontroller.go:229] Error monitoring node status: Get http://127.0.0.1:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:14.604619 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
E0727 06:07:14.604861 1 controllermanager.go:259] Failed to get api versions from server: Get http://127.0.0.1:8080/api: dial tcp 127.0.0.1:8080: connection refused
F0727 06:07:14.604957 1 controllermanager.go:263] Failed to get api versions from server: timed out waiting for the condition
So for my question, how to solve this problem? The problem has troubled me for a long time.
====================================================================
Update:
With the help of Goblin and Lukie, I find the key problem is the Setup Pods is not triggered.
See the manifest of Kubernetes,
{
"name": "controller-manager",
"/hyperkube",
"controller-manager",
"--master=127.0.0.1:8080",
"--service-account-private-key-file=/srv/kubernetes/server.key",
"--root-ca-file=/srv/kubernetes/ca.crt",
"--min-resync-period=3m",
"--v=2"
],
"volumeMounts": [
{
"name": "data",
"mountPath": "/srv/kubernetes"
}
]
}
Option --service-account-private-key-file=/srv/kubernetes/server.key has been added in the manifest file, but it doesn't work. In other words, the controller-manager cannot find this file in the file system. This assumption is supported by following command.
docker exec a82d7f6e4d7d ls -l /srv/kubernetes
ls: cannot access /srv/kubernetes: No such file or directory
Next, we check whether the Setup Pod put the file in the docker volumn. Unfortunately, we find that the Setup Pod is not triggered and worked, therefore no cert file is written in the file system.
docker ps -a | grep setup
54afdd81349e gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" About a minute ago Up About a minute k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_a2edddca
6f714e034098 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" 4 minutes ago Exited (7) 2 minutes ago k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_0d7dab5b
8358f6644d94 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" 6 minutes ago Exited (7) 4 minutes ago k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_41e4c686
Is there any method to do further debug? Or is it a bug in Kubernetes version 1.2?

Answer 1

F0727 06:06:27.593767 1 server.go:410] Invalid Authentication Config: open /srv/kubernetes/basic_auth.csv: no such file or directory
You are missing the basic auth file /srv/kubernetes/basic_auth.csv either createa basic auth file or remove the configuration flag.
Kubernetes authentication

Answer 2

in fact it is W0727 06:06:27.593745 1 server.go:383] No RSA key provided, service account token authentication disabled that is more important in my opinion.
Seems like --service-account-private-key-file is missing on controller-manager so service tokens can not be properly generated.