I'm going through this tutorial
Setting up Jenkins on Container Engine
https://cloud.google.com/solutions/jenkins-on-container-engine-tutorial
and failing on "Creating the Jenkins deployment and services" step
I got this error at one point:
jenkins- 0/1 rpc error: code = 2 desc = failed to start container "": Error response from daemon: {"message":"linux spec user: unable to find user jenkins: no matching entries in passwd file"}
And I get these results for the following commands:
> kubectl apply -f jenkins/k8s/
deployment "jenkins" configured
service "jenkins-ui" configured
service "jenkins-discovery" configured
> get pods --namespace jenkins
NAME READY STATUS RESTARTS AGE
jenkins-<some id> 0/1 CrashLoopBackOff 5 10m
I get it that it is looking for jenkins user in the passwd file, but I'm still not sure why this error took place and what the correct way to fix it is. Any insight would be highly appreciated.
Edit: output of running "kubectl get pods --namespace jenkins"
The very first time running it:
> kubectl get pods --namespace jenkins
NAME READY STATUS RESTARTS AGE
jenkins-1937056428-fp7vr 0/1 ContainerCreating 0 16s
Second time running it:
> kubectl get pods --namespace jenkins
NAME READY STATUS RESTARTS AGE
jenkins-1937056428-fp7vr 0/1 rpc error: code = 2 desc = failed to start container "10a8ab7e3eb0ad153fd6055d86336b1cdfe9642b6993684a7e01fefbeca7a566": Error response from
daemon: {"message":"linux spec user: unable to find user jenkins: no matching entries in passwd file"} 1 39s
Third and after:
> kubectl get pods --namespace jenkins
NAME READY STATUS RESTARTS AGE
jenkins-1937056428-fp7vr 0/1 CrashLoopBackOff 270 22h
It appears that the persistent disk volume for the jenkins is not properly setup. Try running the following commands to reconfigure disk volumes and rerun jenkins pod,
kubectl delete -f jenkins/k8s/
gcloud compute disks delete jenkins-home
gcloud compute images delete jenkins-home-image
gcloud config set compute/zone us-east1-d
gcloud compute images create jenkins-home-image --source-uri https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v3.tar.gz
gcloud compute disks create jenkins-home --image jenkins-home-image --zone us-east1-d
kubectl apply -f jenkins/k8s/
I basically did one step wrong:
Provision a Kubernetes cluster using Container Engine.
gcloud container clusters create jenkins-cd \
--network jenkins \
--scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"
Here make sure the options --network and --scopes actually get passed in. I guess I copied the command without fixing it up and the options got dropped.
Related
I have an on-premise Kubernetes(v1.15.3) cluster(1 Master and 2 Worker nodes). I wanted to run Jenkins agents in the cluster using Kubernetes-plugin feature by keeping Jenkins(version. 2.176.2) master outside the cluster. Therefore, I created a new namespace (jenkins) and followed the configurations mentioned here. Then, I filled my Kubernetes credentials in the cloud field of Jenkins master configuration. The connection was established successfully. Now, when I tried to to run a Jenkins job as a pod in Kubernetes, the pod is not coming online. The logs from Kubernetes shows:
Kubernetes-log
kubectl get sa
NAME SECRETS AGE
default 1 23h
jenkins 1 23h
kubectl get secrets
NAME TYPE DATA AGE
default-token-4gcr4 kubernetes.io/service-account-token 3 23h
jenkins-token-7nwbd kubernetes.io/service-account-token 3 23h
The Console output from Jenkins job shows:
Jenkins-log
Has anyone encountered a similar error before?
I am following all the steps from this link : https://github.com/justmeandopensource/kubernetes
after running the join command in the worker node it's getting added to master, but the status of the worker node is getting changed to ready.
From the logs I got the following :
Container runtime network not ready: NetworkReady=false
reason:NetworkPluginNotReady message:dock
Unable to update cni config: No networks found in /etc/cni/net.d
kubelet.go:2266 -- node "XXXXXXXXX" not found. (xxxxx is the masters
host/node name)
To establish CNI I am using flannel and also tried with weave and many other
CNI networks but the results are the same
points to ponder:
---> worker node kubelet status is healthy
---> trying to run kubeadm init command in the worker node,its showing the status of kubelet might be unhealthy. (Not able to make worker node master by running the kubeadm init command but kubeadm join command is working.After joining kubectl get nodes is showing the worker node but status is notready)
Thank you for the help
I cannot reproduce your issue. I followed exactly the instructions on github`s site you shared, and did not face similar error.
The only extra steps I needed to do, to suppress errors, detected by pre-flight checks of kubeadm init:
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
was to set appropriate flag by running:
echo '1' > /proc/sys/net/ipv4/ip_forward
State of my cluster nodes:
NAME STATUS ROLES AGE VERSION
centos-master Ready master 18h v1.13.1
centos-worker Ready <none> 18h v1.13.1
I verified cluster condition by deploying&exposing sample application and everything seems to be working fine:
kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node
kubectl expose deployment hello-node --port=8080
I`m getting valid response from hello-world node.js app:
curl 10.100.113.255:8080
Hello World!#
What IP address you have put to your /etc/hosts files ?
I am running etcd, kube-apiserver, kube-scheduler, and kube-controllermanager on a master node as well as kubelet and kube-proxy on a minion node as follows (all kube binaries are from kubernetes 1.7.4):
# [master node]
./etcd
./kube-apiserver --logtostderr=true --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.10.10.0/24 --insecure-port 8080 --secure-port=0 --allow-privileged=true --insecure-bind-address 0.0.0.0
./kube-scheduler --address=0.0.0.0 --master=http://127.0.0.1:8080
./kube-controller-manager --address=0.0.0.0 --master=http://127.0.0.1:8080
# [minion node]
./kubelet --logtostderr=true --address=0.0.0.0 --api_servers=http://$MASTER_IP:8080 --allow-privileged=true
./kube-proxy --master=http://$MASTER_IP:8080
After this, if I execute kubectl get all --all-namespaces and kubectl get nodes, I get
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default svc/kubernetes 10.10.10.1 <none> 443/TCP 27m
NAME STATUS AGE VERSION
minion-1 Ready 27m v1.7.4+793658f2d7ca7
Then, I apply flannel as follows:
kubectl apply -f kube-flannel-rbac.yml -f kube-flannel.yml
Now, I see a pod is created, but with error:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-p8tcb 1/2 CrashLoopBackOff 4 2m
When I check the logs inside the failed container in the minion node, I see the following error:
Failed to create SubnetManager: unable to initialize inclusterconfig: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
My question is: how to resolve this? Is this a SSL issue? What step am I missing in setting up my cluster?
Maybe it is your flannel yaml file has something wrong,
you can try this to install your flannel,
check the old ip link
ip link
if it show flannel,please delete it
ip link delete flannel.1
and install , its default pod network cdir is 10.244.0.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.0/Documentation/kube-flannel.yml
You could try to pass --etcd-prefix=/your/prefix and --etcd-endpoints=address to flanneld instead of --kube-subnet-mgr so flannel get net-conf from etcd server and not from api server.
Keep in mind that you must to push net-conf to etcd server.
UPDATE
The problem (/var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory) can appear when execute apiserver without --admission-control=...,ServiceAccount,... or if kubelet is inside a container (eg: hypercube) and this last was my case. If you want execute k8s components inside a container you need to pass 'shared' option to kubelet volume
/var/lib/kubelet/:/var/lib/kubelet:rw,shared
Furthermore enable same option to docker in docker.service
MountFlags=shared
Now the question is: is there a security hole with shared mount?
I am new to kubernetes. I have an issue in the pods. When I run the command
kubectl get pods
Result:
NAME READY STATUS RESTARTS AGE
mysql-apim-db-1viwg 1/1 Running 1 20h
mysql-govdb-qioee 1/1 Running 1 20h
mysql-userdb-l8q8c 1/1 Running 0 20h
wso2am-default-813fy 0/1 ImagePullBackOff 0 20h
Due to an issue of "wso2am-default-813fy" node, I need to restart it. Any suggestion?
In case of not having the yaml file:
kubectl get pod PODNAME -n NAMESPACE -o yaml | kubectl replace --force -f -
Usually in case of "ImagePullBackOff" it's retried after few seconds/minutes. In case you want to try again manually you can delete the old pod and recreate the pod. The one line command to delete and recreate the pod would be:
kubectl replace --force -f <yml_file_describing_pod>
$ kubectl replace --force -f <resource-file>
if all goes well, you should see something like:
<resource-type> <resource-name> deleted
<resource-type> <resource-name> replaced
details of this can be found in the Kubernetes documentation, "manage-deployment" and kubectl-cheatsheet pages at the time of writing.
If the Pod is part of a Deployment or Service, deleting it will restart the Pod and, potentially, place it onto another node:
$ kubectl delete po $POD_NAME
replace it if it's an individual Pod:
$ kubectl get po -n $namespace $POD_NAME -o yaml | kubectl replace -f -
Try with deleting pod it will try to pull image again.
kubectl delete pod <pod_name> -n <namespace_name>
First try to see what's wrong with the pod:
kubectl logs -p <your_pod>
In my case it was a problem with the YAML file.
So, I needed to correct the configuration file and replace it:
kubectl replace --force -f <yml_file_describing_pod>
Most probably the issue of ImagePullBackOff is due to either the image not being present or issue with the pod YAML file.
What I will do is this
kubectl get pod -n $namespace $POD_NAME --export > pod.yaml | kubectl -f apply -
I would also see the pod.yaml to see the why the earlier pod didn't work
There is also possibility that the pull policy is not defined or kubernetes is configured to pull from the hub but fails due network issues. Try setting up a local secure registry and execute a pull . It would work.
I'm newbie of the Kubernetes while I'm using Google Cloud Container. I just follow the tutorials as belows:
https://cloud.google.com/container-engine/docs/tutorials/http-balancer
http://kubernetes.io/docs/hellonode/#create-your-pod
In these tutorials, I'll get the replicacontroller after I run the "kubectl run" but there is no replicacontrollers so that I cannot run the command of "kubectl expose rc" in order to open a port.
Here is my result of the commands:
ChangMatthews-MacBook-Pro:frontend changmatthew$ kubectl run nginx --image=nginx --port=80
deployment "nginx" created
ChangMatthews-MacBook-Pro:frontend changmatthew$ kubectl expose rc nginx --target-port=80 --type=NodePort
Error from server: replicationcontrollers "nginx" not found
Here is my result when I run "kubectl get rc,svc,ingress,deployments,pods":
ChangMatthews-MacBook-Pro:frontend changmatthew$ kubectl get rc,svc,ingress,deployments,pods
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.3.240.1 <none> 443/TCP 12m
NAME RULE BACKEND ADDRESS AGE
basic-ingress - nginx:80 107.178.247.247 12m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx 1 1 1 1 11m
NAME READY STATUS RESTARTS AGE
nginx-198147104-zgo7m 1/1 Running 0 11m
One of my solution is to create yaml file which define the replicacontroller. But is there any way to create replicacontroller via kubectl run command like above tutorials?
Thanks,
Now that kubectl run creates a deployment, you specify that the type being exposed in a deployment rather than a replication controller:
kubectl expose deployment nginx --target-port=80 --type=NodePort
The team might still be updating the docs to reflect 1.2. Note the output you got:
$ kubectl run nginx --image=nginx --port=80
deployment "nginx" created
kubectl run now creates a deployemtn+replica-set.
To view these you can do kubectl get deployment, and get rs respectively.
Deployments are essentially a nicer way to perform rolling update server side, but there's a little more to it. See docs: http://kubernetes.io/docs/user-guide/deployments/
In version 1.15.0, it works as follows.
root#k8smaster ~]# kubectl run guestbook --image=coolguy/k8s_guestbook:1.0 --port=8080 --generator=run/v1
kubectl run --generator=run/v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create
instead.
***replicationcontroller/guestbook created***
In version 1.19.0:
[root#k8smaster ~]# kubectl run guestbook --image=dmsong2008/k8s_guestbook:1.0 --port=8080 --generator=run/v1
***Flag --generator has been deprecated, has no effect and will be removed in the future.***
pod/guestbook created