I'm new to Kubernetes and learning it these days. I'm trying to create a deployment with the help of kubectl and every time I create a deployment, the container is not running and I get ErrImagePull or ImagePullBackOff.
I have tried on two machines, both have the same problem.
kubectl create deployment nginx-depl --image=nginx
Following is the description of one of the deployments I was creating
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23s default-scheduler Successfully assigned default/nginx-85b98978db-z6b2n to minikube
Normal Pulling 22s kubelet Pulling image "nginx"
Warning Failed 7s kubelet Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 7s kubelet Error: ErrImagePull
Normal BackOff 7s kubelet Back-off pulling image "nginx"
Warning Failed 7s kubelet Error: ImagePullBackOff
Looks like you dont have internet connectivity from the cluster nodes. kubelet on the node is trying to pull the docker image from https://registry-1.docker.io/v2/ but connectivity is not established and was timed out. hence you are getting ImagePullBackOff Error
Related
When we tried to install keda on our kubernetes cluster by following instruction mentioned in keda deploy document we are getting image pullbackoff error in keda-operator & keda-operator-metrics-apiserver pod. Please help us on this.
Error:
Type Reason Age From Message
Normal Scheduled 4m23s default-scheduler Successfully assigned keda/keda-metrics-apiserver-5bbcc67cd8-7mm59 to aks-agentpool
Warning Failed 3m3s (x6 over 4m21s) kubelet Error: ImagePullBackOff
Normal Pulling 2m49s (x4 over 4m22s) kubelet Pulling image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.0"
Warning Failed 2m49s (x4 over 4m22s) kubelet Failed to pull image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.0": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.0": failed to resolve reference "ghcr.io/kedacore/keda-metrics-apiserver:2.8.0": failed to do request: Head "https://ghcr.io/v2/kedacore/keda-metrics-apiserver/manifests/2.8.0": EOF
Warning Failed 2m49s (x4 over 4m22s) kubelet Error: ErrImagePull
Normal BackOff 2m38s (x7 over 4m21s) kubelet Back-off pulling image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.0"
I'm using helm chart of falco and here are the commands
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco
Here's the output
NAME READY STATUS RESTARTS AGE
falco-6hlvx 0/1 Init:ImagePullBackOff 0 4m19
Once I type describe pod to know the reason behind this status, I find these events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 22m default-scheduler Successfully assigned default/falco-6hlvx to minikube
Normal Pulling 15m (x4 over 22m) kubelet Pulling image "docker.io/falcosecurity/falco-driver-loader:0.32.1"
Warning Failed 13m (x4 over 20m) kubelet Failed to pull image "docker.io/falcosecurity/falco-driver-loader:0.32.1": rpc error: code = Unknown desc = context deadline exceeded
Warning Failed 13m (x4 over 20m) kubelet Error: ErrImagePull
Warning Failed 12m (x7 over 20m) kubelet Error: ImagePullBackOff
Normal BackOff 2m45s (x34 over 20m) kubelet Back-off pulling image "docker.io/falcosecurity/falco-driver-loader:0.32.1"
Whenever this happens, try to pull the image directly using minikube ssh docker pull to obtain a better error message. Be aware that Docker Hub applies a download rate limit and you might have just hit it.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 51s default-scheduler Successfully assigned jenkins/jenkins-0 to minikube
Normal BackOff 31s kubelet, minikube Back-off pulling image "jenkins/jenkins:2.303.3-jdk11"
Warning Failed 31s kubelet, minikube Error: ImagePullBackOff
Normal Pulling 17s (x2 over 47s) kubelet, minikube Pulling image "jenkins/jenkins:2.303.3-jdk11"
Warning Failed 1s (x2 over 32s) kubelet, minikube Failed to pull image "jenkins/jenkins:2.303.3-jdk11": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 1s (x2 over 32s) kubelet, minikube Error: ErrImagePull
The above error is what I am seeing, when trying to install Jenkins on a minikube cluster. I am using this link and following along: https://www.jenkins.io/doc/book/installing/kubernetes/
appreciate any ideas.
I tried with minikube with virtualbox and that worked out of the box.
But, wanted to get docker working, which I wasn't able to.
Finally, I deleted everything (even reinstalled ubuntu) and re-setup k8s cluster with latest k8s version (before i tried with --version=1.19.0 of k8s)
I used: minikube start --driver=docker. and then followed the official jenkins install with helm3, that too latest, lts –
I have a local kubernetes cluster (minikube), that is trying to load images from my local Docker repo.
When I do a "docker images", I get:
cluster.local/container-images/app-shiny-app-validation-app-converter 1.6.9
cluster.local/container-images/app-shiny-app-validation 1.6.9
Given I know the above images are there, I run some helm commands which uses these images, but I get the below error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 66s (x2 over 2m12s) kubelet Back-off pulling image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9"
Warning Failed 66s (x2 over 2m12s) kubelet Error: ImagePullBackOff
Normal Pulling 51s (x3 over 3m24s) kubelet Pulling image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9"
Warning Failed 11s (x3 over 2m13s) kubelet Failed to pull image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9": rpc error: code = Unknown desc = Error response from daemon: Get https://cluster.local/v2/: dial tcp: lookup cluster.local: Temporary failure in name resolution
Warning Failed 11s (x3 over 2m13s) kubelet Error: ErrImagePull
Anyone know how I can fix this? Seems the biggest problem is Get https://cluster.local/v2/: dial tcp: lookup cluster.local: Temporary failure in name resolution
Since minikube is being used, you can refer to their documentation.
It is recommended that if a imagePullPolicy is being used, it needs to be set to Never. If set to Always, it will try to reach out and pull from the network.
From docs: https://minikube.sigs.k8s.io/docs/handbook/pushing/
"Tip 1: Remember to turn off the imagePullPolicy:Always (use imagePullPolicy:IfNotPresent or imagePullPolicy:Never) in your yaml file. Otherwise Kubernetes won’t use your locally build image and it will pull from the network."
Add cluster.local to your /etc/hosts file in all your kubernetes nodes.
192.168.12.34 cluster.local
Check whether you can login to registry using docker login cluster.local
If your registry has self-signed certificates, copy cluster.local.crt key to all kubernetes worker nodes /etc/docker/certs.d/cluster.local/ca.crt
Trying to create a deployment from an image
kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10
and then do kubectl get pods and check the logs of hello-minikube pod using
kubectl pod describe <pod-name> throws following error
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/hello-minikube-797f975945-dmq26 to minikube
Warning Failed 42s kubelet, minikube Failed to pull image "k8s.gcr.io/echoserver:1.10": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.3:56747->192.168.64.1:53: read: connection refused
Warning Failed 27s kubelet, minikube Failed to pull image "k8s.gcr.io/echoserver:1.10": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.3:48279->192.168.64.1:53: read: connection refused
Normal BackOff 16s (x2 over 42s) kubelet, minikube Back-off pulling image "k8s.gcr.io/echoserver:1.10"
Warning Failed 16s (x2 over 42s) kubelet, minikube Error: ImagePullBackOff
Normal Pulling 4s (x3 over 42s) kubelet, minikube Pulling image "k8s.gcr.io/echoserver:1.10"
Warning Failed 4s (x3 over 42s) kubelet, minikube Error: ErrImagePull
Warning Failed 4s kubelet, minikube Failed to pull image "k8s.gcr.io/echoserver:1.10": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.3:50616->192.168.64.1:53: read: connection refused
This error can be solved using :
Check whether any docker machine is running or not by docker-machine ls
If no machine exist, then create one using docker-machine create <machine-name>
Then get this machine IP using docker-machine ip <machine-name>
Then first delete any existing minikube cluster using minikube delete and start again using minikube start --vm-driver="virtualbox" --insecure-registry="docker-machine IP":80
Then run kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10 and check the logs, it will show image pulled successfully.
From the events it looks like the call to a DNS server at 192.168.64.1:53 to resolve k8s.gcr.io is failing. Check if there's a DNS server running at that IP. It may be configured in /etc/resolv.conf (if Linux) on the minikube host.