I've started k3d with k3d create && k3d start.
All pods fail to start with the following error:
Warning FailedCreatePodSandBox 14s (x2 over 31s) kubelet,
k3d-k3s-default-server Failed to create pod sandbox: rpc error: code
= Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image
"docker.io/rancher/pause:3.1": failed to pull and unpack image
"docker.io/rancher/pause:3.1": failed to resolve reference
"docker.io/rancher/pause:3.1": failed to do request: Head
https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: dial tcp:
lookup registry-1.docker.io: Try again
As recommended by a k3d contributor, I've exec'ed into the k3d server container and attempted to pull the image manually:
$ docker exec -it k3d-k3s-default-server sh
/ # ctr image pull docker.io/rancher/pause:3.1
docker.io/rancher/pause:3.1: resolving |--------------------------------------|
elapsed: 4.9 s total: 0.0 B (0.0 B/s)
ctr: failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: dial tcp: lookup registry-1.docker.io: Try again
In the host environment, docker pull docker.io/rancher/pause:3.1 works just fine.
I've seen a number of people resolve the issue by tweaking various DNS settings. But none described how they arrived at their particular solution.
Solving this issue would make me happy. Discovering a general diagnosis strategy would make me even happier.
What hasn't worked
From here:
I got the issue. I had one entry in
/etc/systemd/network/en0.networking Deleted that file, and everything
is fine.
I have no files in /etc/systemd/network/.
I had the same issue with k3s not being able to pull images and solved it by updating my /etc/resolv.conf to be symlinked from /run/systemd/resolve/stub-resolv.conf on the host machine with
ln -sf /etc/resolv.conf /run/systemd/resolve/stub-resolv.conf
Related
I’m following this guide: Vault Installation to Google Kubernetes Engine via Helm | Vault - HashiCorp Learn: https://learn.hashicorp.com/tutorials/vault/kubernetes-google-cloud-gke
However, after running the Helm install command as below, my vault-agent-injector pod isn’t working as expected.
I ran:
helm install vault hashicorp/vault
–set=‘server.ha.enabled=true’
–set=‘server.ha.raft.enabled=true’
I then see the following events when describing the pod:
Normal Scheduled 51s default-scheduler Successfully assigned default/vault-agent-injector-f59c7f985-n6b72 to gke-test-cluster-test-cluster-np-680d0af5-2lw8
Normal Pulling 51s kubelet Pulling image “hashicorp/vault-k8s:0.16.1”
Warning Failed kubelet Failed to pull image “hashicorp/vault-k8s:0.16.1”: rpc error: code = Unknown desc = failed to pull and unpack image “docker.io/hashicorp/vault-k8s:0.16.1”: failed to resolve reference “docker.io/hashicorp/vault-k8s:0.16.1”: failed to do request: Head "https://registry-1.docker.io/v2/hashicorp/vault-k8s/manifests/0.16.1": dial tcp 44.207.51.64:443: i/o timeout
Warning Failed kubelet Error: ErrImagePull
Normal BackOff kubelet Back-off pulling image “hashicorp/vault-k8s:0.16.1”
Warning Failed kubelet Error: ImagePullBackOff
Normally Helm installs work perfectly fine, so I’m not sure what’s going on here. Could someone please advise?
I have a local kubernetes cluster (minikube), that is trying to load images from my local Docker repo.
When I do a "docker images", I get:
cluster.local/container-images/app-shiny-app-validation-app-converter 1.6.9
cluster.local/container-images/app-shiny-app-validation 1.6.9
Given I know the above images are there, I run some helm commands which uses these images, but I get the below error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 66s (x2 over 2m12s) kubelet Back-off pulling image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9"
Warning Failed 66s (x2 over 2m12s) kubelet Error: ImagePullBackOff
Normal Pulling 51s (x3 over 3m24s) kubelet Pulling image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9"
Warning Failed 11s (x3 over 2m13s) kubelet Failed to pull image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9": rpc error: code = Unknown desc = Error response from daemon: Get https://cluster.local/v2/: dial tcp: lookup cluster.local: Temporary failure in name resolution
Warning Failed 11s (x3 over 2m13s) kubelet Error: ErrImagePull
Anyone know how I can fix this? Seems the biggest problem is Get https://cluster.local/v2/: dial tcp: lookup cluster.local: Temporary failure in name resolution
Since minikube is being used, you can refer to their documentation.
It is recommended that if a imagePullPolicy is being used, it needs to be set to Never. If set to Always, it will try to reach out and pull from the network.
From docs: https://minikube.sigs.k8s.io/docs/handbook/pushing/
"Tip 1: Remember to turn off the imagePullPolicy:Always (use imagePullPolicy:IfNotPresent or imagePullPolicy:Never) in your yaml file. Otherwise Kubernetes won’t use your locally build image and it will pull from the network."
Add cluster.local to your /etc/hosts file in all your kubernetes nodes.
192.168.12.34 cluster.local
Check whether you can login to registry using docker login cluster.local
If your registry has self-signed certificates, copy cluster.local.crt key to all kubernetes worker nodes /etc/docker/certs.d/cluster.local/ca.crt
When executing the following command:
$ sudo docker-compose -f custom-docker-compose.yml up -d
I see this normal output:
Pulling mymd-db (mariadb:10.5.6)...
10.5.6: Pulling from library/mariadb
6a5697faee43: Pulling fs layer
ba13d3bc422b: Pulling fs layer
a254829d9e55: Pulling fs layer
2ee2cadd29fc: Waiting
6915a184049d: Waiting
5ca6ffdb5f56: Waiting
1537f7bbef8b: Waiting
5790e54322d1: Waiting
ea98cb829471: Waiting
But then, out of the blue, I get the following error:
ERROR: error pulling image configuration: Get
https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/c4/c4655f911514fc440...Gmo%3D:
EOF
This happens from time to time. I don't know why. Does anybody know how to fix it or any workaround? I am running a CentOS server.
vim /etc/resolv.conf ,add
nameserver 8.8.8.8
service docker restart
And try again.
Most often than not,this is because of problems with your network
I am very new to Kuberetes and I have done some work with docker previously. I am trying to accomplish following:
Spin up Minikube
Use Kube-ctl to spin up a docker image from docker hub.
I started minikube and things look like they are up and running. Then I pass following command
kubectl run nginx --image=nginx (Please note I do not have this image anywhere on my machine and I am expecting k8 to fetch it for me)
Now, when I do that, it spins up the pod but the status is ImagePullBackOff. So I ran kubectl describe pod command on it and the results look like following:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m default-scheduler Successfully assigned default/ngix-67c6755c86-qm5mv to minikube
Warning Failed 8m kubelet, minikube Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:52133->192.168.64.1:53: read: connection refused
Normal Pulling 8m (x2 over 8m) kubelet, minikube Pulling image "nginx"
Warning Failed 8m (x2 over 8m) kubelet, minikube Error: ErrImagePull
Warning Failed 8m kubelet, minikube Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:40073->192.168.64.1:53: read: connection refused
Normal BackOff 8m (x3 over 8m) kubelet, minikube Back-off pulling image "nginx"
Warning Failed 8m (x3 over 8m) kubelet, minikube Error: ImagePullBackOff
Then I searched around to see if anyone has faced similar issues and it turned out that some people have and they did resolve it by restarting minikube using some more flags which look like below:
minikube start --vm-driver="xhyve" --insecure-registry="$REG_IP":80
when I do nslookup inside Minikube, it does resolve with following information:
Server: 10.12.192.22
Address: 10.12.192.22#53
Non-authoritative answer:
hub.docker.com canonical name = elb-default.us-east-1.aws.dckr.io.
elb-default.us-east-1.aws.dckr.io canonical name = us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.
Name: us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 52.205.36.130
Name: us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 3.217.62.246
Name: us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 35.169.212.184
still no luck. Is there anything that I am doing wrong here?
There error message suggests that the Docker daemon running in the minikube VM can't resolve the registry-1.docker.io hostname because the DNS nameserver it's configured to use for DNS resolution (192.168.64.1:53) is refusing connection. It's strange to me that the Docker deamon is trying to resolve registry-1.docker.io via a nameserver at 192.168.64.1 but when you nslookup on the VM it's using a nameserver at 10.12.192.22. I did an Internet search for "minkube Get registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53" and found an issue where someone made this comment, seems identical to your problem, and seems specific to xhyve.
In that comment the person says:
This issue does look like an xhyve issue not seen with virtualbox.
and
Switching to virtualbox fixed this issue for me.
I stopped minikube, deleted it, started it without --vm-driver=xhyve (minikube uses virtualbox driver by default), and then docker build -t hello-node:v1 . worked fine without errors
In my case it was caused by running dnsmasq, a dns server, on my Mac using Homebrew, which caused the DNS requests to fail inside minikube. After stopping dnsmasq, everything worked.
I got this problem with my local minikube setup and I wasn't able to pull any images I added to a simple deployment manifest.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
test1 0/1 ImagePullBackOff 0 68s
Tried to execute the below test:
apiVersion: v1
kind: Pod
metadata:
name: test1
labels:
site: blog
spec:
containers:
- name: web
image: nginx:latest
It was possible or fixed only after restarting the minikube.
Maybe the dnsmasq was really the cause in this case.
You have:
minukube running with default settings.
docker building your images
(*) configured minikube to point to your docker images local repo
And now minikube can't pull images from public "container" registries, like docker hub.
stop and start minikube, then point it back to your local docker images repo. The commands to do this (and (*) this):
minikube stop
minikube start
minikube -p minikube docker-env
eval $(minikube -p minikube docker-env)
Since running the above I was able to pull nginx, alpine and frens from hub.docker.come just by setting image: alpine in the yaml spec.
The issue was just a short drop in my network connectivity. So if you have no dns/vpn/xhyve complications and it just stops, the fix is easy enough.
I have to install openshift on disconnected system so i followed following steps(original installation requires more image but for sake of understanding i am provided minimum steps)
on system with internet i did following steps
docker pull docker.io/openshift/origin-node:v3.11.0
docker save -o openshift-origin-v3.11.0-images.tar \
docker.io/openshift/origin-node:v3.11.0
on second disconnected system i did following
docker load -i openshift-origin-v3.11.0-images.tar
Now when i start script for installation it pull the images with command docker.io/openshift/origin-node:v3.11.0
which is throwing following error
Error getting v2 registry: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: dial udp [::1]:53: connect: no route to host
When on second system i do docker images
[root#x ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/openshift/origin-node v3.11.0 14d965ab72d5 4 days ago 1.17 GB
Its showing me that image is available. Whats wrong here? My understanding is it should first look locally and then will check from dockerhub
Update1:
if i directly pull it saying
[root#x ~]# docker pull docker.io/openshift/origin-node:v3.11.0
Trying to pull repository docker.io/openshift/origin-node ...
Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.x.x:53: server misbehaving
I am expecting it should say
Status: Image is up to date for
I changed following in /etc/containers/registries.conf and it works
From
[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
To
[registries.search]
registries = []
[registries.block]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']