I am new to Kubernetes. I am using Minikube for Mac with VM hyperkit. I also have docker-desktop installed (in which I have tried both enable/disable Kubernetes).
docker pull is executed smoothly with no error.
but on
kubectl run kubernetes-jenkins --image=jenkins:latest --port=8080
(or any image, be it gcr.io/google-samples/kubernetes-bootcamp:v1) it fails with ImagePullBackOff
Trimming few parts from kubectl cluster-info dump:
I1230 10:20:56.812648 1 serving.go:312] Generated self-signed
cert in-memory W1230 10:20:58.777494 1
configmap_cafile_content.go:102] unable to load initial CA bundle for:
"client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
due to: configmap "extension-apiserver-authentication" not found W1230
10:20:58.778005 1 configmap_cafile_content.go:102] unable to
load initial CA bundle for:
"client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
due to: configmap "extension-apiserver-authentication" not found W1230
10:20:58.849619 1 authorization.go:47] Authorization is disabled
W1230 10:20:58.850375 1 authentication.go:92] Authentication is
disabled
"reason": "Failed",
"message": "Failed to pull image \"jenkins:latest\": rpc error: code = Unknown desc = Error response from daemon: Get
https://registry-1.docker.io/v2/: dial tcp: lookup
registry-1.docker.io on 192.168.64.1:53: read udp
192.168.64.3:38558-\u003e192.168.64.1:53: read: connection refused",
"source": {
"component": "kubelet",
"host": "minikube"
}
Why kubectl is unable to pull image from the repository?
In minikube your local docker registry docker image can't be found,so you have to set your docker env to use minikube registry for local image you build and pull
eval $(minikube docker-env)
if that doesn't solve your problem, you have to start minikube by telling it's registry
minikube start --vm-driver="virtualbox" --insecure-registry=$(docker-machine ip registry):80
Related
I installed vanilla k8s on a Rocky Linux 8.6 with together with docker.
I created the /etc/docker/daemon.json:
{
"insecure-registries":["rocky-master.mfr.org:5000"],
"exec-opts":["native.cgroupdriver=systemd"],
"storage-driver":"overlay2"
}
But my pod on worker1 says:
Warning Failed 5m26s (x4 over 6m48s) kubelet Failed to pull image "rocky-master.mfr.org:5000/sametime-init:20220712-1935": rpc error: code = Unknown desc = failed to pull and unpack image "rocky-master.mfr.org:5000/sametime-init:20220712-1935": failed to resolve reference "rocky-master.mfr.org:5000/sametime-init:20220712-1935": failed to do request: Head "https://rocky-master.mfr.org:5000/v2/sametime-init/manifests/20220712-1935": http: server gave HTTP response to HTTPS client
Any idea?
I am very new to Kuberetes and I have done some work with docker previously. I am trying to accomplish following:
Spin up Minikube
Use Kube-ctl to spin up a docker image from docker hub.
I started minikube and things look like they are up and running. Then I pass following command
kubectl run nginx --image=nginx (Please note I do not have this image anywhere on my machine and I am expecting k8 to fetch it for me)
Now, when I do that, it spins up the pod but the status is ImagePullBackOff. So I ran kubectl describe pod command on it and the results look like following:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m default-scheduler Successfully assigned default/ngix-67c6755c86-qm5mv to minikube
Warning Failed 8m kubelet, minikube Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:52133->192.168.64.1:53: read: connection refused
Normal Pulling 8m (x2 over 8m) kubelet, minikube Pulling image "nginx"
Warning Failed 8m (x2 over 8m) kubelet, minikube Error: ErrImagePull
Warning Failed 8m kubelet, minikube Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:40073->192.168.64.1:53: read: connection refused
Normal BackOff 8m (x3 over 8m) kubelet, minikube Back-off pulling image "nginx"
Warning Failed 8m (x3 over 8m) kubelet, minikube Error: ImagePullBackOff
Then I searched around to see if anyone has faced similar issues and it turned out that some people have and they did resolve it by restarting minikube using some more flags which look like below:
minikube start --vm-driver="xhyve" --insecure-registry="$REG_IP":80
when I do nslookup inside Minikube, it does resolve with following information:
Server: 10.12.192.22
Address: 10.12.192.22#53
Non-authoritative answer:
hub.docker.com canonical name = elb-default.us-east-1.aws.dckr.io.
elb-default.us-east-1.aws.dckr.io canonical name = us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.
Name: us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 52.205.36.130
Name: us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 3.217.62.246
Name: us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 35.169.212.184
still no luck. Is there anything that I am doing wrong here?
There error message suggests that the Docker daemon running in the minikube VM can't resolve the registry-1.docker.io hostname because the DNS nameserver it's configured to use for DNS resolution (192.168.64.1:53) is refusing connection. It's strange to me that the Docker deamon is trying to resolve registry-1.docker.io via a nameserver at 192.168.64.1 but when you nslookup on the VM it's using a nameserver at 10.12.192.22. I did an Internet search for "minkube Get registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53" and found an issue where someone made this comment, seems identical to your problem, and seems specific to xhyve.
In that comment the person says:
This issue does look like an xhyve issue not seen with virtualbox.
and
Switching to virtualbox fixed this issue for me.
I stopped minikube, deleted it, started it without --vm-driver=xhyve (minikube uses virtualbox driver by default), and then docker build -t hello-node:v1 . worked fine without errors
In my case it was caused by running dnsmasq, a dns server, on my Mac using Homebrew, which caused the DNS requests to fail inside minikube. After stopping dnsmasq, everything worked.
I got this problem with my local minikube setup and I wasn't able to pull any images I added to a simple deployment manifest.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
test1 0/1 ImagePullBackOff 0 68s
Tried to execute the below test:
apiVersion: v1
kind: Pod
metadata:
name: test1
labels:
site: blog
spec:
containers:
- name: web
image: nginx:latest
It was possible or fixed only after restarting the minikube.
Maybe the dnsmasq was really the cause in this case.
You have:
minukube running with default settings.
docker building your images
(*) configured minikube to point to your docker images local repo
And now minikube can't pull images from public "container" registries, like docker hub.
stop and start minikube, then point it back to your local docker images repo. The commands to do this (and (*) this):
minikube stop
minikube start
minikube -p minikube docker-env
eval $(minikube -p minikube docker-env)
Since running the above I was able to pull nginx, alpine and frens from hub.docker.come just by setting image: alpine in the yaml spec.
The issue was just a short drop in my network connectivity. So if you have no dns/vpn/xhyve complications and it just stops, the fix is easy enough.
I'm going to setup a local registry by following https://docs.docker.com/registry/deploying/.
docker run -d -p 5000:5000 --restart=always --name reg ubuntu:16.04
When I try to run the following command:
$ docker push localhost:5000/my-ubuntu
I get Error:
Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: connect:connection refused
Any idea?
Connection refused usually means that the service you are trying to connect to isn't actually up and running like it should. There could be other reasons as outlined in this question, but essentially, for your case, it simply means that the registry is not up yet.
Wait for the registry container to be created properly before you do anything else - docker run -d -p 5000:5000 --restart=always --name registry registry:2 that creates a local registry from the official docker image.
Make sure that the registry container is up by running docker ps | grep registry, and then proceed further.
More comments about
Kubenetes(K8s) / Minikube
docker / image / registry, container
If you are using Minikube, and want to pull down an image from 127.0.0.1:5000,
then you meet the errors below:
Failed to pull image "127.0.0.1:5000/nginx_operator:latest": rpc error: code = Unknown desc = Error response from daemon: Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused
Full logs:
$ kubectl describe pod/your_pod
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m29s default-scheduler Successfully assigned tj-blue-whale-05-system/tj-blue-whale-05-controller-manager-6c8f564575-kwxdv to minikube
Normal Pulled 2m25s kubelet Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0" already present on machine
Normal Created 2m24s kubelet Created container kube-rbac-proxy
Normal Started 2m23s kubelet Started container kube-rbac-proxy
Normal BackOff 62s (x5 over 2m22s) kubelet Back-off pulling image "127.0.0.1:5000/nginx_operator:latest"
Warning Failed 62s (x5 over 2m22s) kubelet Error: ImagePullBackOff
Normal Pulling 48s (x4 over 2m23s) kubelet Pulling image "127.0.0.1:5000/nginx_operator:latest"
Warning Failed 48s (x4 over 2m23s) kubelet Failed to pull image "127.0.0.1:5000/nginx_operator:latest": rpc error: code = Unknown desc = Error response from daemon: Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused
Warning Failed 48s (x4 over 2m23s) kubelet Error: ErrImagePull
Possible root cause:
The registry must be setup inside the Minikube side instead of your host side.
i.e.
host: registry (127.0.0.1:5000)
minikube: no registry (the K8s could not find your image)
How to check?
Step1: check your Minikube container
$ docker ps -a
CONTAINER ID IMAGE ... STATUS PORTS NAMES
8c6f49491dd6 gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 ... Up 15 hours 127.0.0.1:49156->22/tcp, 127.0.0.1:49155->2376/tcp, 127.0.0.1:49154->5000/tcp, 127.0.0.1:49153->8443/tcp minikube
# your Minikube is under running
# host:49154 <--> minikube:5000
# where:
# - port 49154 was allocated randomly by the docker service
# - port 22: for ssh
# - port 2376: for docker service
# - port 5000: for registry (image repository)
# - port 8443: for Kubernetes
Step2: login to your Minikube
$ minikube ssh
docker#minikube:~$ curl 127.0.0.1:5000
curl: (7) Failed to connect to 127.0.0.1 port 5000: Connection refused
# setup
# =====
# You did not setup the registry.
# Let's try to setup it.
docker#minikube:~$ docker run --restart=always -d -p 5000:5000 --name registry registry:2
# test
# ====
# test the registry using the following commands
docker#minikube:~$ curl 127.0.0.1:5000
docker#minikube:~$ curl 127.0.0.1:5000/v2
Moved Permanently.
docker#minikube:~$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":[]}
# it's successful
docker#minikube:~$ exit
logout
Step3: build your image, and push it into the registry of your Minikube
# Let's take nginx as an example. (You can build your own image)
$ docker pull nginx
# modify the repository (the source and the name)
$ docker tag nginx 127.0.0.1:49154/nginx_operator
# check the new repository (source and the name)
$ docker images | grep nginx
REPOSITORY TAG IMAGE ID CREATED SIZE
127.0.0.1:49154/nginx_operator latest ae2feff98a0c 3 weeks ago 133MB
# push the image into the registry of your Minikube
$ docker push 127.0.0.1:49154/nginx_operator
Step4: login to your Minikube again
$ minikube ssh
# check the registry
$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":["nginx_operator"]}
# it's successful
# get the image info
$ curl 127.0.0.1:5000/v2/nginx_operator/manifests/latest
docker#minikube:~$ exit
logout
Customize exposed ports of Minikube
if you would like to use the port 5000 on the host side instead of using 49154 (which was allocated randomly by the docker service)
i.e.
host:5000 <--> minikube:5000
you need to recreate a minikube instance with the flag --ports
# delete the old minikube instance
$ minkube delete
# create a new one (with the docker driver)
$ minikube start --ports=5000:5000 --driver=docker
# or
$ minikube start --ports=127.0.0.1:5000:5000 --driver=docker
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5d1e5b61a3bf gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 "/usr/local/bin/entr…" About a minute ago Up About a minute 0.0.0.0:5000->5000/tcp, 127.0.0.1:49162->22/tcp, 127.0.0.1:49161->2376/tcp, 127.0.0.1:49160->5000/tcp, 127.0.0.1:49159->8443/tcp minikube
$ docker port minikube
22/tcp -> 127.0.0.1:49162
2376/tcp -> 127.0.0.1:49161
5000/tcp -> 127.0.0.1:49160
5000/tcp -> 0.0.0.0:5000
8443/tcp -> 127.0.0.1:49159
you can see: 0.0.0.0:5000->5000/tcp
Re-test your registry in the Minikube
# in the host side
$ docker pull nginx
$ docker tag nginx 127.0.0.1:5000/nginx_operator
$ docker ps -a
$ docker push 127.0.0.1:5000/nginx_operator
$ minikube ssh
docker#minikube:~$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":["nginx_operator"]}
# Great!
I am trying to pull images to default docker machine. But I always get
docker#default:/etc$ docker run hellow-world
Unable to find image 'hellow-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
docker#default:/etc$
Even if i give --dns option same error
docker#default:/etc$ docker run hellow-world --dns=8.8.8.8
Unable to find image 'hellow-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
docker#default:/etc$
Tried adding DNS to config.json of the default machine
"EngineOptions": {
"Dns": [
"8.8.8.8",
"8.8.4.4"
]
}
resolv.conf has following entry in default machine
docker#default:/etc$
docker#default:/etc$ cat /etc/resolv.conf
search <company name>
nameserver 10.0.2.3
docker#default:/etc$
This machine sits behind the proxy and I am using kitematic for creating default machine
Kitematic is started by a custom script which will set proxies and start kitematic
Recommended and All images are displayed at the start of kitematic. But can not pull any images
Creating new machine also gives warning
PS C:\Program Files\Docker Toolbox\kitematic> docker-machine create -d virtualbox --engine-opt dns=8.8.8.8 test
Running pre-create checks...
(test) Unable to get the latest Boot2Docker ISO release version: Get https://api.github.com/repos/boot2docker/boot2docker/releases/latest: dial tcp 192.30.253.116:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Creating machine...
(test) Unable to get the latest Boot2Docker ISO release version: Get https://api.github.com/repos/boot2docker/boot2docker/releases/latest: dial tcp 192.30.253.116:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Machine created with --dns option also has /etc/resolv.conf not as 8.8.8.8
docker#test:~$
docker#test:~$ cat /etc/resolv.conf
search <company name>
nameserver 10.0.2.3
docker#test:~$
PS:Issue solved after adding HTTP_PROXY to /var/lib/boot2docker/profile
Found solution
Either
add HTTP_PROXY to /var/lib/boot2docker/profile
or
create docker machine with --engine-env HTTP_PROXY=IP:PORT
I am trying to create a pod using kubernetes with the following simple command
kubectl run example --image=nginx
It runs and assigns the pod to the minion correctly but the status is always in ContainerCreating status due to the following error. I have not hosted GCR or GCloud on my machine. So not sure why its picking from there only.
1h 29m 14s {kubelet centos-minion1} Warning FailedSync Error syncing pod, skipping:
failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed
for gcr.io/google_containers/pause:2.0, this may be because there are no
credentials on this request. details: (unable to ping registry endpoint
https://gcr.io/v0/\nv2 ping attempt failed with error: Get https://gcr.io/v2/:
http: error connecting to proxy http://87.254.212.120:8080: dial tcp
87.254.212.120:8080: i/o timeout\n v1 ping attempt failed with error:
Get https://gcr.io/v1/_ping: http: error connecting to proxy
http://87.254.212.120:8080: dial tcp 87.254.212.120:8080: i/o timeout)
Kubernetes is trying to create a pause container for your pod; this container is used to create the pod's network namespace. See this question and its answers for more general information on the pause container.
To your specific error: Kubernetes tries to pull the pause container's image (which would be gcr.io/google_containers/pause:2.0, according to your error message) from the Google Container Registry (gcr.io). Apparently, your Docker engine tries to connect to GCR using a HTTP proxy located at 87.254.212.120:8080, to which it apparently cannot connect (i/o timeout).
To correct this error, either make sure that you HTTP proxy server is online and does not block HTTP requests to GCR, or (if you do have public Internet access) disable the proxy connection for your Docker engine (this would typically be done using the http_proxy and https_proxy environment variables, which would have been set in /etc/sysconfig/docker or /etc/default/docker, depending on your Linux distribution).