I tried to deploy 'deployments' in kubernetes which is pull docker image from private registry (I don't know who did this setup) but during "docker pull images" through kubernetes i'm getting following error
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 85s default-scheduler Successfully assigned default/trusted-enc-assettag1-deployment-8467b74958-6fbp7 to k8s-node
Normal BackOff 24s (x2 over 61s) kubelet, k8s-node Back-off pulling image "10.105.168.81:5000/simplehttpserverenc:enc_v1"
Warning Failed 24s (x2 over 61s) kubelet, k8s-node Error: ImagePullBackOff
Normal Pulling 12s (x3 over 82s) kubelet, k8s-node Pulling image "10.105.168.81:5000/simplehttpserverenc:enc_v1"
Warning Failed 0s (x3 over 62s) kubelet, k8s-node Failed to pull image "10.105.168.81:5000/simplehttpserverenc:enc_v1": rpc error: code = Unknown desc = Error response from daemon: Get https://10.105.168.81:5000/v2/: net/http: TLS handshake timeout
Warning Failed 0s (x3 over 62s) kubelet, k8s-node Error: ErrImagePull
[root#k8s-master ~]# docker pull 10.105.168.81:5000/simplehttpserverenc:enc_v1
ImagePullBackOff and net/http: TLS handshake timeout error.
Initially this "net/http: TLS handshake timeout" error is observed in docker pull as well. I referred some answers and
configured certificate(/etc/docker/certs.d//ca.crt ) and
proxy (/etc/systemd/system/docker.service.d/proxy.conf)
after that able to perform docker pull from private image.
[root#k8s-master ~]# docker pull 10.105.168.81:5000/simplehttpserverenc:enc_v1
enc_v1: Pulling from simplehttpserverenc
54fec2fa59d0: Pull complete
cd3f35d84cab: Pull complete
a0afc8e92ef0: Pull complete
9691f23efdb7: Pull complete
6512e60b314b: Pull complete
a8ac6632d329: Pull complete
68f4c4e0aa8c: Pull complete
Digest: sha256:0358708cd11e96f6cf6f22b29d46a8eec50d7107597b866e1616a73a198fe797
Status: Downloaded newer image for 10.105.168.81:5000/simplehttpserverenc:enc_v1
10.105.168.81:5000/simplehttpserverenc:enc_v1
[root#k8s-master ~]#
But still unable to perform this docker pull through kubernetes. How to solve this ?
If you use docker as container engine in your k8s, AFAIK it's the same with Understand the configuration. Because the image pulling is conducted by the container engine and it depends the proprietary configuration of each one on the certificates. How about pulling the same image on the worker node in your k8s ? Is it possible to pull the one without errors ?
As your dockerconfigjson is not working properly. Try this method :
kubectl create secret docker-registry regcred --docker-server=10.105.168.81:5000 --docker-username=<your-name> --docker-password=<your-pword>
And in Kubernetes manifest :
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: 10.105.168.81:5000/simplehttpserverenc:enc_v1
imagePullSecrets:
- name: regcred
I had encounted this many times, when I forgot to configure these secrets. Also if you have any othernamespace, you will have to generate secrets for each of these namespaces separately passing -n <your-ns> to above kubectl create secret
Edit : As you can not pull the image from worker node.
Make sure you copied docker-registry ca.crt to /etc/docker/certs.d/ca.crt
and then try docker pull .
Related
I have a local kubernetes cluster (minikube), that is trying to load images from my local Docker repo.
When I do a "docker images", I get:
cluster.local/container-images/app-shiny-app-validation-app-converter 1.6.9
cluster.local/container-images/app-shiny-app-validation 1.6.9
Given I know the above images are there, I run some helm commands which uses these images, but I get the below error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 66s (x2 over 2m12s) kubelet Back-off pulling image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9"
Warning Failed 66s (x2 over 2m12s) kubelet Error: ImagePullBackOff
Normal Pulling 51s (x3 over 3m24s) kubelet Pulling image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9"
Warning Failed 11s (x3 over 2m13s) kubelet Failed to pull image "cluster.local/container-images/app-shiny-app-validation-app-converter:1.6.9": rpc error: code = Unknown desc = Error response from daemon: Get https://cluster.local/v2/: dial tcp: lookup cluster.local: Temporary failure in name resolution
Warning Failed 11s (x3 over 2m13s) kubelet Error: ErrImagePull
Anyone know how I can fix this? Seems the biggest problem is Get https://cluster.local/v2/: dial tcp: lookup cluster.local: Temporary failure in name resolution
Since minikube is being used, you can refer to their documentation.
It is recommended that if a imagePullPolicy is being used, it needs to be set to Never. If set to Always, it will try to reach out and pull from the network.
From docs: https://minikube.sigs.k8s.io/docs/handbook/pushing/
"Tip 1: Remember to turn off the imagePullPolicy:Always (use imagePullPolicy:IfNotPresent or imagePullPolicy:Never) in your yaml file. Otherwise Kubernetes won’t use your locally build image and it will pull from the network."
Add cluster.local to your /etc/hosts file in all your kubernetes nodes.
192.168.12.34 cluster.local
Check whether you can login to registry using docker login cluster.local
If your registry has self-signed certificates, copy cluster.local.crt key to all kubernetes worker nodes /etc/docker/certs.d/cluster.local/ca.crt
Docker Image: -
docker images | grep -i "gcc"
gcc-docker latest 84c4359e6fc9 21 mites ago 1.37GB
docker run -it gcc-docker:latest
hello,world
Kubernetes pod deployed:-
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/hello-world to master-node
Normal Pulling 4s kubelet, master-node Pulling image "gcc-docker:latest"
Warning Failed 0s kubelet, master-node Failed to pull image "gcc-docker:latest": rpc error: code = Unknown desc = Erroresponse from daemon: pull access denied for gcc-docker, repository does not exist or may require 'docker login': denied: requested acce to the resource is denied
Warning Failed 0s kubelet, master-node Error: ErrImagePull
Normal BackOff 0s kubelet, master-node Back-off pulling image "gcc-docker:latest"
Warning Failed 0s kubelet, master-node Error: ImagePullBackOff
-->yaml files used to deploy pod
apiVersion: v1
kind: Pod
metadata:
name: hello-world
labels:
type: hello-world
spec:
containers:
- name: hello-world
image: gcc-docker:latest
command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 60']
ports:
- containerPort: 80
I tried pulling gcc-docker and got the same error.You may have this image present on your system already and now its not on dockerhub.
if you know the repository for this image, try to use the same and for authentication create secrets of docker type and use them as image pull secrets.
Also, one more thing you are running the container on the master node, and I assume it's minikube or some local setup.
Minikube uses a dedicated VM to run Kubernetes which is not the same as the machine on which you have installed minikube.
So images available on your laptop will not be available to minikube.
I am at the initial stage of Kubernetes. I've just created a pod using the command:
kubectl apply -f posts.yaml
It returns me the following:
pod/posts created
After that when I run kubectl get pods
I found the result as following:
NAME READY STATUS RESTARTS AGE
posts 0/1 ErrImagePull 0 2m4s
Here is my posts.yaml file in below:
apiVersion: v1
kind: Pod
metadata:
name: posts
spec:
containers:
- name: posts
image: bappa/posts:0.0.1
This means that kubernetes could not pull the image from the repository. Does the repo maybe need some authorization to allow image pull?
You can do
kubectl describe pod posts
to get some more info.
After applying yaml and looking into the kubectl describe pod posts you can clearly see below error:
Normal BackOff 21s kubelet Back-off pulling image "bappa/posts:0.0.1"
Warning Failed 21s kubelet Error: ImagePullBackOff
Normal Pulling 9s (x2 over 24s) kubelet Pulling image "bappa/posts:0.0.1"
Warning Failed 8s (x2 over 22s) kubelet Failed to pull image "bappa/posts:0.0.1": rpc error: code = Unknown desc = Error response from daemon: pull access denied for bappa/posts, repository does not exist or may require 'docker login'
Warning Failed 8s (x2 over 22s) kubelet Error: ErrImagePull
Failed to pull image "bappa/posts:0.0.1": rpc error: code = Unknown desc = Error response from daemon: pull access denied for bappa/posts, repository does not exist or may require 'docker login'
That means either you have posts image in your PRIVATE bappa repository, or you use non-exist image at all. So if this is your private repo - you should be authorized.
Maybe you wanted to use cleptes/posts:0.01 ?
apiVersion: v1
kind: Pod
metadata:
name: posts
spec:
containers:
- name: posts
image: cleptes/posts:0.01
kubectl get pods posts
NAME READY STATUS RESTARTS AGE
posts 1/1 Running 0 26m10s
kubectl describe pod posts
Normal Pulling 20s kubelet Pulling image "cleptes/posts:0.01"
Normal Pulled 13s kubelet Successfully pulled image "cleptes/posts:0.01"
Normal Created 13s kubelet Created container posts
Normal Started 12s kubelet Started container posts
Basically ErrImagePull means kubernetes is unable to locate the image, bappa/posts:0.0.1 This could either be the registry settings are not correct in the worker nodes or your image name or tags are not correct.
Just like #Henry explained issue a 'kubectl describe pod posts and inspect (and share) the error messages.
If you are using private repository you need to be authorized. If you are authorized and you can't reach the repository I think it might be related you using free account on docker hub and you have more private repositories than one which is for free. If you try to push your repository again you should get an error 'denied: requested access to the resource is denied'.
If you make your repository public it should solve your issue.
I'm deploying my app hosted on Gitlab registry on Kubernetes.
From Gitlab, I generated an access token and created a secret
kubectl create secret docker-registry gitlab-registry \
--docker-server=registry.gitlab.com \
--docker-username=kosted \
--docker-password=HERE-MY-SECRET \
--docker-email=HERE-MY-MAIL
They are all valid.
But when I deploy my app, I got this error :
Warning Failed 17s (x2 over 34s) kubelet, gke-gara-cluster-default-pool-affe594e-hslk Failed to pull image "registry.gitlab.com/gara-project/back-end-micro-services/mebooks:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.gitlab.com/v2/gara-project/back-end-micro-services/mebooks/manifests/latest: denied: access forbidden
Warning Failed 17s (x2 over 34s) kubelet, gke-gara-cluster-default-pool-affe594e-hslk Error: ErrImagePull
Normal BackOff 5s (x3 over 33s) kubelet, gke-gara-cluster-default-pool-affe594e-hslk Back-off pulling image "registry.gitlab.com/gara-project/back-end-micro-services/mebooks:latest"
Warning Failed 5s (x3 over 33s) kubelet, gke-gara-cluster-default-pool-affe594e-hslk Error: ImagePullBackOff
I know that everything works fine, because when I change the access token by my password, kubernetes can access the registry and download the docker image, but I don't want to pu in clear my password.
Please, how could I do ?
Thanks in advance.
Hello Guys hope you well!
I need the that my master machine order the slave to pull the image from my docker hub repo and I get the error below, It doesn't let the slave pull from the repo, but when I go to the slave, manually pull he pulls
This from kubernetes master:
The first lines are a describe from pod my-app-6c99bd7b9c-dqd6l which is running now because I pulled manually the image from the docker hub, but I want Kubernetes to do it.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/my-app2-74969ddd4f-l6d6l to kubeslave.machine.pt
Normal SandboxChanged <invalid> kubelet, kubeslave.machine.pt Pod sandbox changed, it will be killed and re-created.
Warning Failed <invalid> (x3 over <invalid>) kubelet, kubeslave.machine.pt Failed to pull image "bedjase/repository/my-java-app:my-java-app": rpc error: code = Unknown desc = Error response from daemon: pull access denied for bedjase/repository/my-java-app, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed <invalid> (x3 over <invalid>) kubelet, kubeslave.machine.pt Error: ErrImagePull
Normal BackOff <invalid> (x7 over <invalid>) kubelet, kubeslave.machine.pt Back-off pulling image "bedjase/repository/my-java-app:my-java-app"
Warning Failed <invalid> (x7 over <invalid>) kubelet, kubeslave.machine.pt Error: ImagePullBackOff
Normal Pulling <invalid> (x4 over <invalid>) kubelet, kubeslave.machine.pt Pulling image "bedjase/repository/my-java-app:my-java-app"
[root#kubernetes ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-app-6c99bd7b9c-dqd6l 1/1 Running 0 14m
my-app2-74969ddd4f-l6d6l 0/1 ImagePullBackOff 0 2m20s
nginx-86c57db685-bxkpl 1/1 Running 0 8h
This from slave:
[root#kubeslave docker]# docker pull bedjase/repository:my-java-app
my-java-app: Pulling from bedjase/repository
50e431f79093: Already exists
dd8c6d374ea5: Already exists
c85513200d84: Already exists
55769680e827: Already exists
e27ce2095ec2: Already exists
5943eea6cb7c: Already exists
3ed8ceae72a6: Already exists
7ba151cdc926: Already exists
Digest: sha256:c765d09bdda42a4ab682b00f572fdfc4bbcec0b297e9f7716b3e3dbd756ba4f8
Status: Downloaded newer image for bedjase/repository:my-java-app
docker.io/bedjase/repository:my-java-app
I already made the login in both master and slave to docker hub repo and succeed.
Both have /etc/hosts ok, also nodes are connected and ready:
[root#kubernetes ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubernetes.machine.pt Ready master 26h v1.17.4
kubeslave.machine.pt Ready <none> 26h v1.17.4
Am I missing some point here?
For private images you must create a secret with username and password of Docker Hub to Kubernetes be able to pull the image.
The command bellow create a secret name regcred with your Docker Hub credentials, replace the fields <<your-name>>, <your-password> and <your-email>:
kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=<your-name> --docker-password=<your-password> --docker-email=<your-email>
After that you need to add in your pod/deployment spec that you want to use this credentials to pull your private image adding the imagePullSecrets with the credentials created above, see this example:
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: regcred
References:
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-pod-that-uses-your-secret
Just to add to the other answers,
1) Create a secret with the following command:
Create a secret for pulling docker images
2) Create your pod that uses this secret as described here:
use the secret in pod
A detailed script to create the secret and another script to patch all the service accounts can be found in my answer here:
How to pull image from dockerhub in kubernetes?
Patching all the service accounts will allow all your k8s namespaces to pull any image from dockerhub without changing the k8s deploy manifests.