I have set up a Kubernetes cluster on gcloud via gitlab.
I have some trouble pulling my images when I deploy my application.
I use a gcloud cluster with a registry on the same gcloud project. Normally, I'm able to pull my image directly without any modification (supposed to use the Compute Engine default service account?).
But I get a unauthorized on my pod when he try to pull the image :
Warning Failed 3m (x2 over 3m) kubelet, gke-production-default-pool-********-**** Failed to pull image "eu.gcr.io/[My-Project]/services-identity:715bfffa": rpc error: code = Unknown desc = unauthorized: authentication required
Warning Failed 3m (x2 over 3m) kubelet, gke-production-default-pool-********-**** Error: ErrImagePull
Normal BackOff 2m (x6 over 3m) kubelet, gke-production-default-pool-********-**** Back-off pulling image "eu.gcr.io/[My-Project]/services-identity:715bfffa"
Warning Failed 2m (x6 over 3m) kubelet, gke-production-default-pool-********-**** Error: ImagePullBackOff
Normal Pulling 2m (x3 over 3m) kubelet, gke-production-default-pool-********-**** pulling image "eu.gcr.io/[My-Project]/services-identity:715bfffa"
I deploy via gitlab-ci with the following command line:
helm upgrade --install services-identity -f ./deploy/env/production-values.yml ./deploy/ --set image.tag=${CI_COMMIT_SHA:0:8} --namespace=production --wait
For information, I can pull the registry when this one is public, I can also pull the image locally via a docker login(using my gcloud account).
Thanks in advance for your advice.
This is very similar to this: What's the minimal permissions I need to configure for a GKE node pool to pull from a private GCR repo in the same project?, except that you are not mentioning that it's on GKE so I assume is on GCE.
You can use a json_key_file.
On all your nodes (assuming you are using Docker):
$ docker login -u _json_key --password-stdin https://gcr.io
Or the same json_key_file using ImagePullSecrets in the pod spec as described here.
Or you can use on all your Kubernetes nodes:
$ gcloud auth configure-docker
Thanks for your helpful response. On the similar issues you have shared, I have found a solution Use Least Privilege Service Accounts for your Nodes.
The problem is on the Cloud API access scopes level, I have recreated an InstanceGroup with the right scope:
The default scopes for the nodes in GKE are devstorage.read_only,
logging.write, monitoring, service.management.readonly,
servicecontrol, and trace.append. When setting scopes, these are
specified as gke-default. If you are accessing private images in
Google Container Registry, the minimally required scopes are only
logging.write, monitoring, and devstorage.read_only.
Related
I am trying to install Containerized Private Minion but facing error in below
Using a Dockerfile, while making a custom image that allows containers created from them to establish a SSL handshake with endpoints, The error appears as below.
Here I have added command to create /etc/certs since it is not pre existing. Also I am not sure if /opt/java/openjdk/lib/security/cacerts exists on container
FROM quay.io/newrelic/synthetics-minion:latest
RUN mkdir /etc/certs
COPY Cert.pem /etc/certs
RUN keytool -noprompt -import -alias proxy -file /etc/certs/Cert.pem -keystore /opt/java/openjdk/lib/security/cacerts -storepass changeit
I used docker build minionwithcert , but the image created had no repo name & was , I manually tagged the image with a name. It looks like to me the custom image was not properly created, & either the Dockerfile is wrong or a mistake in docker run command.
Error:
Failed to pull image "minionwithcert:3.0.65": rpc error: code = Unknown desc = Error response from daemon: pull access denied for minionwithcert, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed 3m8s (x4 over 4m45s) kubelet Error: ErrImagePull
Warning Failed 2m43s (x6 over 4m44s) kubelet Error: ImagePullBackOff
Normal BackOff 2m31s (x7 over 4m44s) kubelet Back-off pulling image "minionwithcert:3.0.65"
If you want to test an image locally, use its IMAGE ID. Example:
If your image is:
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> a1510947ccde 2 minutes ago 436 MB
You can run it using:
docker run a1510947ccde
Once you tag it, docker tries to download it from a registry.
I'm using JFrog repository as my private jfog repo. And I have specified the secret in order to authenticate it. The pod fails with an ImagePullBackOff error, when I describe the pod I see
Failed to pull image "private_registry/image_name": rpc error: code =
Unknown desc = failed to pull and unpack image
"private_registry/image_name": failed to do request: Head
https://xx.xx.xx.xx:port-number/v2/<docker-registryname>/<application-name>/manifests/<tag>:
http: server gave HTTP response to HTTPS client Warning Failed
23m (x4 over 24m) kubelet, worker01 Error: ErrImagePull
when I pull the same image using docker pull , the image get pulled successfully
While having the HTTP server communicating with the HTTPS server (probably due to the usage of a self-signed certificate) being the private registry, registering the concerned registry as an insecure registry with the docker client could resolve the docker error.
{ "insecure-registries":["IP:PORT"] }
An entry similar to the above need to be included in the /etc/docker/daemon.json file and considering the environment to be in K8s, it needs to be configured on all the nodes.
I have logged in to the Docker hub using the CLI command: docker login. Entered username and password and I can pull and push images to Docker hub.
However, my K8S can't. I am trying to apply a deployment that should pull those images into its pods but I got the following error when running kubectl describe pod POD_NAME:
Warning Failed 9s kubelet Failed to pull image "myprivate/repo:tag": rpc error: code = Unknown desc = Error response from daemon: pull access denied for myprivate/repo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
How to make the docker run in the pods to also be logged to the docker hub as well as doing it from my terminal?
Create "image pull secret" and define on your deployment. Here is how you can do https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
So I have this problem that no matter what I try I get the error below. I know that there are numerous things on this same issue but I have tried everything I see there.
So here is my setup.
Vagrant -> creates 4 nodes (1 Master, 3 nodes) -> puts kubernetes on each then I performed the same thing on all 4 machines just to see if it worked.
document login (successful to my dockerhub account)
docker pull nginx
docker tag nginx nginxy (notice the y)
On Master I run my ngin-deployment.yaml file with image: nginxy
Below is the error EVERYTIME! The funny thing is it only happens on images that I tag. if I use just "nginx" it works. Notice that I can see the tag on all machines and I login. So I believe I meet the requirements of the ERROR.
I thought that only master should have the images but put it in all nodes after nothing worked.
Normal Scheduled <unknown> default-scheduler Successfully assigned default/gyrex- deployment-58f44bf77f-9h6tg to node2
Normal Pulling 2m34s (x4 over 4m11s) kubelet, node2 Pulling image "nginxy:latest"
Warning Failed 2m33s (x4 over 4m10s) kubelet, node2 Failed to pull image "nginxy:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for nginxy, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed 2m33s (x4 over 4m10s) kubelet, node2 Error: ErrImagePull
I went thru this tutorial completely but nothing works.
https://kubernetes.io/docs/concepts/containers/images/#configuring-nodes-to-authenticate-to-a-private-repository
Try adding the imagePullPolicy: Never to the spec section in your container spec in the YAML file.
I'm trying to install Tremolo's OpenUnison Orchestra login portal -
https://github.com/OpenUnison/openunison-k8s-login-activedirectory. However, when I'm running the installation command -
curl https://raw.githubusercontent.com/TremoloSecurity/kubernetes-artifact-deployment/master/src/main/bash/deploy_openunison.sh | bash -s /path/to/orchestra-configmaps /path/to/orchestra-secrets https://raw.githubusercontent.com/OpenUnison/openunison-k8s-login-activedirectory/master/src/main/yaml/artifact-deployment.yaml
K8S tries to deploy docker.io/tremolosecurity/kubernetes-artifact-deployment:1.1.0
But I keep getting the following error(Using kubectl descrive pods -n openunison-deploy):
Failed to pull image "docker.io.tremolosecurity.kubernetes-artifact-deployment:1.1.0": rpc error: code = Unknown desc = context canceled
And then there is an event of Error: ErrImagePull and then Back-off pulling image.. and it goes back on.
How can I fixed it?
EDIT
I tried to simply docker pull docker.io/tremolosecurity/kubernetes-artifact-deployment:1.1.0 and it seems that there is a 100MB which takes a lot of time to download (more than 15 minutes already and not half way done) could this be the problem? As the error states that it failed pulling the image from docker.io...
As mentioned above by community mchawre and Amit Kumar Gupta:
1. For --image-pull-progress-deadline please verify your service by systemctl status kubelet, add option in the section ExecStart=--image-pull-progress-deadline=10m into your kubelet.service and run:
sudo systemctl daemon-reload
sudo systemctl restart restart kubelet
verify kubelet flags by running:
journalctl -u kubelet | grep flag
2. In case you can pull image using docker please setup in your deployment spec:
imagePullPolicy: IfNotPresent
3. Update your docker installation the latest one and Cleanup docker resources
docker images
docker rmi <unused imgaes>
docker system prune
docker image prune -a
docker info
4. Due to other problem with your configuration please consider docker re-installation.
Hope this help.