I am trying to pull an image locally from a gitlab repository.
The yaml file looks like this:
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: tester
image: registry.gitlab.com/<my-project>/<components>
imagePullPolicy: Always
securityContext:
privileged: true
imagePullSecrets:
- name: my-token
---
apiVersion: v1
data:
.dockerconfigjson: <my-key>
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
name: my-token
labels:
app: tester
Then I execute: kubectl apply -f pullImage.yaml
The kubectl describe pod private-reg returns:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m1s default-scheduler Successfully assigned default/private-reg to antonis-dell
Normal Pulled 6m46s kubelet Successfully pulled image "registry.gitlab.com/<my-project>/<components>" in 1m14.136699854s
Normal Pulled 6m43s kubelet Successfully pulled image "registry.gitlab.com/<my-project>/<components>" in 1.808412857s
Normal Pulled 6m27s kubelet Successfully pulled image "registry.gitlab.com/<my-project>/<components>" in 3.046153429s
Normal Pulled 5m56s kubelet Successfully pulled image "registry.gitlab.com/<my-project>/<components>" in 4.143342874s
Normal Created 5m56s (x4 over 6m46s) kubelet Created container ches
Normal Started 5m56s (x4 over 6m46s) kubelet Started container ches
Normal Pulling 5m16s (x5 over 8m1s) kubelet Pulling image "registry.gitlab.com/<my-project>/<components>"
Normal Pulled 5m13s kubelet Successfully pulled image "regregistry.gitlab.com/<my-project>/<components>" in 2.783360345s
Warning BackOff 2m54s (x19 over 6m42s) kubelet Back-off restarting failed container
However I cannot find the image locally.
The docker image ls returns:
REPOSITORY TAG IMAGE ID CREATED SIZE
moby/buildkit buildx-stable-1 440639846006 6 days ago 142MB
registry 2 1fd8e1b0bb7e 12 months ago 26.2MB
I excpect that image registry.gitlab.com/<my-project>/<components> would be there.
Am I missing something here?
Related
How do you deploy from within Kubernetes container - using CI/CD?
Senario:
I am building within Kubernetes using Kaniko
Now how to run kubectl within Kuberneters. And I do have the right serviceAccount for it. First problem is to have a container ready for executing kubectl.
Note: - /bin/cat
I found this, but it give errors:
apiVersion: v1
kind: Pod
metadata:
name: kubectl-deploy
spec:
containers:
- name: kubectl
image: bitnami/kubectl:latest
imagePullPolicy: Always
command:
- /bin/cat
tty: true
Errors:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 78s default-scheduler Successfully assigned default/kubectl-deploy to master
Normal Pulled 76s kubelet Successfully pulled image "bitnami/kubectl:latest" in 874.059036ms
Normal Pulled 74s kubelet Successfully pulled image "bitnami/kubectl:latest" in 860.59161ms
Normal Pulled 60s kubelet Successfully pulled image "bitnami/kubectl:latest" in 859.31958ms
Normal Pulling 33s (x4 over 77s) kubelet Pulling image "bitnami/kubectl:latest"
Normal Created 32s (x4 over 76s) kubelet Created container kubectl
Normal Started 32s (x4 over 76s) kubelet Started container kubectl
Normal Pulled 32s kubelet Successfully pulled image "bitnami/kubectl:latest" in 849.398179ms
Warning BackOff 7s (x7 over 73s) kubelet Back-off restarting failed container
I found this, but it give errors
When you run a Pod in Kubernetes, by default, it expect it to be a long running service. But in your case, you run a one-off command that terminates immediately. To run one-off commands in Kubernetes, it is easiest to run them as Kubernetes Jobs.
First problem is to have a container ready for executing kubectl.
Since you are using Tekton, have a look at the "deploy task" from Tekton Hub, it is configured with an image that includes kubectl.
I have a private Docker Hub registry with a (rather large) image in it that I control.
I also have a Helm deployment chart that specifies an imagePullSecret, after having followed the instructions here https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/.
No matter what I do, though, when installing the Helm chart, I always end up with the following (taken from kubectl describe pod <pod-id>):
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 26m default-scheduler Successfully assigned default/<release>-69584657b7-vkps6 to <node>
Warning Failed 6m28s (x3 over 20m) kubelet Failed to pull image "<registry-username>/<image>:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/<registry-username>/<image>:latest": failed to copy: httpReadSeeker: failed open: server message: invalid_token: authorization failed
Warning Failed 6m28s (x3 over 20m) kubelet Error: ErrImagePull
Normal BackOff 5m50s (x5 over 20m) kubelet Back-off pulling image "<registry-username>/<image>:latest"
Warning Failed 5m50s (x5 over 20m) kubelet Error: ImagePullBackOff
Normal Pulling 5m39s (x4 over 26m) kubelet Pulling image "<registry-username>/<image>:latest"
I have looked high and low on the internet for answers pertaining to this invalid_token output, but have yet to find anything concrete.
I have verified that I can run docker pull manually with the image in question both on the K8s node as well as other boxes. It works just fine.
I have tried using docker.io as the repository URI, as well as (the recommended) https://index.docker.io/v1/.
I have tried using my own Docker Hub password as well as a generated Personal Access Token (I can actually see in Docker Hub that the PAT was, in fact, used, despite the pull failing).
I've examined the secrets via kubectl to verify they're of the expected format and contain the correct data (username, password/token, etc.). They're all fine and match what I'd get when I run docker login on the command line.
I have used this node to deploy other releases via Helm and they have all worked fine (although at least one has been from a different registry).
I am relatively new to K8s and Helm, but I've used Docker for a long while now and I'm at a loss as to this invalid_token issue.
Any help would be greatly appreciated.
Thank you in advance.
UPDATE
Here's the (sanitized) output of helm template:
---
# Source: <deployment>/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: release-name-<deployment>
labels:
helm.sh/chart: <deployment>-0.1.0
app.kubernetes.io/name: <deployment>
app.kubernetes.io/instance: release-name
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: <deployment>
app.kubernetes.io/instance: release-name
template:
metadata:
labels:
app.kubernetes.io/name: <deployment>
app.kubernetes.io/instance: release-name
spec:
imagePullSecrets:
- name: regcred-docker-pat
securityContext:
{}
containers:
- name: <deployment>
securityContext:
{}
image: "<registry-username>/<image>:latest"
imagePullPolicy: IfNotPresent
resources:
{}
I've also confirmed that any secrets I have tried are, in fact, in the same namespace as the pod (in this case, the default namespace).
Is the imagepullsecret created by the helm chart?
Is the imagepullsecret available when the deployment is created?
Do you apply the deployment before the imagepullsecret is available?
I remember the order matters when applying the imagepullsecret; the kube-api does not retry pulling after failure because of authentication.
Docker Image: -
docker images | grep -i "gcc"
gcc-docker latest 84c4359e6fc9 21 mites ago 1.37GB
docker run -it gcc-docker:latest
hello,world
Kubernetes pod deployed:-
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/hello-world to master-node
Normal Pulling 4s kubelet, master-node Pulling image "gcc-docker:latest"
Warning Failed 0s kubelet, master-node Failed to pull image "gcc-docker:latest": rpc error: code = Unknown desc = Erroresponse from daemon: pull access denied for gcc-docker, repository does not exist or may require 'docker login': denied: requested acce to the resource is denied
Warning Failed 0s kubelet, master-node Error: ErrImagePull
Normal BackOff 0s kubelet, master-node Back-off pulling image "gcc-docker:latest"
Warning Failed 0s kubelet, master-node Error: ImagePullBackOff
-->yaml files used to deploy pod
apiVersion: v1
kind: Pod
metadata:
name: hello-world
labels:
type: hello-world
spec:
containers:
- name: hello-world
image: gcc-docker:latest
command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 60']
ports:
- containerPort: 80
I tried pulling gcc-docker and got the same error.You may have this image present on your system already and now its not on dockerhub.
if you know the repository for this image, try to use the same and for authentication create secrets of docker type and use them as image pull secrets.
Also, one more thing you are running the container on the master node, and I assume it's minikube or some local setup.
Minikube uses a dedicated VM to run Kubernetes which is not the same as the machine on which you have installed minikube.
So images available on your laptop will not be available to minikube.
I tried to deploy 'deployments' in kubernetes which is pull docker image from private registry (I don't know who did this setup) but during "docker pull images" through kubernetes i'm getting following error
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 85s default-scheduler Successfully assigned default/trusted-enc-assettag1-deployment-8467b74958-6fbp7 to k8s-node
Normal BackOff 24s (x2 over 61s) kubelet, k8s-node Back-off pulling image "10.105.168.81:5000/simplehttpserverenc:enc_v1"
Warning Failed 24s (x2 over 61s) kubelet, k8s-node Error: ImagePullBackOff
Normal Pulling 12s (x3 over 82s) kubelet, k8s-node Pulling image "10.105.168.81:5000/simplehttpserverenc:enc_v1"
Warning Failed 0s (x3 over 62s) kubelet, k8s-node Failed to pull image "10.105.168.81:5000/simplehttpserverenc:enc_v1": rpc error: code = Unknown desc = Error response from daemon: Get https://10.105.168.81:5000/v2/: net/http: TLS handshake timeout
Warning Failed 0s (x3 over 62s) kubelet, k8s-node Error: ErrImagePull
[root#k8s-master ~]# docker pull 10.105.168.81:5000/simplehttpserverenc:enc_v1
ImagePullBackOff and net/http: TLS handshake timeout error.
Initially this "net/http: TLS handshake timeout" error is observed in docker pull as well. I referred some answers and
configured certificate(/etc/docker/certs.d//ca.crt ) and
proxy (/etc/systemd/system/docker.service.d/proxy.conf)
after that able to perform docker pull from private image.
[root#k8s-master ~]# docker pull 10.105.168.81:5000/simplehttpserverenc:enc_v1
enc_v1: Pulling from simplehttpserverenc
54fec2fa59d0: Pull complete
cd3f35d84cab: Pull complete
a0afc8e92ef0: Pull complete
9691f23efdb7: Pull complete
6512e60b314b: Pull complete
a8ac6632d329: Pull complete
68f4c4e0aa8c: Pull complete
Digest: sha256:0358708cd11e96f6cf6f22b29d46a8eec50d7107597b866e1616a73a198fe797
Status: Downloaded newer image for 10.105.168.81:5000/simplehttpserverenc:enc_v1
10.105.168.81:5000/simplehttpserverenc:enc_v1
[root#k8s-master ~]#
But still unable to perform this docker pull through kubernetes. How to solve this ?
If you use docker as container engine in your k8s, AFAIK it's the same with Understand the configuration. Because the image pulling is conducted by the container engine and it depends the proprietary configuration of each one on the certificates. How about pulling the same image on the worker node in your k8s ? Is it possible to pull the one without errors ?
As your dockerconfigjson is not working properly. Try this method :
kubectl create secret docker-registry regcred --docker-server=10.105.168.81:5000 --docker-username=<your-name> --docker-password=<your-pword>
And in Kubernetes manifest :
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: 10.105.168.81:5000/simplehttpserverenc:enc_v1
imagePullSecrets:
- name: regcred
I had encounted this many times, when I forgot to configure these secrets. Also if you have any othernamespace, you will have to generate secrets for each of these namespaces separately passing -n <your-ns> to above kubectl create secret
Edit : As you can not pull the image from worker node.
Make sure you copied docker-registry ca.crt to /etc/docker/certs.d/ca.crt
and then try docker pull .
I've been trying to create a custom helm chart however I get ErrImagePull no matter what image I add to my chart, I can re-create this really easily:
helm create my-chart
(using default nginx docker image):
helm install my-chart .
NAME: my-chart
LAST DEPLOYED: Fri Jan 17 12:26:13 2020
NAMESPACE: example
STATUS: deployed
REVISION: 1
NOTES:
Change values.yaml for a different image (nginx -> ubuntu):
7 image:
8 repository: ubuntu
9 pullPolicy: IfNotPresent
update helm deployment:
helm upgrade my-chart .
Release "my-chart" has been upgraded. Happy Helming!
NAME: my-chart
LAST DEPLOYED: Fri Jan 17 12:30:13 2020
NAMESPACE: example
STATUS: deployed
REVISION: 2
NOTES:
Pod status:
kubectl get pods
NAME READY STATUS RESTARTS AGE
my-chart-54fb9969dd-gnpt9 0/1 ImagePullBackOff 0 32s
my-chart-56485d7b7-hc25q 1/1 Running 0 4m32s
Describe pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned example/my-chart-54fb9969dd-gnpt9 to aw
Normal Pulling 15s (x3 over 62s) kubelet, aw Pulling image "ubuntu:1.16.0"
Warning Failed 13s (x3 over 59s) kubelet, aw Failed to pull image "ubuntu:1.16.0": rpc error: code = Unknown desc = failed to resolve image "docker.io/library/ubuntu:1.16.0": no available registry endpoint: docker.io/library/ubuntu:1.16.0 not found
Warning Failed 13s (x3 over 59s) kubelet, aw Error: ErrImagePull
Normal BackOff 1s (x3 over 58s) kubelet, aw Back-off pulling image "ubuntu:1.16.0"
Warning Failed 1s (x3 over 58s) kubelet, aw Error: ImagePullBackOff
The issue is caused by the fact that the helm template defaults with the chart.appversion
image: "{{ .Values.image.repository }}:{{ .Chart.AppVersion }}"
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
when you helm create my-chart go to the deployments.yaml and change the image: section to either use the tag variable, then from values.yaml add something like this:
8 repository: ubuntu
9 tag: latest
10 pullPolicy: IfNotPresent