Gitlab CI/CD using Kubernetes agent and the private container registry - docker

Question

I will preface this by saying that I know very little about how to manage Kuberentes clusters, so I am probably doing something dumb. Here goes:
I have created a Kubernetes cluster using GCP's Autopilot mode, and I (think I) registered the cluster to my Gitlab repository using the "Infrastructure->Kubernetes Clusters" menu (It shows as online).
Using Gitlab's CI/CD, I have a build stage which pushes an image to the repo's container registry (I see the the image is indeed there).
I then have a deploy stage with the following script: (censored some data for privacy)
kubectl config use-context <my_username>/<my_reponame>:<my_reponame>
kubectl delete secret regcred --ignore-not-found
kubectl create secret docker-registry regcred --docker-server="$CI_REGISTRY" --docker-username="$CI_REGISTRY_USER" --docker-password="$CI_REGISTRY_PASSWORD"
kubectl delete pod <my_reponame> --ignore-not-found
kubectl apply -f kube.yaml
My kube.yaml contains the following:
apiVersion: v1
kind: Pod
metadata:
name: <my_reponame>
spec:
containers:
- name: <my_reponame>
image: registry.gitlab.com/<my_username>/<my_reponame>:latest
imagePullSecrets:
- name: regcred
I also have a .gitlab/agents/<my_reponame>/config.yaml file with the following:
ci_access:
projects:
- id: <my_username>/<my_reponame>
This is the output of the build stage:
Running with gitlab-runner 14.8.0~beta.44.g57df0d52 (57df0d52)
on blue-2.shared.runners-manager.gitlab.com/default XxUrkriX
Preparing the "docker+machine" executor
00:12
Using Docker executor with image bitnami/kubectl:latest ...
Pulling docker image bitnami/kubectl:latest ...
Using docker image sha256:208d070e071a0165e48ad7bf20b30c054328bcaaad76b0c53a9270a5e8627480 for bitnami/kubectl:latest with digest bitnami/kubectl#sha256:51eb9cb7d811e74bba30f97700cb433424d4025aabe70e8ff80e1289a964ab9c ...
Preparing environment
00:01
Running on runner-xxurkrix-project-32365648-concurrent-0 via runner-xxurkrix-shared-1646113353-c926e727...
Getting source from Git repository
00:02
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/<my_username>/<my_reponame>/.git/
Created fresh repository.
Checking out 23743f89 as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:13
Using docker image sha256:208d070e071a0165e48ad7bf20b30c054328bcaaad76b0c53a9270a5e8627480 for bitnami/kubectl:latest with digest bitnami/kubectl#sha256:51eb9cb7d811e74bba30f97700cb433424d4025aabe70e8ff80e1289a964ab9c ...
$ kubectl config use-context <my_username>/<my_reponame>:<my_reponame>
Switched to context "<my_username>/<my_reponame>:<my_reponame>".
$ kubectl delete secret regcred --ignore-not-found
secret "regcred" deleted
$ kubectl create secret docker-registry regcred --docker-server="$CI_REGISTRY" --docker-username="$CI_REGISTRY_USER" --docker-password="$CI_REGISTRY_PASSWORD"
secret/regcred created
$ kubectl delete pod <my_reponame> --ignore-not-found
$ kubectl apply -f kube.yaml
Warning: Autopilot set default resource requests for Pod default/<my_reponame>, as resource requests were not specified. See http://g.co/gke/autopilot-defaults.
pod/<my_reponame> created
Cleaning up project directory and file based variables
00:00
Job succeeded
However, the deploy to the cluster did not actually work.
Here is some data from running in the gcloud shell
<my_username>#cloudshell:~ (<my_reponame>-342620)$ kubectl get pods
NAME READY STATUS RESTARTS AGE
<my_reponame> 0/1 ImagePullBackOff 0 13h
<my_reponame>-69775f4b-cft8w 0/1 ImagePullBackOff 0 13h
kubectl get pod <my_reponame> -o yaml`
Name: <my_reponame>
Namespace: default
Priority: 0
Node: gk3-<my_reponame>-nap-73l1ao51-a8e3ee2d-wjd8/10.132.0.5
Start Time: Tue, 01 Mar 2022 19:11:18 +0000
Labels: <none>
Annotations: autopilot.gke.io/resource-adjustment:
{"input":{"containers":[{"name":"<my_reponame>"}]},"output":{"containers":[{"limits":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"2Gi"},"requ...
seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status: Pending
IP: 10.34.0.77
IPs:
IP: 10.34.0.77
Containers:
<my_reponame>:
Container ID:
Image: registry.gitlab.com/<my_username>/<my_reponame>:latest
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Limits:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
Requests:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-96wsd (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-96wsd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17s gke.io/optimize-utilization-scheduler Successfully assigned default/<my_reponame> to gk3-<my_reponame>-nap-73l1ao51-a8e3ee2d-wjd8
Warning Failed 14s kubelet Failed to pull image "registry.gitlab.com/<my_username>/<my_reponame>:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/<my_username>/<my_reponame>:latest": failed to copy: httpReaderSeeker: failed open: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
Normal BackOff 14s kubelet Back-off pulling image "registry.gitlab.com/<my_username>/<my_reponame>:latest"
Warning Failed 14s kubelet Error: ImagePullBackOff
Normal Pulling 2s (x2 over 16s) kubelet Pulling image "registry.gitlab.com/<my_username>/<my_reponame>:latest"
Warning Failed 1s (x2 over 14s) kubelet Error: ErrImagePull
Warning Failed 1s kubelet Failed to pull image "registry.gitlab.com/<my_username>/<my_reponame>:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/<my_username>/<my_reponame>:latest": failed to resolve reference "registry.gitlab.com/<my_username>/<my_reponame>:latest": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
kubectl get pod <my_reponame> -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
autopilot.gke.io/resource-adjustment: '{"input":{"containers":[{"name":"<my_reponame>"}]},"output":{"containers":[{"limits":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"2Gi"},"requests":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"2Gi"},"name":"<my_reponame>"}]},"modified":true}'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"<my_reponame>","namespace":"default"},"spec":{"containers":[{"image":"registry.gitlab.com/<my_username>/<my_reponame>:latest","name":"<my_reponame>"}],"imagePullSecrets":[{"name":"regcred"}]}}
seccomp.security.alpha.kubernetes.io/pod: runtime/default
creationTimestamp: "2022-03-01T19:11:18Z"
name: <my_reponame>
namespace: default
resourceVersion: "1299911"
uid: d8a636ee-1dfc-40e9-9ba4-26b5674d3259
spec:
containers:
- image: registry.gitlab.com/<my_username>/<my_reponame>:latest
imagePullPolicy: Always
name: <my_reponame>
resources:
limits:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
requests:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
securityContext:
capabilities:
drop:
- NET_RAW
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-96wsd
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
- name: regcred
nodeName: gk3-<my_reponame>-nap-73l1ao51-a8e3ee2d-wjd8
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: gke.io/optimize-utilization-scheduler
securityContext:
seccompProfile:
type: RuntimeDefault
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-96wsd
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-03-01T19:11:18Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-03-01T19:11:18Z"
message: 'containers with unready status: [<my_reponame>]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-03-01T19:11:18Z"
message: 'containers with unready status: [<my_reponame>]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-03-01T19:11:18Z"
status: "True"
type: PodScheduled
containerStatuses:
- image: registry.gitlab.com/<my_username>/<my_reponame>:latest
imageID: ""
lastState: {}
name: <my_reponame>
ready: false
restartCount: 0
started: false
state:
waiting:
message: 'rpc error: code = Unknown desc = failed to pull and unpack image
"registry.gitlab.com/<my_username>/<my_reponame>:latest": failed to resolve reference
"registry.gitlab.com/<my_username>/<my_reponame>:latest": failed to authorize: failed
to fetch oauth token: unexpected status: 401 Unauthorized'
reason: ErrImagePull
hostIP: 10.132.0.5
phase: Pending
podIP: 10.34.0.77
podIPs:
- ip: 10.34.0.77
qosClass: Guaranteed
startTime: "2022-03-01T19:11:18Z"
It looks like from the last commands log that it was not able to pull from the container registry - I assume I passed the credentials wrong, but I could not find any example or documentation on how to achieve this.
I'm of course willing to give any info if necessary.
Thanks in advance :)

Answer 1

Try https://chris-vermeulen.com/using-gitlab-registry-with-kubernetes/, your cluster cannot pull from the registry because it has no access.