k3s pods are not mounting secrets defined in helm deployment's imagePullSecrets - docker

Question

I am creating a deployment in circlci that deploys my containerized application to a k3s server I have set up. I have set up a secret using the commands found here.
The secret is created using the command:
kubectl create secret docker-registry regkeyname --docker-server=https://index.docker.io/v1/ \
--docker-username=username \
--docker-password=password \
--docker-email=my#email.com \
--namespace=external
My secret is as follows when running kubectl get secret regkeyname --namespace=external --output=yaml:
apiVersion: v1
data:
.dockerconfigjson: secretbase64stuff
kind: Secret
metadata:
creationTimestamp: "2020-11-24T13:11:07Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:.dockerconfigjson: {}
f:type: {}
manager: kubectl
operation: Update
time: "2020-11-24T13:11:07Z"
name: regkeyname
namespace: external
resourceVersion: "16929381"
selfLink: /api/v1/namespaces/external/secrets/regkeyname
uid: 51b87508-9cf2-490b-b871-0b5a342ab64c
type: kubernetes.io/dockerconfigjson
I'm using helm to deploy my application and the Deployment looks like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.labels.app }}
labels:
app: {{ .Values.labels.app }}
spec:
selector:
matchLabels:
app: {{ .Values.labels.app }}
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: {{ .Values.labels.app }}
env: {{ .Values.labels.env }}
spec:
imagePullSecrets:
- name: regkeyname
containers:
- name: my-service
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ .Values.image.imagePullPolicy }}
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 5
successThreshold: 1
after deploying however, the images fail to pull and it appears that my secret "regkeyname" is not used/mounted in the pods. the result is as follows:
Name: my-service-856454c6cd-qcp7w
Namespace: external
Priority: 0
Node: worker-2/192.168.1.13
Start Time: Tue, 24 Nov 2020 07:20:08 -0600
Labels: app=my-service
env=development
pod-template-hash=856454c6cd
Annotations: <none>
Status: Pending
IP: 10.42.2.196
Controlled By: ReplicaSet/my-service-856454c6cd
Containers:
auth-service:
Container ID:
Image: my-repo/my-service:latest
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Readiness: http-get http://:8080/health delay=10s timeout=1s period=10s #success=1 #failure=5
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l9b4k (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-l9b4k:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-l9b4k
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned external/auth-service-856454c6cd-qcp7w to worker-2
Normal Pulling 32m (x4 over 34m) kubelet, worker-2 Pulling image "my-repo/my-service:latest"
Warning Failed 32m (x4 over 34m) kubelet, worker-2 Failed to pull image "my-repo/my-service:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/my-repo/my-service:latest": failed to resolve reference "docker.io/my-repo/my-service:latest": failed to do request: Head https://registry-1.docker.io/v2/my-repo/my-service/manifests/latest: dial tcp: lookup registry-1.docker.io: Try again
Warning Failed 32m (x4 over 34m) kubelet, worker-2 Error: ErrImagePull
Warning Failed 31m (x6 over 34m) kubelet, worker-2 Error: ImagePullBackOff
Normal BackOff 3m54s (x127 over 34m) kubelet, worker-2 Back-off pulling image "my-repo/my-service:latest"
I had this working when running locally with kubernetes so I am assuming the issue must have something to do either with k3s or the fact that now the server is remote rather than local. Any insight would be greatly appreciated. Thanks in advance!

Answer 1

The controller is trying to pull image from the official docker registry:
failed to resolve reference "docker.io/my-repo/my-service:latest"
While creating the imagePullSecret, make sure that you put the correct URL (ie. the URL for your private registry) for performing authentication and pulling image.
$ cat ~/.docker/config.json
{
"auths": {
"https://index.docker.io/v1/": { # <------ change here
"auth": "..........="
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/19.03.5 (linux)"
}