How to make kubernetes docker secret work? - docker

Question

Data showing "xxx" has been masked.
Problem description:
Success Scenario: When i make my image public in docker registry, my pod is getting created successfully.
Failure Scenario: When i make my image private in docker registry. My image pull fails on kubernetes cluster.
Please details below and help.
I have my image published to docker registry.
Following is my kubernetes secret:
c:\xxxxxxx\temp>kubectl get secret regcredx -o yaml
apiVersion: v1
data:
.dockerconfigjson: xxxxxx
kind: Secret
metadata:
creationTimestamp: 2018-10-25T21:38:18Z
name: regcredx
namespace: default
resourceVersion: "1174545"
selfLink: /api/v1/namespaces/default/secrets/regcredx
uid: 49a71ba5-d89e-11e8-8bd2-005056b7126c
type: kubernetes.io/dockerconfigjson
Here is my pod.yaml file:
apiVersion: v1
kind: Pod
metadata:
name: whatever
spec:
containers:
- name: whatever
image: xxxxxxxxx/xxxxxx:123
imagePullPolicy: Always
command: [ "sh", "-c", "tail -f /dev/null" ]
imagePullSecrets:
- name: regcredx
Here is my pod config in cluster:
c:\Sharief\temp>kubectl get pod whatever -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/podIP: 100.96.1.81/32
creationTimestamp: 2018-10-26T20:49:11Z
name: whatever
namespace: default
resourceVersion: "1302024"
selfLink: /api/v1/namespaces/default/pods/whatever
uid: 9783b81f-d960-11e8-94ca-005056b7126c
spec:
containers:
- command:
- sh
- -c
- tail -f /dev/null
image: xxxxxxxxx/xxxxxxx:123
imagePullPolicy: Always
name: whatever
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-4db4c
readOnly: true
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: regcredx
nodeName: xxxx-pvt
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-4db4c
secret:
defaultMode: 420
secretName: default-token-4db4c
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2018-10-26T20:49:33Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2018-10-26T20:49:33Z
message: 'containers with unready status: [whatever]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2018-10-26T20:49:11Z
status: "True"
type: PodScheduled
containerStatuses:
- image: xxxxxxxxx/xxxxxxx:123
imageID: ""
lastState: {}
name: whatever
ready: false
restartCount: 0
state:
waiting:
message: Back-off pulling image "xxxxxxxxx/xxxxxxx:123"
reason: ImagePullBackOff
hostIP: xx.xxx.xx.xx
phase: Pending
podIP: xx.xx.xx.xx
qosClass: BestEffort
startTime: 2018-10-26T20:49:33Z
Here is my pod discription:
c:\xxxxxxx\temp>kubectl describe pod whatever
Name: whatever
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: co2-vmkubwrk01company-pvt/xx.xx.xx.xx
Start Time: Fri, 26 Oct 2018 15:49:33 -0500
Labels: <none>
Annotations: cni.projectcalico.org/podIP=xxx.xx.xx.xx/xx
Status: Pending
IP: xxx.xx.x.xx
Containers:
whatever:
Container ID:
Image: xxxxxxxxx/xxxxxxx:123
Image ID:
Port: <none>
Host Port: <none>
Command:
sh
-c
tail -f /dev/null
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4db4c (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-4db4c:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4db4c
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27m default-scheduler Successfully assigned whatever to xxx
Normal SuccessfulMountVolume 26m kubelet, co2-vmkubwrk01company-pvt MountVolume.SetUp succeeded for volume "default-token-4db4c"
Normal Pulling 25m (x4 over 26m) kubelet, co2-vmkubwrk01company-pvt pulling image "xxxxxxxxx/xxxxxxx:123"
Warning Failed 25m (x4 over 26m) kubelet, co2-vmkubwrk01company-pvt Failed to pull image "xxxxxxxxx/xxxxxxx:123": rpc error: code = Unknown desc = repository docker.io/xxxxxxxxx/xxxxxxx not found: does not exist or no pull access
Warning Failed 25m (x4 over 26m) kubelet, co2-vmkubwrk01company-pvt Error: ErrImagePull
Normal BackOff 16m (x41 over 26m) kubelet, co2-vmkubwrk01company-pvt Back-off pulling image "xxxxxxxxx/xxxxxxx:123"
Warning Failed 1m (x106 over 26m) kubelet, co2-vmkubwrk01company-pvt Error: ImagePullBackOff

Answer 1

Kubernetes could not find your repository , the image path is wrong , you need to fix this:
image: xxxxxxxxx/xxxxxx:123
One thing you can try to test the assumption that pre-pull the image on the node on which the deployment is going to happen. do docker images , note the correct uri/repo:tag and update it in you deployment.