Failed to pull image - unauthorized: authentication required (ImagePullBackOff ) - azure-aks

Question

My release pipeline runs successfully and creates a container in Azure Kubernetes, however when I view in azure Portal>Kubernetes service> Insights screen, it shows a failure.
It fails to pull the image from my private container repository with error message 'ImagePullBackOff'
I did a kubectl describe on the pod and got below error message:
Failed to pull image "myexampleacr.azurecr.io/myacr:13": [rpc error: code = Unknown desc = Error response from daemon: Get https://myexampleacr.azurecr.io/v2/myacr/manifests/53: unauthorized: authentication required.
Below is a brief background on my setup:
I am using Kubernetes secret to access the containers in private container registry.
I generated the Kubernetes secret using clientId and password(secret) from the Service Principle that my DevOps team created.
.
The command used to generate kubernetes secret:
kubectl create secret docker-registry acr-auth --docker-server --docker-username --docker-password --docker-email
I then updated my deployment.yaml with imagePullSecrets: name:acr-auth
After this, I ran my deployment and release pipeline both ran successfully, but they show failure in the kubernetes service with error message 'ImagePullBackOff' error.
Any help will be much appreciated.

Answer 1

As the error shows it required authentication. As I see from your description, the possible reason is that your team does not assign the ACR role to the service principal that your team creates, or you use the wrong service principal. So you need to check two things:
If the service principal you use has the right permission of the ACR.
If the Kubernetes secret was created right in the Kubernetes service.
The way to check if the service principal has the right permission of the ACR is that pull an image in the ACR after you log in with the service principal in docker server. Also, as the comment said, you need to make sure the command is right as below:
kubectl create secret docker-registry acr-auth --docker-server myexampleacr.azurecr.io --docker-username clientId --docker-password password --docker-email yourEmail
Additional, there is a little possibility that you use the wrong image with tag. By the way, check it out.

Answer 2

I had the same error, and I realised that the service principal is expired.
To check the expiration date of your service principal and update your AKS cluster with the new credentials, fallow the following steps:
NOTE: You need the Azure CLI version 2.0.65 or later installed and configured.
1- Get the Client ID of your cluster using the az aks show command.
az aks show --resource-group YOUR_AKS_RESOURCE_GROUP_NAME --name YOUR_AKS_CLUSTER_NAME --query "servicePrincipalProfile.clientId"
2- Check the expiration date of your service principal.
az ad sp credential list --id YOUR_CLIENT_ID --query "[].endDate" -o tsv
If the service principal is expired then, to reset the existing service principal credential fallow the following steps:
1- Reset the credentials using az ad sp credential reset command.
az ad sp credential reset --name YOUR_CLIENT_ID --query password -o tsv
2- Update your AKS cluster with the new service principal credentials.
az aks update-credentials --resource-group YOUR_AKS_RESOURCE_GROUP_NAME --name YOUR_AKS_CLUSTER_NAME --reset-service-principal --service-principal YOUR_CLIENT_ID --client-secret YOUR_NEW_PASSWORD
Source: https://learn.microsoft.com/en-us/azure/aks/update-credentials

Answer 3

It's odd, maybe it shows an old deployment which you didn't delete. It may also be these; incorrect credientials, acr may not be up, image name or tag is wrong. You can also go with aks-acr native authentication and never use a secret: https://learn.microsoft.com/en-gb/azure/container-registry/container-registry-auth-aks

Answer 4

In my case the problem was that my --docker-password had an special character and I was not escaping it using quotes (i.e. --docker-password 'myPwd$')
You can check your password is correct my executing this command:
kubectl get secret < SECRET > -n < NAMESPACE> --output="jsonpath={.data..dockerconfigjson}" | base64 --decode
Reference: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/