Docker Swarm create service cannot pull from remote repository - docker

Question

I have created a two instance docker swarm on Google Compute Engine.
Docker version 18.06.1-ce, build e68fc7a on Ubuntu 18.04.1 LTS
I created a service account:
gcloud iam service-accounts create ${KEY_NAME} --display-name "${KEY_DISPLAY_NAME}"
gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:${KEY_NAME}#${PROJECT}.iam.gserviceaccount.com --role roles/storage.admin
gcloud iam service-accounts keys create --iam-account ${KEY_NAME}#${PROJECT}.iam.gserviceaccount.com key.json
Transferred the key.json to my docker swarm master:
Then I ran the following commands:
gcloud auth configure-docker
cat key.json | tr '\n' ' ' | docker login -u _json_key --password-stdin \
https://eu.gcr.io
I can successfully pull an image from my private eu.gcr.io repository:
docker pull eu.gcr.io/$PROJECT/$IMAGE
So, logging in seems to work and the gcloud helper seems to be properly installed.
But creating a service in my swarm fails:
docker service create --replicas 2 --network overlay --name $NAME eu.gcr.io/$PROJECT/$IMAGE --with-registry-auth
image eu.gcr.io/$PROJECT/$IMAGE:latest could not be accessed on a registry to record
its digest. Each node will access eu.gcr.io/$PROJECT/$IMAGE:latest independently,
possibly leading to different nodes running different versions of the image.
qwdm524vggn50j4lzoe5paknj
overall progress: 0 out of 2 tasks
1/2: No such image: eu.gcr.io/$PROJECT/$IMAGE:latest
2/2: No such image: eu.gcr.io/$PROJECT/$IMAGE:latest
Looking in syslog shows the following:
Aug 25 13:37:15 mgr-1 dockerd[1368]: time="2018-08-25T13:37:15.299064551Z" level=info msg="Attempting next endpoint for pull after error: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"
Aug 25 13:37:15 mgr-1 dockerd[1368]: time="2018-08-25T13:37:15.299168218Z" level=error msg="pulling image failed" error="unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication" module=node/agent/taskmanager node.id=xgozmc8iyjls7ulh4k3tvions service.id=qwdm524vggn50j4lzoe5paknj task.id=qrktpo34iuhiyl1rmbi71y4wg
AFAICS, I use the correct service account JSON to login into the Google Container Repository (as docker pull works), I added the flag --with-registry-auth to docker create service which has been the answer to similar questions, but still it doesn't work. Is docker create service working similar to docker pull?
Any ideas how I might solve this?
UPDATE
Instead of Google Container Registry I tried Gitlab Registry as well. Created a registry deploy token on the Gitlab site and entered the following commands:
docker login registry.gitlab.com -u $USERNAME -p $PASSWORD
Then this just works:
docker pull registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE
But this command fails with a similar error:
docker service create --replicas 2 --network overlay --name $NAME registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE --with-registry-auth
image registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE:latest could not be accessed on a registry to record
its digest. Each node will access registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE:latest independently,
possibly leading to different nodes running different
versions of the image.
r5fqg94jrvt587le0fu779zaw
overall progress: 0 out of 2 tasks
1/2: No such image: $ORGANISATION/$PROJECT/$IMAGE:latest
2/2: No such image: $ORGANISATION/$PROJECT/$IMAGE:latest
And /var/log/syslog contains
Aug 25 21:56:14 mgr-1 dockerd[1368]: time="2018-08-25T21:56:14.615895063Z" level=error msg="pulling image failed" error="Get https://registry.gitlab.com/v2/$ORGANISATION/$PROJECT/$IMAGE/manifests/latest: denied: access forbidden" module=node/agent/taskmanager node.id=xgozmc8iyjls7ulh4k3tvions service.id=r5fqg94jrvt587le0fu779zaw task.id=huwpjtu1wujk527t84y7yvbvd
So it seems docker create service doesn't use the credentials provided and the issue is not related to either Google Container Registry or Gitlab Registry?

Answer 1

OK, I found the problem. I had to use:
docker service create --with-registry-auth --replicas 2 --network overlay --name $NAME registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE
rather than
docker service create --replicas 2 --network overlay --name $NAME registry.gitlab.com/$ORGANISATION/$PROJECT/$IMAGE --with-registry-auth
In the latter case the --with-registry-auth was considered an argument to my image rather than to the docker service create call and hence no authentication was used to pull the images from either private repository.