Pull from GCR inside GCE vm on Ubuntu 20.04 - docker

Question

I haven't set up a GCE stack in a while, and I swear this gets more difficult over time.
So the setup's easy enough: Blank ubuntu VM, installed docker via snap. Now when I try a pull from GCR, I get
> docker pull gcr.io/.../image
Using default tag: latest
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Fair enough. I checked my gcloud command:
> gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* ...-compute#developer.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
So the right service account is there. In IAM it's listed as an editor and for good measure, I added storage admin too.
Now I run
> gcloud auth configure-docker
WARNING: `docker-credential-gcloud` not in system PATH.
gcloud's Docker credential helper can be configured but it will not work until this is corrected.
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
After update, the following will be written to your Docker config file
located at [/home/y/.docker/config.json]:
{
"credHelpers": {
"gcr.io": "gcloud",
"marketplace.gcr.io": "gcloud",
"eu.gcr.io": "gcloud",
"us.gcr.io": "gcloud",
"staging-k8s.gcr.io": "gcloud",
"asia.gcr.io": "gcloud"
}
}
Do you want to continue (Y/n)?
Docker configuration file updated.
And according to gcp's documentation, the warning is fine. gcloud can be used as an alternative to the standalone helper. But still: the pull fails. Bummer.
According to the documentation, sudo is a bad idea. So I tried adding my user to the docker group and apparently that clashes with snap. I ran
> sudo addgroup --system docker
> sudo adduser $USER docker
> newgrp docker
> sudo snap disable docker
> sudo snap enable docker
So now I can use docker with my account.
The issue still persists though. I also tried the standalone helper with
> VERSION=2.0.0
> OS=linux # or "darwin" for OSX, "windows" for Windows.
> ARCH=amd64 # or "386" for 32-bit OSs, "arm64" for ARM 64.
> curl -fsSL "https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${VERSION}/docker-credential-gcr_${OS}_${ARCH}-${VERSION}.tar.gz" | tar xz --to-stdout ./docker-credential-gcr | sudo tee /usr/local/bin/docker-credential-gcr && sudo chmod +x /usr/local/bin/docker-credential-gcr
> docker-credential-gcr configure-docker
I've been troubleshooting this for too long, what's going on here?

Answer 1

Snap seems to have caused the issues here. Somewhere between snap-specific configuration files for the helpers and the snap-install gcloud SDK, the error happened. I went with a fresh installation and apt only:
sudo snap remove google-cloud-sdk
sudo apt update; sudo apt upgrade -y
sudo apt install docker.io
sudo curl -L --fail https://github.com/docker/compose/releases/download/1.25.5/run.sh -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker -v
sudo usermod -a -G docker $USER
## new shell
# exit
curl https://sdk.cloud.google.com | bash
gcloud auth configure-docker
. ~/.bashrc
sudo ln -s $(which gcloud) /usr/bin/
gcloud auth configure-docker