I deleted some images from our registry's storage backend and caused some issues. So, in order to restore the last, "latest"-tagged image, I pushed it from my local Docker, since I know it is the most up-to-date. All of the other systems in the network that need it have suddenly started failing with a new error: "Error response from daemon: missing signature key".
There appeared to be no useful suggestions or documentation online. I've inspected the latest image and its parent, and its parent, etc.. from my local host, and I am able to see information on all of them. This error seems to be specific to pulling/running.
I went back into the storage backend, erased my newly-pushed image and fully restored the original set of images that I had removed. However, I still had the same issue from all tested systems, including my local one. It has to do with some pre-pull lookup that is performed that involves more than a simple "docker inspect" does.
I assumed that there must be something broken with the tip image, so I just used our image-building process to do another "docker build", "docker tag", and "docker push", which both tags (and pushes) a version tag and updates the "latest" tag to point to it. That did it. Suddenly every host/daemon could pull again.
Related
I've been using nexus as a docker repository for a while to mitigate flakey internet. However recently, I've hit an issue that seems a bit weird. So, if I run:
docker pull server:8042/alpine:3.16.2
it works fine and it all gets cached. However, if I try and run
docker pull server:8042/sameersbn/gitlab:15.0.3
I get the following error:
Error response from daemon: unknown image in /v1.41/images/create?fromImage=server%3A8042%2Fsameersbn%2Fgitlab&tag=15.0.3
Running a direct pull from docker works fine, but using the cache, any nested tag with a username fails. I'm using engine 20.10.20 if that helps.
Thanks in advance
This appears to be a bug introduced somewhere between engine 20.10.17 and 20.10.20. Rolling back to an earlier version makes everything work again. I have also reported this to docker, however as I'm not a paid member, I suspect this will go unnoticed.
Everything is exposed via a REST API to publish new edge modules but a nice to have is on publishing we verify the given Docker container URL actually exists. This is to prevent having to publish, test a deployment, check Edge can pull the image, rinse and repeat until working (thinking is to make this idiot proof!). We have achieved this for our own registry and the mcr.micosoft.com (where edgeAgent/Hub live) with: “HEAD: https://mcr.microsoft.com/v2/ azureiotedge-agent/manifests/1.0.8.5”.
However, when trying this on marketplace.azurecr.io (where SQL DB Edge lives) we get an 'unauthorised' response. I have also tried: “https://marketplace.azurecr.io/v2/azure-sql-database-edge/tags/list” but also unauthorised. I am able to pull the container image down with Docker Desktop which seeing as that requires to download the manifest first I am not sure what I can do to simply check the image exists. Do you know a way we could verify the image exists on this container registry?
I have a problem with Docker that seems to happen when I change the machine type of a Google Compute Platform VM instance. Images that were fine fail to run, fail to delete, and fail to pull, all with various obscure messages about missing keys (this on Linux), duplicate or missing layers, and others I don't recall.
The errors don't always happen. One that occurred just now, with an image that ran a couple hundred times yesterday on the same setup, though before a restart, was:
$ docker run --rm -it mbloore/model:conda4.3.1-aq0.1.9
docker: Error response from daemon: layer does not exist.
$ docker pull mbloore/model:conda4.3.1-aq0.1.9
conda4.3.1-aq0.1.9: Pulling from mbloore/model
Digest: sha256:4d203b18fd57f9d867086cc0c97476750b42a86f32d8a9f55976afa59e699b28
Status: Image is up to date for mbloore/model:conda4.3.1-aq0.1.9
$ docker rmi mbloore/model:conda4.3.1-aq0.1.9
Error response from daemon: unrecognized image ID sha256:8315bb7add4fea22d760097bc377dbc6d9f5572bd71e98911e8080924724554e
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
$
So it thinks it has no images, but the Docker folders are full of files, and it does know some hashes. It looks like some index has been damaged.
I restarted that instance, and then Docker seemed to be normal again without any special action on my part.
The only workarounds I have found so far are to restart and hope, or to delete several large Docker directories, and recreate them empty. Then after a restart and pull and run works again. But I'm now not sure that it always will.
I am running with Docker version 17.05.0-ce on Debian 9. My images were built with Docker version 17.03.2-ce on Amazon Linux, and are based on the official Ubuntu image.
Has anyone had this kind of problem, or know a way to reset the state of Docker without deleting almost everything?
Two points:
1) It seems that changing the VM had nothing to do with it. On some boots Docker worked, on others not, with no change in configuration or contents.
2) At Google's suggestion I installed Stackdriver monitoring and logging agents, and I haven't had a problem through seven restarts so far.
My first guess is that there is a race condition on startup, and adding those agents altered it in my favour. Of course, I'd like to have a real fix, but for now I don't have the time to pursue the problem.
I'm trying to push a docker container to a private registry on the Google Cloud Platform:
gcloud docker -- push gcr.io/<project-name>/<container-name>
and a checksum fails:
e9a19ae6509f: Pushing [========================================> ] 610.9 MB/752.4 MB
xxxxxxxxxxxx: Layer already exists
...
xxxxxxxxxxxx: Layer already exists
file integrity checksum failed for "var/lib/postgresql/9.5/main/pg_xlog/000000010000000000000002"
Then I deleted that file (and more) from the container, committed the change, and tried to push the new image. I got the same error.
Is there some way to push up my image without pushing up the commit that contains the broken file? Any insight into why the new commit fails in the same way?
FWIW, that looks like a local daemon error prior to contacting the registry, so I very much doubt there is anything we will be able to do on our side. That said, if you reach out to us as Jake (jsand) suggests, we can hopefully try to help you resolve the issue.
We have been using locked version of the Minio image (RELEASE.2016-10-07T01-16-39Z), but now it seems to have been removed.
I'm getting this from Docker:
Pulling minio (minio/minio:RELEASE.2016-10-07T01-16-39Z)...
Pulling repository docker.io/minio/minio
ERROR: Tag RELEASE.2016-10-07T01-16-39Z not found in repository docker.io/minio/minio
I'm finding Docker hub hard to navigate. Where can I find a list of available versioned images, or a mirror to my exact image?
You can find the available tags for minio/minio on that repository's tag page.
If you have the image you want already downloaded on any of your systems, you can push it to Docker Hub yourself, then pull it onto your other systems. This has the benefit that you can control whether you delete that image (it's your account, not someone else's).
You can also use a private registry, if you want, which would prevent Docker from deleting the image from Docker Hub against your will for some reason. But that is extra work you may not wish to do (you would have to host the registry yourself, set it up, maintain it...)
We removed the docker version due to incompatibilities, from the recent releases it won't happen.