I've tried this on at least 5 different versions of Linux and always hit the same wall:
I can use docker to run hello-world successfully. But whenever I try to pull any other image (e.g. ubuntu, nginx) it pulls the pieces in parallel and then ends up with the filesystem layer verification failure. Has anyone seen this problem or can offer advice?
Components:
CentOS 7.3.1611 (3.10.0-514.el7.x86_64) as a Virtual Box VM
Docker 1.10.3
xfs file system
Configuration steps (CentOS):
# yum install docker
# systemctl start docker
# systemctl status docker
# systemctl enable docker
# docker run hello-world (works)
# docker pull ubuntu (fails)
Note: Yum doesn't install docker 1.12 if I try to manually there are conflicts.
Current questions:
Are there issues with docker in a Virtualbox guest host?
Does Docker require a specify type of filesystem?
I read a comment somewhere that fails when trying to pull multiple
pieces in parallel (hello-world is a single chunk), but I can't
verify that. Is there another tiny image I can try?
The only issues I've seen relate to AWS and I'm not using AWS. Could
it be a SHA key issue?
Answer to comment:
Note: I can run the hello-world example and busybox. The are both one layer. Not sure if that has anything to do with it.sudo docker pull debian
Using default tag: latest
Trying to pull repository docker.io/library/debian ...
latest: Pulling from docker.io/library/debian
75a822cd7888: Verifying Checksum
filesystem layer verification failed for digest sha256:75a822cd7888e394c49828b951061402d31745f596b1f502758570f2d0ee79e2
filesystem layer verification failed for digest sha256:75a822cd7888e394c49828b951061402d31745f596b1f502758570f2d0ee79e2
This turned out to be a Virtualbox bug. It makes sense, since every machine I was trying this on was a Virtualbox VM (see original post). In investigating a work-around to download the pieces manually via wget, wget was getting errors on all machines. Downloads over a few seconds were throwing "SSL3_GET_RECORD:decryption failed or bad record mac". Googling that showed that this is a known (as of 2014 anyway) bug in Virtualbox when the VM's network type is set to Bridged. The solution is to set the network type in the VM to NAT.
Related
Here is my terminal log (Ubuntu 22.04.1, Docker version 20.10.22, build 3a2c30b):
paul#desktop:~/work/arc/code$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
pauljurczak/arc latest 4f3f22791983 35 minutes ago 880MB
sub-1 latest 4f3f22791983 35 minutes ago 880MB
neo-1 latest 3dcf55bb7458 3 days ago 891MB
arc-tut latest 3a9aee91689b 4 days ago 230MB
paul#desktop:~/work/arc/code$ sudo docker push pauljurczak/arc
Using default tag: latest
The push refers to repository [docker.io/pauljurczak/arc]
An image does not exist locally with the tag: pauljurczak/arc
paul#desktop:~/work/arc/code$ sudo docker push pauljurczak/arc:latest
The push refers to repository [docker.io/pauljurczak/arc]
An image does not exist locally with the tag: pauljurczak/arc
pauljurczak/arc
paul#desktop:~/work/arc/code$ sudo docker push latest
Using default tag: latest
The push refers to repository [docker.io/library/latest]
An image does not exist locally with the tag: latest
I created pauljurczak/arc:latest image as shown there. I'm trying to push it to my pauljurczak/arc repository. The error messages don't make any sense to me. Why pauljurczak/arc in my push command is considered a tag? Why An image does not exist locally with the tag: latest, while there exists several images with that tag? What is happening here? I'm following push command description at https://docs.docker.com/docker-hub/repos/#pushing-a-docker-container-image-to-docker-hub.
When viewing my repository with Chrome, I see this info:
That's exactly what I was doing.
It seems that skipping sudo makes it work. Why is that?
It looks like you have both Docker Desktop and the standalone Docker Engine installed. This means you have two different Docker daemons running. The Docker Engine one is accessible via /var/run/docker.sock, given appropriate permissions; Docker Desktop runs a hidden Linux virtual machine (even on native Linux) and makes a similar socket file available in your home directory.
Docker Desktop uses the Docker CLI "context" feature to point docker at the socket in your home directory. That configuration is also stored in a file in your home directory.
This is where sudo makes a difference. When you run sudo docker ..., and it reads $HOME/.docker/contexts/, that home directory is now root's home directory, probably /root. That doesn't have any of the Docker Desktop related configuration in it, and so you use the default /var/run/docker.sock Docker Engine socket instead.
As you note, just consistently not using sudo will resolve this. (You could potentially need sudo to access the Docker Engine socket, which can all but trivially be used to root the host; the Docker Desktop VM is a little more isolated.) Uninstalling Docker Desktop and making sure to only use the standalone Docker Engine on native Linux also would resolve this.
I am trying to pull docker image from Nexus repo without using the registry mirror in the command line and it is throwing an error. If I use the registry mirror in the pull it is succeeding but the image name is not I would like.
My docker version is:
Docker version 20.10.8, build 3967b7d
My nexus version is
Sonatype Nexus Repository ManagerOSS 3.31.1-01
docker system info:
Insecure Registries:
xxx.xxx.x.xxx:8083
127.0.0.0/8
Registry Mirrors:
http://xxx.xxx.x.xxx:8083/
When I run: sudo docker pull xxx.xxx.x.xxx:8083/mongo:4.2.3, it succeeds and the debug info is:
DEBU[2021-08-17T10:37:19.364681226-04:00] Calling HEAD /_ping
DEBU[2021-08-17T10:37:19.365301100-04:00] Calling POST /v1.41/images/create?fromImage=192.168.9.175%3A8083%2Fmongo&tag=4.2.3
DEBU[2021-08-17T10:37:19.367151579-04:00] Trying to pull xxx.xxx.x.xxx:8083/mongo from https://xxx.xxx.x.xxx:8083 v2
WARN[2021-08-17T10:37:19.374915464-04:00] Error getting v2 registry: Get https://xxx.xxx.x.xxx:8083/v2/: http: server gave HTTP response to HTTPS client
INFO[2021-08-17T10:37:19.374944418-04:00] Attempting next endpoint for pull after error: Get https://xxx.xxx.x.xxx:8083/v2/: http: server gave HTTP response to HTTPS client
DEBU[2021-08-17T10:37:19.374964188-04:00] Trying to pull xxx.xxx.x.xxx:8083/mongo from http://xxx.xxx.x.xxx:8083 v2
DEBU[2021-08-17T10:37:19.398630498-04:00] Fetching manifest from remote digest="sha256:92814bb60dc673bb68b6aca0b24bcb8738d7b2c267b97ce62fa92adc3746a0ea" error="<nil>" remote="192.168.9.175:8083/mongo:4.2.3"
DEBU[2021-08-17T10:37:19.429454057-04:00] Pulling ref from V2 registry: xxx.xxx.x.xxx:8083/mongo:4.2.3
When I run: sudo docker pull mongo:4.2.3 it fails to pull the image from Nexus with an error and pulls from docker.io on the next try. Debug info as below:
DEBU[2021-08-17T10:26:25.078886904-04:00] Calling HEAD /_ping
DEBU[2021-08-17T10:26:25.079306196-04:00] Calling GET /v1.41/info
DEBU[2021-08-17T10:26:25.097994642-04:00] Calling POST /v1.41/images/create?fromImage=mongo&tag=4.2.3
DEBU[2021-08-17T10:26:25.099642151-04:00] Trying to pull mongo from http://xxx.xxx.x.xxx:8083/ v2
INFO[2021-08-17T10:26:25.116000813-04:00] **Attempting next endpoint for pull after error: manifest unknown: manifest unknown**
DEBU[2021-08-17T10:26:25.116039299-04:00] Trying to pull mongo from https://registry-1.docker.io v2
DEBU[2021-08-17T10:26:25.305043063-04:00] Fetching manifest from remote digest="sha256:58b25d51baa11a85b6aedf7c4e05710d12a27ddc2883e2692e7d58527d98bd73" error="<nil>" remote="docker.io/library/mongo:4.2.3"
DEBU[2021-08-17T10:26:25.360955030-04:00] Pulling ref from V2 registry: mongo:4.2.3
DEBU[2021-08-17T10:26:25.361036645-04:00] docker.io/library/mongo:4.2.3 resolved to a manifestList object with 5 entries; looking for a unknown/amd64 match
Issue with Image name:
REPOSITORY TAG IMAGE ID CREATED SIZE
xxx.xxx.x.xxx:8083/mongo 4.2.3 97a9a3e85158 17 months ago 386MB
Any guidance on this would help.
Nexus Docker ( xxx.xxx.x.xxx:8083) is pointed to hosted Type on port 8083 and the mongo:4.2.3 is uploaded into this docker type. We ultimately want to use this in a air gapped system where there is no internet connection.
There are three things going on here:
I am trying to pull docker image from Nexus repo without using the registry mirror in the command line and it is throwing an error. If I use the registry mirror in the pull it is succeeding but the image name is not I would like.
I'm going to recommend changing your likes. :)
If you want to pull from a specific registry, then use that registry in the image name. Trying to refer to your local registry with short names is merging two different image registry namespaces, which means it's trivial to run an image from the wrong namespace and result in a security breach. This was a large issue for other package repositories (see "dependency confusion" attacks) that docker was not susceptible to because they require the registry name as part of the image name (the only exception being Docker Hub). Even RedHat who tried to get options like add-registry and block-registry into the upstream docker engine (and failed, these options only ever appeared in a RedHat specific fork) is now telling users that it was a very bad idea and now their users are exposed to security vulnerabilities they can't easily fix because removing the feature will break lots of user environments.
Next, why doesn't the pull go to your registry? Because your image name doesn't match that of Docker Hub. Official images without a username are actually under the library repository. This is typically hidden from view, but you can do things like docker pull library/alpine or even docker pull docker.io/library/alpine instead of docker pull alpine, and all 3 will be pulling from the same place.
The fix is to run
docker pull xxx.xxx.x.xxx:8083/mongo:4.2.3
docker tag xxx.xxx.x.xxx:8083/mongo:4.2.3 xxx.xxx.x.xxx:8083/library/mongo:4.2.3
docker push xxx.xxx.x.xxx:8083/library/mongo:4.2.3
The last issue I actually can't help you with, it comes from the error message you're seeing when pulling from Hub, which should work:
docker.io/library/mongo:4.2.3 resolved to a manifestList object with 5 entries; looking for a unknown/amd64 match
The unknown/amd64 is unexpected to me, typically that would be linux/amd64 so there is something unexpected with the platform you're running your commands on. If you want to get into debugging that, update your question with docker info. You can try working around that with:
docker pull --platform linux/amd64 mongo:4.2.3
to force the platform, but that still doesn't explain why it doesn't know your current platform.
I guess you are trying to set your nexus docker repository to be the default one for the machine in the sealed network.
that needs changing because of the following from docker documentation:
Tag an image for a private repository
To push an image to a private registry and not the central Docker registry you must tag it with the registry hostname and port (if needed).
$ docker tag 0e5574283393 myregistryhost:5000/fedora/httpd:version1.0
with more upfront configuration and upkeep but no changes requiered for the client machines
Is if you have a DNS server in your network you could point docker.io to your nexus host ip address and put a proxy to intercept the communication and redirect and adapt the requests as they were to the nexus docker registry
Hopes this solves your pickle :)
Update 1:
It could be that you need to also change /etc/containers/registries.conf like specified here to only or also specify your nexus docker registry.
Update 2:
Before letting Gopi give up entirely, I would suggest using Podman as an alternative to Docker. Podman is a daemon-less container engine that works by forking processes to handle each running container. It seamlessly works with docker images thanks to the OCI standard, and on top of that, the only change when using it is replacing the docker command prefix with podman since all the commands are exactly the same. Podman was created by RedHat so by default it searches RedHat repos and you can add your own too as shown in this article that I mentioned before.
I have a CentOS 7 with Docker installed on a VirtualBox. I am also new to Docker. When I run:
docker pull microsoft/aspnet
I get an error:
no matching manifest for unknown in the manifest list entries
I can see the image listed if I do:
docker search aspnet
Do I understand this correctly that this image is not suitable to run in Docker on Linux? Or is there an error getting the correct manifest - thus 'unknown' in the error message?
Running below command produces:
$ docker info -f '{{.OSType}}/{{.Architecture}}'
linux/x86_64
Some additional information - I was able to pull and successfully use microsoft/dotnet image.
That docker image is based on Windows Server Core as the base OS and also uses IIS as the web server, that makes it Windows only, you can use docker pull microsoft/aspnetcore on Linux. As for microsoft/dotnet it's cross platform and i guess it's basically the same as microsoft/aspnetcore but without the ASP stuff (sorry i'm not that familiar with Windows SDKs and Frameworks).
I am refering to this link - Docker pull.
By default, docker pull pulls images from Docker Hub (https://hub.docker.com).
I would like to know where this link is configured on our local machine setup.
I am using Docker on Windows 10.
You cannot change the default domain of a docker image. This is by design:
Your Docker installation with this "private registry defined in the config file" would be incompatible with every other Docker installation out there. Running docker pull debian needs to pull from the same place on every Docker install.
A developer using Docker on their box will use debian, centos and ubuntu official images. Your hacked up Docker install would just serve your own versions of those images (if they're present) and this will break things.
You should identify your image through the full URL:
<your-private-registry>/<repository>/<image>:<tag>
The default domain docker.io (the "docker hub") is hardcoded in docker's code.
For example here:
https://github.com/docker/distribution/blob/master/reference/normalize.go
Check the function splitDockerDomain which sets docker.io as registry if it's not provided by the user.
I'm unable to pull docker images in my environment. I think it's blocked by company firewall, but I'm not sure why It gets layer info and later It prints that repository is not found.
sudo docker pull hello-world
latest: Pulling from hello-world
50a54e1f9180: Pulling fs layer
7a5a2d73abce: Pulling fs layer
Pulling repository hello-world
Repository not found
Docker version: (I cannot upgrade to newest docker on RHEL 6.9)
Docker version 1.7.1, build 786b29d/1.7.1
Could somebody explain me which protocols (https only?) are used during docker image pulling phase and what addresses are contacted ("https://registry-1.docker.io/v2" only?) ?
Docker images can consist of multiple layers. By default, the Docker daemon will pull three layers of an image at a time but will pull less in case an image has lesser layers. Also, if no tag is provided, Docker Engine uses the :latest tag as a default. Above is a basic log of your pull request indicating docker trying to pull layers of the image but failing, may be due to firewall restrictions or older docker version.
Docker uses the https:// protocol to communicate with a registry,
unless the registry is allowed to be accessed over an insecure
connection.
Not sure what all addresses it tries to connect to pull an image.
Problem was that firewall was blocking connections during pulling images.
Docker registry uses CDN so more URLs need to be allowed and not only registry URL.
I requested to allow the following URLs on company firewall and it is working now.
dseasb33srnrn.cloudfront.net
auth.docker.io
elb-registry.us-east-1.aws.dckr.io.
us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
registry-1.docker.io
registry-origin.docker.io
index.docker.io
elb-io.us-east-1.aws.dckr.io
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com
Docker log file (/var/log/docker) help me to identify root problem.
There were the following errors:
level=error msg="Error from V2 registry: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/78/78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3/data?Expires=1493033299&Signature=DiEmffSxF1F9z-SRoGyX3NwzfeQY3BhE2Du3aPb1qy9VglXyn1mus7Xy9Y~DQnwaQ9IIN71FboK5lOAiN1Qj-x662qhioi72CJ-v02fiMHqC03FDb0l4LyULquU8GaalW3uZG4hdfuSqOBQ1qo9HEcxhMyQGqOqpfPUKjUlHqm8_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: read tcp 52.85.173.110:443: connection reset by peer"
The list of URLs which needs to be allowed I found here:
https://forums.docker.com/t/list-of-docker-hub-mirror-sites-to-configure-proxy-whitelist/20845/2