I have a strange problem with docker and pulling microsoft images.
Used command docker pull mcr.microsoft.com/dotnet/sdk:6.0
Server 1: no issue
Server 2: Error response from daemon: Get "https://mcr.microsoft.com/v2/": dial tcp 13.69.64.80:443: connect: connection refused
Both servers are
behind the firewall
configured with same /etc/docker/daemon.json whicht points to the internal artifactory instance
commands like docker pull sonarsource/sonar-scanner-cli works on both servers without an issue
also pulling privat images from artifactory works on both servers
any idea on that behavior?
Is your remote repo proxying mcr.microsoft.com and is configured to cache foreign layers?
If there is a local image, when the docker client builds the image and publishes it to Artifactory, Artifactory gets the manifest but has reference to foreign layers which are remote, so Artifactory is not going out and fetch them in a local repository. This behavior we observed appears to be related to this RTFACT-22938 Jira. If the foreign layers from the base image are not cached locally, Artifactory will not fetch these layers. In order for Artifactory to retrieve the foreign layers as well, all of the base image’s layers need to be cached in Artifactory.
Since Microsoft changes the tags of the images constantly, and the manifest is changing, the layers that were pulled will not be used since they are no longer referenced in the manifest. Re-tagging the images with custom tags can be one way to avoid this situation.
Related
I am trying to push a docker image to a repository and having several problems. I have my local development environment set up with Docker desktop on OSX. Now, I need to to push the updated image to dockerhub, so it can be pulled into Kubernetes. I can't be sure, but I think I received security warnings from my workplace after trying that action.
Are there known security risks with connecting to dockerhub?
Do I need to protect the daemon socket? Our desktop guy send this to me, but I am not connecting to a host other than Dockerhub, so is the following still relevant? https://docs.docker.com/engine/security/protect-access/
I have two docker repositories running on the same JFrog cloud account/instance. One for internal release candidates and the other for potentially external GC releases. I want to be able to build the docker images and push to the internal repository, let QA/UAT go to town, and then copy the image to the release repository. I don't want to rebuild the image from source. Unfortunately, when I try to pull, tag and then push the image, I'm getting an error:
unauthorized: Pushing Docker images with manifest v2 schema 1 to this repository is blocked.
Both repositories block schema 1 manifests, but I am pushing fine to the internal repository, so it doesn't make much sense I wouldn't be able to push the same image to the release repository.
I've setup a pretty simple test to confirm (actual repository URLs censored):
% docker pull hello-world:latest
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
...
% docker tag hello-world:latest internal-rc.jfrog.io/hello-world:1.0.0-beta
% docker push internal-rc.jfrog.io/hello-world:1.0.0-beta
The push refers to repository [internal-rc.jfrog.io/hello-world]
9c27e219663c: Pushed
...
% docker system prune -a
...
Total reclaimed space: 131.8MB
% docker image pull internal-rc.jfrog.io/hello-world:1.0.0-beta
1.0.0-beta: Pulling from hello-world
0e03bdcc26d7: Pull complete
...
% docker image tag internal-rc.jfrog.io/hello-world:1.0.0-beta docker-release.jfrog.io/hello-world:1.0.0
% docker image push docker-release.jfrog.io/hello-world:1.0.0
The push refers to repository [docker-release.jfrog.io/hello-world]
9c27e219663c: Layer already exists
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker-release.jfrog.io registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
unauthorized: Pushing Docker images with manifest v2 schema 1 to this repository is blocked. For more information visit https://www.jfrog.com/confluence/display/RTF/Advanced+Topics#AdvancedTopics-DockerManifestV2Schema1Deprecation
So I can upload the image fine to the first repository, and confirm that it is using schema 2:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 7004,
"digest": "sha256:66f750f4871ba45724699d7341ee7135caba46f63fb205351197464a66b55eff"
...
Does that mediaType being v1 matter? It seems like the manifest itself is version 2... But I don't know how I would change that, or why it would be allowed in one repository but not the other.
I'm using I believe the latest version of docker Docker version 19.03.8, build afacb8b
Anyone have any idea what's going on there? Is the schema version being changed between when I upload it the first time and when I download it? Or is it when I tag it or upload it the second time?
The root of this problem can probably be classified as user error. Specifically the user I'm using somehow had permissions removed from the release repository. Once that was restored everything works as expected.
I say "probably" because the error message has nothing to do with the actual problem, and cost me 2-3 hours worth of wild goose chasing.
So... If you see this error, go ahead and double check everything else around permissions/access before trying to figure out if there's something actually wrong with your image schema version.
We had a different case today with a similar error. I'm adding here because this is the top google result at the moment.
Pulling Docker images with manifest v2 schema 1 to this repository is blocked.
The fix was to change a setting on the remote repository.
Via UI:
Artifactory Admin -> Repositories -> Repositories -> Remote tab
Then select your Docker Hub repo, whatever you named it, then under Basic settings -> Docker Settings, uncheck the checkbox labeled
Block pulling of image manifest v2 schema 1
After that our images began pulling properly again.
There is a similar checkbox on local repos for pushing.
For what it's worth, we're on Artifactory version 7.18.5 rev 71805900
edit: The surprisingness of our particular issue is (potentially) explained in some more detail here: https://www.jfrog.com/jira/browse/RTFACT-2591
Docker pull requests fail due to a change in Docker Hub behavior. Now Docker Hub HTTP response headers return in lower case, for example, 'content-type' instead of 'Content-Type', causing Artifactory to fail to download and cache Docker images from Docker Hub.
but we have not yet tested whether an upgrade allows us to re-enable the aforementioned checkbox.
I have been getting the below errors while either pulling or pushing docker images from build servers. i have a proxy in env which used to connect docker registry. My DNS server while resolving the proxy FQDN it was returning a non functional IP-address. i have 4 DNS servers and multiple proxy servers based on region. Once DNS is updated and working/functional proxy returned it started working. Just check on network side too , it may solve the issue. the error messages were misleading initially i though to be docker layer issue, credential issue ..but no network issue. for the below errors
error pulling image configuration: unknown blob or
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
manifest invalid: manifest invalid
. Will start No.6 try.
GitHub packages started returning error pulling image configuration: unknown blob this weekend when trying to pull docker images. It still works to push images to the registry. I haven't found any infromation pointing to problems at GitHub.
000eee12ec04: Pulling fs layer
db438065d064: Pulling fs layer
e345d85b1d3e: Pulling fs layer
f6285e273036: Waiting
2354ee191574: Waiting
69189c7cf8d6: Waiting
771c701acbb7: Waiting
error pulling image configuration: unknown blob
How do I troubleshoot this?
This is the result of a failed push where the push appears to have been successful but something went wrong on the registry side and something is missing.
To fix it build your container again and push it again.
While this is likely a rare situation it would be possible to test for this by deleting your image locally after pushing and pulling it again to ensure pulls work as expected.
One possible cause of failure of pulling or pushing image layer is the unreliable network connection outlined in this blog. By default docker engine has 5 parallel upload operations.
You can update the docker engine to only use single upload or download operation by specifying values for max-concurrent-downloads for download or max-concurrent-uploads for upload.
On windows, you should update via C:\Users\{username}\.docker\daemon.json or via the Docker for Desktop GUI:
{
...
"max-concurrent-uploads": 1
}
On *Nix, open /etc/docker/daemon.json (If the daemon.json file doesn’t exist in /etc/docker/, create it.) and add following values as needed:
{
...
"max-concurrent-uploads": 1
}
And restart daemon.
Note: Currently this is not possible to specify these options in docker push or docker pull command as per this post.
We're setting up a server to host Windows containers.
This server gets the images from an internal Docker registry we have setup.
The issue is that the server is unable to pull down images because it's trying to get a base image from the internet, and the server has no internet connection.
I found a troubleshooting script from Microsoft and notice one passage:
At least one of 'microsoft/windowsservercore' or
'microsoft/nanoserver' should be installed
Try docker pull microsoft/nanoserver or docker pull
microsoft/windowsservercore to pull a Windows container image
Since my PC has internet connection, I downloaded these images, pushed them to the registry, but pulling the images on the new server fails:
The description for Event ID '1' in Source 'docker' cannot be found. The local computer may not have the necessary registry information or message DLL files to display the message, or you may not have permission to access them. The following information is part of the event:'Error initiating layer download: Get https://go.microsoft.com/fwlink/?linkid=860052: dial tcp 23.207.173.222:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.'
That link it's trying to get is a base image on the internet, but I thought the registry was storing the complete image, so what gives? Is it really not possible to store the base images in a registry?
Doing some reading I found this: https://docs.docker.com/registry/deploying/#considerations-for-air-gapped-registries
Certain images, such as the official Microsoft Windows base images,
are not distributable. This means that when you push an image based on
one of these images to your private registry, the non-distributable
layers are not pushed, but are always fetched from their authorized
location. This is fine for internet-connected hosts, but will not work
in an air-gapped set-up.
The doc then details how to setup the registry to store non-distributable layers, but they also say to be mindful of the terms of use for non-distributable layers.
So two possible solutions are:
Make sure you can store the non-distributable layers, then reconfigure the registry to store the non-distributable layers
Connect the server to the internet, download the base images, then use those images
We have been using locked version of the Minio image (RELEASE.2016-10-07T01-16-39Z), but now it seems to have been removed.
I'm getting this from Docker:
Pulling minio (minio/minio:RELEASE.2016-10-07T01-16-39Z)...
Pulling repository docker.io/minio/minio
ERROR: Tag RELEASE.2016-10-07T01-16-39Z not found in repository docker.io/minio/minio
I'm finding Docker hub hard to navigate. Where can I find a list of available versioned images, or a mirror to my exact image?
You can find the available tags for minio/minio on that repository's tag page.
If you have the image you want already downloaded on any of your systems, you can push it to Docker Hub yourself, then pull it onto your other systems. This has the benefit that you can control whether you delete that image (it's your account, not someone else's).
You can also use a private registry, if you want, which would prevent Docker from deleting the image from Docker Hub against your will for some reason. But that is extra work you may not wish to do (you would have to host the registry yourself, set it up, maintain it...)
We removed the docker version due to incompatibilities, from the recent releases it won't happen.