Why docker push registry errors on certificate file? - docker

As described here How to setup docker private registry on ubuntu 16.04,
I changed /etc/hosts like this:
192.168.1.154 registry-server
192.168.1.90 registry-client
Then I pulled the registry image:
docker pull registry
Then I made certificate files
mkdir /etc/certs
cd /etc/certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crt
I copied the ca.crt to these pathes in client host:
/etc/certs/
/etc/docker/certs.d/registry-server:5000/
Then I ran the container on the server host:
docker run -d -p 5000:5000 --restart=always --name registry -v /etc/certs:/etc/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/certs/ca.crt -e REGISTRY_HTTP_TLS_KEY=/etc/certs/ca.key registry
I tagged the image
docker tag phpmyadmin/phpmyadmin:latest registry-server:5000/pma-test
But when I want to push the image registry-server:5000/pma-test to the server:
docker push registry-server:5000/pma-test:latest
The following error occurs:
Error response from daemon: open /etc/docker/certs.d/registry-server:5000: permission denied
======================================
Update:
I ran journalctl -xe, and found these errors:
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.229097561Z" level=debug msg="Calling GET /_ping"
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238248010Z" level=debug msg="Calling POST /v1.38/images/registry-server:5000/pma-test/push?tag="
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238670117Z" level=debug msg="hostDir: /etc/docker/certs.d/registry-server:5000"
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238797277Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: open /etc/docker/certs.d/registry-server:5000: permission denied" error_type="*os.PathError" module=api
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238831133Z" level=error msg="Handler for POST /v1.38/images/registry-server:5000/pma-test/push returned error: open /etc/docker/certs.d/registry-server:5000: permission denied"
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238861895Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: open /etc/docker/certs.d/registry-server:5000: permission denied" error_type="*os.PathError" module=api
Sep 30 13:58:37 audit[926]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/docker/certs.d/registry-server:5000/" pid=926 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Sep 30 13:58:37 kernel: audit: type=1400 audit(1569851917.234:53): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/docker/certs.d/registry-server:5000/" pid=926 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Why did this error happen?
Docker version 19.03.2, build 6a30dfc
docker-compose version 1.24.0, build 0aa59064
Server and client host: Ubuntu 18.04

Finally, I found it:
I added the following line to /var/lib/snapd/apparmor/profiles/snap.docker.docker
/etc/docker/certs.d/** r,
Then I ran:
apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.docker.dockerd
And the problem solved.

Related

Docker pull hangs on "extracting"

Every pull I run hangs on "Extracting"
# docker -v pull ansible/awx_rabbitmq:3.7.4
3.7.4: Pulling from ansible/awx_rabbitmq
ff3a5c916c92: Extracting [> ] 32.77kB/2.066MB
5387f4b4c52b: Download complete
dba8c403a5b6: Download complete
4258fc50c523: Download complete
41e241289d30: Download complete
7a8ab8823f42: Download complete
21ac0c0b3f13: Download complete
56b9421a89dc: Download complete
e69676b835e9: Download complete
13a893bc00f9: Download complete
e7cf78370af8: Download complete
363d82081450: Download complete
bc716b889e7b: Download complete
ebd8aacef79e: Download complete
839d6f7a7803: Download complete
this is the syslog output (docker daemon verbosity in debug):
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.216589013+01:00" level=debug msg="Calling HEAD /_ping"
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.216881715+01:00" level=debug msg="Calling GET /v1.41/info"
Jan 26 13:10:56 XXXMACHINENAME dbus-daemon[24071]: [session uid=0 pid=24069] Activating service name='org.freedesktop.secrets' requested by ':1.0' (uid=0 pid=24060 comm="docker-credential-secretservice get " label="unconfined")
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.274306708+01:00" level=debug msg="Calling POST /v1.41/images/create?fromImage=ansible%2Fawx_rabbitmq&tag=3.7.4"
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.275348977+01:00" level=debug msg="Trying to pull ansible/awx_rabbitmq from https://registry-1.docker.io v2"
Jan 26 13:10:57 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:57.668102153+01:00" level=debug msg="Pulling ref from V2 registry: ansible/awx_rabbitmq:3.7.4"
Jan 26 13:10:57 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:57.668519475+01:00" level=debug msg="pulling blob \"sha256:dba8c403a5b6fbb5651fd71cc7e2c96605165864b4ee509d2b6676e2958b8164\""
Jan 26 13:10:57 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:57.668554305+01:00" level=debug msg="pulling blob \"sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28\""
...
Jan 26 13:10:58 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:58.548777211+01:00" level=debug msg="Using /usr/bin/unpigz to decompress"
...
Jan 26 13:11:04 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:11:04.347774528+01:00" level=debug msg="Downloaded ebd8aacef79e to tempfile /var/lib/docker/tmp/GetImageBlob810059367"
Not a disk space problem, the filesystem is almost empty.

In Sonatype Nexus Docker proxy, how do I debug "Bad request" error?

I've been following the instructions to set up a Proxy Repository for Docker. I am intending to set up a proxy for Docker hub, that is for https://index.docker.io/.
My setup details on Nexus
Nexus version 3.36.0-01
Installed via docker-compose
Generated/installed self-signed cert
Using built-in https/jetty, NOT reverse proxy
http listening on port 80
https listening on port 443
My setup details on Nexus docker proxy repo
Configured for https, port 8443
Proxy remote storage: https://registry-1.docker.io
Proxy docker index: "use docker hub", pre-filled as https://index.docker.io/
Allowing anonymous docker pull
Enabled Docker Bearer Token Realm
Enabled docker v1 API
Enabled foreign layer caching
My setup details on Ubuntu docker client
Trusted self-signed cert in /etc/docker/certs.d
Trusted self-signed cert in /usr/local/share/ca-certificates + update-ca-certificates
Enabled Docker daemon debugging in /etc/docker/daemon.json
Enabled Docker proxy via httpsProxy in /home/myuser/.docker/config.json
Enabled Docker proxy via httpsProxy in /etc/systemd/system/docker.service.d/https-proxy.conf, reloaded/restarted Docker daemon
My test from the client
docker pull hello-world:latest
returns error Error response from daemon: Get https://registry-1.docker.io/v2/: Bad Request
In debug logs:
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.190545462Z" level=debug msg="Calling HEAD /_ping"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.190878019Z" level=debug msg="Calling GET /v1.40/info"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.213218413Z" level=debug msg="Calling POST /v1.40/images/create?fromImage=hello-world&tag=latest"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.213290250Z" level=debug msg="Trying to pull hello-world from https://registry-1.docker.io v2"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234803592Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: Bad Request"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234865780Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: Bad Request"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234976364Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://registry-1.docker.io/v2/: Bad Request"
Next Steps
I'm watching the logs on the server while this is happening. It shows no errors. However the client side seems to indicate the request is partly working.
I tried increasing org.apache.http.wire to DEBUG as per this other SO question/answer, but that also showed nothing.
How do I continue debugging?
If you examine the docker output you’ll notice it isn’t going to nexus, it is making the request to https://registry-1.docker.io. To pull from Nexus you need to prepend the host and port to the pull request.
docker pull hostname:8443/hello-world:latest
There isn’t any way in docker to have it default to a private registry btw, so you’ll always need to prepend host:port.

Almost `docker pull` command hanging - retry(after 5min) - complete

$ docker pull python:3.6.4-stretch
3.6.4-stretch: Pulling from library/python
c73ab1c6897b: Pull complete
1ab373b3deae: Downloading [=============================================> ] 10.13MB/11.11MB
b542772b4177: Download complete
57c8de432dbe: Download complete
1ab373b3deae: Pull complete
b542772b4177: Pull complete
57c8de432dbe: Pull complete
1785850988c5: Pull complete
676ef2d8682b: Pull complete
56321bcc2d38: Pull complete
4788c366a216: Pull complete
0d970fbfeb26: Pull complete
Digest: sha256:db22cb78ba16cb6a0632eead1e48a239636a5a77c9f8cf343087acf309ad0248
Status: Downloaded newer image for python:3.6.4-stretch
Time: 0h:05m:33s
Download hangs with a probability of 80% or more like above ouput. Then hold this state for 5 minutes and pull will succeed when retry starts.
For more detail, this problem occurs in three ubuntu pc.
Two are Ubuntu 16.04 and one is 18.04. All machine are on the same office network.
At first I tried changing the docker and ubuntu versions but it failed. service docker restart was also useless. I noticed that I installed a new gigabit switch hub(https://iptime.com/iptime/?page_id=11&pf=12&page=2&pt=311&pd=1), and I suspected the hub device. It works well when machine connects directly to the LAN without a switching hub and When I changed the switching hub to the old 100Mb/s thing, it worked well also.
It can be judged as a problem of the gigabit switching hub, but it is difficult to find out because all other internet use is work well with gigabit switching hub. So I wonder if there is no other problem with docker pull or there is no other solution.
$ uname -a
Linux my-ubuntu18.04 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:48:57 2018
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:16:44 2018
OS/Arch: linux/amd64
Experimental: false
Please search failed keyword in below my docker daemon log.
12:13:32 level=debug msg="Calling GET /_ping"
12:13:32 level=debug msg="Calling GET /v1.39/info"
12:13:32 level=debug msg="Calling POST /v1.39/images/create?fromImage=python&tag=3.6.6-stretch"
12:13:32 level=debug msg="Trying to pull python from https://registry-1.docker.io v2"
12:13:35 level=debug msg="Pulling ref from V2 registry: python:3.6.6-stretch"
12:13:35 level=debug msg="docker.io/library/python:3.6.6-stretch resolved to a manifestList object with 7 entries; looking for a unknown/amd64 match"
12:13:35 level=debug msg="found match for linux/amd64 with media type application/vnd.docker.distribution.manifest.v2+json, digest sha256:c306863aa2e858ccf00958c625ca2ffdbf8845da76e266b0b0d9c4760170aff3"
12:13:36 level=debug msg="Layer already exists: bc9ab73e5b14"
12:13:36 level=debug msg="Layer already exists: 193a6306c92a"
12:13:36 level=debug msg="Layer already exists: e5c3f8c317dc"
12:13:36 level=debug msg="Layer already exists: a587a86c9dcb"
12:13:36 level=debug msg="pulling blob \"sha256:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29\""
12:13:36 level=debug msg="pulling blob \"sha256:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c\""
12:13:37 level=debug msg="Downloaded 6598fc9d11d1 to tempfile /var/lib/docker/tmp/GetImageBlob402585872"
12:13:37 level=debug msg="pulling blob \"sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f\""
12:13:40 level=debug msg="Downloaded 72744d0a318b to tempfile /var/lib/docker/tmp/GetImageBlob083083061"
12:13:40 level=debug msg="pulling blob \"sha256:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a\""
12:13:40 level=debug msg="Applying tar in /var/lib/docker/overlay2/9eaab31d9a1f108ba8a5c712cf23f36a9097140d09c76e6b966667fba2cc014b/diff" storage-driver=overlay2
12:13:42 level=debug msg="Downloaded 93612f47cdc3 to tempfile /var/lib/docker/tmp/GetImageBlob281534658"
12:13:42 level=debug msg="pulling blob \"sha256:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b\""
12:13:43 level=debug msg="Downloaded 1bc4b4b50870 to tempfile /var/lib/docker/tmp/GetImageBlob872144708"
12:13:48 level=debug msg="Applied tar sha256:9978d084fd771e0b3d1acd7f3525d1b25288ababe9ad8ed259b36101e4e3addd to 9eaab31d9a1f108ba8a5c712cf23f36a9097140d09c76e6b966667fba2cc014b, size: 556457027"
12:13:48 level=debug msg="Applying tar in /var/lib/docker/overlay2/3f91f78b3bb3f2cb6096472759bb84ae2f30f0825a7f935cad3b420c5cd71bee/diff" storage-driver=overlay2
12:13:48 level=debug msg="Applied tar sha256:2f4f74d3821ecbdd60b5d932452ea9e30cecf902334165c4a19837f6ee636377 to 3f91f78b3bb3f2cb6096472759bb84ae2f30f0825a7f935cad3b420c5cd71bee, size: 16849952"
12:18:46 level=error msg="Download failed, retrying: read tcp 10.251.12.218:48728->104.18.121.25:443: read: connection timed out"
12:18:51 level=debug msg="pulling blob \"sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f\""
12:18:51 level=debug msg="attempting to resume download of \"sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f\" from 20499209 bytes"
12:18:53 level=debug msg="Downloaded 4b1d9004d467 to tempfile /var/lib/docker/tmp/GetImageBlob954105135"
12:18:53 level=debug msg="Applying tar in /var/lib/docker/overlay2/458b54b72a80967b2ba5dfca870ed5de222677fc98910538674fbf15ce958dda/diff" storage-driver=overlay2
12:18:54 level=debug msg="Applied tar sha256:003bb6178bc3218242d73e51d5e9ab2f991dc607780194719c6bd4c8c412fe8c to 458b54b72a80967b2ba5dfca870ed5de222677fc98910538674fbf15ce958dda, size: 65191894"
12:18:54 level=debug msg="Applying tar in /var/lib/docker/overlay2/0f4a3bdc5aa6c4428d3368143b0b26c92dd19e12c7c536d20a95a3fdc8a221d3/diff" storage-driver=overlay2
12:18:54 level=debug msg="Applied tar sha256:15b32d849da2239b1af583f9381c7a75d7aceba12f5ddfffa7a059116cf05ab9 to 0f4a3bdc5aa6c4428d3368143b0b26c92dd19e12c7c536d20a95a3fdc8a221d3, size: 32"
12:18:54 level=debug msg="Applying tar in /var/lib/docker/overlay2/c7905eabea23cd147b6772ce255d536b0cdcb759d4387b8282259b338d392c34/diff" storage-driver=overlay2
12:18:54 level=debug msg="Applied tar sha256:6e5c5f6bf043bc634378b1e4b61af09be74741f2ac80204d7a373713b1fd5a40 to c7905eabea23cd147b6772ce255d536b0cdcb759d4387b8282259b338d392c34, size: 5918893"

Unable to start the Docker daemon if file /etc/docker/daemon.json exist

Wheh the /etc/docker/daemon.json exists, docker no longer started!
I installed docker from snap, so service docker restart will not work.
I start docker with
sudo snap start docker
The output from journalctl -xe is:
Aug 20 09:08:44 user-TV kernel: aufs aufs_fill_super:912:mount[1404]: no arg
Aug 20 09:08:44 user-TV kernel: overlayfs: missing 'lowerdir'
Aug 20 08:55:29 user-TV audit[644]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/docker/daemon.json" pid=644 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
Aug 20 08:55:29 user-TV docker.dockerd[644]: unable to configure the Docker daemon with file /etc/docker/daemon.json: open /etc/docker/daemon.json: permission denied
Aug 20 08:55:29 user-TV kernel: audit: type=1400 audit(1534726529.513:7216): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/docker/daemon.json" pid=644 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
Aug 20 08:55:29 user-TV systemd[1]: snap.docker.dockerd.service: Main process exited, code=exited, status=1/FAILURE
Why it failed with open /etc/docker/daemon.json: permission denied in line 4 even I chmod 777 to it?
The content of the file is:
{
"experimental": true
}
The docker version is 17.06.2-ce
You have an AppArmor policy which is blocking access to this file.
Aug 20 08:55:29 user-TV audit[644]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/docker/daemon.json" pid=644 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
Modify the apparmor to allow this and you should be ok
I configure Docker to listen to both a Unix socket (default) and to an all-interfaces TCP port. After updating to Docker version 18.09.0, build 4d60db4 I could no longer start Docker while using /etc/docker/daemon.json on Centos 7.5.
journalctl -xe reported:
Nov 12 08:58:45 my.dev dockerd[6778]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts: (from flag: [unix://],
On systemd systems the daemon.json config option is no longer supported. Instead:
sudo rm /etc/daemon/daemon.json
sudo systemctl edit docker.service
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:///var/run/docker.sock -H tcp://0.0.0.0:1234
sudo systemctl daemon-reload
sudo systemctl restart docker.service

Artifactory: "docker service create" does not work with images of 0 byte

docker service create ... works even though compressed image size is 0B in Docker Hub. On the other hand, when I use Artifactory as private registry, it fails with No such image error. Docker daemons' debug logs say manifest verification failed for digest ...
As an example, compressed size of portainer's latest tag and main release tags (1.13.1, 1.13.2, etc.) are 0 B: https://hub.docker.com/r/portainer/portainer/tags/
Following command works:
docker service create \
--name portainer \
--publish 9000:9000 \
--constraint 'node.role == manager' \
--mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
portainer/portainer \
-H unix:///var/run/docker.sock
but following command does not work:
docker service create \
--name portainer \
--publish 9000:9000 \
--constraint 'node.role == manager' \
--mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
artifactory.mycompany.com/portainer/portainer \
-H unix:///var/run/docker.sock
Service's state:
[myuser#rose1]$ docker service ps --no-trunc portainer
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
gzk05p5x89w9pcvenuyio8pu8 portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Ready Rejected 2 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
fcovqtudbv3zmgo4von01y5wv \_ portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Shutdown Rejected 7 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
jfy8lr2prypcx72dryse5vmwx \_ portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Shutdown Rejected 12 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
3ovw7pwgr6srhvqocrqayiuqx \_ portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Shutdown Rejected 12 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
Docker daemon's debug logs:
...
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.137611299+03:00" level=debug msg="Trying to pull artifactory.mycompany.com/portainer/portainer from https://artifactory.mycompany.com v2"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.169441596+03:00" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher node.id=xdn6m020ugsnbfqfk2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.169573572+03:00" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher node.id=xdn6m020ugsnbfqfk2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175689648+03:00" level=debug msg="Pulling ref from V2 registry: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b0349
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175757143+03:00" level=error msg="manifest verification failed for digest sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175783178+03:00" level=info msg="Attempting next endpoint for pull after error: manifest verification failed for digest sha256:5393dc7fc9
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175800969+03:00" level=debug msg="Skipping v1 endpoint https://artifactory.mycompany.com because v2 registry was detected"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175878617+03:00" level=debug msg="pull in progress"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175909141+03:00" level=error msg="pulling image failed" error="manifest verification failed for digest sha256:5393dc7fc9e93f8ca8b034941a2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176596565+03:00" level=error msg="fatal task error" error="No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176643801+03:00" level=debug msg="state changed" module="node/agent/taskmanager" node.id=xdn6m020ugsnbfqfk2f5g74jx service.id=ve3ipsb1cx3
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176882355+03:00" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" node.id=xdn6m020ugsnbfqfk2f5g74jx task.id=3rzww5i46b8sv3
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.177387272+03:00" level=debug msg="task status reported" module="node/agent" node.id=xdn6m020ugsnbfqfk2f5g74jx
...
Artifactory logs:
...
2017-06-09 14:00:11,725 [http-nio-8081-exec-1] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:14,940 [http-nio-8081-exec-1] [INFO ] (o.a.r.HttpRepo :420) - registry-1.docker.io downloading https://registry-1.docker.io/v2/portainer/portainer/manifests/latest 944 bytes
2017-06-09 14:00:14,948 [http-nio-8081-exec-1] [INFO ] (o.a.r.HttpRepo :433) - registry-1.docker.io downloaded https://registry-1.docker.io/v2/portainer/portainer/manifests/latest 944 bytes at 125.43 KB/sec
2017-06-09 14:00:15,194 [http-nio-8081-exec-5] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:15,529 [http-nio-8081-exec-7] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:20,526 [http-nio-8081-exec-8] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
...
Update 1:
docker pull ... works properly:
docker pull artifactory.mycompany.com/portainer/portainer
and docker run ... also works properly:
docker run \
-v /var/lib/docker.sock:/var/lib/docker.sock \
-p 9000:9000 \
artifactory.mycompany.com/portainer/portainer \
-H unix:///var/run/docker.sock
The problem only exists with swarm mode.
Update 2:
As #Tony pointed out, if image is a multi-arch manifest (hence the 0B size) I have issues with Artifactory. For example, all the images under https://hub.docker.com/u/trollin are multi-arch and each tag of each image seem 0 Byte. I can reproduce the same issue with these images & tags. Take trollin/nginx, as an example.
Following commands work:
1)
docker pull artifactory.mycompany/trollin/nginx
2)
docker run --name trollin_nginx \
--publish 9991:80 \
artifactory.mycompany/trollin/nginx
3)
docker service create \
--name trollin_nginx \
--publish 9991:80 \
trollin/nginx
Following command does not work:
docker service create \
--name trollin_nginx \
--publish 9991:80 \
artifactory.mycompany.com/trollin/nginx

Resources