Every pull I run hangs on "Extracting"
# docker -v pull ansible/awx_rabbitmq:3.7.4
3.7.4: Pulling from ansible/awx_rabbitmq
ff3a5c916c92: Extracting [> ] 32.77kB/2.066MB
5387f4b4c52b: Download complete
dba8c403a5b6: Download complete
4258fc50c523: Download complete
41e241289d30: Download complete
7a8ab8823f42: Download complete
21ac0c0b3f13: Download complete
56b9421a89dc: Download complete
e69676b835e9: Download complete
13a893bc00f9: Download complete
e7cf78370af8: Download complete
363d82081450: Download complete
bc716b889e7b: Download complete
ebd8aacef79e: Download complete
839d6f7a7803: Download complete
this is the syslog output (docker daemon verbosity in debug):
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.216589013+01:00" level=debug msg="Calling HEAD /_ping"
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.216881715+01:00" level=debug msg="Calling GET /v1.41/info"
Jan 26 13:10:56 XXXMACHINENAME dbus-daemon[24071]: [session uid=0 pid=24069] Activating service name='org.freedesktop.secrets' requested by ':1.0' (uid=0 pid=24060 comm="docker-credential-secretservice get " label="unconfined")
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.274306708+01:00" level=debug msg="Calling POST /v1.41/images/create?fromImage=ansible%2Fawx_rabbitmq&tag=3.7.4"
Jan 26 13:10:56 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:56.275348977+01:00" level=debug msg="Trying to pull ansible/awx_rabbitmq from https://registry-1.docker.io v2"
Jan 26 13:10:57 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:57.668102153+01:00" level=debug msg="Pulling ref from V2 registry: ansible/awx_rabbitmq:3.7.4"
Jan 26 13:10:57 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:57.668519475+01:00" level=debug msg="pulling blob \"sha256:dba8c403a5b6fbb5651fd71cc7e2c96605165864b4ee509d2b6676e2958b8164\""
Jan 26 13:10:57 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:57.668554305+01:00" level=debug msg="pulling blob \"sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28\""
...
Jan 26 13:10:58 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:10:58.548777211+01:00" level=debug msg="Using /usr/bin/unpigz to decompress"
...
Jan 26 13:11:04 XXXMACHINENAME dockerd[23893]: time="2022-01-26T13:11:04.347774528+01:00" level=debug msg="Downloaded ebd8aacef79e to tempfile /var/lib/docker/tmp/GetImageBlob810059367"
Not a disk space problem, the filesystem is almost empty.
Related
As described here How to setup docker private registry on ubuntu 16.04,
I changed /etc/hosts like this:
192.168.1.154 registry-server
192.168.1.90 registry-client
Then I pulled the registry image:
docker pull registry
Then I made certificate files
mkdir /etc/certs
cd /etc/certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crt
I copied the ca.crt to these pathes in client host:
/etc/certs/
/etc/docker/certs.d/registry-server:5000/
Then I ran the container on the server host:
docker run -d -p 5000:5000 --restart=always --name registry -v /etc/certs:/etc/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/certs/ca.crt -e REGISTRY_HTTP_TLS_KEY=/etc/certs/ca.key registry
I tagged the image
docker tag phpmyadmin/phpmyadmin:latest registry-server:5000/pma-test
But when I want to push the image registry-server:5000/pma-test to the server:
docker push registry-server:5000/pma-test:latest
The following error occurs:
Error response from daemon: open /etc/docker/certs.d/registry-server:5000: permission denied
======================================
Update:
I ran journalctl -xe, and found these errors:
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.229097561Z" level=debug msg="Calling GET /_ping"
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238248010Z" level=debug msg="Calling POST /v1.38/images/registry-server:5000/pma-test/push?tag="
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238670117Z" level=debug msg="hostDir: /etc/docker/certs.d/registry-server:5000"
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238797277Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: open /etc/docker/certs.d/registry-server:5000: permission denied" error_type="*os.PathError" module=api
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238831133Z" level=error msg="Handler for POST /v1.38/images/registry-server:5000/pma-test/push returned error: open /etc/docker/certs.d/registry-server:5000: permission denied"
Sep 30 13:58:37 docker.dockerd[926]: time="2019-09-30T13:58:37.238861895Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: open /etc/docker/certs.d/registry-server:5000: permission denied" error_type="*os.PathError" module=api
Sep 30 13:58:37 audit[926]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/docker/certs.d/registry-server:5000/" pid=926 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Sep 30 13:58:37 kernel: audit: type=1400 audit(1569851917.234:53): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/docker/certs.d/registry-server:5000/" pid=926 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Why did this error happen?
Docker version 19.03.2, build 6a30dfc
docker-compose version 1.24.0, build 0aa59064
Server and client host: Ubuntu 18.04
Finally, I found it:
I added the following line to /var/lib/snapd/apparmor/profiles/snap.docker.docker
/etc/docker/certs.d/** r,
Then I ran:
apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.docker.dockerd
And the problem solved.
Hi all i have found a strange issue in my docker swarm standalone mode infrastucture: on one server and only one service is automatically paused after 5 second of starting . docker unpause also does not helps .
020450186dc1 myservice:latest "/bin/sh -c '/app/os…" 13 minutes ago Up 13 minutes (Paused) server-15.myservice_2
After some time this service is in pause mode again . I have no idea who can stop this service and where can i see some logging info about it . I know that the service can be killed but i have never heard about pause . The other instances of the same service are working stable.
UPD:
I dont see any special information on this issue even in docker debug mode . I just see that somebody pauses my container:
Apr 29 11:50:01 dockerd[1069]: time="2019-04-29T11:50:01.539113504+02:00" level=debug msg="Calling GET /_ping"
Apr 29 11:50:01 dockerd[1069]: time="2019-04-29T11:50:01.539687070+02:00" level=debug msg="Calling POST /v1.38/containers/debee0c7acb9aa0db2021719ee6ee3d084b88e47d9f2250b1af920f5271bf353/pause"
Apr 29 11:50:01 dockerd[1069]: time="2019-04-29T11:50:01+02:00" level=debug msg="event published" ns=moby topic="/tasks/paused" type=containerd.events.TaskPaused
Apr 29 11:50:01 dockerd[1069]: time="2019-04-29T11:50:01.570125095+02:00" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/paused
$ docker pull python:3.6.4-stretch
3.6.4-stretch: Pulling from library/python
c73ab1c6897b: Pull complete
1ab373b3deae: Downloading [=============================================> ] 10.13MB/11.11MB
b542772b4177: Download complete
57c8de432dbe: Download complete
1ab373b3deae: Pull complete
b542772b4177: Pull complete
57c8de432dbe: Pull complete
1785850988c5: Pull complete
676ef2d8682b: Pull complete
56321bcc2d38: Pull complete
4788c366a216: Pull complete
0d970fbfeb26: Pull complete
Digest: sha256:db22cb78ba16cb6a0632eead1e48a239636a5a77c9f8cf343087acf309ad0248
Status: Downloaded newer image for python:3.6.4-stretch
Time: 0h:05m:33s
Download hangs with a probability of 80% or more like above ouput. Then hold this state for 5 minutes and pull will succeed when retry starts.
For more detail, this problem occurs in three ubuntu pc.
Two are Ubuntu 16.04 and one is 18.04. All machine are on the same office network.
At first I tried changing the docker and ubuntu versions but it failed. service docker restart was also useless. I noticed that I installed a new gigabit switch hub(https://iptime.com/iptime/?page_id=11&pf=12&page=2&pt=311&pd=1), and I suspected the hub device. It works well when machine connects directly to the LAN without a switching hub and When I changed the switching hub to the old 100Mb/s thing, it worked well also.
It can be judged as a problem of the gigabit switching hub, but it is difficult to find out because all other internet use is work well with gigabit switching hub. So I wonder if there is no other problem with docker pull or there is no other solution.
$ uname -a
Linux my-ubuntu18.04 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:48:57 2018
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:16:44 2018
OS/Arch: linux/amd64
Experimental: false
Please search failed keyword in below my docker daemon log.
12:13:32 level=debug msg="Calling GET /_ping"
12:13:32 level=debug msg="Calling GET /v1.39/info"
12:13:32 level=debug msg="Calling POST /v1.39/images/create?fromImage=python&tag=3.6.6-stretch"
12:13:32 level=debug msg="Trying to pull python from https://registry-1.docker.io v2"
12:13:35 level=debug msg="Pulling ref from V2 registry: python:3.6.6-stretch"
12:13:35 level=debug msg="docker.io/library/python:3.6.6-stretch resolved to a manifestList object with 7 entries; looking for a unknown/amd64 match"
12:13:35 level=debug msg="found match for linux/amd64 with media type application/vnd.docker.distribution.manifest.v2+json, digest sha256:c306863aa2e858ccf00958c625ca2ffdbf8845da76e266b0b0d9c4760170aff3"
12:13:36 level=debug msg="Layer already exists: bc9ab73e5b14"
12:13:36 level=debug msg="Layer already exists: 193a6306c92a"
12:13:36 level=debug msg="Layer already exists: e5c3f8c317dc"
12:13:36 level=debug msg="Layer already exists: a587a86c9dcb"
12:13:36 level=debug msg="pulling blob \"sha256:72744d0a318b0788001cc4f5f83c6847ba4b753307fadd046b508bbc41eb9e29\""
12:13:36 level=debug msg="pulling blob \"sha256:6598fc9d11d10365ac9281071a87930a2382ee31d026f1b6d432717b31db387c\""
12:13:37 level=debug msg="Downloaded 6598fc9d11d1 to tempfile /var/lib/docker/tmp/GetImageBlob402585872"
12:13:37 level=debug msg="pulling blob \"sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f\""
12:13:40 level=debug msg="Downloaded 72744d0a318b to tempfile /var/lib/docker/tmp/GetImageBlob083083061"
12:13:40 level=debug msg="pulling blob \"sha256:93612f47cdc374d0b33057b9e71eac173ac469da3e1a631dc8a32ba6986a408a\""
12:13:40 level=debug msg="Applying tar in /var/lib/docker/overlay2/9eaab31d9a1f108ba8a5c712cf23f36a9097140d09c76e6b966667fba2cc014b/diff" storage-driver=overlay2
12:13:42 level=debug msg="Downloaded 93612f47cdc3 to tempfile /var/lib/docker/tmp/GetImageBlob281534658"
12:13:42 level=debug msg="pulling blob \"sha256:1bc4b4b508703799ef67a807dacce4736045e642e87bcd49871cd0f23e7f5b8b\""
12:13:43 level=debug msg="Downloaded 1bc4b4b50870 to tempfile /var/lib/docker/tmp/GetImageBlob872144708"
12:13:48 level=debug msg="Applied tar sha256:9978d084fd771e0b3d1acd7f3525d1b25288ababe9ad8ed259b36101e4e3addd to 9eaab31d9a1f108ba8a5c712cf23f36a9097140d09c76e6b966667fba2cc014b, size: 556457027"
12:13:48 level=debug msg="Applying tar in /var/lib/docker/overlay2/3f91f78b3bb3f2cb6096472759bb84ae2f30f0825a7f935cad3b420c5cd71bee/diff" storage-driver=overlay2
12:13:48 level=debug msg="Applied tar sha256:2f4f74d3821ecbdd60b5d932452ea9e30cecf902334165c4a19837f6ee636377 to 3f91f78b3bb3f2cb6096472759bb84ae2f30f0825a7f935cad3b420c5cd71bee, size: 16849952"
12:18:46 level=error msg="Download failed, retrying: read tcp 10.251.12.218:48728->104.18.121.25:443: read: connection timed out"
12:18:51 level=debug msg="pulling blob \"sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f\""
12:18:51 level=debug msg="attempting to resume download of \"sha256:4b1d9004d467b4e710d770a881df027df7e5e7e4629f6e473760893ffc1a667f\" from 20499209 bytes"
12:18:53 level=debug msg="Downloaded 4b1d9004d467 to tempfile /var/lib/docker/tmp/GetImageBlob954105135"
12:18:53 level=debug msg="Applying tar in /var/lib/docker/overlay2/458b54b72a80967b2ba5dfca870ed5de222677fc98910538674fbf15ce958dda/diff" storage-driver=overlay2
12:18:54 level=debug msg="Applied tar sha256:003bb6178bc3218242d73e51d5e9ab2f991dc607780194719c6bd4c8c412fe8c to 458b54b72a80967b2ba5dfca870ed5de222677fc98910538674fbf15ce958dda, size: 65191894"
12:18:54 level=debug msg="Applying tar in /var/lib/docker/overlay2/0f4a3bdc5aa6c4428d3368143b0b26c92dd19e12c7c536d20a95a3fdc8a221d3/diff" storage-driver=overlay2
12:18:54 level=debug msg="Applied tar sha256:15b32d849da2239b1af583f9381c7a75d7aceba12f5ddfffa7a059116cf05ab9 to 0f4a3bdc5aa6c4428d3368143b0b26c92dd19e12c7c536d20a95a3fdc8a221d3, size: 32"
12:18:54 level=debug msg="Applying tar in /var/lib/docker/overlay2/c7905eabea23cd147b6772ce255d536b0cdcb759d4387b8282259b338d392c34/diff" storage-driver=overlay2
12:18:54 level=debug msg="Applied tar sha256:6e5c5f6bf043bc634378b1e4b61af09be74741f2ac80204d7a373713b1fd5a40 to c7905eabea23cd147b6772ce255d536b0cdcb759d4387b8282259b338d392c34, size: 5918893"
Environment
a) Cent OS VM's created in Hyper V.
b) Behind Corporate proxy.
I am experiencing strange issue when doing docker push.
I am getting below error
"denied: requested access to the resource is denied"
Docker pull is working without any issues.
I have attached daemon logs generated during docker push.
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.886973000+10:00" level=debug msg="Calling GET /_ping"
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.887518100+10:00" level=debug msg="Calling GET /v1.27/info"
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.893846700+10:00" level=debug msg="Calling POST /v1.27/images/vishnuvpotti/test/push?tag="
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.893969800+10:00" level=debug msg="Trying to push docker.io/vishnuvpotti/test to https://registry-1.docker.io v2"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.863929300+10:00" level=debug msg="Pushing repository: vishnuvpotti/test:latest"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.864814900+10:00" level=debug msg="Pushing layer: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865066700+10:00" level=debug msg="Pushing layer: sha256:defd90aaa156603d9362f29909f6f5539df841b6398b3836a5d559066295ec2b"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865155500+10:00" level=debug msg="Pushing layer: sha256:be4ee57de94e9c212ac27cf3b2f7ec826617a622d1e10f5b28558cce1f71ec24"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865233700+10:00" level=debug msg="Pushing layer: sha256:f971a434fe54442b68fe4d8bb9dc58cc51bfef3d960985fd1d2243a36abe89c2"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865308500+10:00" level=debug msg="Pushing layer: sha256:68078adbc0cb48954d27352b190afc2b011b3bb3c2adf61c45991669ea02134e"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.691569600+10:00" level=error msg="Upload failed: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.691661000+10:00" level=debug msg="Pushing layer: sha256:fb479b796fe5e208adece47fc14092d57c43b1dc0978ddd6a645cbdc7e85bbff"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694653100+10:00" level=error msg="Upload failed: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694753300+10:00" level=debug msg="Pushing layer: sha256:f32dba9a1eadb592cceb96d7b3fdc4a1c5213044fd7b30ae41e17673ccdf385d"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694875200+10:00" level=error msg="Attempting next endpoint for push after error: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.699788800+10:00" level=debug msg="Pushing layer: sha256:19cbee8a76396ae7b80f835da4be7229dc8b094535ebf50d35ab2bf941039f7a"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.701883600+10:00" level=debug msg="Pushing layer: sha256:5a29dcc5d0b483d880a4736853253433c16103b9e03f9f52e94577cf1895563b"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.707591200+10:00" level=debug msg="Pushing layer: sha256:92f9849bf0840ff917901665d775da4454d84a23439ef2ccbb93c6cc3363ab82"
Aug 15 10:02:44 testmachine dockerd[40449]: time="2017-08-15T10:02:44.588052200+10:00" level=debug msg="Pushing layer: sha256:74351e413c54d36337dff701c3568dc291f45523fe74261281b9ec9372528092"
Docker Version
root#testmachine# docker version
Client:
Version: 17.03.2-ce
API version: 1.27
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 02:21:36 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.2-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 02:21:36 2017
OS/Arch: linux/amd64
Experimental: false
Docker Info
root#testmachine # docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 2
Server Version: 17.03.2-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.549 GiB
Name: testmachine
ID: ATWM:MTHC:B2BA:6E4E:RAWL:E5VD:OUV2:2QHV:IBVQ:K525:DIPF:ICHC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 21
Goroutines: 26
System Time: 2017-08-15T11:21:02.8957667+10:00
EventsListeners: 0
Http Proxy: http://proxy
Https Proxy: https://proxy
No Proxy: localhost,127.0.0.1
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
Reformat the filesystem with ftype=1 to enable d_type support.
Running without d_type support will not be supported in future releases.
Note:
1) I have checked my repo settings in docker hub and Default Repository Visibility is public.
2) I deleted ~/.docker/config.json, restarted docker and tried.
3) I also tried docker login -u myusername
Can anyone please let me know what is going wrong. Is it an issue with new docker version?
Update 1:
1) I have updated docker version to the latest i.e : 17.06.0-ce . Still not working.
Update 2
2) I created a ubuntu VM in virtual box and gave a try on latest docker version. Installed ca certificate,edited proxy infromation for both machine and docker. Still facing same issues.
3)Update 3 (16/08/2017)
Turned off proxy, switched to a network without proxy,restarted docker and tried. Still same issue.
4) Update 4 (17/08/2017)
I created an instance in AWS and was able to push to docker registry. So it must be some issue in corporate network. Any comments appreciated.
docker service create ... works even though compressed image size is 0B in Docker Hub. On the other hand, when I use Artifactory as private registry, it fails with No such image error. Docker daemons' debug logs say manifest verification failed for digest ...
As an example, compressed size of portainer's latest tag and main release tags (1.13.1, 1.13.2, etc.) are 0 B: https://hub.docker.com/r/portainer/portainer/tags/
Following command works:
docker service create \
--name portainer \
--publish 9000:9000 \
--constraint 'node.role == manager' \
--mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
portainer/portainer \
-H unix:///var/run/docker.sock
but following command does not work:
docker service create \
--name portainer \
--publish 9000:9000 \
--constraint 'node.role == manager' \
--mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock \
artifactory.mycompany.com/portainer/portainer \
-H unix:///var/run/docker.sock
Service's state:
[myuser#rose1]$ docker service ps --no-trunc portainer
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
gzk05p5x89w9pcvenuyio8pu8 portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Ready Rejected 2 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
fcovqtudbv3zmgo4von01y5wv \_ portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Shutdown Rejected 7 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
jfy8lr2prypcx72dryse5vmwx \_ portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Shutdown Rejected 12 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
3ovw7pwgr6srhvqocrqayiuqx \_ portainer.1 artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40 rose1 Shutdown Rejected 12 seconds ago "No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d0558cd40"
Docker daemon's debug logs:
...
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.137611299+03:00" level=debug msg="Trying to pull artifactory.mycompany.com/portainer/portainer from https://artifactory.mycompany.com v2"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.169441596+03:00" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher node.id=xdn6m020ugsnbfqfk2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.169573572+03:00" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher node.id=xdn6m020ugsnbfqfk2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175689648+03:00" level=debug msg="Pulling ref from V2 registry: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93f8ca8b0349
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175757143+03:00" level=error msg="manifest verification failed for digest sha256:5393dc7fc9e93f8ca8b034941a2c6af0ae176c89c92728d4ff0e110d
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175783178+03:00" level=info msg="Attempting next endpoint for pull after error: manifest verification failed for digest sha256:5393dc7fc9
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175800969+03:00" level=debug msg="Skipping v1 endpoint https://artifactory.mycompany.com because v2 registry was detected"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175878617+03:00" level=debug msg="pull in progress"
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.175909141+03:00" level=error msg="pulling image failed" error="manifest verification failed for digest sha256:5393dc7fc9e93f8ca8b034941a2
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176596565+03:00" level=error msg="fatal task error" error="No such image: artifactory.mycompany.com/portainer/portainer:latest#sha256:5393dc7fc9e93
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176643801+03:00" level=debug msg="state changed" module="node/agent/taskmanager" node.id=xdn6m020ugsnbfqfk2f5g74jx service.id=ve3ipsb1cx3
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.176882355+03:00" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" node.id=xdn6m020ugsnbfqfk2f5g74jx task.id=3rzww5i46b8sv3
Jun 08 12:29:58 rose1 dockerd[14289]: time="2017-06-08T12:29:58.177387272+03:00" level=debug msg="task status reported" module="node/agent" node.id=xdn6m020ugsnbfqfk2f5g74jx
...
Artifactory logs:
...
2017-06-09 14:00:11,725 [http-nio-8081-exec-1] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:14,940 [http-nio-8081-exec-1] [INFO ] (o.a.r.HttpRepo :420) - registry-1.docker.io downloading https://registry-1.docker.io/v2/portainer/portainer/manifests/latest 944 bytes
2017-06-09 14:00:14,948 [http-nio-8081-exec-1] [INFO ] (o.a.r.HttpRepo :433) - registry-1.docker.io downloaded https://registry-1.docker.io/v2/portainer/portainer/manifests/latest 944 bytes at 125.43 KB/sec
2017-06-09 14:00:15,194 [http-nio-8081-exec-5] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:15,529 [http-nio-8081-exec-7] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
2017-06-09 14:00:20,526 [http-nio-8081-exec-8] [INFO ] (o.a.a.d.r.v.r.v.DockerV2VirtualRepoHandler:105) - Fetching docker manifest for repo 'portainer/portainer' and tag 'latest'
...
Update 1:
docker pull ... works properly:
docker pull artifactory.mycompany.com/portainer/portainer
and docker run ... also works properly:
docker run \
-v /var/lib/docker.sock:/var/lib/docker.sock \
-p 9000:9000 \
artifactory.mycompany.com/portainer/portainer \
-H unix:///var/run/docker.sock
The problem only exists with swarm mode.
Update 2:
As #Tony pointed out, if image is a multi-arch manifest (hence the 0B size) I have issues with Artifactory. For example, all the images under https://hub.docker.com/u/trollin are multi-arch and each tag of each image seem 0 Byte. I can reproduce the same issue with these images & tags. Take trollin/nginx, as an example.
Following commands work:
1)
docker pull artifactory.mycompany/trollin/nginx
2)
docker run --name trollin_nginx \
--publish 9991:80 \
artifactory.mycompany/trollin/nginx
3)
docker service create \
--name trollin_nginx \
--publish 9991:80 \
trollin/nginx
Following command does not work:
docker service create \
--name trollin_nginx \
--publish 9991:80 \
artifactory.mycompany.com/trollin/nginx