Why would `docker stop` kill the docker service? - jenkins

Our CI infrastructure (Jenkins) builds and tests docker containers. We have found that stopping a container used during test takes the docker service offline, disconnecting the slave from our Jenkins master. Why would stopping the memcached container kill and restart the docker service?
What we see in the job:
+ docker stop memcached55c7e2ab363be9425039412edc81be1e8601c196
FATAL: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:295)
at hudson.remoting.Channel.terminate(Channel.java:814)
/var/log/docker:
time="2015-07-29T11:43:07-04:00" level=info msg="POST /v1.18/containers/memcached55c7e2ab363be9425039412edc81be1e8601c196/stop?t=10"
time="2015-07-29T11:43:07-04:00" level=info msg="+job stop(memcached55c7e2ab363be9425039412edc81be1e8601c196)"
\nWed Jul 29 11:43:52 EDT 2015\n
time="2015-07-29T11:43:53-04:00" level=warning msg="You are running linux kernel version 2.6.32-504.30.3.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0."
time="2015-07-29T11:43:53-04:00" level=info msg="+job serveapi(unix:///var/run/docker.sock)"
time="2015-07-29T11:43:53-04:00" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
time="2015-07-29T11:43:53-04:00" level=info msg="+job init_networkdriver()"
time="2015-07-29T11:43:53-04:00" level=info msg="-job init_networkdriver() = OK (0)"
time="2015-07-29T11:43:53-04:00" level=info msg="Loading containers: start."
...time="2015-07-29T11:43:53-04:00" level=error msg="Error unmounting device 23607b95df9a186d80bb346c914cd01e7631c4fec3968bde29df7bc327f0b820: UnmountDevice: device not-mounted id 23607b95df9a186d80bb346c914cd01e7631c4fec3968bde29df7bc327f0b820"
.time="2015-07-29T11:43:53-04:00" level=error msg="Error unmounting device d01f73b589a9d598cc97752ce554b5c5dcf1b51d3991c1dee54880bb7d718fb7: UnmountDevice: device not-mounted id d01f73b589a9d598cc97752ce554b5c5dcf1b51d3991c1dee54880bb7d718fb7"
time="2015-07-29T11:43:53-04:00" level=info msg="Loading containers: done."
time="2015-07-29T11:43:53-04:00" level=info msg="docker daemon: 1.6.2 7c8fca2/1.6.2; execdriver: native-0.2; graphdriver: devicemapper"
time="2015-07-29T11:43:53-04:00" level=info msg="+job acceptconnections()"
time="2015-07-29T11:43:53-04:00" level=info msg="-job acceptconnections() = OK (0)"
time="2015-07-29T11:43:53-04:00" level=info msg="Daemon has completed initialization"
Versions:
-bash-4.1$ uname -a
Linux ip-172-17-70-142.ec2.internal 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
-bash-4.1$ docker --version
Docker version 1.6.2, build 7c8fca2/1.6.2
-bash-4.1$ docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2/1.6.2
OS/Arch (server): linux/amd64
-bash-4.1$ docker info
Containers: 2
Images: 65
Storage Driver: devicemapper
Pool Name: docker-202:64-936887-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.597 GB
Data Space Total: 107.4 GB
Data Space Available: 12.44 GB
Metadata Space Used: 3.584 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.30.3.el6.x86_64
Operating System: <unknown>
CPUs: 2
Total Memory: 7.166 GiB
Name: ip-172-17-70-142.ec2.internal
ID: X4UG:BKKW:5ZQX:VXRV:3GYP:FU3J:7FJC:4FFM:CK4D:SWSH:KIHW:IPNE
Username: dxbuildmaster
Registry: [https://index.docker.io/v1/]

Are you sure that this is caused by docker? We were using a Jenkins version in our environment that does not deal with docker at all and had the same error message. There is a known Jenkins bug about that which has been fixed in version 1.532.1. So which Jenkins version are you using?

Related

Docker failed to start daemon: Devices cgroup isn't mounted [Debian GNU/Linux 9 (stretch) ] armv7l

docker.log
time="2020-02-18T10:05:51.740757000+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded." storage-driver=overlay2
time="2020-02-18T10:05:51.747446000+08:00" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
time="2020-02-18T10:05:51.758181000+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded." storage-driver=overlay
time="2020-02-18T10:05:51.990661000+08:00" level=error msg="Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/dockerd/#storage-driver-options" storage-driver=devicemapper
time="2020-02-18T10:05:52.146274000+08:00" level=warning msg="Your kernel does not support cgroup memory limit"
time="2020-02-18T10:05:52.146389000+08:00" level=warning msg="Unable to find cpu cgroup in mounts"
time="2020-02-18T10:05:52.146445000+08:00" level=warning msg="Unable to find blkio cgroup in mounts"
time="2020-02-18T10:05:52.146504000+08:00" level=warning msg="Unable to find cpuset cgroup in mounts"
time="2020-02-18T10:05:52.146553000+08:00" level=warning msg="mountpoint for pids not found"
time="2020-02-18T10:05:52.149506000+08:00" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2020-02-18T10:05:52.160535000+08:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Devices cgroup isn't mounted
docker version
Client: Docker Engine - Community
Version: 19.03.6
API version: 1.40
Go version: go1.12.16
Git commit: 369ce74
Built: Thu Feb 13 01:40:25 2020
OS/Arch: linux/arm
Experimental: false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I have this literally bookmarked, and I go here every time I need to start docker on my netbooting Alpine. Single command to mount the cgroup pseudofs, w/o touching fstab:
sudo mount -t cgroup cgroup /sys/fs/cgroup
FYI: this may be also caused by your kernel not having cgroup support compiled in, but it's highly unlikely.

Docker 1.6.2 : "System error: no such device" When starting hello-world image

I'm trying to enable docker on arm32v7 target (Kernel 3.10.17 ) . I enabled needed components in kernel(AUFS, cgroup..) side but when I try to run Hello-world image to test docker integration I get this error :
docker run hello-world
FATA[0000] Error response from daemon:
Cannot start container ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9: [8] System error: no such device
Can anyone help please?
Thank you in advance !
Here docker version & info :
docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.3
Git commit (client): 7c8fca2-dirty
OS/Arch (client): linux/arm
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.3
Git commit (server): 7c8fca2-dirty
OS/Arch (server): linux/arm
docker info
Containers: 1
Images: 2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: tmpfs
Dirs: 4
Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.10.17-rt12+pamela-dual-lite-04.09.03+g46a870e
Operating System:
CPUs: 1
Total Memory: 497.2 MiB
Name: xxxxxxx
ID: QT4I:2EDK:MIRS:HV75:4JOW:PRAC:NS2F:JTAD:TQ7P:6MOO:O6VP:NM4K
WARNING: No swap limit support
Here logs :
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="-job log(start, ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9, hello-world:latest) = OK (0)"
systemd[1]: Starting docker container ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9.
systemd[1]: Started docker container ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9.
kernel: device vethe9ae91c entered promiscuous mode
kernel: IPv6: ADDRCONF(NETDEV_UP): vethe9ae91c: link is not ready
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethe9ae91c: link becomes ready
kernel: docker0: port 1(vethe9ae91c) entered forwarding state
docker0: port 1(vethe9ae91c) entered forwarding state
docker[4661]: time="2018-10-31T15:41:35Z" level=warning msg="signal: killed"
kernel: docker0: port 1(vethe9ae91c) entered disabled state
kernel: device vethe9ae91c left promiscuous mode
kernel: docker0: port 1(vethe9ae91c) entered disabled state
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="+job release_interface(ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9)"
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="-job release_interface(ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9) = OK (0)"
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="-job attach(ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9) = OK (0)"
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="+job release_interface(ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9)"
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="-job release_interface(ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9) = OK (0)"
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="+job log(die, ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9, hello-world:latest)"
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="-job log(die, ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9, hello-world:latest) = OK (0)"
docker[4661]: Cannot start container ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9: [8] System error: no such device
docker[4661]: time="2018-10-31T15:41:35Z" level=info msg="-job start(ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9) = ERR (1)"
docker[4661]: time="2018-10-31T15:41:35Z" level=error msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9: [8] System error: no such device"
docker[4661]: time="2018-10-31T15:41:35Z" level=error msg="HTTP Error: statusCode=404 Cannot start container ea3a71d1025ac733ebb9214045cdaffebfb0d8fb434c7a8bd83cdb93e4d7f5b9: [8] System error: no such device"
I fixed the issue by enabling CONFIG_POSIX_MQUEUE in kernel side . The error no such device is coming from the fact docker wants /dev/mqueue for container fs mounting.
Maybe there was another config option on runtime to select device , so we can avoid this kernel config .
docker run arm32v7/busybox ls
bin
dev
etc
home
proc
root
sys
tmp
usr
var
docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.3
Git commit (client): 7c8fca2-dirty
OS/Arch (client): linux/arm
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.3
Git commit (server): 7c8fca2-dirty
OS/Arch (server): linux/arm

Docker takes 4 minutes to start on debian

I observed that the docker takes 4 minutes to start each time after reboot. I have provided some details as attached. Any thoughts/tips are appreciated. Thanks.
Debian
-> docker -v
Docker version 18.03.0-ce, build 0520e24
-> uname -a
Linux x 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux
Other logs:
-> systemd-analyze blame
4min 21.855s docker.service
2.863s cloud-final.service
2.021s cloud-init-local.service
1.698s cloud-init.service
1.476s cloud-config.service
…
21ms systemd-tmpfiles-clean.service
19ms systemd-update-utmp.service
18ms docker.socket
-> sudo journalctl -u docker.service
– Logs begin at Sun 2018-05-06 20:49:58 PDT, end at Sun 2018-05-06 21:11:45 PDT. –
May 06 20:50:02 x systemd[1]: Starting Docker Application Container Engine…
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23.757674968-07:00” level=debug msg=“Listener created for HTTP on fd ()”
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23.769857584-07:00” level=info msg=“libcontainerd: started new docker-containerd process” pid=1328
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“starting containerd” module=containerd revision=cfd04396dc68220d1cecbe686a6cc3aa5ce3667c version=v1.0.2
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=debug msg=“changing OOM score to -500” module=containerd
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.content.v1.content”…” module=containerd type=io.containerd.content.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.snapshotter.v1.btrfs”…” module=containerd type=io.containerd.snapshotter.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=warning msg=“failed to load plugin io.containerd.snapshotter.v1.btrfs” error="path /var/lib/docker/containerd/daemon/io.containerd.sn
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.snapshotter.v1.overlayfs”…” module=containerd type=io.containerd.snapshotter.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.metadata.v1.bolt”…” module=containerd type=io.containerd.metadata.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=warning msg=“could not use snapshotter btrfs in metadata plugin” error="path /var/lib/docker/containerd/daemon/io.containerd.snapshot
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.differ.v1.walking”…” module=containerd type=io.containerd.differ.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.gc.v1.scheduler”…” module=containerd type=io.containerd.gc.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.grpc.v1.containers”…” module=containerd type=io.containerd.grpc.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.grpc.v1.content”…” module=containerd type=io.containerd.grpc.v1
I posted the question in docker forum, but got no luck.
https://forums.docker.com/t/docker-takes-4-minutes-to-start-on-debian/50441
====== updated #1
System info: 1G RAM, 25 GB SSD, Clean system (just installed)
> docker info
xiejw:x -> docker info
Containers: 8
Running: 3
Paused: 0
Stopped: 5
Images: 15
Server Version: 18.03.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-6-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.5MiB
Name: x
ID: Y3MP:DHWK:4VES:2LEZ:RHJK:UFCB:HX2W:4CK2:XHBI:CG7B:PBT3:2LTG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 37
Goroutines: 45
System Time: 2018-05-09T10:44:20.217001219-07:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support

When running the 'hello world' test container for new installation of Docker, i get an OCI runtime error

I've installed Docker on Centos 7 using the yum process. Below is the output for systemctl status docker.service -l:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-01-02 14:11:09 GMT; 1h 15min ago
Docs: https://docs.docker.com
Main PID: 136537 (dockerd)
Memory: 71.3M
CGroup: /system.slice/docker.service
├─136537 /usr/bin/dockerd
└─136556 docker-containerd --config /var/run/docker/containerd/containerd.toml
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157175048Z" level=error msg="c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5 cleanup: failed to delete container from containerd: no such container"
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157230927Z" level=error msg="Handler for POST /v1.35/containers/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.837926048Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/shim.sock" debug=false module="containerd/tasks" pid=145298
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim reaped" id=3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f module="containerd/tasks"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899560066Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899585062Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.921068216Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309201262Z" level=error msg="3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f cleanup: failed to delete container from containerd: no such container"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309254679Z" level=error msg="Handler for POST /v1.35/containers/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
The error i get (sudo docker run hello-world) is:
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown.
ERRO[0001] error waiting for container: context canceled
docker --version
Docker version 17.12.0-ce, build c97c6d6
and here is the docker info
Containers: 27
Running: 0
Paused: 0
Stopped: 27
Images: 47
Server Version: 17.12.0-ce
Storage Driver: devicemapper
Pool Name: docker-253:0-135160107-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 14.4GB
Data Space Total: 107.4GB
Data Space Available: 25.82GB
Metadata Space Used: 12.6MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.135GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: N/A (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 88
Total Memory: 125.6GiB
Name: colossus
ID: HV7I:KP2J:N6SQ:SAEP:MWSO:O5T3:Y2PE:L6TL:2D6L:IHVD:SQ3E:TM5P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
You most likely have an old package installed that is breaking runc. Make sure to upgrade your operating system to the current release (for CentOS, this is 1708, or the RHEL equivalent of 7.4). Others have reported similar issues related to packages being out of date.
I had 2 systems with identical CentOS 7.2 versions and kernel. Had this problem with one but not the other. Comparing package versions showed a difference in the version of containerd.io -- I downgraded that on the non-working version and it worked:
$ sudo yum downgrade containerd.io-1.2.0-3.el7
Removed:
containerd.io.x86_64 0:1.2.4-3.1.el7
Installed:
containerd.io.x86_64 0:1.2.0-3.el7

Error on Docker push “denied: requested access to the resource is denied”

Environment
a) Cent OS VM's created in Hyper V.
b) Behind Corporate proxy.
I am experiencing strange issue when doing docker push.
I am getting below error
"denied: requested access to the resource is denied"
Docker pull is working without any issues.
I have attached daemon logs generated during docker push.
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.886973000+10:00" level=debug msg="Calling GET /_ping"
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.887518100+10:00" level=debug msg="Calling GET /v1.27/info"
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.893846700+10:00" level=debug msg="Calling POST /v1.27/images/vishnuvpotti/test/push?tag="
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.893969800+10:00" level=debug msg="Trying to push docker.io/vishnuvpotti/test to https://registry-1.docker.io v2"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.863929300+10:00" level=debug msg="Pushing repository: vishnuvpotti/test:latest"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.864814900+10:00" level=debug msg="Pushing layer: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865066700+10:00" level=debug msg="Pushing layer: sha256:defd90aaa156603d9362f29909f6f5539df841b6398b3836a5d559066295ec2b"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865155500+10:00" level=debug msg="Pushing layer: sha256:be4ee57de94e9c212ac27cf3b2f7ec826617a622d1e10f5b28558cce1f71ec24"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865233700+10:00" level=debug msg="Pushing layer: sha256:f971a434fe54442b68fe4d8bb9dc58cc51bfef3d960985fd1d2243a36abe89c2"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865308500+10:00" level=debug msg="Pushing layer: sha256:68078adbc0cb48954d27352b190afc2b011b3bb3c2adf61c45991669ea02134e"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.691569600+10:00" level=error msg="Upload failed: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.691661000+10:00" level=debug msg="Pushing layer: sha256:fb479b796fe5e208adece47fc14092d57c43b1dc0978ddd6a645cbdc7e85bbff"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694653100+10:00" level=error msg="Upload failed: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694753300+10:00" level=debug msg="Pushing layer: sha256:f32dba9a1eadb592cceb96d7b3fdc4a1c5213044fd7b30ae41e17673ccdf385d"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694875200+10:00" level=error msg="Attempting next endpoint for push after error: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.699788800+10:00" level=debug msg="Pushing layer: sha256:19cbee8a76396ae7b80f835da4be7229dc8b094535ebf50d35ab2bf941039f7a"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.701883600+10:00" level=debug msg="Pushing layer: sha256:5a29dcc5d0b483d880a4736853253433c16103b9e03f9f52e94577cf1895563b"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.707591200+10:00" level=debug msg="Pushing layer: sha256:92f9849bf0840ff917901665d775da4454d84a23439ef2ccbb93c6cc3363ab82"
Aug 15 10:02:44 testmachine dockerd[40449]: time="2017-08-15T10:02:44.588052200+10:00" level=debug msg="Pushing layer: sha256:74351e413c54d36337dff701c3568dc291f45523fe74261281b9ec9372528092"
Docker Version
root#testmachine# docker version
Client:
Version: 17.03.2-ce
API version: 1.27
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 02:21:36 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.2-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 02:21:36 2017
OS/Arch: linux/amd64
Experimental: false
Docker Info
root#testmachine # docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 2
Server Version: 17.03.2-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.549 GiB
Name: testmachine
ID: ATWM:MTHC:B2BA:6E4E:RAWL:E5VD:OUV2:2QHV:IBVQ:K525:DIPF:ICHC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 21
Goroutines: 26
System Time: 2017-08-15T11:21:02.8957667+10:00
EventsListeners: 0
Http Proxy: http://proxy
Https Proxy: https://proxy
No Proxy: localhost,127.0.0.1
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
Reformat the filesystem with ftype=1 to enable d_type support.
Running without d_type support will not be supported in future releases.
Note:
1) I have checked my repo settings in docker hub and Default Repository Visibility is public.
2) I deleted ~/.docker/config.json, restarted docker and tried.
3) I also tried docker login -u myusername
Can anyone please let me know what is going wrong. Is it an issue with new docker version?
Update 1:
1) I have updated docker version to the latest i.e : 17.06.0-ce . Still not working.
Update 2
2) I created a ubuntu VM in virtual box and gave a try on latest docker version. Installed ca certificate,edited proxy infromation for both machine and docker. Still facing same issues.
3)Update 3 (16/08/2017)
Turned off proxy, switched to a network without proxy,restarted docker and tried. Still same issue.
4) Update 4 (17/08/2017)
I created an instance in AWS and was able to push to docker registry. So it must be some issue in corporate network. Any comments appreciated.

Resources