I observed that the docker takes 4 minutes to start each time after reboot. I have provided some details as attached. Any thoughts/tips are appreciated. Thanks.
Debian
-> docker -v
Docker version 18.03.0-ce, build 0520e24
-> uname -a
Linux x 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux
Other logs:
-> systemd-analyze blame
4min 21.855s docker.service
2.863s cloud-final.service
2.021s cloud-init-local.service
1.698s cloud-init.service
1.476s cloud-config.service
…
21ms systemd-tmpfiles-clean.service
19ms systemd-update-utmp.service
18ms docker.socket
-> sudo journalctl -u docker.service
– Logs begin at Sun 2018-05-06 20:49:58 PDT, end at Sun 2018-05-06 21:11:45 PDT. –
May 06 20:50:02 x systemd[1]: Starting Docker Application Container Engine…
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23.757674968-07:00” level=debug msg=“Listener created for HTTP on fd ()”
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23.769857584-07:00” level=info msg=“libcontainerd: started new docker-containerd process” pid=1328
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“starting containerd” module=containerd revision=cfd04396dc68220d1cecbe686a6cc3aa5ce3667c version=v1.0.2
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=debug msg=“changing OOM score to -500” module=containerd
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.content.v1.content”…” module=containerd type=io.containerd.content.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.snapshotter.v1.btrfs”…” module=containerd type=io.containerd.snapshotter.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=warning msg=“failed to load plugin io.containerd.snapshotter.v1.btrfs” error="path /var/lib/docker/containerd/daemon/io.containerd.sn
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.snapshotter.v1.overlayfs”…” module=containerd type=io.containerd.snapshotter.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.metadata.v1.bolt”…” module=containerd type=io.containerd.metadata.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=warning msg=“could not use snapshotter btrfs in metadata plugin” error="path /var/lib/docker/containerd/daemon/io.containerd.snapshot
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.differ.v1.walking”…” module=containerd type=io.containerd.differ.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.gc.v1.scheduler”…” module=containerd type=io.containerd.gc.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.grpc.v1.containers”…” module=containerd type=io.containerd.grpc.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.grpc.v1.content”…” module=containerd type=io.containerd.grpc.v1
I posted the question in docker forum, but got no luck.
https://forums.docker.com/t/docker-takes-4-minutes-to-start-on-debian/50441
====== updated #1
System info: 1G RAM, 25 GB SSD, Clean system (just installed)
> docker info
xiejw:x -> docker info
Containers: 8
Running: 3
Paused: 0
Stopped: 5
Images: 15
Server Version: 18.03.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-6-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.5MiB
Name: x
ID: Y3MP:DHWK:4VES:2LEZ:RHJK:UFCB:HX2W:4CK2:XHBI:CG7B:PBT3:2LTG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 37
Goroutines: 45
System Time: 2018-05-09T10:44:20.217001219-07:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Related
I am facing a strange issue with the docker image pull command where the command fails with this error;
[desai#brilp0017 ~]$ docker image pull nginx:latest
latest: Pulling from library/nginx
d121f8d1c412: Extracting [==================================================>]
27.09MB/27.09MB
ebd81fc8c071: Download complete
655316c160af: Download complete
d15953c0e0f8: Download complete
2ee525c5c3cc: Download complete
failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root534731447: device or resource busy
After this error the docker daemon is no longer accessible and all docker commands return following error;
[desai#brilp0017 ~]$ docker info
Client:
Debug Mode: false
Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
The command systemctl status docker however shows it as running;
[desai#brilp0017 ~]$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-09-11 14:25:53 BST; 14min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 2380 (dockerd)
Tasks: 14
Memory: 249.5M
CGroup: /system.slice/docker.service
└─2380 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
The journalctl log has the same error line as encountered in the pull command;
Sep 11 14:25:52 brilp0017 systemd[1]: Starting Docker Application Container Engine...
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.345006155+01:00" level=info msg="Starting up"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348597478+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348667479+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348733420+01:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348765306+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353865701+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353908904+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353944835+01:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353988191+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.497701794+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816295801+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816318357+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816442165+01:00" level=info msg="Loading containers: start."
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.101411528+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.125378601+01:00" level=info msg="Loading containers: done."
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.291896277+01:00" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.292711063+01:00" level=info msg="Docker daemon" commit=48a66213fe graphdriver(s)=overlay2 version=19.03.12-ce
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.293190069+01:00" level=info msg="Daemon has completed initialization"
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.340381428+01:00" level=info msg="API listen on /run/docker.sock"
Sep 11 14:25:53 brilp0017 systemd[1]: Started Docker Application Container Engine.
Sep 11 14:32:38 brilp0017 dockerd[2380]: time="2020-09-11T14:32:38.011501405+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 11 14:33:11 brilp0017 dockerd[2380]: time="2020-09-11T14:33:11.592234770+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 11 14:34:52 brilp0017 dockerd[2380]: time="2020-09-11T14:34:52.864254519+01:00" level=info msg="Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root534731447: device or resource busy"
After this the error remains the same even after stopping and starting docker service multiple times with systemctl. After restarting the laptop fully and starting the docker service it starts work as expected until the next time docker pull command is used.
I have searched for solution on the internet but majority of them point to the user not being in docker group but that is not the case for me;
[desai#brilp0017 ~]$ groups
sys network power vboxusers wireshark sambashare docker lp wheel desai
Here is the output of docker version before it crashes for version details;
[desai#brilp0017 ~]$ docker version
Client:
Version: 19.03.12-ce
API version: 1.40
Go version: go1.14.5
Git commit: 48a66213fe
Built: Sat Jul 18 01:33:21 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.12-ce
API version: 1.40 (minimum version 1.12)
Go version: go1.14.5
Git commit: 48a66213fe
Built: Sat Jul 18 01:32:59 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.0.m
GitCommit: 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a.m
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.18.0
GitCommit: fec3683
I am using Manjaro linux OS;
Operating System: Manjaro Linux
KDE Plasma Version: 5.19.4
KDE Frameworks Version: 5.73.0
Qt Version: 5.15.0
Kernel Version: 4.19.141-2-MANJARO
OS Type: 64-bit
Processors: 8 × Intel® Core™ i7-8550U CPU # 1.80GHz
Memory: 31.2 GiB of RAM
Graphics Processor: Mesa Intel® UHD Graphics 620
Any help on this would be appreciated.
This issue was resolved by updating the kernel to 5.4 version.
I have docker installed on centos & system.
I am able to download docker images using docker pull command.
but when I run the container using docker run alpine, the server restarts.
this is happening every time.
I found this from /var/log/messages | grep docker
CODE
Below is my configuration:
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 18.09.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-957.1.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 251.7GiB
Name: CHMCISPRDOCKENG
ID: XFPC:SYGF:Q3P7:M32Z:VRTX:TFGZ:YA43:NYSY:UGVK:PC2M:HVAU:TIM2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
some-registry
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: API is accessible on http://127.0.0.1:2375 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information:
https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Now when I run docker pull alpine. image is downloaded successfully.
But when I run docker run -it alpine , I am logged out from the server & server is restarted.
below are the logs when I ran cat /var/log/messages | grep docker.
Dec 17 19:28:12 CHMCISPRDOCKENG systemd: Started docker.service.
Dec 17 19:28:12 CHMCISPRDOCKENG audispd: node=CHMCISPRDOCKENG type=SERVICE_START msg=audit(1545055092.725:167): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=docker comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.023820157+05:30" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.043715041+05:30" level=info msg="libcontainerd: started new containerd process" pid=15564
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.043788743+05:30" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.043810434+05:30" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.044206037+05:30" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}]" module=grpc
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.044227097+05:30" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.044283337+05:30" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420963f00, CONNECTING" module=grpc
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.726282962+05:30" level=info msg="starting containerd" revision=c4446665cb9c30056f4998ed953e6d4ff22c7c39 version=1.2.0
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.727202682+05:30" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.733393760+05:30" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.733593273+05:30" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.733607841+05:30" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.737142848+05:30" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.737165057+05:30" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.740458140+05:30" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.747800098+05:30" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.748343654+05:30" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.748359281+05:30" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.748378883+05:30" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.748386577+05:30" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Dec 17 19:28:16 CHMCISPRDOCKENG dockerd: time="2018-12-17T19:28:16.748392893+05:30" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
Any suggestions?
Found the answer to be storage driver problem.
the configuration of the server was not supporting the default overlay2 storage driver for docker. changing it to devicemapper fixed things.
I've installed Docker on Centos 7 using the yum process. Below is the output for systemctl status docker.service -l:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-01-02 14:11:09 GMT; 1h 15min ago
Docs: https://docs.docker.com
Main PID: 136537 (dockerd)
Memory: 71.3M
CGroup: /system.slice/docker.service
├─136537 /usr/bin/dockerd
└─136556 docker-containerd --config /var/run/docker/containerd/containerd.toml
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157175048Z" level=error msg="c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5 cleanup: failed to delete container from containerd: no such container"
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157230927Z" level=error msg="Handler for POST /v1.35/containers/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.837926048Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/shim.sock" debug=false module="containerd/tasks" pid=145298
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim reaped" id=3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f module="containerd/tasks"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899560066Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899585062Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.921068216Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309201262Z" level=error msg="3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f cleanup: failed to delete container from containerd: no such container"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309254679Z" level=error msg="Handler for POST /v1.35/containers/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
The error i get (sudo docker run hello-world) is:
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown.
ERRO[0001] error waiting for container: context canceled
docker --version
Docker version 17.12.0-ce, build c97c6d6
and here is the docker info
Containers: 27
Running: 0
Paused: 0
Stopped: 27
Images: 47
Server Version: 17.12.0-ce
Storage Driver: devicemapper
Pool Name: docker-253:0-135160107-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 14.4GB
Data Space Total: 107.4GB
Data Space Available: 25.82GB
Metadata Space Used: 12.6MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.135GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: N/A (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 88
Total Memory: 125.6GiB
Name: colossus
ID: HV7I:KP2J:N6SQ:SAEP:MWSO:O5T3:Y2PE:L6TL:2D6L:IHVD:SQ3E:TM5P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
You most likely have an old package installed that is breaking runc. Make sure to upgrade your operating system to the current release (for CentOS, this is 1708, or the RHEL equivalent of 7.4). Others have reported similar issues related to packages being out of date.
I had 2 systems with identical CentOS 7.2 versions and kernel. Had this problem with one but not the other. Comparing package versions showed a difference in the version of containerd.io -- I downgraded that on the non-working version and it worked:
$ sudo yum downgrade containerd.io-1.2.0-3.el7
Removed:
containerd.io.x86_64 0:1.2.4-3.1.el7
Installed:
containerd.io.x86_64 0:1.2.0-3.el7
Environment
a) Cent OS VM's created in Hyper V.
b) Behind Corporate proxy.
I am experiencing strange issue when doing docker push.
I am getting below error
"denied: requested access to the resource is denied"
Docker pull is working without any issues.
I have attached daemon logs generated during docker push.
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.886973000+10:00" level=debug msg="Calling GET /_ping"
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.887518100+10:00" level=debug msg="Calling GET /v1.27/info"
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.893846700+10:00" level=debug msg="Calling POST /v1.27/images/vishnuvpotti/test/push?tag="
Aug 15 10:02:40 testmachine dockerd[40449]: time="2017-08-15T10:02:40.893969800+10:00" level=debug msg="Trying to push docker.io/vishnuvpotti/test to https://registry-1.docker.io v2"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.863929300+10:00" level=debug msg="Pushing repository: vishnuvpotti/test:latest"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.864814900+10:00" level=debug msg="Pushing layer: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865066700+10:00" level=debug msg="Pushing layer: sha256:defd90aaa156603d9362f29909f6f5539df841b6398b3836a5d559066295ec2b"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865155500+10:00" level=debug msg="Pushing layer: sha256:be4ee57de94e9c212ac27cf3b2f7ec826617a622d1e10f5b28558cce1f71ec24"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865233700+10:00" level=debug msg="Pushing layer: sha256:f971a434fe54442b68fe4d8bb9dc58cc51bfef3d960985fd1d2243a36abe89c2"
Aug 15 10:02:41 testmachine dockerd[40449]: time="2017-08-15T10:02:41.865308500+10:00" level=debug msg="Pushing layer: sha256:68078adbc0cb48954d27352b190afc2b011b3bb3c2adf61c45991669ea02134e"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.691569600+10:00" level=error msg="Upload failed: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.691661000+10:00" level=debug msg="Pushing layer: sha256:fb479b796fe5e208adece47fc14092d57c43b1dc0978ddd6a645cbdc7e85bbff"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694653100+10:00" level=error msg="Upload failed: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694753300+10:00" level=debug msg="Pushing layer: sha256:f32dba9a1eadb592cceb96d7b3fdc4a1c5213044fd7b30ae41e17673ccdf385d"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.694875200+10:00" level=error msg="Attempting next endpoint for push after error: denied: requested access to the resource is denied"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.699788800+10:00" level=debug msg="Pushing layer: sha256:19cbee8a76396ae7b80f835da4be7229dc8b094535ebf50d35ab2bf941039f7a"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.701883600+10:00" level=debug msg="Pushing layer: sha256:5a29dcc5d0b483d880a4736853253433c16103b9e03f9f52e94577cf1895563b"
Aug 15 10:02:43 testmachine dockerd[40449]: time="2017-08-15T10:02:43.707591200+10:00" level=debug msg="Pushing layer: sha256:92f9849bf0840ff917901665d775da4454d84a23439ef2ccbb93c6cc3363ab82"
Aug 15 10:02:44 testmachine dockerd[40449]: time="2017-08-15T10:02:44.588052200+10:00" level=debug msg="Pushing layer: sha256:74351e413c54d36337dff701c3568dc291f45523fe74261281b9ec9372528092"
Docker Version
root#testmachine# docker version
Client:
Version: 17.03.2-ce
API version: 1.27
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 02:21:36 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.2-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: f5ec1e2
Built: Tue Jun 27 02:21:36 2017
OS/Arch: linux/amd64
Experimental: false
Docker Info
root#testmachine # docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 2
Server Version: 17.03.2-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.549 GiB
Name: testmachine
ID: ATWM:MTHC:B2BA:6E4E:RAWL:E5VD:OUV2:2QHV:IBVQ:K525:DIPF:ICHC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 21
Goroutines: 26
System Time: 2017-08-15T11:21:02.8957667+10:00
EventsListeners: 0
Http Proxy: http://proxy
Https Proxy: https://proxy
No Proxy: localhost,127.0.0.1
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
Reformat the filesystem with ftype=1 to enable d_type support.
Running without d_type support will not be supported in future releases.
Note:
1) I have checked my repo settings in docker hub and Default Repository Visibility is public.
2) I deleted ~/.docker/config.json, restarted docker and tried.
3) I also tried docker login -u myusername
Can anyone please let me know what is going wrong. Is it an issue with new docker version?
Update 1:
1) I have updated docker version to the latest i.e : 17.06.0-ce . Still not working.
Update 2
2) I created a ubuntu VM in virtual box and gave a try on latest docker version. Installed ca certificate,edited proxy infromation for both machine and docker. Still facing same issues.
3)Update 3 (16/08/2017)
Turned off proxy, switched to a network without proxy,restarted docker and tried. Still same issue.
4) Update 4 (17/08/2017)
I created an instance in AWS and was able to push to docker registry. So it must be some issue in corporate network. Any comments appreciated.
Our CI infrastructure (Jenkins) builds and tests docker containers. We have found that stopping a container used during test takes the docker service offline, disconnecting the slave from our Jenkins master. Why would stopping the memcached container kill and restart the docker service?
What we see in the job:
+ docker stop memcached55c7e2ab363be9425039412edc81be1e8601c196
FATAL: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:295)
at hudson.remoting.Channel.terminate(Channel.java:814)
/var/log/docker:
time="2015-07-29T11:43:07-04:00" level=info msg="POST /v1.18/containers/memcached55c7e2ab363be9425039412edc81be1e8601c196/stop?t=10"
time="2015-07-29T11:43:07-04:00" level=info msg="+job stop(memcached55c7e2ab363be9425039412edc81be1e8601c196)"
\nWed Jul 29 11:43:52 EDT 2015\n
time="2015-07-29T11:43:53-04:00" level=warning msg="You are running linux kernel version 2.6.32-504.30.3.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0."
time="2015-07-29T11:43:53-04:00" level=info msg="+job serveapi(unix:///var/run/docker.sock)"
time="2015-07-29T11:43:53-04:00" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
time="2015-07-29T11:43:53-04:00" level=info msg="+job init_networkdriver()"
time="2015-07-29T11:43:53-04:00" level=info msg="-job init_networkdriver() = OK (0)"
time="2015-07-29T11:43:53-04:00" level=info msg="Loading containers: start."
...time="2015-07-29T11:43:53-04:00" level=error msg="Error unmounting device 23607b95df9a186d80bb346c914cd01e7631c4fec3968bde29df7bc327f0b820: UnmountDevice: device not-mounted id 23607b95df9a186d80bb346c914cd01e7631c4fec3968bde29df7bc327f0b820"
.time="2015-07-29T11:43:53-04:00" level=error msg="Error unmounting device d01f73b589a9d598cc97752ce554b5c5dcf1b51d3991c1dee54880bb7d718fb7: UnmountDevice: device not-mounted id d01f73b589a9d598cc97752ce554b5c5dcf1b51d3991c1dee54880bb7d718fb7"
time="2015-07-29T11:43:53-04:00" level=info msg="Loading containers: done."
time="2015-07-29T11:43:53-04:00" level=info msg="docker daemon: 1.6.2 7c8fca2/1.6.2; execdriver: native-0.2; graphdriver: devicemapper"
time="2015-07-29T11:43:53-04:00" level=info msg="+job acceptconnections()"
time="2015-07-29T11:43:53-04:00" level=info msg="-job acceptconnections() = OK (0)"
time="2015-07-29T11:43:53-04:00" level=info msg="Daemon has completed initialization"
Versions:
-bash-4.1$ uname -a
Linux ip-172-17-70-142.ec2.internal 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
-bash-4.1$ docker --version
Docker version 1.6.2, build 7c8fca2/1.6.2
-bash-4.1$ docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2/1.6.2
OS/Arch (server): linux/amd64
-bash-4.1$ docker info
Containers: 2
Images: 65
Storage Driver: devicemapper
Pool Name: docker-202:64-936887-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.597 GB
Data Space Total: 107.4 GB
Data Space Available: 12.44 GB
Metadata Space Used: 3.584 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.30.3.el6.x86_64
Operating System: <unknown>
CPUs: 2
Total Memory: 7.166 GiB
Name: ip-172-17-70-142.ec2.internal
ID: X4UG:BKKW:5ZQX:VXRV:3GYP:FU3J:7FJC:4FFM:CK4D:SWSH:KIHW:IPNE
Username: dxbuildmaster
Registry: [https://index.docker.io/v1/]
Are you sure that this is caused by docker? We were using a Jenkins version in our environment that does not deal with docker at all and had the same error message. There is a known Jenkins bug about that which has been fixed in version 1.532.1. So which Jenkins version are you using?