I've installed Docker on Centos 7 using the yum process. Below is the output for systemctl status docker.service -l:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-01-02 14:11:09 GMT; 1h 15min ago
Docs: https://docs.docker.com
Main PID: 136537 (dockerd)
Memory: 71.3M
CGroup: /system.slice/docker.service
├─136537 /usr/bin/dockerd
└─136556 docker-containerd --config /var/run/docker/containerd/containerd.toml
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157175048Z" level=error msg="c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5 cleanup: failed to delete container from containerd: no such container"
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157230927Z" level=error msg="Handler for POST /v1.35/containers/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.837926048Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/shim.sock" debug=false module="containerd/tasks" pid=145298
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim reaped" id=3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f module="containerd/tasks"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899560066Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899585062Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.921068216Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309201262Z" level=error msg="3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f cleanup: failed to delete container from containerd: no such container"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309254679Z" level=error msg="Handler for POST /v1.35/containers/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
The error i get (sudo docker run hello-world) is:
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown.
ERRO[0001] error waiting for container: context canceled
docker --version
Docker version 17.12.0-ce, build c97c6d6
and here is the docker info
Containers: 27
Running: 0
Paused: 0
Stopped: 27
Images: 47
Server Version: 17.12.0-ce
Storage Driver: devicemapper
Pool Name: docker-253:0-135160107-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 14.4GB
Data Space Total: 107.4GB
Data Space Available: 25.82GB
Metadata Space Used: 12.6MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.135GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: N/A (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 88
Total Memory: 125.6GiB
Name: colossus
ID: HV7I:KP2J:N6SQ:SAEP:MWSO:O5T3:Y2PE:L6TL:2D6L:IHVD:SQ3E:TM5P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
You most likely have an old package installed that is breaking runc. Make sure to upgrade your operating system to the current release (for CentOS, this is 1708, or the RHEL equivalent of 7.4). Others have reported similar issues related to packages being out of date.
I had 2 systems with identical CentOS 7.2 versions and kernel. Had this problem with one but not the other. Comparing package versions showed a difference in the version of containerd.io -- I downgraded that on the non-working version and it worked:
$ sudo yum downgrade containerd.io-1.2.0-3.el7
Removed:
containerd.io.x86_64 0:1.2.4-3.1.el7
Installed:
containerd.io.x86_64 0:1.2.0-3.el7
Related
docker.log
time="2020-02-18T10:05:51.740757000+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded." storage-driver=overlay2
time="2020-02-18T10:05:51.747446000+08:00" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
time="2020-02-18T10:05:51.758181000+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded." storage-driver=overlay
time="2020-02-18T10:05:51.990661000+08:00" level=error msg="Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/dockerd/#storage-driver-options" storage-driver=devicemapper
time="2020-02-18T10:05:52.146274000+08:00" level=warning msg="Your kernel does not support cgroup memory limit"
time="2020-02-18T10:05:52.146389000+08:00" level=warning msg="Unable to find cpu cgroup in mounts"
time="2020-02-18T10:05:52.146445000+08:00" level=warning msg="Unable to find blkio cgroup in mounts"
time="2020-02-18T10:05:52.146504000+08:00" level=warning msg="Unable to find cpuset cgroup in mounts"
time="2020-02-18T10:05:52.146553000+08:00" level=warning msg="mountpoint for pids not found"
time="2020-02-18T10:05:52.149506000+08:00" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2020-02-18T10:05:52.160535000+08:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Devices cgroup isn't mounted
docker version
Client: Docker Engine - Community
Version: 19.03.6
API version: 1.40
Go version: go1.12.16
Git commit: 369ce74
Built: Thu Feb 13 01:40:25 2020
OS/Arch: linux/arm
Experimental: false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I have this literally bookmarked, and I go here every time I need to start docker on my netbooting Alpine. Single command to mount the cgroup pseudofs, w/o touching fstab:
sudo mount -t cgroup cgroup /sys/fs/cgroup
FYI: this may be also caused by your kernel not having cgroup support compiled in, but it's highly unlikely.
I observed that the docker takes 4 minutes to start each time after reboot. I have provided some details as attached. Any thoughts/tips are appreciated. Thanks.
Debian
-> docker -v
Docker version 18.03.0-ce, build 0520e24
-> uname -a
Linux x 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux
Other logs:
-> systemd-analyze blame
4min 21.855s docker.service
2.863s cloud-final.service
2.021s cloud-init-local.service
1.698s cloud-init.service
1.476s cloud-config.service
…
21ms systemd-tmpfiles-clean.service
19ms systemd-update-utmp.service
18ms docker.socket
-> sudo journalctl -u docker.service
– Logs begin at Sun 2018-05-06 20:49:58 PDT, end at Sun 2018-05-06 21:11:45 PDT. –
May 06 20:50:02 x systemd[1]: Starting Docker Application Container Engine…
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23.757674968-07:00” level=debug msg=“Listener created for HTTP on fd ()”
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23.769857584-07:00” level=info msg=“libcontainerd: started new docker-containerd process” pid=1328
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“starting containerd” module=containerd revision=cfd04396dc68220d1cecbe686a6cc3aa5ce3667c version=v1.0.2
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=debug msg=“changing OOM score to -500” module=containerd
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.content.v1.content”…” module=containerd type=io.containerd.content.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.snapshotter.v1.btrfs”…” module=containerd type=io.containerd.snapshotter.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=warning msg=“failed to load plugin io.containerd.snapshotter.v1.btrfs” error="path /var/lib/docker/containerd/daemon/io.containerd.sn
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.snapshotter.v1.overlayfs”…” module=containerd type=io.containerd.snapshotter.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.metadata.v1.bolt”…” module=containerd type=io.containerd.metadata.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=warning msg=“could not use snapshotter btrfs in metadata plugin” error="path /var/lib/docker/containerd/daemon/io.containerd.snapshot
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.differ.v1.walking”…” module=containerd type=io.containerd.differ.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.gc.v1.scheduler”…” module=containerd type=io.containerd.gc.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.grpc.v1.containers”…” module=containerd type=io.containerd.grpc.v1
May 06 20:54:23 x dockerd[654]: time=“2018-05-06T20:54:23-07:00” level=info msg=“loading plugin “io.containerd.grpc.v1.content”…” module=containerd type=io.containerd.grpc.v1
I posted the question in docker forum, but got no luck.
https://forums.docker.com/t/docker-takes-4-minutes-to-start-on-debian/50441
====== updated #1
System info: 1G RAM, 25 GB SSD, Clean system (just installed)
> docker info
xiejw:x -> docker info
Containers: 8
Running: 3
Paused: 0
Stopped: 5
Images: 15
Server Version: 18.03.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-6-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.5MiB
Name: x
ID: Y3MP:DHWK:4VES:2LEZ:RHJK:UFCB:HX2W:4CK2:XHBI:CG7B:PBT3:2LTG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 37
Goroutines: 45
System Time: 2018-05-09T10:44:20.217001219-07:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Description
I am experiencing some intermittent communications issues between containers in the same overlay network. I have been struggling to find a solution to this for weeks but everything I see in Google relating to communications issues dosen't quite match what I am seeing. So I am hoping someone here can help me figure out what is going on.
We are using Docker 17.06
We are using standalone swarm with three masters and one node.
We have multiple overlay networks
Containers attached to each overlay network:
1 container running Apache Tomcat 8.5 and HAproxy 1.7 (called the controller)
1 container just running Apache Tomcat 8.5 (called the apps container)
3 containers running Postgresql 9.6
1 container running an FTP service
1 container running Logstash
Steps to reproduce the issue:
Create a new overlay network
Attach containers
Look at the logs and after a short while you see the errors
Describe the results you received:
The "controller" polls a servlet on "apps" container every few seconds.
Every 15 minutes or so we see a connect timed out error in the log files of the "controller". And perodically we see connection attempt failed when the controller tries to access its database in one of the Postgresql containers.
Error when polling apps container
org.apache.http.conn.ConnectTimeoutException: Connect to srvpln50-webapp_1.0-1:5050 [srvpln50-webapp_1.0-1/10.0.1.6] failed: connect timed out
Error when trying to connect to database
JavaException: com.ebasetech.xi.exceptions.FormRuntimeException: Error getting connection using Database Connection CONTROLLER, SQLEx
ception in StandardPoolDataSource:getConnection exception: java.sql.SQLException: SQLException in StandardPoolDataSource:getConnection no connection available java.sql.SQLException: Cannot
get connection for URL jdbc:postgresql://srvpln50-controller-db_latest:5432/ctrldata : The connection attempt failed.
I turned on debug mode on the docker deamon node.
Every time these errors occur I see the following corrosponding entry in the docker logs:
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.422797691Z" level=debug msg="Name To resolve: srvpln50-webapp_1.0-1."
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.422905040Z" level=debug msg="Lookup for srvpln50-webapp_1.0-1.: IP [10.0.1.6]"
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.648262289Z" level=debug msg="miss notification: dest IP 10.0.0.3, dest MAC 02:42:0a:00:00:03"
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.716329366Z" level=debug msg="miss notification: dest IP 10.0.0.6, dest MAC 02:42:0a:00:00:06"
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.716952000Z" level=debug msg="miss notification: dest IP 10.0.0.6, dest MAC 02:42:0a:00:00:06"
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.802320875Z" level=debug msg="miss notification: dest IP 10.0.0.3, dest MAC 02:42:0a:00:00:03"
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.944189349Z" level=debug msg="miss notification: dest IP 10.0.0.9, dest MAC 02:42:0a:00:00:09"
Feb 09 14:27:26 swarm-node-1 dockerd[12193]: time="2018-02-09T14:27:26.944770233Z" level=debug msg="miss notification: dest IP 10.0.0.9, dest MAC 02:42:0a:00:00:09"
IP 10.0.0.3 is the "controller" container
IP 10.0.0.6 is the "apps" container
IP 10.0.0.9 is the "postgresql" container that the "controller" is trying to connect to.
Describe the results you expected:
Not to have the connection errors
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client:
Version: 17.06.1-ce
API version: 1.30
Go version: go1.8.3
Git commit: 874a737
Built: Thu Aug 17 22:51:12 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.1-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 874a737
Built: Thu Aug 17 22:50:04 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 19
Running: 19
Paused: 0
Stopped: 0
Images: 18
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 385
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-108-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.784GiB
Name: swarm-node-1
ID: O5ON:VQE7:IRV6:WCB7:RQO4:RIZ4:XFHE:AUCX:ZLM2:GPZL:DXQO:BCIX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 217
Goroutines: 371
System Time: 2018-02-09T15:50:01.902816981Z
EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
name=swarm-node-1
Experimental: false
Cluster Store: etcd://localhost:2379/store
Cluster Advertise: 10.80.120.13:2376
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
Swarm masters, node and containers are running Ubuntu 16.04 on bare metal servers
If there is anything I have missed that would aid diagnose please let me know.
Having read many comments from the Docker folks on Google about many communication issues being fixed in the latest version of Docker we upgraded to 17.12 CE and all the issues we were experiencing went away.
Would love to know what the issue was but am more than happy to see them gone.
I use official version of docker-ce at centos7, start a docker daemon in container:
[root#5cae7be526b4 /]# rpm -qa docker-ce
docker-ce-17.09.0.ce-1.el7.centos.x86_64
Here is my daemon config
{
"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:5555"],
"live-restore": true,
"insecure-registries": ["172.17.0.6:9980"]
}
Without changing config, docker daemon can start and restart in container:
[root#5cae7be526b4 /]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: CentOS Linux 7 (Core) (containerized)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.26GiB
Name: 5cae7be526b4
ID: N3Y4:VTIJ:WCHK:AQL3:MU3F:DNHE:BIXO:7ISI:4D4V:Q4IG:VYIT:FOH3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
But change the config, it failed:
[root#5cae7be526b4 /]# systemctl restart docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
[root#5cae7be526b4 /]# systemctl status docker -l
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Thu 2017-11-02 05:51:02 UTC; 2s ago
Docs: https://docs.docker.com
Process: 260 ExecStart=/usr/bin/dockerd (code=exited, status=1/FAILURE)
Main PID: 260 (code=exited, status=1/FAILURE)
Nov 02 05:51:02 5cae7be526b4 systemd[1]: Failed to start Docker Application Container Engine.
Nov 02 05:51:02 5cae7be526b4 systemd[1]: Unit docker.service entered failed state.
Nov 02 05:51:02 5cae7be526b4 systemd[1]: docker.service failed.
Nov 02 05:51:02 5cae7be526b4 systemd[1]: docker.service holdoff time over, scheduling restart.
Nov 02 05:51:02 5cae7be526b4 systemd[1]: start request repeated too quickly for docker.service
Nov 02 05:51:02 5cae7be526b4 systemd[1]: Failed to start Docker Application Container Engine.
Nov 02 05:51:02 5cae7be526b4 systemd[1]: Unit docker.service entered failed state.
Nov 02 05:51:02 5cae7be526b4 systemd[1]: docker.service failed.
Of course, this daemon config can run at host.
I has start container with --privileged and -v /sys/fs/cgroup:/sys/fs/cgroup to enable use systemctl in container.
The root cause is "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:5555"], i do not know how to fix it but i need to set host indeed.
I want to make this container as a repo and start other container to do docker action like pull from this docker daemon.
How can i enable it?
I share my solution here, please tell me if wrong or has a better way.
First, check docker version, only new official version can run in container. For centos, it means docker-ce, refer to https://docs.docker.com/engine/installation/linux/docker-ce/centos/
Second, check the host port doesn't be set as -p when start container. docker daemon can listen it and specify with -p would make conflicts.
Third, start docker daemon in container should assign a volume for storage. If not, the storage option only can be vfs.
Our CI infrastructure (Jenkins) builds and tests docker containers. We have found that stopping a container used during test takes the docker service offline, disconnecting the slave from our Jenkins master. Why would stopping the memcached container kill and restart the docker service?
What we see in the job:
+ docker stop memcached55c7e2ab363be9425039412edc81be1e8601c196
FATAL: java.io.IOException: Unexpected termination of the channel
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:295)
at hudson.remoting.Channel.terminate(Channel.java:814)
/var/log/docker:
time="2015-07-29T11:43:07-04:00" level=info msg="POST /v1.18/containers/memcached55c7e2ab363be9425039412edc81be1e8601c196/stop?t=10"
time="2015-07-29T11:43:07-04:00" level=info msg="+job stop(memcached55c7e2ab363be9425039412edc81be1e8601c196)"
\nWed Jul 29 11:43:52 EDT 2015\n
time="2015-07-29T11:43:53-04:00" level=warning msg="You are running linux kernel version 2.6.32-504.30.3.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0."
time="2015-07-29T11:43:53-04:00" level=info msg="+job serveapi(unix:///var/run/docker.sock)"
time="2015-07-29T11:43:53-04:00" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
time="2015-07-29T11:43:53-04:00" level=info msg="+job init_networkdriver()"
time="2015-07-29T11:43:53-04:00" level=info msg="-job init_networkdriver() = OK (0)"
time="2015-07-29T11:43:53-04:00" level=info msg="Loading containers: start."
...time="2015-07-29T11:43:53-04:00" level=error msg="Error unmounting device 23607b95df9a186d80bb346c914cd01e7631c4fec3968bde29df7bc327f0b820: UnmountDevice: device not-mounted id 23607b95df9a186d80bb346c914cd01e7631c4fec3968bde29df7bc327f0b820"
.time="2015-07-29T11:43:53-04:00" level=error msg="Error unmounting device d01f73b589a9d598cc97752ce554b5c5dcf1b51d3991c1dee54880bb7d718fb7: UnmountDevice: device not-mounted id d01f73b589a9d598cc97752ce554b5c5dcf1b51d3991c1dee54880bb7d718fb7"
time="2015-07-29T11:43:53-04:00" level=info msg="Loading containers: done."
time="2015-07-29T11:43:53-04:00" level=info msg="docker daemon: 1.6.2 7c8fca2/1.6.2; execdriver: native-0.2; graphdriver: devicemapper"
time="2015-07-29T11:43:53-04:00" level=info msg="+job acceptconnections()"
time="2015-07-29T11:43:53-04:00" level=info msg="-job acceptconnections() = OK (0)"
time="2015-07-29T11:43:53-04:00" level=info msg="Daemon has completed initialization"
Versions:
-bash-4.1$ uname -a
Linux ip-172-17-70-142.ec2.internal 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
-bash-4.1$ docker --version
Docker version 1.6.2, build 7c8fca2/1.6.2
-bash-4.1$ docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2/1.6.2
OS/Arch (server): linux/amd64
-bash-4.1$ docker info
Containers: 2
Images: 65
Storage Driver: devicemapper
Pool Name: docker-202:64-936887-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.597 GB
Data Space Total: 107.4 GB
Data Space Available: 12.44 GB
Metadata Space Used: 3.584 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.30.3.el6.x86_64
Operating System: <unknown>
CPUs: 2
Total Memory: 7.166 GiB
Name: ip-172-17-70-142.ec2.internal
ID: X4UG:BKKW:5ZQX:VXRV:3GYP:FU3J:7FJC:4FFM:CK4D:SWSH:KIHW:IPNE
Username: dxbuildmaster
Registry: [https://index.docker.io/v1/]
Are you sure that this is caused by docker? We were using a Jenkins version in our environment that does not deal with docker at all and had the same error message. There is a known Jenkins bug about that which has been fixed in version 1.532.1. So which Jenkins version are you using?