I am using Ubuntu 16.04 with docker 1.11.2. I have configured systemd to automatically restart docker daemon. When I kill the docker daemon, docker daemon restarts, but container will not even it has RestartPolicy set to always. From the logs I can read that it failed to create directory because it exists. I personally think that it related to stopping containerd.
Any help would be appreciated.
Aug 25 19:20:19 api-31 systemd[1]: docker.service: Main process exited, code=killed, status=9/KILL
Aug 25 19:20:19 api-31 docker[17617]: time="2016-08-25T19:20:19Z" level=info msg="stopping containerd after receiving terminated"
Aug 25 19:21:49 api-31 systemd[1]: docker.service: State 'stop-sigterm' timed out. Killing.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Unit entered failed state.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Failed with result 'timeout'.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 25 19:21:49 api-31 systemd[1]: Stopped Docker Application Container Engine.
Aug 25 19:21:49 api-31 systemd[1]: Closed Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Stopping Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Starting Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Listening on Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Starting Docker Application Container Engine...
Aug 25 19:21:49 api-31 docker[19023]: time="2016-08-25T19:21:49.913162167Z" level=info msg="New containerd process, pid: 19029\n"
Aug 25 19:21:50 api-31 kernel: [87066.742831] audit: type=1400 audit(1472152910.946:23): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=19043 comm="apparmor_parser"
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.952073973Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.956693893Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.961641996Z" level=info msg="Firewalld running: false"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.016582850Z" level=info msg="Removing stale sandbox 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596 (547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613)"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.046227326Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.081106790Z" level=warning msg="Your kernel does not support swap memory limit."
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.081650610Z" level=info msg="Loading containers: start."
Aug 25 19:22:01 api-31 kernel: [87076.922492] docker0: port 1(vethbbc1192) entered disabled state
Aug 25 19:22:01 api-31 kernel: [87076.927128] device vethbbc1192 left promiscuous mode
Aug 25 19:22:01 api-31 kernel: [87076.927131] docker0: port 1(vethbbc1192) entered disabled state
Aug 25 19:22:03 api-31 docker[19023]: .time="2016-08-25T19:22:03.085800458Z" level=warning msg="error locating sandbox id 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596: sandbox 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596 not found"
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.085907328Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613/shm: invalid argument"
Aug 25 19:22:03 api-31 kernel: [87078.882836] device veth5c6999c entered promiscuous mode
Aug 25 19:22:03 api-31 kernel: [87078.882984] IPv6: ADDRCONF(NETDEV_UP): veth5c6999c: link is not ready
Aug 25 19:22:03 api-31 systemd-udevd[19128]: Could not generate persistent MAC address for veth5c6999c: No such file or directory
Aug 25 19:22:03 api-31 systemd-udevd[19127]: Could not generate persistent MAC address for veth39fb4d3: No such file or directory
Aug 25 19:22:03 api-31 kernel: [87078.944218] docker0: port 1(veth5c6999c) entered disabled state
Aug 25 19:22:03 api-31 kernel: [87078.948636] device veth5c6999c left promiscuous mode
Aug 25 19:22:03 api-31 kernel: [87078.948640] docker0: port 1(veth5c6999c) entered disabled state
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219677059Z" level=error msg="Failed to start container 547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613: rpc error: code = 6 desc = \"mkdir /run/containerd/547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613: file exists\""
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219750430Z" level=info msg="Loading containers: done."
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219776593Z" level=info msg="Daemon has completed initialization"
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219847738Z" level=info msg="Docker daemon" commit=b9f10c9 graphdriver=overlay version=1.11.2
Aug 25 19:22:03 api-31 systemd[1]: Started Docker Application Container Engine.
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.226116336Z" level=info msg="API listen on /var/run/docker.sock"
#VonC - Thank you for pointing me at the right direction. I researched the thread, but in my case the apparmor is not an issue. There are some other issues mentioned in the thread, so I followed them and I found the solution.
SOLUTION:
On Ubuntu 16.04 the problem is that systemd kills process containerd with the docker daemon process. In order to prevent it, you need to add
KillMode=process
to /lib/systemd/system/docker.service and that fixes the issue.
Here are the sources I used:
https://github.com/docker/docker/issues/25246
https://github.com/docker/docker/blob/master/contrib/init/systemd/docker.service#L25
That seems to be followed by issue 25487 (August 2016), and was reported even before (April 2016) in issue 22195.
Check if you are in the situation mentioned in issue 21702 by Tõnis Tiigi:
This seems to be caused by the apparmor profile for docker daemon we have in docker/contrib/apparmor.
If this profile is applied in v1.11 (at least ubuntu wily) then container starting does not work.
I'm not sure if users have just manually enforced this profile or apparently we also accidentally installed this profile in 1.10.0-rc1 (#19707).
So the workaround, until we figure out how to deal with this, is to unload the profile with something like apparmor_parser -R /etc/apparmor.d/docker-engine ,delete it and restart daemon.
/etc/apparmor.d/docker is the profile for the containers and does not need to be changed.
Related
I can't restart docker service. After command system just getting stuck.
docker service status is here:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: deactivating (stop-sigterm)
Docs: https://docs.docker.com
Main PID: 1216 (dockerd)
Tasks: 9
CGroup: /system.slice/docker.service
└─1216 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Aug 25 15:08:21 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:21.179342033-07:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=
Aug 25 15:08:21 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:21.631714600-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.607526073-07:00" level=warning msg="Your kernel does not support swap memory limit"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.608984581-07:00" level=warning msg="Your kernel does not support cgroup rt period"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.609119466-07:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.609435097-07:00" level=info msg="Loading containers: start."
Aug 25 15:08:32 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:32.266035465-07:00" level=info msg="Removing stale sandbox f2295d6e0a7aec8569c42470bbccbc5a1
Aug 25 15:08:32 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:32.677509372-07:00" level=warning msg="Error (Unable to complete atomic operation, key modif
Aug 25 15:08:33 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:33.339048947-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address
Aug 27 05:12:40 hq-rproxy02 dockerd[1216]: time="2020-08-27T05:12:40.330007214-07:00" level=info msg="Processing signal 'terminated'"
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-08-25 15:07:44 PDT; 5 days ago
Listen: /var/run/docker.sock (Stream)
Tasks: 0 (limit: 1113)
CGroup: /system.slice/docker.socket
Aug 25 15:07:44 hq-rproxy02 systemd[1]: Starting Docker Socket for the API.
Aug 25 15:07:44 hq-rproxy02 systemd[1]: Listening on Docker Socket for the API.
What can I check?
I'm trying to install Kubernetes on CentOS 7.7, therefore, I have to install docker first.
I followed Kubernetes Documentation to install docker-ce and modify daemon.json file.
$ yum install yum-utils device-mapper-persistent-data lvm2
$ yum-config-manager --add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
$ yum update && yum install \
containerd.io-1.2.10 \
docker-ce-19.03.4 \
docker-ce-cli-19.03.4
$ mkdir /etc/docker
$ cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
$ mkdir -p /etc/systemd/system/docker.service.d
$ systemctl daemon-reload
$ systemctl start docker
When started docker service, it said:
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
$ systemctl status -l docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2020-01-07 14:44:11 UTC; 7min ago
Docs: https://docs.docker.com
Process: 9879 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 9879 (code=exited, status=1/FAILURE)
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: docker.service failed.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: docker.service holdoff time over, scheduling restart.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Stopped Docker Application Container Engine.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: start request repeated too quickly for docker.service
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: docker.service failed.
$ journalctl -xe
.
.
-- Unit docker.service has begun starting up.
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.722780008Z" level=info msg="Starting up"
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728447514Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728479813Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728510943Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728526075Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.732325726Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733844225Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733880664Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733898044Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.743421350Z" level=warning msg="Using pre-4.0.0 kernel for overlay2, mount failures may require
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: failed to start daemon: error initializing graphdriver: overlay2: the backing xfs filesystem is formatted without d_type
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: docker.service failed.
Could anyone tell me why docker service start failed after modifying daemon.json file? And how to specify cgroupdriver, default log-driver and default storage-driver in the right way?
Any suggestion will be greatly appreciated.
Thanks.
This error is pointing to an issue forcing docker to use overlay2 without the proper backing filesystem:
failed to start daemon: error initializing graphdriver: overlay2: the backing xfs filesystem is formatted without d_type
See docker's table for details on backing filesystem requirements for the different storage drivers: https://docs.docker.com/storage/storagedriver/#supported-backing-filesystems
The fix is to remove the storage driver settings, or fix the backing filesystem with the needed options to support overlay2:
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
}
}
For details on changing the xfs options, that appears to require rebuilding the filesystem. See this answer for more details on the needed steps.
execute below command on slave node in kubernetes cluster.
for SERVICES in docker; do systemctl restart $SERVICES; systemctl enable $SERVICES; systemctl status $SERVICES; done
got below error
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/docker.service.d
└─flannel.conf
Active: failed (Result: exit-code) since Wed 2018-04-11 11:20:10 GMT; 67ms ago
Docs: http://docs.docker.com
Main PID: 2315 (code=exited, status=1/FAILURE)
Apr 11 11:20:10 LX01511L docker[2315]: time="2018-04-11T11:20:10Z" level=error msg="WARNING: No --storage-opt dm.thinpooldev specified, using loopback; thi...ction use"
Apr 11 11:20:10 LX01511L docker[2315]: time="2018-04-11T11:20:10Z" level=warning msg="Docker could not enable SELinux on the host system"
Apr 11 11:20:10 LX01511L docker[2315]: time="2018-04-11T11:20:10Z" level=info msg="+job init_networkdriver()"
Apr 11 11:20:10 LX01511L docker[2315]: Bridge ip (172.17.42.1) does not match existing bridge configuration 172.30.81.1
Apr 11 11:20:10 LX01511L docker[2315]: time="2018-04-11T11:20:10Z" level=info msg="-job init_networkdriver() = ERR (1)"
Apr 11 11:20:10 LX01511L docker[2315]: time="2018-04-11T11:20:10Z" level=fatal msg="Shutting down daemon due to errors: Bridge ip (172.17.42.1) does not ma...2.30.81.1"
Apr 11 11:20:10 LX01511L systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 11 11:20:10 LX01511L systemd[1]: Failed to start Docker Application Container Engine.
Apr 11 11:20:10 LX01511L systemd[1]: Unit docker.service entered failed state.
Apr 11 11:20:10 LX01511L systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
msg="Shutting down daemon due to errors: Bridge ip (172.17.42.1) does not ma...2.30.81.1"
Following is the error I am getting while trying to start the docker daemon service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─50-docker-service.conf
Active: failed (Result: exit-code) since Tue 2017-08-22 02:09:40 UTC; 15min ago
Docs: http://docs.docker.com
Main PID: 3571 (code=exited, status=1/FAILURE)
CPU: 292ms
Aug 22 02:09:40 systemd[1]: docker.service: Unit entered failed state.
Aug 22 02:09:40 systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 22 02:09:40 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 22 02:09:40 systemd[1]: Stopped Docker Application Container Engine.
Aug 22 02:09:40 systemd[1]: docker.service: Start request repeated too quickly.
Aug 22 02:09:40 systemd[1]: Failed to start Docker Application Container Engine.
Aug 22 02:09:40 systemd[1]: docker.service: Unit entered failed state.
Aug 22 02:09:40 systemd[1]: docker.service: Failed with result 'exit-code'.
Below is the config files I have
50-docker-service.conf
[Service]
Environment="DOCKER_OPTS=--bip=A.B.C.D"
what could be the cause?
$ls -ltr /etc/systemd/system/docker.service.d
total 16
-rw-r--r--. 1 root root 125 Aug 22 02:09 50-docker-service.conf
journalctl logs
Jul 14 13:55:52 systemd[1]: Starting Docker Application Container Engine...
Jul 14 13:55:52 dockerd[1274]: time="2017-07-14T13:55:52.925276313Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Jul 14 13:55:53 dockerd[1274]: time="2017-07-14T13:55:53.378204522Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jul 14 13:55:53 dockerd[1274]: time="2017-07-14T13:55:53.379367854Z" level=info msg="Loading containers: start."
Jul 14 13:55:53 dockerd[1274]: ..time="2017-07-14T13:55:53.507972850Z" level=info msg="Firewalld running: false"
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.013379242Z" level=info msg="Loading containers: done."
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.021206395Z" level=info msg="Daemon has completed initialization"
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.021283711Z" level=info msg="Docker daemon" commit=a82d35e graphdriver=overlay version=1.12.6
Jul 14 13:55:54 systemd[1]: Started Docker Application Container Engine.
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.039479153Z" level=info msg="API listen on /var/run/docker.sock"
Jul 14 13:56:03 dockerd[1274]: time="2017-07-14T13:56:03.565234227Z" level=error msg="Handler for POST /v1.24/containers/7019b26d0cb3/start returned error: Container already started"
Jul 14 13:56:09 dockerd[1274]: time="2017-07-14T13:56:09.660967581Z" level=error msg="Handler for POST /v1.24/containers/7019b26d0cb3/start returned error: Container already started"
Jul 14 13:56:14 dockerd[1274]: time="2017-07-14T13:56:14.741806551Z" level=error msg="Handler for POST /v1.24/containers/d9a3cb2b66e0/start returned error: Container already started"
Jul 14 21:05:16 dockerd[1274]: time="2017-07-14T21:05:16.992138499Z" level=info msg="Container 7019b26d0cb31412f40f8ab7f971f26896debcce09a58c39679dbaf62f6caa0b failed to exit within 0 s
Jul 14 21:07:20 dockerd[1274]: time="2017-07-14T21:07:20.897682536Z" level=info msg="Container d9a3cb2b66e0395086decd444f7ef52775f76f64b8d4dc291ee66cca48e53535 failed to exit within 2 s
Jul 14 21:08:08 systemd[1]: Stopping Docker Application Container Engine...
Jul 14 21:08:08 dockerd[1274]: time="2017-07-14T21:08:08.243279424Z" level=info msg="Processing signal 'terminated'"
Jul 14 21:08:18 dockerd[1274]: time="2017-07-14T21:08:18.244896088Z" level=info msg="Container 9966c97ca301ef593a68ab8e50730552dbe945c42e6b530d4c6339d3ffa8f544 failed to exit within 10
Jul 14 21:08:18 dockerd[1274]: time="2017-07-14T21:08:18.244909995Z" level=info msg="Container d94a3d0c45471753da85fce062e3c56c79c14ad40b6870281515333b98d0807e failed to exit within 10
Jul 14 21:08:18 dockerd[1274]: time="2017-07-14T21:08:18.244936931Z" level=info msg="Container b49d40479583c121ae7abe14489c7121a25a56da00f3d25961f981c918d3257e failed to exit within 10
Jul 14 21:08:18 systemd[1]: Stopped Docker Application Container Engine.
Docker bridge ip is not configured properly.
Environment="DOCKER_OPTS=--bip=A.B.C.D
Instead configured bip with a fully qualified CIDR
Environment="DOCKER_OPTS=--bip=A.B.C.D/size
that solved my problem
Check ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
is not modified in /lib/systemd/system/docker.service
I am trying to set docker up to connect all containers to my own manually created bridge (br0), I don't want docker to create or edit anything in my bridge, because I have other services which uses and depends on my bridge (like OpenVPN) therefore I prefer to create the bridge using my own bash script.
The problem comes when I start docker service, docker changes my bridge IP address from what I want (192.168.1.10) to something else address(169.254.x.x)!!!
My Docker version 1.12.1, build 23cf638
The steps I did
Bridge creation:
sudo brctl addbr br0
sudo brctl addif br0 eth0
sudo ip addr del 192.168.1.10/24 dev eth0
sudo ip addr add 192.168.1.10/24 dev br0
sudo ip route add default via 192.168.1.1 dev br0
I also deleted the default docker0 brdige.
Tell docker to use my br0 instead of the default docker0:
Passing -b br0 parameter to dockerd.service starting script to tell docker that I want him to use my br0:
sudo vi /etc/systemd/system/docker.service.d/overlay.conf
I edited ExecStart to be like this:
ExecStart=/usr/bin/dockerd --storage-driver=overlay -H fd:// -b=br0
and then:
sudo systemctl daemon-reload
sudo systemctl restart docker
And now when I check my br0 IP, it is NOT 192.168.1.10 any more, it is back to 172.17.x.x, and when I try to change it now manually back to 192.168.1.10, the interfaces in containers keeps using 169.254.x.x instead of the IP I want.
P.s. when I check where are the interfaces of my containers: brctl show, they are really in my br0 (that means docker accepted -b br0 paramter, but it just ignores or override my intended IP address).
Could some one help me please to over come that problem? it looks for me like a bug maybe. I just want docker to use my br0 with the intended IP address 192.168.1.10.
My need is that all my containers get and IP address in the range I want.
Thanks in advance.
Edited:
My /var/log/daemon.log
Oct 10 20:41:12 raspberrypi systemd[1]: Stopping Docker Application Container Engine...
Oct 10 20:41:12 raspberrypi dockerd[976]: time="2016-10-10T20:41:12.067551389Z" level=info msg="Processing signal 'terminated'"
Oct 10 20:41:12 raspberrypi dockerd[976]: time="2016-10-10T20:41:12.128388194Z" level=info msg="stopping containerd after receiving terminated"
Oct 10 20:41:13 raspberrypi systemd[1]: Stopped Docker Application Container Engine.
Oct 10 20:41:13 raspberrypi systemd[1]: Stopping Docker Socket for the API.
Oct 10 20:41:13 raspberrypi systemd[1]: Closed Docker Socket for the API.
Oct 10 20:41:13 raspberrypi systemd[1]: Stopped Docker Application Container Engine.
Oct 10 20:41:50 raspberrypi avahi-daemon[440]: Withdrawing address record for 169.254.124.135 on br0.
Oct 10 20:41:50 raspberrypi dhcpcd[698]: br0: removing IP address 169.254.124.135/16
Oct 10 20:41:50 raspberrypi avahi-daemon[440]: Leaving mDNS multicast group on interface br0.IPv4 with address 169.254.124.135.
Oct 10 20:41:50 raspberrypi avahi-daemon[440]: Interface br0.IPv4 no longer relevant for mDNS.
Oct 10 20:41:50 raspberrypi dhcpcd[698]: br0: deleting route to 169.254.0.0/16
Oct 10 20:41:52 raspberrypi ntpd[723]: Deleting interface #7 br0, 169.254.124.135#123, interface stats: received=0, sent=0, dropped=0, active_time=516 secs
Oct 10 20:41:52 raspberrypi ntpd[723]: peers refreshed
Oct 10 20:42:58 raspberrypi avahi-daemon[440]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.1.19.
Oct 10 20:42:58 raspberrypi avahi-daemon[440]: New relevant interface br0.IPv4 for mDNS.
Oct 10 20:42:58 raspberrypi avahi-daemon[440]: Registering new address record for 192.168.1.19 on br0.IPv4.
Oct 10 20:43:00 raspberrypi ntpd[723]: Listen normally on 8 br0 192.168.1.19 UDP 123
Oct 10 20:43:00 raspberrypi ntpd[723]: peers refreshed
Oct 10 20:43:15 raspberrypi systemd[1]: getty#tty1.service has no holdoff time, scheduling restart.
Oct 10 20:43:15 raspberrypi systemd[1]: Stopping Getty on tty1...
Oct 10 20:43:15 raspberrypi systemd[1]: Starting Getty on tty1...
Oct 10 20:43:15 raspberrypi systemd[1]: Started Getty on tty1.
Oct 10 20:43:21 raspberrypi systemd[1]: getty#tty1.service has no holdoff time, scheduling restart.
Oct 10 20:43:21 raspberrypi systemd[1]: Stopping Getty on tty1...
Oct 10 20:43:21 raspberrypi systemd[1]: Starting Getty on tty1...
Oct 10 20:43:21 raspberrypi systemd[1]: Started Getty on tty1.
Oct 10 20:44:31 raspberrypi systemd[1]: Starting Docker Socket for the API.
Oct 10 20:44:31 raspberrypi systemd[1]: Listening on Docker Socket for the API.
Oct 10 20:44:31 raspberrypi systemd[1]: Starting Docker Application Container Engine...
Oct 10 20:44:31 raspberrypi dockerd[1536]: time="2016-10-10T20:44:31.887581128Z" level=info msg="libcontainerd: new containerd process, pid: 1543"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.903109872Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.950908429Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951611338Z" level=warning msg="Your kernel does not support swap memory limit."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951800086Z" level=warning msg="Your kernel does not support kernel memory limit."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951906179Z" level=warning msg="Your kernel does not support cgroup cfs period"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951993522Z" level=warning msg="Your kernel does not support cgroup cfs quotas"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.952173520Z" level=warning msg="Unable to find cpuset cgroup in mounts"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.952372059Z" level=warning msg="mountpoint for pids not found"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.953406319Z" level=info msg="Loading containers: start."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.970612440Z" level=info msg="Firewalld running: false"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.953406319Z" level=info msg="Loading containers: start."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.970612440Z" level=info msg="Firewalld running: false"
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Withdrawing address record for 192.168.1.19 on br0.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.19.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Interface br0.IPv4 no longer relevant for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Joining mDNS multicast group on interface br0.IPv4 with address 169.254.124.135.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: New relevant interface br0.IPv4 for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Registering new address record for 169.254.124.135 on br0.IPv4.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715576231Z" level=info msg="Loading containers: done."
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715837582Z" level=info msg="Daemon has completed initialization"
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715921435Z" level=info msg="Docker daemon" commit=23cf638 graphdriver=overlay version=1.12.1
Oct 10 20:44:33 raspberrypi systemd[1]: Started Docker Application Container Engine.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.754984356Z" level=info msg="API listen on /var/run/docker.sock"
Oct 10 20:44:34 raspberrypi ntpd[723]: Listen normally on 9 br0 169.254.124.135 UDP 123
Oct 10 20:44:34 raspberrypi ntpd[723]: Deleting interface #8 br0, 192.168.1.19#123, interface stats: received=0, sent=0, dropped=0, active_time=94 secs
Oct 10 20:44:34 raspberrypi ntpd[723]: peers refreshed
The interesting part is the last part (I recopied it here bellow):
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Withdrawing address record for 192.168.1.19 on br0.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.19.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Interface br0.IPv4 no longer relevant for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Joining mDNS multicast group on interface br0.IPv4 with address 169.254.124.135.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: New relevant interface br0.IPv4 for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Registering new address record for 169.254.124.135 on br0.IPv4.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715576231Z" level=info msg="Loading containers: done."
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715837582Z" level=info msg="Daemon has completed initialization"
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715921435Z" level=info msg="Docker daemon" commit=23cf638 graphdriver=overlay version=1.12.1
Oct 10 20:44:33 raspberrypi systemd[1]: Started Docker Application Container Engine.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.754984356Z" level=info msg="API listen on /var/run/docker.sock"
Oct 10 20:44:34 raspberrypi ntpd[723]: Listen normally on 9 br0 169.254.124.135 UDP 123
Oct 10 20:44:34 raspberrypi ntpd[723]: Deleting interface #8 br0, 192.168.1.19#123, interface stats: received=0, sent=0, dropped=0, active_time=94
Once the docker container is running the network configuration is not editable. Try running your docker container with --bip=CIDR and set your bridge ip manually. For more info follow here.