I am trying to set docker up to connect all containers to my own manually created bridge (br0), I don't want docker to create or edit anything in my bridge, because I have other services which uses and depends on my bridge (like OpenVPN) therefore I prefer to create the bridge using my own bash script.
The problem comes when I start docker service, docker changes my bridge IP address from what I want (192.168.1.10) to something else address(169.254.x.x)!!!
My Docker version 1.12.1, build 23cf638
The steps I did
Bridge creation:
sudo brctl addbr br0
sudo brctl addif br0 eth0
sudo ip addr del 192.168.1.10/24 dev eth0
sudo ip addr add 192.168.1.10/24 dev br0
sudo ip route add default via 192.168.1.1 dev br0
I also deleted the default docker0 brdige.
Tell docker to use my br0 instead of the default docker0:
Passing -b br0 parameter to dockerd.service starting script to tell docker that I want him to use my br0:
sudo vi /etc/systemd/system/docker.service.d/overlay.conf
I edited ExecStart to be like this:
ExecStart=/usr/bin/dockerd --storage-driver=overlay -H fd:// -b=br0
and then:
sudo systemctl daemon-reload
sudo systemctl restart docker
And now when I check my br0 IP, it is NOT 192.168.1.10 any more, it is back to 172.17.x.x, and when I try to change it now manually back to 192.168.1.10, the interfaces in containers keeps using 169.254.x.x instead of the IP I want.
P.s. when I check where are the interfaces of my containers: brctl show, they are really in my br0 (that means docker accepted -b br0 paramter, but it just ignores or override my intended IP address).
Could some one help me please to over come that problem? it looks for me like a bug maybe. I just want docker to use my br0 with the intended IP address 192.168.1.10.
My need is that all my containers get and IP address in the range I want.
Thanks in advance.
Edited:
My /var/log/daemon.log
Oct 10 20:41:12 raspberrypi systemd[1]: Stopping Docker Application Container Engine...
Oct 10 20:41:12 raspberrypi dockerd[976]: time="2016-10-10T20:41:12.067551389Z" level=info msg="Processing signal 'terminated'"
Oct 10 20:41:12 raspberrypi dockerd[976]: time="2016-10-10T20:41:12.128388194Z" level=info msg="stopping containerd after receiving terminated"
Oct 10 20:41:13 raspberrypi systemd[1]: Stopped Docker Application Container Engine.
Oct 10 20:41:13 raspberrypi systemd[1]: Stopping Docker Socket for the API.
Oct 10 20:41:13 raspberrypi systemd[1]: Closed Docker Socket for the API.
Oct 10 20:41:13 raspberrypi systemd[1]: Stopped Docker Application Container Engine.
Oct 10 20:41:50 raspberrypi avahi-daemon[440]: Withdrawing address record for 169.254.124.135 on br0.
Oct 10 20:41:50 raspberrypi dhcpcd[698]: br0: removing IP address 169.254.124.135/16
Oct 10 20:41:50 raspberrypi avahi-daemon[440]: Leaving mDNS multicast group on interface br0.IPv4 with address 169.254.124.135.
Oct 10 20:41:50 raspberrypi avahi-daemon[440]: Interface br0.IPv4 no longer relevant for mDNS.
Oct 10 20:41:50 raspberrypi dhcpcd[698]: br0: deleting route to 169.254.0.0/16
Oct 10 20:41:52 raspberrypi ntpd[723]: Deleting interface #7 br0, 169.254.124.135#123, interface stats: received=0, sent=0, dropped=0, active_time=516 secs
Oct 10 20:41:52 raspberrypi ntpd[723]: peers refreshed
Oct 10 20:42:58 raspberrypi avahi-daemon[440]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.1.19.
Oct 10 20:42:58 raspberrypi avahi-daemon[440]: New relevant interface br0.IPv4 for mDNS.
Oct 10 20:42:58 raspberrypi avahi-daemon[440]: Registering new address record for 192.168.1.19 on br0.IPv4.
Oct 10 20:43:00 raspberrypi ntpd[723]: Listen normally on 8 br0 192.168.1.19 UDP 123
Oct 10 20:43:00 raspberrypi ntpd[723]: peers refreshed
Oct 10 20:43:15 raspberrypi systemd[1]: getty#tty1.service has no holdoff time, scheduling restart.
Oct 10 20:43:15 raspberrypi systemd[1]: Stopping Getty on tty1...
Oct 10 20:43:15 raspberrypi systemd[1]: Starting Getty on tty1...
Oct 10 20:43:15 raspberrypi systemd[1]: Started Getty on tty1.
Oct 10 20:43:21 raspberrypi systemd[1]: getty#tty1.service has no holdoff time, scheduling restart.
Oct 10 20:43:21 raspberrypi systemd[1]: Stopping Getty on tty1...
Oct 10 20:43:21 raspberrypi systemd[1]: Starting Getty on tty1...
Oct 10 20:43:21 raspberrypi systemd[1]: Started Getty on tty1.
Oct 10 20:44:31 raspberrypi systemd[1]: Starting Docker Socket for the API.
Oct 10 20:44:31 raspberrypi systemd[1]: Listening on Docker Socket for the API.
Oct 10 20:44:31 raspberrypi systemd[1]: Starting Docker Application Container Engine...
Oct 10 20:44:31 raspberrypi dockerd[1536]: time="2016-10-10T20:44:31.887581128Z" level=info msg="libcontainerd: new containerd process, pid: 1543"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.903109872Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.950908429Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951611338Z" level=warning msg="Your kernel does not support swap memory limit."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951800086Z" level=warning msg="Your kernel does not support kernel memory limit."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951906179Z" level=warning msg="Your kernel does not support cgroup cfs period"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.951993522Z" level=warning msg="Your kernel does not support cgroup cfs quotas"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.952173520Z" level=warning msg="Unable to find cpuset cgroup in mounts"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.952372059Z" level=warning msg="mountpoint for pids not found"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.953406319Z" level=info msg="Loading containers: start."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.970612440Z" level=info msg="Firewalld running: false"
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.953406319Z" level=info msg="Loading containers: start."
Oct 10 20:44:32 raspberrypi dockerd[1536]: time="2016-10-10T20:44:32.970612440Z" level=info msg="Firewalld running: false"
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Withdrawing address record for 192.168.1.19 on br0.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.19.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Interface br0.IPv4 no longer relevant for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Joining mDNS multicast group on interface br0.IPv4 with address 169.254.124.135.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: New relevant interface br0.IPv4 for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Registering new address record for 169.254.124.135 on br0.IPv4.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715576231Z" level=info msg="Loading containers: done."
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715837582Z" level=info msg="Daemon has completed initialization"
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715921435Z" level=info msg="Docker daemon" commit=23cf638 graphdriver=overlay version=1.12.1
Oct 10 20:44:33 raspberrypi systemd[1]: Started Docker Application Container Engine.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.754984356Z" level=info msg="API listen on /var/run/docker.sock"
Oct 10 20:44:34 raspberrypi ntpd[723]: Listen normally on 9 br0 169.254.124.135 UDP 123
Oct 10 20:44:34 raspberrypi ntpd[723]: Deleting interface #8 br0, 192.168.1.19#123, interface stats: received=0, sent=0, dropped=0, active_time=94 secs
Oct 10 20:44:34 raspberrypi ntpd[723]: peers refreshed
The interesting part is the last part (I recopied it here bellow):
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Withdrawing address record for 192.168.1.19 on br0.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.19.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Interface br0.IPv4 no longer relevant for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Joining mDNS multicast group on interface br0.IPv4 with address 169.254.124.135.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: New relevant interface br0.IPv4 for mDNS.
Oct 10 20:44:33 raspberrypi avahi-daemon[440]: Registering new address record for 169.254.124.135 on br0.IPv4.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715576231Z" level=info msg="Loading containers: done."
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715837582Z" level=info msg="Daemon has completed initialization"
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.715921435Z" level=info msg="Docker daemon" commit=23cf638 graphdriver=overlay version=1.12.1
Oct 10 20:44:33 raspberrypi systemd[1]: Started Docker Application Container Engine.
Oct 10 20:44:33 raspberrypi dockerd[1536]: time="2016-10-10T20:44:33.754984356Z" level=info msg="API listen on /var/run/docker.sock"
Oct 10 20:44:34 raspberrypi ntpd[723]: Listen normally on 9 br0 169.254.124.135 UDP 123
Oct 10 20:44:34 raspberrypi ntpd[723]: Deleting interface #8 br0, 192.168.1.19#123, interface stats: received=0, sent=0, dropped=0, active_time=94
Once the docker container is running the network configuration is not editable. Try running your docker container with --bip=CIDR and set your bridge ip manually. For more info follow here.
Related
question
A virtual machine of mine is ubuntu, docker is installed when install ubuntu from iso. After that, I installed again from apt, because previous one is not registered in "systemctl". Now , there is two space for both docker, just like following.
Two docker have their own space for "docker ps / image CMD" .
Reboot start with "SPACE docker A", and "systemctl status" works good. Once "systemctl restart", move to "SPACE docker B".
If I want remove "SPACE docker A". How can I do it? I would be very grateful if any suggestion is given.
For now, my workaround is manually systemctl restart every time.
appendix
root#wzhjworkmachine:~/gitRoot# echo "now I reboot"
now I reboot
root#wzhjworkmachine:~/gitRoot# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-08-24 04:12:19 UTC; 1min 10s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 941 (dockerd)
Tasks: 11
Memory: 118.5M
CGroup: /system.slice/docker.service
└─941 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.395653457Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.395739427Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.395827870Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.396046259Z" level=info msg="Loading containers: start."
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.631892982Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option -->
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.666930410Z" level=info msg="Loading containers: done."
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.719311959Z" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.720376021Z" level=info msg="Daemon has completed initialization"
Aug 24 04:12:19 wzhjworkmachine systemd[1]: Started Docker Application Container Engine.
Aug 24 04:12:19 wzhjworkmachine dockerd[941]: time="2020-08-24T04:12:19.752861257Z" level=info msg="API listen on /run/docker.sock"
root#wzhjworkmachine:~/gitRoot# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root#wzhjworkmachine:~/gitRoot# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
wzhjv001 v1 0127641f0d76 2 days ago 867MB
hub.ark.jd.com/wzhj-compile-image4jenkins/quickcompile gov1.14 0127641f0d76 2 days ago 867MB
root#wzhjworkmachine:~/gitRoot# systemctl restart docker
root#wzhjworkmachine:~/gitRoot# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-08-24 04:13:45 UTC; 6s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 2103 (dockerd)
Tasks: 12
Memory: 38.4M
CGroup: /system.slice/docker.service
└─2103 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.183475793Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.183480494Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.183485240Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.183615822Z" level=info msg="Loading containers: start."
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.269534194Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option ->
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.303140798Z" level=info msg="Loading containers: done."
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.313875267Z" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.314078704Z" level=info msg="Daemon has completed initialization"
Aug 24 04:13:45 wzhjworkmachine dockerd[2103]: time="2020-08-24T04:13:45.326129124Z" level=info msg="API listen on /run/docker.sock"
Aug 24 04:13:45 wzhjworkmachine systemd[1]: Started Docker Application Container Engine.
root#wzhjworkmachine:~/gitRoot# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1d09c0866672 9f266d35e02c "/bin/bash" About a minute ago Exited (0) About a minute ago intelligent_rhodes
root#wzhjworkmachine:~/gitRoot# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
centos 7.4.1708 9f266d35e02c 17 months ago 197MB
I can't restart docker service. After command system just getting stuck.
docker service status is here:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: deactivating (stop-sigterm)
Docs: https://docs.docker.com
Main PID: 1216 (dockerd)
Tasks: 9
CGroup: /system.slice/docker.service
└─1216 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Aug 25 15:08:21 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:21.179342033-07:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=
Aug 25 15:08:21 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:21.631714600-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.607526073-07:00" level=warning msg="Your kernel does not support swap memory limit"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.608984581-07:00" level=warning msg="Your kernel does not support cgroup rt period"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.609119466-07:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Aug 25 15:08:23 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:23.609435097-07:00" level=info msg="Loading containers: start."
Aug 25 15:08:32 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:32.266035465-07:00" level=info msg="Removing stale sandbox f2295d6e0a7aec8569c42470bbccbc5a1
Aug 25 15:08:32 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:32.677509372-07:00" level=warning msg="Error (Unable to complete atomic operation, key modif
Aug 25 15:08:33 hq-rproxy02 dockerd[1216]: time="2020-08-25T15:08:33.339048947-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address
Aug 27 05:12:40 hq-rproxy02 dockerd[1216]: time="2020-08-27T05:12:40.330007214-07:00" level=info msg="Processing signal 'terminated'"
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-08-25 15:07:44 PDT; 5 days ago
Listen: /var/run/docker.sock (Stream)
Tasks: 0 (limit: 1113)
CGroup: /system.slice/docker.socket
Aug 25 15:07:44 hq-rproxy02 systemd[1]: Starting Docker Socket for the API.
Aug 25 15:07:44 hq-rproxy02 systemd[1]: Listening on Docker Socket for the API.
What can I check?
Following is the error I am getting while trying to start the docker daemon service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─50-docker-service.conf
Active: failed (Result: exit-code) since Tue 2017-08-22 02:09:40 UTC; 15min ago
Docs: http://docs.docker.com
Main PID: 3571 (code=exited, status=1/FAILURE)
CPU: 292ms
Aug 22 02:09:40 systemd[1]: docker.service: Unit entered failed state.
Aug 22 02:09:40 systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 22 02:09:40 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 22 02:09:40 systemd[1]: Stopped Docker Application Container Engine.
Aug 22 02:09:40 systemd[1]: docker.service: Start request repeated too quickly.
Aug 22 02:09:40 systemd[1]: Failed to start Docker Application Container Engine.
Aug 22 02:09:40 systemd[1]: docker.service: Unit entered failed state.
Aug 22 02:09:40 systemd[1]: docker.service: Failed with result 'exit-code'.
Below is the config files I have
50-docker-service.conf
[Service]
Environment="DOCKER_OPTS=--bip=A.B.C.D"
what could be the cause?
$ls -ltr /etc/systemd/system/docker.service.d
total 16
-rw-r--r--. 1 root root 125 Aug 22 02:09 50-docker-service.conf
journalctl logs
Jul 14 13:55:52 systemd[1]: Starting Docker Application Container Engine...
Jul 14 13:55:52 dockerd[1274]: time="2017-07-14T13:55:52.925276313Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Jul 14 13:55:53 dockerd[1274]: time="2017-07-14T13:55:53.378204522Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jul 14 13:55:53 dockerd[1274]: time="2017-07-14T13:55:53.379367854Z" level=info msg="Loading containers: start."
Jul 14 13:55:53 dockerd[1274]: ..time="2017-07-14T13:55:53.507972850Z" level=info msg="Firewalld running: false"
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.013379242Z" level=info msg="Loading containers: done."
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.021206395Z" level=info msg="Daemon has completed initialization"
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.021283711Z" level=info msg="Docker daemon" commit=a82d35e graphdriver=overlay version=1.12.6
Jul 14 13:55:54 systemd[1]: Started Docker Application Container Engine.
Jul 14 13:55:54 dockerd[1274]: time="2017-07-14T13:55:54.039479153Z" level=info msg="API listen on /var/run/docker.sock"
Jul 14 13:56:03 dockerd[1274]: time="2017-07-14T13:56:03.565234227Z" level=error msg="Handler for POST /v1.24/containers/7019b26d0cb3/start returned error: Container already started"
Jul 14 13:56:09 dockerd[1274]: time="2017-07-14T13:56:09.660967581Z" level=error msg="Handler for POST /v1.24/containers/7019b26d0cb3/start returned error: Container already started"
Jul 14 13:56:14 dockerd[1274]: time="2017-07-14T13:56:14.741806551Z" level=error msg="Handler for POST /v1.24/containers/d9a3cb2b66e0/start returned error: Container already started"
Jul 14 21:05:16 dockerd[1274]: time="2017-07-14T21:05:16.992138499Z" level=info msg="Container 7019b26d0cb31412f40f8ab7f971f26896debcce09a58c39679dbaf62f6caa0b failed to exit within 0 s
Jul 14 21:07:20 dockerd[1274]: time="2017-07-14T21:07:20.897682536Z" level=info msg="Container d9a3cb2b66e0395086decd444f7ef52775f76f64b8d4dc291ee66cca48e53535 failed to exit within 2 s
Jul 14 21:08:08 systemd[1]: Stopping Docker Application Container Engine...
Jul 14 21:08:08 dockerd[1274]: time="2017-07-14T21:08:08.243279424Z" level=info msg="Processing signal 'terminated'"
Jul 14 21:08:18 dockerd[1274]: time="2017-07-14T21:08:18.244896088Z" level=info msg="Container 9966c97ca301ef593a68ab8e50730552dbe945c42e6b530d4c6339d3ffa8f544 failed to exit within 10
Jul 14 21:08:18 dockerd[1274]: time="2017-07-14T21:08:18.244909995Z" level=info msg="Container d94a3d0c45471753da85fce062e3c56c79c14ad40b6870281515333b98d0807e failed to exit within 10
Jul 14 21:08:18 dockerd[1274]: time="2017-07-14T21:08:18.244936931Z" level=info msg="Container b49d40479583c121ae7abe14489c7121a25a56da00f3d25961f981c918d3257e failed to exit within 10
Jul 14 21:08:18 systemd[1]: Stopped Docker Application Container Engine.
Docker bridge ip is not configured properly.
Environment="DOCKER_OPTS=--bip=A.B.C.D
Instead configured bip with a fully qualified CIDR
Environment="DOCKER_OPTS=--bip=A.B.C.D/size
that solved my problem
Check ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
is not modified in /lib/systemd/system/docker.service
I have updated docker in my OpenSUSE 13.2.
After some tests I see that -H flag in /etc/sysconfig/docker is causing dockerd not to start, but I need it to enable port 2375 or 2376 (as it has been working OK for months). With it, TSL or not TSL, all or any port, docker will not start. I have tried binding to 0.0.0.0, localhost, ...
-- Logs begin at Tue 2016-10-25 12:48:00 CEST, end at Thu 2017-02-02 23:02:35 CET. --
Feb 02 23:01:35 ezequiel dockerd[22661]: time="2017-02-02T23:01:35.134216922+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Feb 02 23:01:35 ezequiel dockerd[22661]: time="2017-02-02T23:01:35.247510727+01:00" level=info msg="Loading containers: done."
Feb 02 23:01:35 ezequiel dockerd[22661]: time="2017-02-02T23:01:35.247659069+01:00" level=info msg="Daemon has completed initialization"
Feb 02 23:01:35 ezequiel dockerd[22661]: time="2017-02-02T23:01:35.247709386+01:00" level=info msg="Docker daemon" commit=78d1802 graphdriver=btrfs version=1.12.6
Feb 02 23:01:35 ezequiel dockerd[22661]: time="2017-02-02T23:01:35.267370317+01:00" level=info msg="API listen on 192.168.100.1:2375"
Feb 02 23:02:35 ezequiel docker_service_helper.sh[22662]: Docker is dead
Feb 02 23:02:35 ezequiel systemd[1]: docker.service: control process exited, code=exited status=1
Feb 02 23:02:35 ezequiel dockerd[22661]: time="2017-02-02T23:02:35.810756005+01:00" level=info msg="Processing signal 'terminated'"
Feb 02 23:02:35 ezequiel systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Feb 02 23:02:35 ezequiel systemd[1]: Unit docker.service entered failed state.
If I remove it, docker starts, but I can't access it from outside the host (I used to use TSL through port 2376)
I have tried dockerd directly and it binds to tcp port:
# /usr/bin/dockerd --containerd /run/containerd/containerd.sock --add-runtime oci=/usr/bin/docker-runc --label provider=generic -g /optLVM/varLibDocker -H tcp://127.0.0.1:2375
WARN[0000] [!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]
INFO[0000] [graphdriver] using prior storage driver "btrfs"
INFO[0000] Graph migration to content-addressability took 0.00 seconds
WARN[0000] Your kernel does not support swap memory limit.
WARN[0000] Your kernel does not support kernel memory limit.
WARN[0000] mountpoint for pids not found
INFO[0000] Loading containers: start.
.................INFO[0000] Firewalld running: false
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[0000] Loading containers: done.
INFO[0000] Daemon has completed initialization
INFO[0000] Docker daemon commit=78d1802 graphdriver=btrfs version=1.12.6
INFO[0000] API listen on 127.0.0.1:2375
So it seems something has changed in the configuration.
My old version was:
docker-1.12.1-152.3.x86_64
And new one:
docker-1.12.6-176.1.x86_64
Thanks for any help... I do need TCP, with or without TSL, to access docker remotely.
I got the same Problem after updating.
From 1.12.1 to 1.12.6 they changed somthing with the "fd://". For me it did not work anymore. I'm using tcp with tls.
In my config file (/etc/docker/daemon.json)
{
"tls" : true,
"tlsverify": true,
"tlscacert": "/etc/docker/ca.pem",
"tlscert" : "/etc/docker/server/server-cert.pem",
"tlskey" : "/etc/docker/server/server-key.pem",
"hosts" : ["unix:///var/run/docker.sock", "tcp://10.10.1.1:2376"]
}
I am adding the "host": unix:///var/run/docker.sock
I think the local communication is handled via the unix socket and remote connection runs over tcp.
You can find the Infos here ...
I am using Ubuntu 16.04 with docker 1.11.2. I have configured systemd to automatically restart docker daemon. When I kill the docker daemon, docker daemon restarts, but container will not even it has RestartPolicy set to always. From the logs I can read that it failed to create directory because it exists. I personally think that it related to stopping containerd.
Any help would be appreciated.
Aug 25 19:20:19 api-31 systemd[1]: docker.service: Main process exited, code=killed, status=9/KILL
Aug 25 19:20:19 api-31 docker[17617]: time="2016-08-25T19:20:19Z" level=info msg="stopping containerd after receiving terminated"
Aug 25 19:21:49 api-31 systemd[1]: docker.service: State 'stop-sigterm' timed out. Killing.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Unit entered failed state.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Failed with result 'timeout'.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 25 19:21:49 api-31 systemd[1]: Stopped Docker Application Container Engine.
Aug 25 19:21:49 api-31 systemd[1]: Closed Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Stopping Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Starting Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Listening on Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Starting Docker Application Container Engine...
Aug 25 19:21:49 api-31 docker[19023]: time="2016-08-25T19:21:49.913162167Z" level=info msg="New containerd process, pid: 19029\n"
Aug 25 19:21:50 api-31 kernel: [87066.742831] audit: type=1400 audit(1472152910.946:23): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=19043 comm="apparmor_parser"
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.952073973Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.956693893Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.961641996Z" level=info msg="Firewalld running: false"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.016582850Z" level=info msg="Removing stale sandbox 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596 (547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613)"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.046227326Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.081106790Z" level=warning msg="Your kernel does not support swap memory limit."
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.081650610Z" level=info msg="Loading containers: start."
Aug 25 19:22:01 api-31 kernel: [87076.922492] docker0: port 1(vethbbc1192) entered disabled state
Aug 25 19:22:01 api-31 kernel: [87076.927128] device vethbbc1192 left promiscuous mode
Aug 25 19:22:01 api-31 kernel: [87076.927131] docker0: port 1(vethbbc1192) entered disabled state
Aug 25 19:22:03 api-31 docker[19023]: .time="2016-08-25T19:22:03.085800458Z" level=warning msg="error locating sandbox id 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596: sandbox 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596 not found"
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.085907328Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613/shm: invalid argument"
Aug 25 19:22:03 api-31 kernel: [87078.882836] device veth5c6999c entered promiscuous mode
Aug 25 19:22:03 api-31 kernel: [87078.882984] IPv6: ADDRCONF(NETDEV_UP): veth5c6999c: link is not ready
Aug 25 19:22:03 api-31 systemd-udevd[19128]: Could not generate persistent MAC address for veth5c6999c: No such file or directory
Aug 25 19:22:03 api-31 systemd-udevd[19127]: Could not generate persistent MAC address for veth39fb4d3: No such file or directory
Aug 25 19:22:03 api-31 kernel: [87078.944218] docker0: port 1(veth5c6999c) entered disabled state
Aug 25 19:22:03 api-31 kernel: [87078.948636] device veth5c6999c left promiscuous mode
Aug 25 19:22:03 api-31 kernel: [87078.948640] docker0: port 1(veth5c6999c) entered disabled state
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219677059Z" level=error msg="Failed to start container 547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613: rpc error: code = 6 desc = \"mkdir /run/containerd/547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613: file exists\""
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219750430Z" level=info msg="Loading containers: done."
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219776593Z" level=info msg="Daemon has completed initialization"
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219847738Z" level=info msg="Docker daemon" commit=b9f10c9 graphdriver=overlay version=1.11.2
Aug 25 19:22:03 api-31 systemd[1]: Started Docker Application Container Engine.
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.226116336Z" level=info msg="API listen on /var/run/docker.sock"
#VonC - Thank you for pointing me at the right direction. I researched the thread, but in my case the apparmor is not an issue. There are some other issues mentioned in the thread, so I followed them and I found the solution.
SOLUTION:
On Ubuntu 16.04 the problem is that systemd kills process containerd with the docker daemon process. In order to prevent it, you need to add
KillMode=process
to /lib/systemd/system/docker.service and that fixes the issue.
Here are the sources I used:
https://github.com/docker/docker/issues/25246
https://github.com/docker/docker/blob/master/contrib/init/systemd/docker.service#L25
That seems to be followed by issue 25487 (August 2016), and was reported even before (April 2016) in issue 22195.
Check if you are in the situation mentioned in issue 21702 by Tõnis Tiigi:
This seems to be caused by the apparmor profile for docker daemon we have in docker/contrib/apparmor.
If this profile is applied in v1.11 (at least ubuntu wily) then container starting does not work.
I'm not sure if users have just manually enforced this profile or apparently we also accidentally installed this profile in 1.10.0-rc1 (#19707).
So the workaround, until we figure out how to deal with this, is to unload the profile with something like apparmor_parser -R /etc/apparmor.d/docker-engine ,delete it and restart daemon.
/etc/apparmor.d/docker is the profile for the containers and does not need to be changed.