I currently have a docker setup working with haproxy as a load balancer directing traffic to containers running my web app. I'm trying to add SSL termination to HAProxy and have run into some trouble. When I add DEFAULT_SSL_CERT as an environment variable to my haproxy container I get these errors:
Mar 20 20:15:03 escapes-artist kernel: [3804709.167813] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 20:15:03 escapes-artist kernel: [3804709.213993] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 20:15:04 escapes-artist kernel: [3804709.674840] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 20:15:04 escapes-artist kernel: [3804709.688631] device vethebd7d1d entered promiscuous mode
Mar 20 20:15:04 escapes-artist kernel: [3804709.688767] IPv6: ADDRCONF(NETDEV_UP): vethebd7d1d: link is not ready
Mar 20 20:15:04 escapes-artist systemd-udevd: Could not generate persistent MAC address for veth5c0585c: No such file or directory
Mar 20 20:15:04 escapes-artist systemd-udevd: Could not generate persistent MAC address for vethebd7d1d: No such file or directory
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04.671620998Z" level=warning msg="Your kernel does not support swap memory limit."
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04.672345380Z" level=warning msg="Your kernel does not support cgroup rt period"
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04.672732724Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Mar 20 20:15:04 escapes-artist dockerd: time="2017-03-21T02:15:04Z" level=info msg="Firewalld running: false"
Mar 20 20:15:05 escapes-artist kernel: [3804710.392546] eth0: renamed from veth5c0585c
Mar 20 20:15:05 escapes-artist kernel: [3804710.395273] IPv6: ADDRCONF(NETDEV_CHANGE): vethebd7d1d: link becomes ready
Mar 20 20:15:05 escapes-artist kernel: [3804710.395303] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
Mar 20 20:15:05 escapes-artist kernel: [3804710.395313] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
Mar 20 20:15:05 escapes-artist kernel: [3804711.072047] br-5c6735a37ece: port 2(vethbaf33bd) entered forwarding state
Mar 20 20:15:08 escapes-artist kernel: [3804713.819317] haproxy[29684]: segfault at 7f560000003b ip 00007f56f6ac74bb sp 00007ffe45011290 error 4 in libcrypto.so.1.0.0[7f56f69ce000+3f3000]
Mar 20 20:15:11 escapes-artist sshd: Received disconnect from 122.194.229.7 port 21903:11: [preauth]
Mar 20 20:15:11 escapes-artist sshd: Disconnected from 122.194.229.7 port 21903 [preauth]
Mar 20 20:15:13 escapes-artist kernel: [3804718.789238] haproxy[29686]: segfault at 7fbb0000003b ip 00007fbb747b74bb sp 00007ffc944fcc10 error 4 in libcrypto.so.1.0.0[7fbb746be000+3f3000]
Mar 20 20:15:17 escapes-artist kernel: [3804722.944073] br-5c6735a37ece: port 1(veth610d1f4) entered forwarding state
Mar 20 20:15:18 escapes-artist kernel: [3804723.790663] haproxy[29688]: segfault at 7ff10000003b ip 00007ff1ad6004bb sp 00007fffa6f03cb0 error 4 in libcrypto.so.1.0.0[7ff1ad507000+3f3000]
Mar 20 20:15:20 escapes-artist kernel: [3804725.408060] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
Mar 20 20:15:23 escapes-artist kernel: [3804728.792134] haproxy[29690]: segfault at 7f130000003b ip 00007f13210c54bb sp 00007ffcbe3f7670 error 4 in libcrypto.so.1.0.0[7f1320fcc000+3f3000]
Mar 20 20:15:28 escapes-artist kernel: [3804733.823940] haproxy[29692]: segfault at 7f500000003b ip 00007f500b9d94bb sp 00007ffe6d044f10 error 4 in libcrypto.so.1.0.0[7f500b8e0000+3f3000]
Mar 20 20:15:33 escapes-artist kernel: [3804738.780797] haproxy[29694]: segfault at 7f000000003b ip 00007f00310124bb sp 00007fffd6e979b0 error 4 in libcrypto.so.1.0.0[7f0030f19000+3f3000]
Does anyone know how to fix this? I've experimented for hours trying different formats for the cert file, environment variables, etc. and can't seem to figure anything out. Here is the docker-compose.yml file I'm using:
version: '2'
services:
db:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: docker
MYSQL_USER: admin
MYSQL_PASSWORD: password
volumes:
- /storage/docker/mysql-datadir:/var/lib/mysql
ports:
- 3306:3306
web:
image: myimage
restart: always
depends_on:
- db
volumes:
- /home/docker/persistent/media/:/home/docker/code/media/
lb:
image: dockercloud/haproxy
links:
- web
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /etc/haproxy/certs:/certs
environment:
STATS_AUTH: admin:password
RSYSLOG_DESTINATION: logs5.papertrailapp.com:41747
DEFAULT_SSL_CERT: (I've tried both pasting cert here directly and a path to cert)
ports:
- 80:80
- 443:443
- 1936:1936
I have Letsencrypt setup on the host machine to autorenew. The cert that I've been trying to use is a combination of the privkey.pem and fullchian.pem. I've tried concatenating them, using awk 1 ORS='\\n' like the dockercloud/haproxy docs suggest, and just about every other configuration I can think of. Any help would be greatly appreciated.
Also, if I use CERT_FOLDER: /certs/ instead of DEFAULT_SSL_CERT and have my certificate stored in /certs/cert0.pem I get this error instead...
Mar 20 21:19:38 escapes-artist dockerd: time="2017-03-21T03:19:38.840340234Z" level=error msg="containerd: deleting container" error="exit status 1: \"container ce6c0b6df31419691b6593be6744d01c8ccecf5f38851106aa4bb8fac915a63a does not exist\\none or more of the container deletions failed\\n\""
Mar 20 21:19:38 escapes-artist kernel: [3808584.302038] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
Mar 20 21:19:38 escapes-artist kernel: [3808584.302192] veth0bcd06c: renamed from eth0
Mar 20 21:19:38 escapes-artist kernel: [3808584.320863] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
Mar 20 21:19:38 escapes-artist kernel: [3808584.321869] device veth8b1ea8e left promiscuous mode
Mar 20 21:19:38 escapes-artist kernel: [3808584.321874] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
Mar 20 21:19:39 escapes-artist dockerd: time="2017-03-21T03:19:39.055316431Z" level=error msg="Handler for GET /v1.25/exec/c79e3c9b77f0c84d849cc641a425950d55fcbb22bf566922d3fd12e6a0e12e07/json returned error: Container ce6c0b6df31419691b6593be6744d01c8ccecf5f38851106aa4bb8fac915a63a is not running: Exited (0) Less than a second ago"
Mar 20 21:19:39 escapes-artist kernel: [3808584.964578] aufs au_opts_verify:1597:dockerd[23058]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 21:19:39 escapes-artist kernel: [3808585.005699] aufs au_opts_verify:1597:dockerd[23058]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 21:19:40 escapes-artist kernel: [3808585.489799] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
Mar 20 21:19:40 escapes-artist kernel: [3808585.500609] device veth24d6316 entered promiscuous mode
Mar 20 21:19:40 escapes-artist systemd-udevd: Could not generate persistent MAC address for veth24d6316: No such file or directory
Mar 20 21:19:40 escapes-artist kernel: [3808585.505055] IPv6: ADDRCONF(NETDEV_UP): veth24d6316: link is not ready
Mar 20 21:19:40 escapes-artist systemd-udevd: Could not generate persistent MAC address for vethedaad7c: No such file or directory
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40.259076690Z" level=warning msg="Your kernel does not support swap memory limit."
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40.260183880Z" level=warning msg="Your kernel does not support cgroup rt period"
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40.260663645Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Mar 20 21:19:40 escapes-artist dockerd: time="2017-03-21T03:19:40Z" level=info msg="Firewalld running: false"
Mar 20 21:19:40 escapes-artist kernel: [3808585.904671] eth0: renamed from vethedaad7c
Mar 20 21:19:40 escapes-artist kernel: [3808585.918744] IPv6: ADDRCONF(NETDEV_CHANGE): veth24d6316: link becomes ready
Mar 20 21:19:40 escapes-artist kernel: [3808585.919040] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
Mar 20 21:19:40 escapes-artist kernel: [3808585.919058] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
Mar 20 21:19:44 escapes-artist kernel: [3808589.585674] haproxy[32235]: segfault at 341 ip 0000000000000341 sp 00007ffe732fe5b8 error 14 in haproxy[55f6998b1000+d1000]
Mar 20 21:19:49 escapes-artist kernel: [3808594.704226] haproxy[32237]: segfault at 341 ip 0000000000000341 sp 00007ffcb4d1aa08 error 14 in haproxy[563827d10000+d1000]
Mar 20 21:19:54 escapes-artist kernel: [3808599.669540] haproxy[32239]: segfault at 341 ip 0000000000000341 sp 00007ffd1e8bb1b8 error 14 in haproxy[562d926fa000+d1000]
Mar 20 21:19:55 escapes-artist kernel: [3808600.928110] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
Mar 20 21:19:59 escapes-artist kernel: [3808604.602704] haproxy[32241]: segfault at 341 ip 0000000000000341 sp 00007fff142d0898 error 14 in haproxy[5592e3a63000+d1000]
Ok, figured out what the issue was. the dockercloud/haproxy image creates cert files and puts them in /certs/. I had mounted a volume into /certs/, which was messing things up. I moved my mounted volume to /shared-certs/ and everything works!
Related
Thank you for checking this.
Ubuntu 18 server on AWS EC2, docker-compose up was running just fine, suddenly it stopped building after a reboot. Not sure what changed.
Here is the docker-compose.yml
version: '2'
services:
web:
build: .
restart: "no"
command: gulp serve --max_new_space_size=8192 --max-old-space-size=8192 -LLLL
env_file:
- .env
volumes:
- .:/app/code
ports:
- "8050:8000"
- "8005:8005"
- "8888:8888"
Here is the Dockerfile
FROM node:6.10.3
RUN mkdir /app
RUN mkdir /app/code
WORKDIR /app
# Install JavaScript requirements
COPY package.json /app/
COPY package-lock.json /app/
RUN npm install -d
RUN npm rebuild node-sass
# Link gulp
RUN ln -s /app/node_modules/.bin/gulp /usr/bin/gulp
COPY . /app/code/
WORKDIR /app/code
RUN export NODE_OPTIONS="--max-old-space-size=8192"
# Build webpack files
RUN gulp build
EXPOSE 8000
CMD gulp serve
I see some errors in the syslog, not sure if it is related.
Jun 2 15:25:24 ip-10-0-1-194 kernel: [52500.188965] docker0: port 1(veth638f141) entered blocking state
Jun 2 15:25:24 ip-10-0-1-194 kernel: [52500.188968] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:24 ip-10-0-1-194 kernel: [52500.189101] device veth638f141 entered promiscuous mode
Jun 2 15:25:24 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Link UP
Jun 2 15:25:24 ip-10-0-1-194 networkd-dispatcher[947]: WARNING:Unknown index 338 seen, reloading interface list
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5940]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5940]: Could not generate persistent MAC address for veth3a08f68: No such file or directory
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5941]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 2 15:25:24 ip-10-0-1-194 systemd-udevd[5941]: Could not generate persistent MAC address for veth638f141: No such file or directory
Jun 2 15:25:24 ip-10-0-1-194 containerd[993]: time="2021-06-02T15:25:24.995557031Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/7712e133ca3de4a4d407341b7a51428e984c4bcbf2311e27ffbd43cbff56ef44 pid=6001
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.489989] eth0: renamed from veth3a08f68
Jun 2 15:25:25 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Gained carrier
Jun 2 15:25:25 ip-10-0-1-194 systemd-networkd[734]: docker0: Gained carrier
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.509809] IPv6: ADDRCONF(NETDEV_CHANGE): veth638f141: link becomes ready
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.509869] docker0: port 1(veth638f141) entered blocking state
Jun 2 15:25:25 ip-10-0-1-194 kernel: [52500.509870] docker0: port 1(veth638f141) entered forwarding state
Jun 2 15:25:26 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Gained IPv6LL
Jun 2 15:25:27 ip-10-0-1-194 containerd[993]: time="2021-06-02T15:25:27.979112078Z" level=info msg="shim disconnected" id=7712e133ca3de4a4d407341b7a51428e984c4bcbf2311e27ffbd43cbff56ef44
Jun 2 15:25:27 ip-10-0-1-194 dockerd[1010]: time="2021-06-02T15:25:27.979239439Z" level=info msg="ignoring event" container=7712e133ca3de4a4d407341b7a51428e984c4bcbf2311e27ffbd43cbff56ef44 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.292104] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.292214] veth3a08f68: renamed from eth0
Jun 2 15:25:28 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Lost carrier
Jun 2 15:25:28 ip-10-0-1-194 systemd-udevd[6146]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 2 15:25:28 ip-10-0-1-194 systemd-networkd[734]: veth638f141: Link DOWN
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.350623] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.353895] device veth638f141 left promiscuous mode
Jun 2 15:25:28 ip-10-0-1-194 kernel: [52503.353912] docker0: port 1(veth638f141) entered disabled state
Jun 2 15:25:28 ip-10-0-1-194 networkd-dispatcher[947]: WARNING:Unknown index 337 seen, reloading interface list
Jun 2 15:25:28 ip-10-0-1-194 networkd-dispatcher[947]: **ERROR:Unknown interface index 337 seen even after reload**
Jun 2 15:25:29 ip-10-0-1-194 systemd-networkd[734]: docker0: Lost carrier
I am trying to run multiple squid containers whose configs are built at container run time. Each container needs to route traffic independently from the other. Aside from where traffic is forwarded on, the configs are the same.
I can get a single squid container running and doing what I need it to with no problems.
docker run -v /var/log/squid:/var/log/squid -p 3133-3138:3133-3138 my_images/squid_test:version1.0
Trying to run a second container with:
docker run -v /var/log/squid:/var/log/squid -p 4133-4138:3133-3138 my_images/squid_test:version1.0
This instantly spits out: Aborted (core dumped)
I have one other container running on port 9000 but thats it.
This is a syslog dump from the host at the time the second container launch is attempted
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356170] docker0: port 3(veth89ab0c1) entered blocking state
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356172] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356209] device veth89ab0c1 entered promiscuous mode
Jun 18 04:45:17 dockerdevr1 kernel: [84821.356252] IPv6: ADDRCONF(NETDEV_UP): veth89ab0c1: link is not ready
Jun 18 04:45:17 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Link UP
Jun 18 04:45:17 dockerdevr1 networkd-dispatcher[1048]: WARNING:Unknown index 421 seen, reloading interface list
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25899]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25900]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25899]: Could not generate persistent MAC address for vethb0dffb8: No such file or directory
Jun 18 04:45:17 dockerdevr1 systemd-udevd[25900]: Could not generate persistent MAC address for veth89ab0c1: No such file or directory
Jun 18 04:45:17 dockerdevr1 containerd[1119]: time="2020-06-18T04:45:17.567627817Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/85f0acae4a948ed16b3b29988291b5df3d052b10d1965f1198745966e63c3732/shim.sock" debug=false pid=25920
Jun 18 04:45:17 dockerdevr1 kernel: [84821.841905] eth0: renamed from vethb0dffb8
Jun 18 04:45:17 dockerdevr1 kernel: [84821.858172] IPv6: ADDRCONF(NETDEV_CHANGE): veth89ab0c1: link becomes ready
Jun 18 04:45:17 dockerdevr1 kernel: [84821.858263] docker0: port 3(veth89ab0c1) entered blocking state
Jun 18 04:45:17 dockerdevr1 kernel: [84821.858265] docker0: port 3(veth89ab0c1) entered forwarding state
Jun 18 04:45:17 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Gained carrier
Jun 18 04:45:19 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Gained IPv6LL
Jun 18 04:45:19 dockerdevr1 containerd[1119]: time="2020-06-18T04:45:19.221654620Z" level=info msg="shim reaped" id=85f0acae4a948ed16b3b29988291b5df3d052b10d1965f1198745966e63c3732
Jun 18 04:45:19 dockerdevr1 dockerd[1171]: time="2020-06-18T04:45:19.232623257Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 18 04:45:19 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Lost carrier
Jun 18 04:45:19 dockerdevr1 kernel: [84823.251203] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:19 dockerdevr1 kernel: [84823.254402] vethb0dffb8: renamed from eth0
Jun 18 04:45:19 dockerdevr1 systemd-networkd[765]: veth89ab0c1: Link DOWN
Jun 18 04:45:19 dockerdevr1 kernel: [84823.293507] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:19 dockerdevr1 kernel: [84823.294577] device veth89ab0c1 left promiscuous mode
Jun 18 04:45:19 dockerdevr1 kernel: [84823.294580] docker0: port 3(veth89ab0c1) entered disabled state
Jun 18 04:45:19 dockerdevr1 networkd-dispatcher[1048]: WARNING:Unknown index 420 seen, reloading interface list
Jun 18 04:45:19 dockerdevr1 networkd-dispatcher[1048]: ERROR:Unknown interface index 420 seen even after reload
Jun 18 04:45:19 dockerdevr1 systemd-udevd[26041]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 18 04:45:19 dockerdevr1 systemd-udevd[26041]: link_config: could not get ethtool features for vethb0dffb8
Jun 18 04:45:19 dockerdevr1 systemd-udevd[26041]: Could not set offload features of vethb0dffb8: No such device
Has anyone tried something similar to this? I know I can get multiple nginx containers running on different ports. Any insight would be greatly appreciated!
Off recently we are facing the below issue while performing a CI/CD build from gitlab runner.
Below is the log snippet from /var/log/syslog.
pr 22 03:02:04 cirunner dockerd[1103]: time="2019-04-22T03:02:04.136857571Z" level=error msg="Handler for DELETE /v1.18/containers/runner-301e5f4d-project-786-concurrent-0-build-4 returned error: No such container: runner-301e5f4d-project-786-concurrent-0-build-4"
Apr 22 03:02:04 cirunner kernel: [1616845.656927] aufs au_opts_verify:1597:dockerd[1568]: dirperm1 breaks the protection by the permission bits on the lower branch
Apr 22 03:02:04 cirunner kernel: [1616846.186616] aufs au_opts_verify:1597:dockerd[1568]: dirperm1 breaks the protection by the permission bits on the lower branch
Apr 22 03:02:05 cirunner kernel: [1616846.383784] aufs au_opts_verify:1597:dockerd[1568]: dirperm1 breaks the protection by the permission bits on the lower branch
Apr 22 03:02:05 cirunner systemd-udevd[1187]: Could not generate persistent MAC address for veth0675b93: No such file or directory
Apr 22 03:02:05 cirunner kernel: [1616846.385245] device veth8b64bcd entered promiscuous mode
Apr 22 03:02:05 cirunner kernel: [1616846.385299] IPv6: ADDRCONF(NETDEV_UP): veth8b64bcd: link is not ready
Apr 22 03:02:05 cirunner systemd-udevd[1188]: Could not generate persistent MAC address for veth8b64bcd: No such file or directory
Apr 22 03:02:05 cirunner kernel: [1616846.788755] eth0: renamed from veth0675b93
Apr 22 03:02:05 cirunner kernel: [1616846.804716] IPv6: ADDRCONF(NETDEV_CHANGE): veth8b64bcd: link becomes ready
Apr 22 03:02:05 cirunner kernel: [1616846.804739] docker0: port 3(veth8b64bcd) entered forwarding state
Apr 22 03:02:05 cirunner kernel: [1616846.804747] docker0: port 3(veth8b64bcd) entered forwarding state
Apr 22 03:02:20 cirunner kernel: [1616861.819201] docker0: port 3(veth8b64bcd) entered forwarding state
Apr 22 03:37:13 cirunner dockerd[1103]: time="2019-04-22T03:37:13.298195303Z" level=error msg="Handler for GET
/v1.18/containers/6f6b71442b5bbc70f980cd05272c8f05d514735f39e9b73b52a094a0e87db475/json returned error: No such container: 6f6b71442b5bbc70f980cd05272c8f05d514735f39e9b73b52a094a0e87db475"
Could you please help me out what exactly is the issue and how can to trouble shoot.
Let me know if you require additional details from my side.
One of the containers hanged and became unresponsive with the following error message in syslogs. How can I pinpoint the issue?
sys log error:
Jul 20 12:58:26 B2BTestServer dockerd[1140]: time=“2017-07-20T12:58:26.879111489Z” level=error msg=“containerd: deleting container” error=“exit status 1: “container 5f090b50b3dd7840dd296bd3eede2c4db171b5787944317f0a0e52d71c368361 does not exist\none or more of the container deletions failed\n””
Jul 20 13:30:01 B2BTestServer CRON[42898]: (xxxx) CMD (docker exec xxApp_worker_1 ./artisan omni:key b2b_fwad login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42899]: (xxxx) CMD (docker exec xxApp_app_1 ./artisan omni:key b2b_fwad login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42901]: (xxxx) CMD (docker exec xxApp_worker_1 ./artisan omni:key b2b_yww login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42900]: (xxxx) CMD (docker exec xxApp_app_1 ./artisan omni:key b2b_yww login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.382668472Z” level=error msg="Error setting up exec command in container xxApp_app_1: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.382714110Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/exec returned error: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.392014905Z” level=error msg="Error setting up exec command in container xxApp_app_1: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.392237089Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/exec returned error: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.429292849Z” level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.463663587Z” level=error msg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n”
Jul 20 13:30:36 B2BTestServer kernel: [262136.359656] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262136.583654] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.120768757Z” level=error msg=“containerd: deleting container” error="exit status 1: “container f80c3f7d41bcbb2133036c795068ea2200b8043bce1a043f5fb5e09d09c2f720 does not exist\none or more of the container deletions failed\n”"
Jul 20 13:30:37 B2BTestServer kernel: [262136.651251] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.668608] vethebfd8d9: renamed from eth0
Jul 20 13:30:37 B2BTestServer kernel: [262136.716289] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262136.738679] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.753388] device vethe12115e left promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262136.762750] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.804865] device vethdb0a6ac entered promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262136.814982] IPv6: ADDRCONF(NETDEV_UP): vethdb0a6ac: link is not ready
Jul 20 13:30:37 B2BTestServer kernel: [262136.827291] br-e80847589bfa: port 1(vethdb0a6ac) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262136.840256] br-e80847589bfa: port 1(vethdb0a6ac) entered forwarding state
Jul 20 13:30:37 B2BTestServer systemd-udevd[43041]: Could not generate persistent MAC address for vethdb0a6ac: No such file or directory
Jul 20 13:30:37 B2BTestServer systemd-udevd[43040]: Could not generate persistent MAC address for vethf7a98cb: No such file or directory
Jul 20 13:30:37 B2BTestServer kernel: [262137.026630] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579534632Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579622593Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579632100Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:37 B2BTestServer kernel: [262137.101169] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37Z” level=info msg="Firewalld running: false"
Jul 20 13:30:37 B2BTestServer kernel: [262137.207004] aufs au_opts_verify:1597:dockerd[25633]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262137.248708] eth0: renamed from vethf7a98cb
Jul 20 13:30:37 B2BTestServer systemd-udevd[43171]: Could not generate persistent MAC address for veth91e8259: No such file or directory
Jul 20 13:30:37 B2BTestServer systemd-udevd[43172]: Could not generate persistent MAC address for veth3fff19d: No such file or directory
Jul 20 13:30:37 B2BTestServer kernel: [262137.273642] device veth3fff19d entered promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262137.287783] IPv6: ADDRCONF(NETDEV_UP): veth3fff19d: link is not ready
Jul 20 13:30:37 B2BTestServer kernel: [262137.307604] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262137.320987] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262137.335059] IPv6: ADDRCONF(NETDEV_CHANGE): vethdb0a6ac: link becomes ready
Jul 20 13:30:37 B2BTestServer kernel: [262137.349736] br-e80847589bfa: port 2(veth3fff19d) entered disabled state
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883496197Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883793205Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883937606Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37Z” level=info msg="Firewalld running: false"
Jul 20 13:30:38 B2BTestServer kernel: [262137.524751] eth0: renamed from veth91e8259
Jul 20 13:30:38 B2BTestServer kernel: [262137.544382] IPv6: ADDRCONF(NETDEV_CHANGE): veth3fff19d: link becomes ready
Jul 20 13:30:38 B2BTestServer kernel: [262137.556891] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262137.569301] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262137.704615] aufs au_opts_verify:1597:dockerd[3229]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.803344] aufs au_opts_verify:1597:dockerd[3229]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.902578] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.926880] device veth4ff9aca entered promiscuous mode
Jul 20 13:30:38 B2BTestServer systemd-udevd[43302]: Could not generate persistent MAC address for veth4ff9aca: No such file or directory
Jul 20 13:30:38 B2BTestServer kernel: [262137.938135] IPv6: ADDRCONF(NETDEV_UP): veth4ff9aca: link is not ready
Jul 20 13:30:38 B2BTestServer systemd-udevd[43301]: Could not generate persistent MAC address for vetha4d818b: No such file or directory
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495129396Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495215954Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495226161Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38Z” level=info msg="Firewalld running: false"
Jul 20 13:30:38 B2BTestServer kernel: [262138.116861] eth0: renamed from vetha4d818b
Jul 20 13:30:38 B2BTestServer kernel: [262138.136455] IPv6: ADDRCONF(NETDEV_CHANGE): veth4ff9aca: link becomes ready
Jul 20 13:30:38 B2BTestServer kernel: [262138.149540] br-e80847589bfa: port 4(veth4ff9aca) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262138.162430] br-e80847589bfa: port 4(veth4ff9aca) entered forwarding state
Jul 20 13:35:31 B2BTestServer dockerd[1140]: time=“2017-07-20T13:35:31.489837293Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/start returned error: Container already started"
Jul 20 13:36:05 B2BTestServer dockerd[1140]: time=“2017-07-20T13:36:05.230362142Z” level=error msg=“Handler for POST /v1.25/containers/xxApp_worker_1/start returned error: Container already started”
Jul 20 13:50:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:50:01.606622410Z” level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n"
Jul 20 13:50:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:50:01.687809674Z” level=error msg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n”
The following is Docker version information:
sudo docker info
Containers: 8
Running: 3
Paused: 0
Stopped: 5
Images: 14
Server Version: 1.13.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 126
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-83-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Name: TestServer
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
I am using Ubuntu 16.04 with docker 1.11.2. I have configured systemd to automatically restart docker daemon. When I kill the docker daemon, docker daemon restarts, but container will not even it has RestartPolicy set to always. From the logs I can read that it failed to create directory because it exists. I personally think that it related to stopping containerd.
Any help would be appreciated.
Aug 25 19:20:19 api-31 systemd[1]: docker.service: Main process exited, code=killed, status=9/KILL
Aug 25 19:20:19 api-31 docker[17617]: time="2016-08-25T19:20:19Z" level=info msg="stopping containerd after receiving terminated"
Aug 25 19:21:49 api-31 systemd[1]: docker.service: State 'stop-sigterm' timed out. Killing.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Unit entered failed state.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Failed with result 'timeout'.
Aug 25 19:21:49 api-31 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 25 19:21:49 api-31 systemd[1]: Stopped Docker Application Container Engine.
Aug 25 19:21:49 api-31 systemd[1]: Closed Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Stopping Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Starting Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Listening on Docker Socket for the API.
Aug 25 19:21:49 api-31 systemd[1]: Starting Docker Application Container Engine...
Aug 25 19:21:49 api-31 docker[19023]: time="2016-08-25T19:21:49.913162167Z" level=info msg="New containerd process, pid: 19029\n"
Aug 25 19:21:50 api-31 kernel: [87066.742831] audit: type=1400 audit(1472152910.946:23): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=19043 comm="apparmor_parser"
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.952073973Z" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.956693893Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Aug 25 19:21:50 api-31 docker[19023]: time="2016-08-25T19:21:50.961641996Z" level=info msg="Firewalld running: false"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.016582850Z" level=info msg="Removing stale sandbox 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596 (547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613)"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.046227326Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.081106790Z" level=warning msg="Your kernel does not support swap memory limit."
Aug 25 19:21:51 api-31 docker[19023]: time="2016-08-25T19:21:51.081650610Z" level=info msg="Loading containers: start."
Aug 25 19:22:01 api-31 kernel: [87076.922492] docker0: port 1(vethbbc1192) entered disabled state
Aug 25 19:22:01 api-31 kernel: [87076.927128] device vethbbc1192 left promiscuous mode
Aug 25 19:22:01 api-31 kernel: [87076.927131] docker0: port 1(vethbbc1192) entered disabled state
Aug 25 19:22:03 api-31 docker[19023]: .time="2016-08-25T19:22:03.085800458Z" level=warning msg="error locating sandbox id 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596: sandbox 66ef9e1af997a1090fac0c89bf96c2631bea32fbe3c238c4349472987957c596 not found"
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.085907328Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613/shm: invalid argument"
Aug 25 19:22:03 api-31 kernel: [87078.882836] device veth5c6999c entered promiscuous mode
Aug 25 19:22:03 api-31 kernel: [87078.882984] IPv6: ADDRCONF(NETDEV_UP): veth5c6999c: link is not ready
Aug 25 19:22:03 api-31 systemd-udevd[19128]: Could not generate persistent MAC address for veth5c6999c: No such file or directory
Aug 25 19:22:03 api-31 systemd-udevd[19127]: Could not generate persistent MAC address for veth39fb4d3: No such file or directory
Aug 25 19:22:03 api-31 kernel: [87078.944218] docker0: port 1(veth5c6999c) entered disabled state
Aug 25 19:22:03 api-31 kernel: [87078.948636] device veth5c6999c left promiscuous mode
Aug 25 19:22:03 api-31 kernel: [87078.948640] docker0: port 1(veth5c6999c) entered disabled state
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219677059Z" level=error msg="Failed to start container 547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613: rpc error: code = 6 desc = \"mkdir /run/containerd/547bceaad5d121444ddc6effbac3f472d0c232d693d8cc076027e238cf253613: file exists\""
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219750430Z" level=info msg="Loading containers: done."
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219776593Z" level=info msg="Daemon has completed initialization"
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.219847738Z" level=info msg="Docker daemon" commit=b9f10c9 graphdriver=overlay version=1.11.2
Aug 25 19:22:03 api-31 systemd[1]: Started Docker Application Container Engine.
Aug 25 19:22:03 api-31 docker[19023]: time="2016-08-25T19:22:03.226116336Z" level=info msg="API listen on /var/run/docker.sock"
#VonC - Thank you for pointing me at the right direction. I researched the thread, but in my case the apparmor is not an issue. There are some other issues mentioned in the thread, so I followed them and I found the solution.
SOLUTION:
On Ubuntu 16.04 the problem is that systemd kills process containerd with the docker daemon process. In order to prevent it, you need to add
KillMode=process
to /lib/systemd/system/docker.service and that fixes the issue.
Here are the sources I used:
https://github.com/docker/docker/issues/25246
https://github.com/docker/docker/blob/master/contrib/init/systemd/docker.service#L25
That seems to be followed by issue 25487 (August 2016), and was reported even before (April 2016) in issue 22195.
Check if you are in the situation mentioned in issue 21702 by Tõnis Tiigi:
This seems to be caused by the apparmor profile for docker daemon we have in docker/contrib/apparmor.
If this profile is applied in v1.11 (at least ubuntu wily) then container starting does not work.
I'm not sure if users have just manually enforced this profile or apparently we also accidentally installed this profile in 1.10.0-rc1 (#19707).
So the workaround, until we figure out how to deal with this, is to unload the profile with something like apparmor_parser -R /etc/apparmor.d/docker-engine ,delete it and restart daemon.
/etc/apparmor.d/docker is the profile for the containers and does not need to be changed.