Docker failed to restart with data on mount hard drive - docker

I mount a hard drive on /mnt/data with mount /dev/vdb /mnt/data
Change default docker folder inside : /etc/docker/daemon.json by
{
"graph": "/mnt/data/docker"
}
Everything work fine.
but after a computer restart :
docker daemon can't restart
I got these error :
$ service docker start
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
and
$ systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since Thu 2017-10-05 13:58:58 UTC; 45s ago
Docs: https://docs.docker.com
Process: 5896 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
Main PID: 5896 (code=exited, status=1/FAILURE)
Oct 05 13:58:58 mycomputer systemd[1]: Failed to start Docker Application Container Engine.
Oct 05 13:58:58 mycomputer systemd[1]: docker.service: Unit entered failed state.
Oct 05 13:58:58 mycomputer systemd[1]: docker.service: Failed with result 'exit-code'.
Oct 05 13:58:58 mycomputer systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Oct 05 13:58:58 mycomputer systemd[1]: Stopped Docker Application Container Engine.
Oct 05 13:58:58 mycomputer systemd[1]: docker.service: Start request repeated too quickly.
Oct 05 13:58:58 mycomputer systemd[1]: Failed to start Docker Application Container Engine.

Docker is starting before your non-OS disk (mnt/data/docker) is mounted. You can ask docker to wait for a successful mount before starting docker.
Find the mount, edit your Docker systemd service file to require and wait for the mount before starting.
systemctl list-units --type==mount # Lists all the mounts.
systemctl status <example.mount> # Get details about mount. Find the one which matches your mount directory
sudo nano /lib/systemd/system/docker.service # Edit your systemd service file
Add the <example.mount> name at the end of the "After" and "Requires" row.
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service <example.mount>
Wants=network-online.target
Requires=docker.socket <example.mount>
...
sudo systemctl daemon-reload
sudo systemctl restart docker.service

Related

docker.service failed to run 'start' task: No such file or directory

When I using this script to start docker on CentOS 7.4:
systemctl start docker.service
It shows this error:
[root#ops001 docker]# systemctl start docker.service
JJob for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.
Then I use this command to check docker status:
systemctl status docker.service -l
It show me like this:
[root#ops001 docker]# systemctl status docker.service -l
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Sun 2020-04-12 21:31:35 CST; 3min 17s ago
Docs: https://docs.docker.com
Apr 12 21:31:33 ops001.example.com systemd[1]: docker.service failed to run 'start' task: No such file or directory
Apr 12 21:31:33 ops001.example.com systemd[1]: Failed to start Docker Application Container Engine.
Apr 12 21:31:33 ops001.example.com systemd[1]: Unit docker.service entered failed state.
Apr 12 21:31:33 ops001.example.com systemd[1]: docker.service failed.
Apr 12 21:31:35 ops001.example.com systemd[1]: docker.service holdoff time over, scheduling restart.
Apr 12 21:31:35 ops001.example.com systemd[1]: Stopped Docker Application Container Engine.
Apr 12 21:31:35 ops001.example.com systemd[1]: start request repeated too quickly for docker.service
Apr 12 21:31:35 ops001.example.com systemd[1]: Failed to start Docker Application Container Engine.
Apr 12 21:31:35 ops001.example.com systemd[1]: Unit docker.service entered failed state.
Apr 12 21:31:35 ops001.example.com systemd[1]: docker.service failed.
where is going wrong? What should I do to make docker work?
I am check the docker service define file vim /usr/lib/systemd/system/docker.service:
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
that using file /run/docker_opts.env, but do not find the file in the path. so using this command to generate the file:
/opt/k8s/bin/mk-docker-opts.sh -d /run/docker_opts.env -c
then using this command to start:
systemctl start docker
works.

Does dockerd support WatchdogSec sd_notify health checks?

We've been having issues where the Docker daemon will occasionally stop responding on one of our Kubernetes systems, but Systemd still thinks the service is running:
systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-04-15 20:40:57 UTC; 3 months 22 days ago
Docs: https://docs.docker.com
Main PID: 1281 (dockerd)
Tasks: 1409
Memory: 31.0G
CPU: 5d 17h 3min 4.758s
CGroup: /system.slice/docker.service
├─ 1281 /usr/bin/dockerd -H fd://
...
There isn't anything in the journalctl -u docker or syslog files to indicate what the issue is, but the Docker daemon no longer responds to requests (docker ps just hangs). We are currently using the 17.03.2~ce-0~ubuntu-xenial package for Ubuntu 16.04, which has the following service unit:
cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket firewalld.service
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I noticed that even though it is a Type=notify service, there isn't a WatchdogSec= defined in the service unit.
Does the Docker daemon support setting a watchdog timeout for sd_notify based health checks?
No, currently the components/engine/cmd/dockerd/daemon_linux.go file only implements systemdDaemon.SdNotifyReady to notify Systemd when the process has started. For watchdog support it would have to use something like SdWatchdogEnabled to continually send SdNotifyWatchdog = "WATCHDOG=1" notifications.
If you try and set WatchdogSec=60s on the docker.service file it will kill and restart the service because the daemon doesn't send the required notifications.
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-08-08 02:09:52 UTC; 50s ago
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: deactivating (stop-sigabrt) (Result: watchdog) since Thu 2019-08-08 02:10:02 UTC; 45ms ago
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: activating (start) since Thu 2019-08-08 02:10:04 UTC; 777ms ago
# Log entries:
Aug 08 02:09:14 kam1 systemd[1]: Starting Docker Application Container Engine...
Aug 08 02:09:15 kam1 systemd[1]: Started Docker Application Container Engine.
Aug 08 02:10:15 kam1 systemd[1]: docker.service: Watchdog timeout (limit 60s)!
Aug 08 02:10:15 kam1 systemd[1]: docker.service: Killing process 12383 (dockerd) with signal SIGABRT.
Aug 08 02:10:16 kam1 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 08 02:10:16 kam1 systemd[1]: docker.service: Failed with result 'watchdog'.
Aug 08 02:10:18 kam1 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 08 02:10:18 kam1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 08 02:10:18 kam1 systemd[1]: Stopped Docker Application Container Engine.
Aug 08 02:10:18 kam1 systemd[1]: Starting Docker Application Container Engine...

Cannot connect to the Docker daemon at unix:///var/run/docker.sock on Ubuntu 16.04

Accidentally, my docker goes wrong, and cannot be started.
docker psgives the following error
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
And systemctl status docker.service gives the following outputs:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/docker.service.d
└─override.conf
Active: failed (Result: start-limit-hit) since Tue 2018-11-20 18:27:11 PST; 11s ago
Docs: https://docs.docker.com
Process: 16519 ExecStart=/usr/bin/dockerd --host=fd:// --add-runtime=nvidia=/usr/bin/nvidia-container-runtime (code=exited, status=1/FAILURE)
Main PID: 16519 (code=exited, status=1/FAILURE)
Nov 20 18:27:09 loccalhost systemd[1]: Failed to start Docker Application Container Engine.
Nov 20 18:27:09 loccalhost systemd[1]: docker.service: Unit entered failed state.
Nov 20 18:27:09 loccalhost systemd[1]: docker.service: Failed with result 'exit-code'.
Nov 20 18:27:11 loccalhost systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Nov 20 18:27:11 loccalhost systemd[1]: Stopped Docker Application Container Engine.
Nov 20 18:27:11 loccalhost systemd[1]: docker.service: Start request repeated too quickly.
Nov 20 18:27:11 loccalhost systemd[1]: Failed to start Docker Application Container Engine.
Nov 20 18:27:11 loccalhost systemd[1]: docker.service: Unit entered failed state.
Nov 20 18:27:11 loccalhost systemd[1]: docker.service: Failed with result 'start-limit-hit'.
A more recent answer in the same thread mentioned by skyuua suggests the following (it changes the -H fd:// to -H unix://)
sudo systemctl edit docker.service
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:// -H tcp://127.0.0.1:2375
followed by
sudo systemctl daemon-reload
sudo systemctl restart docker.service
On my Ubuntu 16.04 with docker 18.0.3 this rewrites the file /etc/systemd/system/docker.service.d/override.conf which allowed docker to start correctly.
We ran sudo apt-get update, our docker version got updated too, it changed from 18.09.1 to 18.09.3, and docker went haywire. After digging online, I fould following solution that solved issue for us.
sudo systemctl edit docker.service
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd-ce -H unix://var/run/docker.sock -H tcp://0.0.0.0:2375
then restart service
sudo systemctl restart docker.service

start docker error when add --insecure-registry

i want create private registry and follow this answer Setting up a remote private Docker registry
docker run -p 5000:5000 registry
and other servre modify /usr/lib/systemd/system/docker.service file to disable https
ExecStart=/usr/bin/dockerd --insecure-registry 172.24.54.41:5000
i can curl 172.24.54.41:5000
sudo service docker stop
sudo systemctl daemon-reload
sudo systemctl start docker
the error log in systemctl status docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Mon 2017-12-11 16:52:23 CST; 6min ago
Docs: https://docs.docker.com
Process: 13300 ExecStart=/usr/bin/dockerd --insecure-registry 172.24.54.41:5000 (code=exited, status=1/FAILURE)
Main PID: 13300 (code=exited, status=1/FAILURE)
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: Failed to start Docker Application Container Engine.
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: Unit docker.service entered failed state.
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: docker.service failed.
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: docker.service holdoff time over, scheduling restart.
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: start request repeated too quickly for docker.service
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: Failed to start Docker Application Container Engine.
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: Unit docker.service entered failed state.
Dec 11 16:52:23 iZhp39y3a2bx21oz0waijyZ systemd[1]: docker.service failed.
how to solve it?
It seems that the issue does not lie with your running of a registry but with your configuration of the docker service.
Could you link the content of your usr/lib/systemd/system/docker.service file?

Warning: Stopping docker.service, but it can still be activated by: docker.socket

I've reinstalled Docker. When I'm trying to start Docker, everything is fine:
# /etc/init.d/docker start
[ ok ] Starting docker (via systemctl): docker.service.
until I want to stop Docker service and many times restart it:
# /etc/init.d/docker stop
[....] Stopping docker (via systemctl): docker.serviceWarning: Stopping docker.service, but it can still be activated by:
docker.socket
. ok
Finally, I've got error:
# /etc/init.d/docker start
[....] Starting docker (via systemctl): docker.serviceJob for docker.service failed.
See "systemctl status docker.service" and "journalctl -xe" for details.
failed!
# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Sat 2017-11-25 20:04:20 CET; 2min 4s ago
Docs: https://docs.docker.com
Process: 12845 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=0/SUCCESS)
Main PID: 12845 (code=exited, status=0/SUCCESS)
CPU: 326ms
Nov 25 20:04:18 example.com systemd[1]: Started Docker Application Container Engine.
Nov 25 20:04:18 example.com dockerd[12845]: time="2017-11-25T20:04:18.191949863+01:00" level=inf
Nov 25 20:04:19 example.com systemd[1]: Stopping Docker Application Container Engine...
Nov 25 20:04:19 example.com dockerd[12845]: time="2017-11-25T20:04:19.368990531+01:00" level=inf
Nov 25 20:04:19 example.com dockerd[12845]: time="2017-11-25T20:04:19.37953454+01:00" level=info
Nov 25 20:04:20 example.com systemd[1]: Stopped Docker Application Container Engine.
Nov 25 20:04:21 example.com systemd[1]: docker.service: Start request repeated too quickly.
Nov 25 20:04:21 example.com systemd[1]: Failed to start Docker Application Container Engine.
Nov 25 20:04:21 example.com systemd[1]: docker.service: Unit entered failed state.
Nov 25 20:04:21 example.com systemd[1]: docker.service: Failed with result 'start-limit-hit'.
I've installed Docker on Debian 9 Stretch.
Can anyone help me get rid of this warning and resolve an error "Failed with result 'start-limit-hit'"?
Simply start and stop the socket if the docker is triggered by the socket
sudo systemctl stop docker.socket
This is because in addition to the docker.service unit file, there is a docker.socket unit file... this is for socket activation. The warning means if you try to connect to the docker socket while the docker service is not running, then systemd will automatically start docker for you.
You can get rid of this by removing /lib/systemd/system/docker.socket... you may also need to remove -H fd:// from the docker.service unit file.

Resources