My ultimate goal is to have docker wait for my mergerfs mount before it starts docker and for mergerfs to wait for 3 rclone mounts before it starts itself. I've tried 'requires' and bash script for 'execprestart' and 'requiresmountsfor', 'wants=', 'after=', and a combination of all of these things. I've even tried mounting mergerfs via /etc/fstab. None of it works. The systemd services just 'fail' and don't try again because of requirements etc. Nothing waits or keeps trying.
What's the proper way to mount rclone, have mergerfs wait until those mounts are successful, then have docker override wait for mergerfs to successfully mount to start?
docker.service.d/override.conf
[Unit]
After=merge.service
{tv,movies,music}.service
[Unit]
Description=tv
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
Environment=RCLONE_CONFIG=/data/rclone/config/rclone.conf
RestartSec=5
ExecStart=/usr/bin/rclone mount tv: /mnt/media/tv \
--<options>
ExecStop=/bin/fusermount -uz /mnt/media/tv
ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5572 _async=true
Restart=on-failure
User=user
Group=user
[Install]
WantedBy=multi-user.target
merge.service
[Unit]
Description=merge
After=tv.service movies.service music.service
[Service]
Type=forking
ExecStart=/usr/bin/mergerfs /mnt/local:/mnt/media=NC /mnt/merge -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full,nonempty
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
A quick way to accomplish the goal is to test for existence of the mount via pre start script and have systemd retry on failure.
[Service]
ExecStartPre=/usr/bin/test -f /mnt/media/tv
RestartSec=5s
Restart=on-failure
Related
My VM crashed because it was out of memory. After rebooting the machine docker was not running:
systemctl status docker
● docker.service
Loaded: error (Reason: Invalid argument)
Active: inactive (dead)
Dec 19 08:18:21 my-vm-single-instance systemd[1]: [/lib/systemd/system/docker.service:1] Assignment outside of section. Ignoring.
Dec 19 08:18:21 my-vm-single-instance systemd[1]: docker.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
I installed docker using the offical documentation: https://docs.docker.com/engine/install/debian/
The VM is running:
Debian GNU/Linux 9 (stretch)
Docker version 19.03.14, build 5eb3275d40
docker-compose version 1.25.4, build 8d51620a
I got docker up and running again with
dockerd
However I would like to get it running again through systemctl.
The contents of /lib/systemd/system/docker.service are:
Environment="GOOGLE_APPLICATION_CREDENTIALS=/etc/docker/key.json"
Any ideas how to fix this problem?
If the docker.service contains only one line as it mentioned it's bogus.
As it says
docker.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
the executions scripts are missing at least.
Here is a sample service file:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
This is my default service file. I've never modified it after installation.
When I am trying to start docker using the command:
sudo systemctl start docker
I am getting below error
Failed to start docker.service: Unit not found.
I tried finding some suggestions over the web to resolve this issue and followed that but it didn't solve the issue.
Cannot start docker daemon in CentOS7
This is my docker.socket file [which is just a copy-paste of one of the answer]
[Unit]
Description=Docker Socket for the API
PartOf=docker.service
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
This is the error I am getting error while starting docker.socket
sudo systemctl start docker.socket
See "systemctl status docker.socket" and "journalctl -xe" for details.
output of "systemctl status docker.socket"
systemctl status docker.socket
systemd[1]: Socket service docker.service not loaded, refusing.
systemd[1]: Failed to listen on Docker Socket for the API.
docker version details
Client: Docker Engine - Community
Version: 19.03.2
API version: 1.40
Go version: go1.12.8
Git commit: 6a30dfca03
Built: Thu Aug 29 05:26:30 2019
OS/Arch: linux/amd64
Experimental: false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
To me, it looks like that docker.service is an issue. Could you please suggest how I can resolve it.
There should be a docker.service unit file at either /lib/systemd/system or /etc/systemd/system. Mine looks like what's shown below.
If you have one there, you can try to make sure it's enabled via:
sudo systemctl enable docker.service
Here's an exmaple of the docker.service unit file:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I have the following systemd script:
[Unit]
Description=Hub docker container
After=docker.service
[Service]
User=root
ExecStart=/home/hub/hub.sh
ExecStop=/bin/docker stop hub
ExecStopPost=/bin/docker rm hub
[Install]
WantedBy=multi-user.target
Running the command: systemctl start/stop hub works fine. I also created the symlink by using systemctl enable hub. Why doesn't my service start up after I reboot the entire laptop? I followed the docker guide so that Docker starts up on reboot, but for some reason my container doesn't start up. Am I missing a field in my script?
The command I am using my ExecStart, "/home/hub/hub.sh" script is:
docker run --net=host --restart=always --name hub -t hub
After reboot I get the following when I type systemctl status hub:
● hub.service - Hub docker container
Loaded: loaded (/etc/systemd/system/hub.service; enabled; vendor preset: disabled)
Active: inactive (dead)
In my case, I already had the containers set to restart=always (btw you can inspect a container's restart policy with docker inspect -f "{{ .HostConfig.RestartPolicy.Name }}" <container> and/or change it with docker update --restart=always <container>) but the containers still were not starting up until I ran a command like docker ps.
It turns out that the socket was enabled in systemd, but the service itself was disabled and so wouldn't start until a command was issued against it.
Inspecting via systemctl status docker.socket and systemctl status docker.service verified this:
root#poke:~# systemctl status docker.socket
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-30 18:28:38 EDT; 18h ago
Listen: /var/run/docker.sock (Stream)
Tasks: 0 (limit: 4647)
CGroup: /system.slice/docker.socket
root#poke:~# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2020-07-31 13:19:53 EDT; 5min ago
Docs: https://docs.docker.com
Main PID: 3094 (dockerd)
Tasks: 20
CGroup: /system.slice/docker.service
├─3094 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
└─3426 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6379 -container-ip 172.17.0.3 -container-
(Note the "disabled" for docker.service, even though it was running at the time.)
I was able to fix this by running systemctl enable --now docker.service:
root#poke:~# systemctl enable --now docker.service
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
Many thanks to this reddit user's reply for tipping me off.
In order to start container after reboot you need to add this property: --restart=always to your container start script. For example:
docker run -d -p 80:5000 --restart=always image_name
Running COS on GCE
Any ideas on how to get COS to do a graceful docker shutdown?
My innermost process is celery, which says he wants a SIGTERM to stop gracefully
http://docs.celeryproject.org/en/latest/userguide/workers.html#stopping-the-worker
My entrypoint is something like
exec celery -A some_app worker -c some_concurrency
On COS I am running my docker a service, something like
write_files:
- path: /etc/systemd/system/servicename.service
permissions: 0644
owner: root
content: |
[Unit]
Description=Some service
[Service]
Environment="HOME=/home/some_home"
RestartSec=10
Restart=always
ExecStartPre=/usr/share/google/dockercfg_update.sh
ExecStart=/usr/bin/docker run -u 2000 --name=somename --restart always some_image param_1 param_2
ExecStopPost=/usr/bin/docker stop servicename
KillMode=processes
KillSignal=SIGTERM
But ultimately when my COS instance it shut down, it just yanks the plug.
Do I need to add a shutdown script to do a docker stop? Do I need to do something more advanced?
What is the expected exit status of your container process when when it receives SIGTERM?
Running systemctl stop <service> then systemctl status -l <service> should show the exit code of the main process. Example:
Main PID: 21799 (code=exited, status=143)
One possibility is that the process does receive SIGTERM and shuts down gracefully, but returns non-zero exit code.
This would make the systemd believe that it didn't shutdown correctly. If that is the case, adding
SuccessExitStatus=143
to your systemd service should help. (Replace 143 with the actual exit code of your main process.)
Docker not picking up flannel subnet. Any help will be greatful. I am
using coreos as my container Linux and docker version is 1.12.6
my docker startup file look like below.
Flannel is working as expected
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=containerd.service docker.socket network.target
Requires=containerd.service docker.socket
[Service]
Type=notify
EnvironmentFile=-/run/flannel/flannel_docker_opts.env
# the default is not to use systemd for cgroups because the delegate issues
still
# exists and systemd currently does not support the cgroup feature set
required
# for containers run by docker
ExecStart=/usr/lib/coreos/dockerd --host=fd:// --
containerd=/var/run/docker/libcontainerd/docker-containerd.sock
$DOCKER_OPTS $DOCKER_CGROUPS $
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting
overhead
# in the kernel. We recommend using cgroups to do container-local
accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker
containers
Delegate=yes
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/docker.service.d/40-flannel.conf
[Unit]
Requires=flanneld.service
After=flanneld.service
[Service]
EnvironmentFile=/etc/kubernetes/cni/docker_opts_cni.env
# /etc/systemd/system/docker.service.d/40-storage.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --host=fd:// --graph="/abc/docker" $DOCKER_OPTS
$DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
# /etc/systemd/system/docker.service.d/50-insecure-registry.conf
[Service]
Environment=DOCKER_OPTS='--insecure-registry="10.x.x.x:5000"'
# /etc/systemd/system/docker.service.d/50-require-flannel.conf
[Unit]
Requires=flanneld.service
After=flanneld.service
Check if /run/flannel/flannel_docker_opts.env exists and its content.
My /run/flannel/subnet.env looks like:
FLANNEL_NETWORK=10.252.0.0/16
FLANNEL_SUBNET=10.252.127.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
Add --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} to dockerd args.
If this cannot work, rm -rf /var/lib/docker/overlay2/* and restart docker.service