Docker container doesn't start after reboot with enabling systemd script - docker

I have the following systemd script:
[Unit]
Description=Hub docker container
After=docker.service
[Service]
User=root
ExecStart=/home/hub/hub.sh
ExecStop=/bin/docker stop hub
ExecStopPost=/bin/docker rm hub
[Install]
WantedBy=multi-user.target
Running the command: systemctl start/stop hub works fine. I also created the symlink by using systemctl enable hub. Why doesn't my service start up after I reboot the entire laptop? I followed the docker guide so that Docker starts up on reboot, but for some reason my container doesn't start up. Am I missing a field in my script?
The command I am using my ExecStart, "/home/hub/hub.sh" script is:
docker run --net=host --restart=always --name hub -t hub
After reboot I get the following when I type systemctl status hub:
● hub.service - Hub docker container
Loaded: loaded (/etc/systemd/system/hub.service; enabled; vendor preset: disabled)
Active: inactive (dead)

In my case, I already had the containers set to restart=always (btw you can inspect a container's restart policy with docker inspect -f "{{ .HostConfig.RestartPolicy.Name }}" <container> and/or change it with docker update --restart=always <container>) but the containers still were not starting up until I ran a command like docker ps.
It turns out that the socket was enabled in systemd, but the service itself was disabled and so wouldn't start until a command was issued against it.
Inspecting via systemctl status docker.socket and systemctl status docker.service verified this:
root#poke:~# systemctl status docker.socket
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-30 18:28:38 EDT; 18h ago
Listen: /var/run/docker.sock (Stream)
Tasks: 0 (limit: 4647)
CGroup: /system.slice/docker.socket
root#poke:~# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2020-07-31 13:19:53 EDT; 5min ago
Docs: https://docs.docker.com
Main PID: 3094 (dockerd)
Tasks: 20
CGroup: /system.slice/docker.service
├─3094 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
└─3426 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6379 -container-ip 172.17.0.3 -container-
(Note the "disabled" for docker.service, even though it was running at the time.)
I was able to fix this by running systemctl enable --now docker.service:
root#poke:~# systemctl enable --now docker.service
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
Many thanks to this reddit user's reply for tipping me off.

In order to start container after reboot you need to add this property: --restart=always to your container start script. For example:
docker run -d -p 80:5000 --restart=always image_name

Related

Can nerdctl/crictl be used to list containers started by docker

I'm using version 20.10.21 of docker, in my understanding docker with this version uses containerd to manage image and container lifecycle, but why cannot I use crictl/nerdctl to list the containers which I started by docker cli?
What I've tried:
Check if docker uses containerd to manage contianers, ths is the result of systemctl status docker
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─http-proxy.conf
Active: active (running) since Sun 2022-12-04 22:44:27 CST; 1min 18s ago TriggeredBy: ● docker.socket
Docs: https://docs.docker.com Main PID: 1821 (dockerd)
Tasks: 91 (limit: 38297)
Memory: 229.6M
CPU: 1.214s
CGroup: /system.slice/docker.service
├─1821 /usr/bin/dockerd -H fd://
├─1845 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
I guess this means containerd is started by docker daemon. And the unix socket is located at /var/run/docker/containerd/containerd.sock
Try nerdctl to list containers but got error message:
$ nerdctl --address unix:///var/run/docker/containerd/containerd.sock ps
FATA[0000] rootless containerd not running? (hint: use `containerd-rootless-setuptool.sh install` to start rootless containerd): stat /run/user/1000/containerd-rootless: no such file or directory
Then I tried it again with sudo
sudo nerdctl --address unix:///var/run/docker/containerd/containerd.sock ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
As you can see, there's no container listed, but docker ps shows many containers I started.
Try crictl to check result, but got errors:
sudo crictl --r unix:///var/run/docker/containerd/containerd.sock ps
E1204 22:47:27.190569 3925 remote_runtime.go:557] "ListContainers with filter from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" filter="&ContainerFilter{Id:,State:&ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map[string]string{},}"
FATA[0000] listing containers: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
So my questions is: Why can't I get the same results of docker cli by nerdctl/crictl? Is there anything wrong I've done? or anything wrong in my understanding?
Thanks for any tips.

docker.socket: Failed with result 'service-start-limit-hit' after protecting docker daemon socket

I followed the steps provided in the documentation here to add tls security for docker api. Certificates are located in ~/.docker/ as well as /etc/docker/ssl/ folders. I added override.conf to /etc/systemd/system/docker.service.d/ with content
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem
Then, I used daemon-reload and docker start
$ systemctl daemon-reload
$ service docker start
The errors in journalctl -xe is:
-- Unit docker.socket has finished starting up.
--
-- The start-up result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Start request repeated too quickly.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 15 21:43:24 cynicalplyaground systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has failed.
--
-- The result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 15 21:45:01 cynicalplyaground CRON[12769]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session closed for user root
How can I sort this issue?
In the present case the same error occured after the latest manjaro update (2020-01-20).
Tried to change the systemd docker service, as adviced in other cases, but I reverted those changes and finally this was solved with:
a reboot of the system
(like advised here: https://www.reddit.com/r/archlinux/comments/7ya4ug/installing_docker_on_arch_linux/)
Getting to the root of the problem;
systemctl status docker.service
has this:
/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Trying to run that command, it complains about
unable to configure the Docker daemon with file /etc/docker/daemon.json: EOF
ls -l /etc/docker/daemon.json
-rw-r--r-- 1 root root 0 Jul 30 10:32 /etc/docker/daemon.json
NOTE that the JSON file is empty. Delete it.
For me it was because the docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables.
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
dockerd, should start fine after switching to iptables-legacy.
I have the same issue and just modify the "/usr/bin/dockerd" to "/usr/sbin/dockerd", then it works.
You can check the dockerd path first.
in my case... the host was part of a docker swarm...but the IPv6 was no longer reachable or automatically assigned to the host...
I manually add the old_IPv6
ip -6 address add 28xx:xxxx:x:x:xx:ebff:fe14:xxx dev ens3x
the journalctl -u docker.service mention:
level=fatal msg="Error starting cluster component: could not find local IP address: dial udp [2xxx:xxx:xxxx:xxx]:2377: connect: network is unreachable"
after add manually the IPv6 I was able to start docker so with docker running I leave the "swarm" and reboot
docker swarm leave --force
after reboot the docker services run as usual
For me it was missing disk space. Reboot also helped, but I was stillnot able to build any container.
After pruning some outdated stuff from the docker volumes I was able to continue.
I faced a similar issue on Ubuntu because I added the hosts option to /etc/docker/daemon.json file. That's ok, but for systems that use systemd it may cause conflict with the arguments passed to dockerd on start.
The solution was to delete the /etc/docker/daemon.json's hosts entry and set this config on file /etc/systemd/system/docker.service.d/options.conf.
$ cat /etc/systemd/system/docker.service.d/options.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://
After that, restart the service.
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
You may check that your changes has been applied by running docker info. Also, you may note on the docker service status that Drop-In field is using the options.conf created, and dockerd was executed with the specified host list.
$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset>
Drop-In: /etc/systemd/system/docker.service.d
└─options.conf
Active: active (running) since Fri 2022-11-18 01:02:18 EST; 1h 50min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 1111 (dockerd)
Tasks: 18
Memory: 58.5M
CPU: 1.294s
CGroup: /system.slice/docker.service
└─1111 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://
References:
Daemon configuration file
Control Docker with systemd
I had a similar issue on nixOS installed in a btrfs filesystem.
For me the solution was to add virtualisation.docker.storageDriver = "btrfs"; to my /etc/nixos/configuration.nix
Which according to the docker docs should equate to adding the following to /etc/docker/daemon.json in most other distros:
{
"storage-driver": "btrfs"
}
I was able to solve the problem by disabling the firewalld
systemctl disable firewalld
systemctl stop firewalld

Error building a ManageIQ container from git

I am trying to build a ManageIQ container from its source code
and I am seeing errors doing so..
I am using the following:
sudo docker build -t darga-container-28072016 .
and receiving the following error:
Step 11 : RUN curl -sSLko /etc/yum.repos.d/rhscl-rh-postgresql94-epel-7.repo https://copr-fe.cloud.fedoraproject.org/coprs/rhscl/rh-postgresql94/repo/epel-7/rhscl-rh-postgresql94-epel-7.repo && curl -sSLko /etc/yum.repos.d/ncarboni-pglogical-SCL-epel-7.repo https://copr.fedorainfracloud.org/coprs/ncarboni/pglogical-SCL/repo/epel-7/ncarboni-pglogical-SCL-epel-7.repo
---> Running in 9bca2fce10d7
Cannot start container 9bca2fce10d74375a70250a4ab4c6d3d38cdf510c943e90cab4235400f1b053a: [9] System error: exit status 1
What I am not 100% sure about is whether I should run it with sudo or without. I tried without and I got:
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Which is weird because I know that the service indeed runs:
$ sudo service docker status
Redirecting to /bin/systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2016-08-04 16:13:37 IDT; 12min ago
Docs: http://docs.docker.com
Main PID: 3460 (sh)
CGroup: /system.slice/docker.service
├─3460 /bin/sh -c /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgro...
├─3462 /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --selinux-enable...
└─3463 /usr/bin/forward-journald -tag docker
Any pointers to what should I do here?
Thanks!
You need to add your users to the docker group to be able to use docker from a user that is not root.
If you want to use the docker container with a socket under 1024 you will have to be in dockerroot too.
https://forums.docker.com/t/unable-to-use-docker-without-sudo/8982/2

Docker command can't connect docker daemon using Hypriot Docker

I installed Docker for RPi from Hypriot.
When I first run systemctl -l status docker I get this response:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled)
Active: inactive (dead) since Mon 2016-06-13 06:32:50 UTC; 18min ago
Docs: https://docs.docker.com
Process: 1327 ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS (code=exited, status=0/SUCCESS)
Main PID: 1327 (code=exited, status=0/SUCCESS)
After sudo service docker start and running systemctl -l status docker again, I get this response:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled)
Active: active (running) since Mon 2016-06-13 06:52:55 UTC; 6s ago
Docs: https://docs.docker.com
Main PID: 1434 (docker)
CGroup: /system.slice/docker.service
└─1434 /usr/bin/docker daemon -H fd:// --storage-driver=overlay -D
"Horray! It is working, I am running Docker on my RPi lets pull an image!!!"
docker pull resin/rpi-raspbian, but:
Using default tag: latest
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon. Is the docker daemon running on this host?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
What am I doing wrong?
Okey, so this might have solved my issue (see link below). Don't know about the versions on hypriot-docker though, but I will look it up :)
https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-(ARMv6)-in-three-steps
I used some faulty instructions on how to install using apt-get. The instructions in the link below do work, but I am not sure about the hypriot-docker versions though.
https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-(ARMv6)-in-three-steps
I had this issue I when I forgot to run my Docker commands with sudon.
The error is happening because you don't have permission to access the socket Docker is listening on unless you are root or are in the "docker" group.

Docker containers shut down after systemd start

For some reason when using systemd unit files my docker containers start but get shut down instantly. I have tried finding logs but can not see any indication on why this is happening. Is there someone that knows how to solve this / find the logs that show what is happening?
Note: When starting them manually after boot with docker start containername then it works (also when using systemctl start nginx)
After some more digging I found this error: could not find udev device: No such device it could have something to do with this?
Unit Service file:
[Unit]
Description=nginx-container
Requires=docker.service
After=docker.service
[Service]
Restart=always
RestartSec=2
StartLimitInterval=3600
StartLimitBurst=5
TimeoutStartSec=5
ExecStartPre=-/usr/bin/docker kill nginx
ExecStartPre=-/usr/bin/docker rm nginx
ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend: nginx
ExecStop=/usr/bin/docker stop -t 2 nginx
[Install]
WantedBy=multi-user.target
Journalctl output:
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="-job start(d757f83d4a13f876140ae008da943e8c5c3a0765c1fe5bc4a4e2599b70c30626) = OK (0)"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="POST /v1.18/containers/nginx/stop?t=2"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="+job stop(nginx)"
Docker logs: empty (docker logs nginx)
Systemctl output: (systemctl status nginx, nginx.service)
● nginx.service - nginx-container
Loaded: loaded (/etc/systemd/system/multi-user.target.wants/nginx.service)
Active: failed (Result: start-limit) since Thu 2015-05-28 11:18:20 UTC; 12min ago
Process: 3378 ExecStop=/usr/bin/docker stop -t 2 nginx (code=exited, status=0/SUCCESS)
Process: 3281 ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend:/nginx (code=exited, status=0/SUCCESS)
Process: 3258 ExecStartPre=/usr/bin/docker rm nginx (code=exited, status=0/SUCCESS)
Process: 3246 ExecStartPre=/usr/bin/docker kill nginx (code=exited, status=0/SUCCESS)
Main PID: 3281 (code=exited, status=0/SUCCESS)
May 28 11:18:20,frontend systemd[1]: nginx.service holdoff time over, scheduling restart.
May 28 11:18:20 frontend systemd[1]: start request repeated too quickly for nginx.service
May 28 11:18:20 frontend systemd[1]: Failed to start nginx-container.
May 28 11:18:20 frontend systemd[1]: Unit nginx.service entered failed state.
May 28 11:18:20 frontend systemd[1]: nginx.service failed.
Because you have not specified a Type in your systemd unit file, systemd is using the default, simple. From systemd.service:
If set to simple (the default if neither Type= nor BusName=, but
ExecStart= are specified), it is expected that the process
configured with ExecStart= is the main process of the service.
This means that if the process started by ExecStart exits, systemd
will assume your service has exited and will clean everything up.
Because you are running the docker client with -d, it exits
immediately...thus, systemd cleans up the service.
Typically, when starting containers with systemd, you would not use
the -d flag. This means that the client will continue running, and
will allow systemd to collect any output produced by your application.
That said, there are fundamental problems in starting Docker containers with systemd. Because of the way Docker operates, there really is no way for systemd to monitor the status of your container. All it can really do is track the status of the docker client, which is not the same thing (the client can exit/crash/etc without impacting your container). This isn't just relevant to systemd; any sort of process supervisor (upstart, runit, supervisor, etc) will have the same problem.

Resources