CoreOS Unit Failed on Launched - docker

I tried launching a service using chat.service unit file on a CoreOS and it failed:
// chat.service
[Unit]
Description=ChatApp
[Service]
ExecStartPre=-/usr/bin/docker kill simplechat1
ExecStartPre=-/usr/bin/docker rm simplechat1
ExecStartPre=-/usr/bin/docker pull jochasinga/socketio-chat
ExecStart=/usr/bin/docker run -p 3000:3000 --name simplechat1 jochasinga/socketio-chat
fleetctl list-units shows:
UNIT MACHINE ACTIVE SUB
chat.service cfe13a03.../<virtual-ip> failed failed
However, if I changed the chat.service to just:
// chat.service
[Service]
ExecStart=/usr/bin/docker run -p 3000:3000 <mydockerhubuser>/socketio-chat
It ran just fine. fleetctl list-units shows:
UNIT MACHINE ACTIVE SUB
chat.service 8df7b42d.../<virtual-ip> active running
EDIT
Using journalctl -u chat.service I got:
Jun 02 00:02:47 core-01 systemd[1]: Started chat.service.
Jun 02 00:02:47 core-01 systemd[1]: chat.service: Main process exited, code=exited, status=125/n/a
Jun 02 00:02:47 core-01 docker[8924]: docker: Error response from daemon: failed to create endpoint clever_tesla on network brid
Jun 02 00:02:47 core-01 systemd[1]: chat.service: Unit entered failed state.
Jun 02 00:02:47 core-01 systemd[1]: chat.service: Failed with result 'exit-code'.
Jun 02 00:02:58 core-01 systemd[1]: Stopped chat.service.
Jun 02 00:03:08 core-01 systemd[1]: Stopped chat.service.
What had I done wrong in the first chat.service unit file? Any guidance is appreciated.
Running Vagrant version of CoreOS (stable) on Mac OS X.

Your ExecStartPre= command doesn't seem to have a docker subcommand in it. Did you mean to use pull?
Reading the journal for the unit should get you more information: journactl -u chat.service

After looking into the journal using #Rob suggestion and some research, it appears that docker couldn't create an endpoint at the port 3000 because on the OS there was a running docker process on that port.
Simply stop the process with docker stop <processname> and re-launching with fleetctl start chat solved the problem.

Related

Docker service doesn't auto start after moving the docker image data directory to external drive location

Following this page, I have moved the docker data directory and created a symbolic link to it. It works. But everytime after rebooting my computer, the Docker service doesn't start automatically any more. How can I solve this problem?
journalctl -u docker.service returns:
Jun 30 10:29:55 ubuntu systemd[1]: Starting Docker Application Container Engine...
Jun 30 10:29:55 ubuntu dockerd[2358]: time="2022-06-30T10:29:55.426467188+10:00" level=info msg="S>
Jun 30 10:29:55 ubuntu dockerd[2358]: mkdir /var/lib/docker: file exists
Jun 30 10:29:55 ubuntu systemd[1]: docker.service: Main process exited, code=exited, status=1/FAIL>
Jun 30 10:29:55 ubuntu systemd[1]: docker.service: Failed with result 'exit-code'.
Jun 30 10:29:55 ubuntu systemd[1]: Failed to start Docker Application Container Engine.
Jun 30 10:29:57 ubuntu systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jun 30 10:29:57 ubuntu systemd[1]: Stopped Docker Application Container Engine.
Jun 30 10:29:57 ubuntu systemd[1]: docker.service: Start request repeated too quickly.
Jun 30 10:29:57 ubuntu systemd[1]: docker.service: Failed with result 'exit-code'.
Jun 30 10:29:57 ubuntu systemd[1]: Failed to start Docker Application Container Engine.
Before moving the data directory "/var/lib/docker", it was a directory used by Docker, now it is a symbolic link that points to the external directory where the docker image data is stored. Why there is a mkdir command?
If I run dockerd, it returns:
INFO[2022-06-30T20:53:05.143671302+10:00] Starting up
dockerd needs to be started with root privileges. To run dockerd in rootless mode as an unprivileged user, see https://docs.docker.com/go/rootless/
If I run sudo service docker start, docker can start without error. But I don't want to run this everyday. Docker used to start automatically. Any ideas?
I was able to reproduce the error message with the same configuration:
systemd[1]: Starting Docker Application Container Engine...
dockerd[47623]: time="2022-06-30T16:36:20.047741616Z" level=in..
dockerd[47623]: mkdir /data/docker: file exists
systemd[1]: docker.service: Main process exited, code=exited, ..
The reason was that my external drive wasn't mounted yet.
Adding systemd mount/automount units resolve the issue. Or you can add your external drive to your /etc/fstab (Add nofail for avoid the 90 seconds wait when you don't have it with you).
Also from Docker doc:
You can configure the Docker daemon to use a different directory, using the data-root configuration option.
So editing your /etc/docker/daemon.json with:
{
"data-root": "/data/docker"
}
is probably better than using symlinks.

Detect if docker should run or start a container on startup - using systemd but it keeps propagating errors - how to stop?

I'm trying to use a startup script on a Google Compute Engine instance to either:
If the docker container called rstudio is present but in stopped state, run docker start rstudio
If the docker container is not present, run rstudio run --name=rstudio rocker/rstudio
From this SO I thought this could be achieved via docker top rstudio || docker run --name=rstudio rocker/rstudio but it seems to always error at the docker top rstudio part. In that case, I have tried piping docker top rstudio &>/dev/null but no effect.
I have a cloud-config that runs when the instance boots up.
My problem is that the script to run or start the container keeps registering as an error, and doesn't go on to the logic of pulling the image. I have tried putting it in a seperate bash script and directly via ExecStart - also putting "-" in front of the ExecStart command (which is supposed to ignore errors?) but this also seems to have no effect. This is where I have ended up:
#cloud-config
users:
- name: gcer
uid: 2000
write_files:
- path: /home/gcer/docker-rstudio.sh
permissions: 0755
owner: root
content: |
#!/bin/bash
echo "Docker RStudio launch script"
if ! docker top rstudio &>/dev/null
then
echo "Pulling new rstudio"
docker run -p 80:8787 \
-e ROOT=TRUE \
-e USER=%s -e PASSWORD=%s \
-v /home/gcer:/home/rstudio \
--name=rstudio \
%s
else
echo "Starting existing rstudio"
docker start rstudio
fi
- path: /etc/systemd/system/rstudio.service
permissions: 0644
owner: root
content: |
[Unit]
Description=RStudio Server
Requires=docker.service
After=docker.service
[Service]
Restart=always
Environment="HOME=/home/gcer"
ExecStartPre=/usr/share/google/dockercfg_update.sh
ExecStart=-/home/gcer/docker-rstudio.sh
ExecStop=/usr/bin/docker stop rstudio
runcmd:
- systemctl daemon-reload
- systemctl start rstudio.service
Whatever I try, I end up with this error log when I run sudo journalctl -u rstudio.service
Feb 14 23:26:09 test-9 systemd[1]: Started RStudio Server.
Feb 14 23:26:09 test-9 docker[770]: Error response from daemon: No such container: rstudio
Feb 14 23:26:09 test-9 systemd[1]: rstudio.service: Control process exited, code=exited status=1
Feb 14 23:26:09 test-9 systemd[1]: rstudio.service: Unit entered failed state.
Feb 14 23:26:09 test-9 systemd[1]: rstudio.service: Failed with result 'exit-code'.
Feb 14 23:26:09 test-9 systemd[1]: rstudio.service: Service hold-off time over, scheduling restart.
Feb 14 23:26:09 test-9 systemd[1]: Stopped RStudio Server.
Feb 14 23:26:09 test-9 systemd[1]: Starting RStudio Server...
...
Feb 14 23:26:09 test-9 systemd[1]: Started RStudio Server.
Feb 14 23:26:09 test-9 docker[809]: Error response from daemon: No such container: rstudio
Feb 14 23:26:09 test-9 systemd[1]: rstudio.service: Control process exited, code=exited status=1
Feb 14 23:26:09 test-9 systemd[1]: rstudio.service: Unit entered failed state.
Feb 14 23:26:09 test-9 systemd[1]: rstudio.service: Failed with result 'exit-code'.
Feb 14 23:26:10 test-9 systemd[1]: rstudio.service: Service hold-off time over, scheduling restart.
Feb 14 23:26:10 test-9 systemd[1]: Stopped RStudio Server.
Feb 14 23:26:10 test-9 systemd[1]: rstudio.service: Start request repeated too quickly.
Feb 14 23:26:10 test-9 systemd[1]: Failed to start RStudio Server.
Feb 14 23:26:10 test-9 systemd[1]: rstudio.service: Unit entered failed state.
Feb 14 23:26:10 test-9 systemd[1]: rstudio.service: Failed with result 'exit-code'.
Can anyone help me get this working?
I would delete the container when you stop it. Then your startup script reduces to making extra sure the container is deleted, and unconditionally docker running it anew.
This would make the entire contents of the script be:
#!/bin/sh
docker stop rstudio
docker rm rstudio
docker run -p 80:8787 \
--name=rstudio \
... \
rstudio run --name=rstudio rocker/rstudio
Without the set -e option, even if the earlier commands fail (because the container doesn't exist) the script will still go on to the docker run command. This avoids any testing of trying to figure out whether a container is there or not and always leaves you in a consistent state.
Similarly, to clean up a little better, I'd change the systemd unit file to delete the container after it stops
ExecStop=/usr/bin/docker stop rstudio
ExecStopPost=/usr/bin/docker rm rstudio
(Your setup has three possible states: the container is running; the container exists but is stopped; and the container doesn't exist. My setup removes the "exists but is stopped" state, which doesn't have a whole lot of value, especially since you use a docker run -v option to store data outside of container space.)

Failed to start Docker Application Container Engine

I am new to Docker, so don't have much idea about it.
I tried restarting Docker service using command
service docker restart
As the command was taking too much of time I did a CTL+C
Now I am not able to start docker deamon
Any docker command gives following op
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
I tried starting Docker deamon using
systemctl start docker
But it outputs:
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
Output of command
**systemctl status docker.service**
`● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─docker.conf, http-proxy.conf, https-proxy.conf
Active: failed (Result: exit-code) since Mon 2018-03-05 17:17:54 IST; 2min 23s ago
Docs: https://docs.docker.com
Process: 11331 ExecStart=/usr/bin/dockerd --graph=/app/dockerRT (code=exited, status=1/FAILURE)
Main PID: 11331 (code=exited, status=1/FAILURE)
Memory: 76.9M
CGroup: /system.slice/docker.service
└─4593 docker-containerd-shim 3bda33eac892d14adda9f3b1fc8dc52173e26ce60ca949075227d903399c7517 /var/run/docker/libcontainerd/3bda33eac892d14adda9f3b1fc8dc52173e26c...
Mar 05 17:17:05 hj-fsbfsd9761.persistent.co.in systemd[1]: Starting Docker Application Container Engine...
Mar 05 17:17:05 hj-fsbfsd9761.persistent.co.in dockerd[11331]: time="2018-03-05T17:17:05.126009059+05:30" level=info msg="libcontainerd: new containerd process, pid: 11337"
Mar 05 17:17:06 hj-fsbfsd9761.persistent.co.in dockerd[11331]: time="2018-03-05T17:17:06.346599571+05:30" level=warning msg="devmapper: Usage of loopback devices is ...section."
Mar 05 17:17:10 hj-fsbfsd9761.persistent.co.in dockerd[11331]: time="2018-03-05T17:17:10.889378989+05:30" level=warning msg="devmapper: Base device already exists an...ignored."
Mar 05 17:17:10 hj-fsbfsd9761.persistent.co.in dockerd[11331]: time="2018-03-05T17:17:10.976695025+05:30" level=info msg="[graphdriver] using prior storage driver \"...mapper\""
Mar 05 17:17:54 hj-fsbfsd9761.persistent.co.in dockerd[11331]: time="2018-03-05T17:17:54.312812069+05:30" level=fatal msg="Error starting daemon: timeout"
Mar 05 17:17:54 hj-fsbfsd9761.persistent.co.in systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 05 17:17:54 hj-fsbfsd9761.persistent.co.in systemd[1]: **Failed to start Docker Application Container Engine.**
Mar 05 17:17:54 hj-fsbfsd9761.persistent.co.in systemd[1]: Unit docker.service entered failed state.
Mar 05 17:17:54 hj-fsbfsd9761.persistent.co.in systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
journalctl -xe
loop: Write error at byte offset 63585648640, length 4096.
How would I be able to start Docker without losing any containers and using previous configurations?
I had the same issue (Fedora 30 x86_64, kernel 5.2.9) and it turned out that being connected to a VPN was the problem. Apparently having a changed gateway address causes an "Error initializing network controller" error which I was able to see when I tried starting docker via sudo dockerd instead of sudo systemctl start docker.
I found the note here about the VPN being a possible problem, disconnecting immediately allowed me to start docker with systemctl start docker.
"Failed to start Docker Application Container Engine" is a general error message.
You should inspect journal for more details:
journalctl -eu docker
In my case it was: "error initializing graphdriver: /var/lib/docker contains several valid graphdrivers: devicemapper, overlay2"
Changing graphdriver to overlay2, fixed it:
$ sudo systemctl stop docker
$ vi /etc/docker/daemon.json # Create the file if it does not exist, and add:
{
"storage-driver": "overlay2"
}
$ sudo systemctl start docker
$ systemctl status docker.service # Hopefully it's running now
I removed the file /etc/docker/daemon.json and started it with sudo systemctl start docker and it worked!!
For the googlers:
For me what worked was doing a killall dockerd and a sudo rm /var/run/docker.pid
I got the error message that recommended that from running dockerd
I ran into this too but all I did after receiving the error was sudo systemctl start docker and then ran sudo systemctl status docker again and that took care of it while on a vpn.
You may have the issue with the current docker installation.
If you don't have much done already you may want to try to reinstall using the install script provided by Docker: https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-using-the-convenience-script this will help you to investigate the errors.
I faced the same problem, in my case the disk was full. I removed docker volumes from /var/lib/docker/volumes/ and started docker with sudo systemctl start docker.

Warning: Stopping docker.service, but it can still be activated by: docker.socket

I've reinstalled Docker. When I'm trying to start Docker, everything is fine:
# /etc/init.d/docker start
[ ok ] Starting docker (via systemctl): docker.service.
until I want to stop Docker service and many times restart it:
# /etc/init.d/docker stop
[....] Stopping docker (via systemctl): docker.serviceWarning: Stopping docker.service, but it can still be activated by:
docker.socket
. ok
Finally, I've got error:
# /etc/init.d/docker start
[....] Starting docker (via systemctl): docker.serviceJob for docker.service failed.
See "systemctl status docker.service" and "journalctl -xe" for details.
failed!
# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Sat 2017-11-25 20:04:20 CET; 2min 4s ago
Docs: https://docs.docker.com
Process: 12845 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=0/SUCCESS)
Main PID: 12845 (code=exited, status=0/SUCCESS)
CPU: 326ms
Nov 25 20:04:18 example.com systemd[1]: Started Docker Application Container Engine.
Nov 25 20:04:18 example.com dockerd[12845]: time="2017-11-25T20:04:18.191949863+01:00" level=inf
Nov 25 20:04:19 example.com systemd[1]: Stopping Docker Application Container Engine...
Nov 25 20:04:19 example.com dockerd[12845]: time="2017-11-25T20:04:19.368990531+01:00" level=inf
Nov 25 20:04:19 example.com dockerd[12845]: time="2017-11-25T20:04:19.37953454+01:00" level=info
Nov 25 20:04:20 example.com systemd[1]: Stopped Docker Application Container Engine.
Nov 25 20:04:21 example.com systemd[1]: docker.service: Start request repeated too quickly.
Nov 25 20:04:21 example.com systemd[1]: Failed to start Docker Application Container Engine.
Nov 25 20:04:21 example.com systemd[1]: docker.service: Unit entered failed state.
Nov 25 20:04:21 example.com systemd[1]: docker.service: Failed with result 'start-limit-hit'.
I've installed Docker on Debian 9 Stretch.
Can anyone help me get rid of this warning and resolve an error "Failed with result 'start-limit-hit'"?
Simply start and stop the socket if the docker is triggered by the socket
sudo systemctl stop docker.socket
This is because in addition to the docker.service unit file, there is a docker.socket unit file... this is for socket activation. The warning means if you try to connect to the docker socket while the docker service is not running, then systemd will automatically start docker for you.
You can get rid of this by removing /lib/systemd/system/docker.socket... you may also need to remove -H fd:// from the docker.service unit file.

Can't start docker after reboot Ubuntu 16.05

I'm trying run docker in Ubuntu 16.04 after system reboot . I created service for it "/etc/systemd/system/openvpnBOX.service":
[Unit]
Description=Openvpn Docker
[Service]
User=root
ExecStart=/etc/init/openvpn.conf
[Install]
WantedBy=multi-user.target
Alias=openvpnBOX.service
openvpn.conf:
#!/bin/bash
exec docker run --volumes-from ovpn-data --rm -p 1194:1194/udp --cap- add=NET_ADMIN kylemanna/openvpn
When i'm running this service "sudo service openvpnBOX start i see that service is run, but when i'm rebooting my system, after reboot i see that service can't start:
"sudo service openvpnBOX status"
● openvpnBOX.service - Openvpn Docker
Loaded: loaded (/etc/systemd/system/openvpnBOX.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2017-10-01 21:35:48 SST; 2min 51s ago
Process: 1771 ExecStart=/etc/init/openvpn.conf (code=exited, status=1/FAILURE)
Main PID: 1771 (code=exited, status=1/FAILURE)
Oct 01 21:35:48 systemd[1]: openvpnBOX.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 21:35:48 systemd[1]: openvpnBOX.service: Unit entered failed state.
Oct 01 21:35:48 systemd[1]: openvpnBOX.service: Failed with result 'exit-code'.
Oct 01 21:35:48 systemd[1]: Started Openvpn Docker.
Oct 01 21:35:48 openvpn.conf[1771]: Error response from daemon: 404 page not found
Oct 01 21:35:48 systemd[1]: openvpnBOX.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 21:35:48 systemd[1]: openvpnBOX.service: Unit entered failed state.
Oct 01 21:35:48 systemd[1]: openvpnBOX.service: Failed with result 'exit-code'.
Oct 01 21:35:48 systemd[1]: openvpnBOX.service: Start request repeated too quickly.
Oct 01 21:35:48 systemd[1]: Failed to start Openvpn Docker.
I can use "sudo docker run --restart=always --volumes-from ovpn-data -p 1194:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn" but it doesn't solve my problem, because i woud like understand why my service doesn't work after reboot.
Any idea?

Resources