Although Docker is running, daemon cannot be connected - docker

System: Ubuntu 20.02:
Whe I run systemctl status docker, I got:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-10-20 14:52:17 JST; 26s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 811329 (dockerd)
Tasks: 40
Memory: 26.7M
CGroup: /system.slice/docker.service
└─811329 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
but when I run docker container list, I got:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
What is the matter?

Related

Can nerdctl/crictl be used to list containers started by docker

I'm using version 20.10.21 of docker, in my understanding docker with this version uses containerd to manage image and container lifecycle, but why cannot I use crictl/nerdctl to list the containers which I started by docker cli?
What I've tried:
Check if docker uses containerd to manage contianers, ths is the result of systemctl status docker
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─http-proxy.conf
Active: active (running) since Sun 2022-12-04 22:44:27 CST; 1min 18s ago TriggeredBy: ● docker.socket
Docs: https://docs.docker.com Main PID: 1821 (dockerd)
Tasks: 91 (limit: 38297)
Memory: 229.6M
CPU: 1.214s
CGroup: /system.slice/docker.service
├─1821 /usr/bin/dockerd -H fd://
├─1845 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
I guess this means containerd is started by docker daemon. And the unix socket is located at /var/run/docker/containerd/containerd.sock
Try nerdctl to list containers but got error message:
$ nerdctl --address unix:///var/run/docker/containerd/containerd.sock ps
FATA[0000] rootless containerd not running? (hint: use `containerd-rootless-setuptool.sh install` to start rootless containerd): stat /run/user/1000/containerd-rootless: no such file or directory
Then I tried it again with sudo
sudo nerdctl --address unix:///var/run/docker/containerd/containerd.sock ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
As you can see, there's no container listed, but docker ps shows many containers I started.
Try crictl to check result, but got errors:
sudo crictl --r unix:///var/run/docker/containerd/containerd.sock ps
E1204 22:47:27.190569 3925 remote_runtime.go:557] "ListContainers with filter from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" filter="&ContainerFilter{Id:,State:&ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map[string]string{},}"
FATA[0000] listing containers: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
So my questions is: Why can't I get the same results of docker cli by nerdctl/crictl? Is there anything wrong I've done? or anything wrong in my understanding?
Thanks for any tips.

Docker won't startup

If I try to start docker server like so:
systemctl start docker
I get:
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
Running:
systemctl status docker.service
Shows:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor p>
Active: failed (Result: exit-code) since Thu 2020-06-04 03:25:10 EDT; 36s >
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Process: 84866 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/F>
Main PID: 84866 (code=exited, status=1/FAILURE)
lines 1-7/7 (END)...skipping...
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2020-06-04 03:25:10 EDT; 36s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Process: 84866 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
Main PID: 84866 (code=exited, status=1/FAILURE)
I'm on Solus linux and installed docker normally like sudo eopkg install docker. How should I go about trying to get it to work ?
This seems to have been a Solus linux specific issue due to an update. I fixed it by running sudo rm -rf /var/lib/docker/overlay as suggested here (the issue is tracked here).

Does dockerd support WatchdogSec sd_notify health checks?

We've been having issues where the Docker daemon will occasionally stop responding on one of our Kubernetes systems, but Systemd still thinks the service is running:
systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-04-15 20:40:57 UTC; 3 months 22 days ago
Docs: https://docs.docker.com
Main PID: 1281 (dockerd)
Tasks: 1409
Memory: 31.0G
CPU: 5d 17h 3min 4.758s
CGroup: /system.slice/docker.service
├─ 1281 /usr/bin/dockerd -H fd://
...
There isn't anything in the journalctl -u docker or syslog files to indicate what the issue is, but the Docker daemon no longer responds to requests (docker ps just hangs). We are currently using the 17.03.2~ce-0~ubuntu-xenial package for Ubuntu 16.04, which has the following service unit:
cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket firewalld.service
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I noticed that even though it is a Type=notify service, there isn't a WatchdogSec= defined in the service unit.
Does the Docker daemon support setting a watchdog timeout for sd_notify based health checks?
No, currently the components/engine/cmd/dockerd/daemon_linux.go file only implements systemdDaemon.SdNotifyReady to notify Systemd when the process has started. For watchdog support it would have to use something like SdWatchdogEnabled to continually send SdNotifyWatchdog = "WATCHDOG=1" notifications.
If you try and set WatchdogSec=60s on the docker.service file it will kill and restart the service because the daemon doesn't send the required notifications.
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-08-08 02:09:52 UTC; 50s ago
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: deactivating (stop-sigabrt) (Result: watchdog) since Thu 2019-08-08 02:10:02 UTC; 45ms ago
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: activating (start) since Thu 2019-08-08 02:10:04 UTC; 777ms ago
# Log entries:
Aug 08 02:09:14 kam1 systemd[1]: Starting Docker Application Container Engine...
Aug 08 02:09:15 kam1 systemd[1]: Started Docker Application Container Engine.
Aug 08 02:10:15 kam1 systemd[1]: docker.service: Watchdog timeout (limit 60s)!
Aug 08 02:10:15 kam1 systemd[1]: docker.service: Killing process 12383 (dockerd) with signal SIGABRT.
Aug 08 02:10:16 kam1 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 08 02:10:16 kam1 systemd[1]: docker.service: Failed with result 'watchdog'.
Aug 08 02:10:18 kam1 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 08 02:10:18 kam1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 08 02:10:18 kam1 systemd[1]: Stopped Docker Application Container Engine.
Aug 08 02:10:18 kam1 systemd[1]: Starting Docker Application Container Engine...

Docker container doesn't start after reboot with enabling systemd script

I have the following systemd script:
[Unit]
Description=Hub docker container
After=docker.service
[Service]
User=root
ExecStart=/home/hub/hub.sh
ExecStop=/bin/docker stop hub
ExecStopPost=/bin/docker rm hub
[Install]
WantedBy=multi-user.target
Running the command: systemctl start/stop hub works fine. I also created the symlink by using systemctl enable hub. Why doesn't my service start up after I reboot the entire laptop? I followed the docker guide so that Docker starts up on reboot, but for some reason my container doesn't start up. Am I missing a field in my script?
The command I am using my ExecStart, "/home/hub/hub.sh" script is:
docker run --net=host --restart=always --name hub -t hub
After reboot I get the following when I type systemctl status hub:
● hub.service - Hub docker container
Loaded: loaded (/etc/systemd/system/hub.service; enabled; vendor preset: disabled)
Active: inactive (dead)
In my case, I already had the containers set to restart=always (btw you can inspect a container's restart policy with docker inspect -f "{{ .HostConfig.RestartPolicy.Name }}" <container> and/or change it with docker update --restart=always <container>) but the containers still were not starting up until I ran a command like docker ps.
It turns out that the socket was enabled in systemd, but the service itself was disabled and so wouldn't start until a command was issued against it.
Inspecting via systemctl status docker.socket and systemctl status docker.service verified this:
root#poke:~# systemctl status docker.socket
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-30 18:28:38 EDT; 18h ago
Listen: /var/run/docker.sock (Stream)
Tasks: 0 (limit: 4647)
CGroup: /system.slice/docker.socket
root#poke:~# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2020-07-31 13:19:53 EDT; 5min ago
Docs: https://docs.docker.com
Main PID: 3094 (dockerd)
Tasks: 20
CGroup: /system.slice/docker.service
├─3094 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
└─3426 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6379 -container-ip 172.17.0.3 -container-
(Note the "disabled" for docker.service, even though it was running at the time.)
I was able to fix this by running systemctl enable --now docker.service:
root#poke:~# systemctl enable --now docker.service
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
Many thanks to this reddit user's reply for tipping me off.
In order to start container after reboot you need to add this property: --restart=always to your container start script. For example:
docker run -d -p 80:5000 --restart=always image_name

Docker command can't connect docker daemon using Hypriot Docker

I installed Docker for RPi from Hypriot.
When I first run systemctl -l status docker I get this response:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled)
Active: inactive (dead) since Mon 2016-06-13 06:32:50 UTC; 18min ago
Docs: https://docs.docker.com
Process: 1327 ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS (code=exited, status=0/SUCCESS)
Main PID: 1327 (code=exited, status=0/SUCCESS)
After sudo service docker start and running systemctl -l status docker again, I get this response:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled)
Active: active (running) since Mon 2016-06-13 06:52:55 UTC; 6s ago
Docs: https://docs.docker.com
Main PID: 1434 (docker)
CGroup: /system.slice/docker.service
└─1434 /usr/bin/docker daemon -H fd:// --storage-driver=overlay -D
"Horray! It is working, I am running Docker on my RPi lets pull an image!!!"
docker pull resin/rpi-raspbian, but:
Using default tag: latest
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon. Is the docker daemon running on this host?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
What am I doing wrong?
Okey, so this might have solved my issue (see link below). Don't know about the versions on hypriot-docker though, but I will look it up :)
https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-(ARMv6)-in-three-steps
I used some faulty instructions on how to install using apt-get. The instructions in the link below do work, but I am not sure about the hypriot-docker versions though.
https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-(ARMv6)-in-three-steps
I had this issue I when I forgot to run my Docker commands with sudon.
The error is happening because you don't have permission to access the socket Docker is listening on unless you are root or are in the "docker" group.

Resources