Docker containers terminate on shell logout - docker

I'm running a debian backed docker container created with a docker-compose up -d detached command. It starts without problems and serves requests without issues, but it requires I keep logged-in with an active shell in order to keep containers running.
My docker-compose.yml file is as follows:
services:
db:
image: postgres:alpine
restart: always
ports:
- '5432:5432'
volumes:
- /data/docker/postgresql/data:/var/lib/postgresql/data:Z
pgadmin:
image: dpage/pgadmin4:latest
restart: always
ports:
- "5050:80"
volumes:
- /data/docker/postgresql/pgadmin-data:/var/lib/pgadmin:Z
Everytime I restart the server it doesn't appears to start up until I log in.
The status of docker service, as per systemctl status docker.service is:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-08-09 22:37:46 -03; 21min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 4359 (dockerd)
Tasks: 11
CPU: 1.396s
CGroup: /system.slice/docker.service
└─4359 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Even when all looks good, the moment I logout the shell session, the containers stop running.

The problem I was having can be searched for:
Rootless containers exit once the user session exits
I finally could fix it after realizing i was using the rootless configuration for docker, and both containers where started using a normal user.
Linux stops processes started by a normal user if loginctl is configured to not use lingering, to prevent normal users to keep long-running processes executing in the system.
I've found this link that helped: 17) rootless containers exit once the user session exits
You need to set lingering mode through loginctl to prevent user
processes to be killed once the user session completed.
Symptom
Once the user logs out all the containers exit.
Solution
You'll need to either:
# loginctl enable-linger $UID

Related

rootless docker - containers do not start after a power cut but starts again when host reboot

I have debian 10.5 host with docker running in rootless mode (followed this guide : https://docs.docker.com/engine/security/rootless/)
When there is a power cut (I don't have a UPS), my debian 10.5 VM starts automatically when power is restored.
Everything works fine except my docker daemon. There's no error with the service:
systemctl --user status docker
● docker.service - Docker Application Container Engine (Rootless)
Loaded: loaded (/home/dockerprod/.config/systemd/user/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-01-08 12:04:40 +04; 4min 43s ago
Docs: https://docs.docker.com
Main PID: 770 (rootlesskit)
CGroup: /user.slice/user-1001.slice/user#1001.service/docker.service
├─770 rootlesskit --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run /
├─805 /proc/self/exe --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/ru
├─816 vpnkit --ethernet /tmp/rootlesskit308973386/vpnkit-ethernet.sock --mtu 1500 --host-ip 0.0.0.0
├─896 dockerd --experimental --storage-driver=vfs
└─936 containerd --config /run/user/1001/docker/containerd/containerd.toml --log-level info
But the containers did not start for some reason.
I am not sure what logs to look at.
sudo journalctl -u docker.service
returns nothing
If I restart the host, the containers start as normal. So I always need to restart the host after a power cut which is not ideal when I am not at home.
Any idea what to look at?
Maybe a clue; my docker lib folder (where containers and images are stored) is on another HDD mounted automatically in /etc/fstab
Maybe after a power cut upon reboot, docker daemon is started before the HDD on which the docker lib folder is mounted? Does not know if this makes sense.
edit:
I moved the mounting command in /etc/fstab of HDD on which the docker lib is on to the top.
Does not solve the issue.
Another note, /lib/docker/containers/ is empty after the power cut
If I restart the host, /lib/docker/containers/ contains the containers again..

Cannot use docker-compose with overlay network

I'm pretty baffled what's going on here, but I've narrowed it down to a very small test case. Here's my docker-compose file:
version: "3.7"
networks:
cl_net_overlay:
driver: overlay
services:
redis:
image: "redis:alpine"
networks:
- cl_net_overlay
The cl_net_overlay network doesn't exist. When I run this with:
docker-compose up
It stalls for a little while, then says:
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Creating network "tmp_cl_net_overlay" with driver "overlay"
Recreating tmp_redis_1 ... error
ERROR: for tmp_redis_1 Cannot start service redis: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded
ERROR: for redis Cannot start service redis: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded
ERROR: Encountered errors while bringing up the project.
This file was working fine for me on my previous laptop. My docker and docker-compose should be up to date since this is a brand new laptop. Is there some piece of the puzzle I'm missing?
05:01:11::mlissner#gabbro::/tmp
↪ docker --version
Docker version 19.03.1, build 74b1e89
05:01:57::mlissner#gabbro::/tmp
↪ docker-compose --version
docker-compose version 1.24.1, build 4667896b
Any ideas what's going on here? I've been trying to get it to work all day and I'm feeling a little like I'm losing my mind.
Small follow up. The message says:
make sure your network options are correct and check manager logs
I have no idea how to check the manager logs. That might be a useful first step?
Another follow up, per comments. If I try to deploy this I get no logs and it's unable to start up:
05:44:32::mlissner#gabbro::~/Programming/courtlistener/docker/courtlistener
↪ docker stack deploy --compose-file /tmp/docker-compose.yml test2
Creating network test2_cl_net_overlay2
Creating service test2_redis
05:44:50::mlissner#gabbro::~/Programming/courtlistener/docker/courtlistener
↪ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
5y7o01o5mifn test2_redis replicated 0/1 redis:alpine
05:44:57::mlissner#gabbro::~/Programming/courtlistener/docker/courtlistener
↪ docker service ps 5y
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
0kbph0ie8qth test2_redis.1 redis:alpine gabbro Ready Rejected 4 seconds ago "mkdir /var/lib/docker: read-o…"
inr81c3r4un7 \_ test2_redis.1 redis:alpine gabbro Shutdown Rejected 9 seconds ago "mkdir /var/lib/docker: read-o…"
tl1h6dp90ur2 \_ test2_redis.1 redis:alpine gabbro Shutdown Rejected 14 seconds ago "mkdir /var/lib/docker: read-o…"
jacv2yvkspix \_ test2_redis.1 redis:alpine gabbro Shutdown Rejected 19 seconds ago "mkdir /var/lib/docker: read-o…"
7cm6e8snf517 \_ test2_redis.1 redis:alpine gabbro Shutdown Rejected 19 seconds ago "mkdir /var/lib/docker: read-o…"
Another idea: Running as root. Same issue.
Do you have the right plugins (see more bellow on the docker info command)?
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
It works on:
$ docker swarm init
$ docker-compose up
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Creating network "stackoverflow-57701373_cl_net_overlay" with driver "overlay"
Pulling redis (redis:alpine)...
alpine: Pulling from library/redis
9d48c3bd43c5: Pull complete
(...)
redis_1 | 1:M 29 Aug 2019 01:27:31.969 * Ready to accept connection
When:
$ docker --version
Docker version 19.03.1-ce, build 74b1e89e8a
and info:
$ docker info
Client:
Debug Mode: false
Server:
(...)
Server Version: 19.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: ff5mogx0ph4pgmwm2zrbhmjb4
Is Manager: true
ClusterID: vloixv7g75jflw5i1k81neul1
Managers: 1
Nodes: 1
(...)

Error building a ManageIQ container from git

I am trying to build a ManageIQ container from its source code
and I am seeing errors doing so..
I am using the following:
sudo docker build -t darga-container-28072016 .
and receiving the following error:
Step 11 : RUN curl -sSLko /etc/yum.repos.d/rhscl-rh-postgresql94-epel-7.repo https://copr-fe.cloud.fedoraproject.org/coprs/rhscl/rh-postgresql94/repo/epel-7/rhscl-rh-postgresql94-epel-7.repo && curl -sSLko /etc/yum.repos.d/ncarboni-pglogical-SCL-epel-7.repo https://copr.fedorainfracloud.org/coprs/ncarboni/pglogical-SCL/repo/epel-7/ncarboni-pglogical-SCL-epel-7.repo
---> Running in 9bca2fce10d7
Cannot start container 9bca2fce10d74375a70250a4ab4c6d3d38cdf510c943e90cab4235400f1b053a: [9] System error: exit status 1
What I am not 100% sure about is whether I should run it with sudo or without. I tried without and I got:
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Which is weird because I know that the service indeed runs:
$ sudo service docker status
Redirecting to /bin/systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2016-08-04 16:13:37 IDT; 12min ago
Docs: http://docs.docker.com
Main PID: 3460 (sh)
CGroup: /system.slice/docker.service
├─3460 /bin/sh -c /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgro...
├─3462 /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --selinux-enable...
└─3463 /usr/bin/forward-journald -tag docker
Any pointers to what should I do here?
Thanks!
You need to add your users to the docker group to be able to use docker from a user that is not root.
If you want to use the docker container with a socket under 1024 you will have to be in dockerroot too.
https://forums.docker.com/t/unable-to-use-docker-without-sudo/8982/2

Docker command can't connect docker daemon using Hypriot Docker

I installed Docker for RPi from Hypriot.
When I first run systemctl -l status docker I get this response:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled)
Active: inactive (dead) since Mon 2016-06-13 06:32:50 UTC; 18min ago
Docs: https://docs.docker.com
Process: 1327 ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS (code=exited, status=0/SUCCESS)
Main PID: 1327 (code=exited, status=0/SUCCESS)
After sudo service docker start and running systemctl -l status docker again, I get this response:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled)
Active: active (running) since Mon 2016-06-13 06:52:55 UTC; 6s ago
Docs: https://docs.docker.com
Main PID: 1434 (docker)
CGroup: /system.slice/docker.service
└─1434 /usr/bin/docker daemon -H fd:// --storage-driver=overlay -D
"Horray! It is working, I am running Docker on my RPi lets pull an image!!!"
docker pull resin/rpi-raspbian, but:
Using default tag: latest
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon. Is the docker daemon running on this host?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
What am I doing wrong?
Okey, so this might have solved my issue (see link below). Don't know about the versions on hypriot-docker though, but I will look it up :)
https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-(ARMv6)-in-three-steps
I used some faulty instructions on how to install using apt-get. The instructions in the link below do work, but I am not sure about the hypriot-docker versions though.
https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-(ARMv6)-in-three-steps
I had this issue I when I forgot to run my Docker commands with sudon.
The error is happening because you don't have permission to access the socket Docker is listening on unless you are root or are in the "docker" group.

CoreOS Fleet could not get container

I have 3 containers running on 3 machines. One is called graphite, one is called back and one is called front. The front container needs both the others to run, so i link them separately like this:
[Unit]
Description=front hystrix
[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill front
ExecStartPre=-/usr/bin/docker rm -v front
ExecStartPre=/usr/bin/docker pull blurio/hystrixfront
ExecStart=/usr/bin/docker run --name front --link graphite:graphite --link back:back -p 8080:8080 blurio/hystrixfront
ExecStop=/usr/bin/docker stop front
I start both the other containers, wait till they're up and running, then start this one with fleetctl and it just instantly fails with this message:
fleetctl status front.service
? front.service - front hystrix
Loaded: loaded (/run/fleet/units/front.service; linked-runtime; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2015-05-12 13:46:08 UTC; 24s ago
Process: 922 ExecStop=/usr/bin/docker stop front (code=exited, status=0/SUCCESS)
Process: 912 ExecStart=/usr/bin/docker run --name front --link graphite:graphite --link back:back -p 8080:8080 blurio/hystrixfront (code=exited, status=1/FAILURE)
Process: 902 ExecStartPre=/usr/bin/docker pull blurio/hystrixfront (code=exited, status=0/SUCCESS)
Process: 892 ExecStartPre=/usr/bin/docker rm -v front (code=exited, status=1/FAILURE)
Process: 885 ExecStartPre=/usr/bin/docker kill front (code=exited, status=1/FAILURE)
Main PID: 912 (code=exited, status=1/FAILURE)
May 12 13:46:08 core-04 docker[902]: 8b9853c10955: Download complete
May 12 13:46:08 core-04 docker[902]: 0dc7a355f916: Download complete
May 12 13:46:08 core-04 docker[902]: 0dc7a355f916: Download complete
May 12 13:46:08 core-04 docker[902]: Status: Image is up to date for blurio/hystrixfront:latest
May 12 13:46:08 core-04 systemd[1]: Started front hystrix.
May 12 13:46:08 core-04 docker[912]: time="2015-05-12T13:46:08Z" level="fatal" msg="Error response from daemon: Could not get container for graphite"
May 12 13:46:08 core-04 systemd[1]: front.service: main process exited, code=exited, status=1/FAILURE
May 12 13:46:08 core-04 docker[922]: front
May 12 13:46:08 core-04 systemd[1]: Unit front.service entered failed state.
May 12 13:46:08 core-04 systemd[1]: front.service failed.
I also want to include the fleetctl list-units output, where you can see that the other two are running without problems.
fleetctl list-units
UNIT MACHINE ACTIVE SUB
back.service 0ff08b11.../172.17.8.103 active running
front.service 69ab2600.../172.17.8.104 failed failed
graphite.service 2886cedd.../172.17.8.101 active running
there are a couple issues here. first, you can't use the --link argument for docker. this is a docker specific instruction for linking one container to another on the same docker engine. in your example, you have multiple engines, so this technique won't work. If you want to use that technique, you will need to employ the ambassador pattern: coreos ambassador, either that, you you can use the X-Fleet directive MachineOf: to make all of the docker containers run on the same machine, however, I think that would defeat your goals.
Often with cloud services one service needs another, like in your case. If the other service is not running (yet), then the services that need it should be well behaved and either exit, or wait for the needed service to be ready. So the needed service must be discovered. There are many techniques for the discovery phase, and the waiting phase. For example, you can write a 'wrapper' script in each of your containers. That wrapper can do these duties. In your case, you could have a script in the back.service and graphite.service which writes information to the etcd database, like:
ExecStartPre=/usr/bin/env etcdctl set /graphite/status ready }'
Then in the startup script in front you can do a etcdctl get /graphite/status to see when the container becomes ready (and not continue until it is). If you like you can store the ip address and port in the graphite script so that the front script can pick up the place to connect to.
Another technique for discovery is to use registrator. This is a super handy docker container that updates a directory structure in etcd everytime a container comes and goes. This makes it easier to use a discovery technique like I listed above without having each container having to announce itself, it becomes automatic. You still need the 'front' container to have a startup script that waits for the service to appear in the etcd database. I usually start registrator on coreos boot. In fact, I start two copies, one for discovering internal addresses (flannel ones) and one for external (services that are available outside my containers). Here is an example of the database registrator manages on my machines:
core#fo1 ~/prs $ etcdctl ls --recursive /skydns
/skydns/net
/skydns/net/tacodata
/skydns/net/tacodata/services
/skydns/net/tacodata/services/cadvisor-4194
/skydns/net/tacodata/services/cadvisor-4194/fo2:cadvisor:4194
/skydns/net/tacodata/services/cadvisor-4194/fo1:cadvisor:4194
/skydns/net/tacodata/services/cadvisor-4194/fo3:cadvisor:4194
/skydns/net/tacodata/services/internal
/skydns/net/tacodata/services/internal/cadvisor-4194
/skydns/net/tacodata/services/internal/cadvisor-4194/fo2:cadvisor:4194
/skydns/net/tacodata/services/internal/cadvisor-4194/fo1:cadvisor:4194
/skydns/net/tacodata/services/internal/cadvisor-4194/fo3:cadvisor:4194
/skydns/net/tacodata/services/internal/cadvisor-8080
/skydns/net/tacodata/services/internal/cadvisor-8080/fo2:cadvisor:8080
/skydns/net/tacodata/services/internal/cadvisor-8080/fo1:cadvisor:8080
/skydns/net/tacodata/services/internal/cadvisor-8080/fo3:cadvisor:8080
You can see the internal and external available ports for cadvisor. If I get one of the records:
etcdctl get /skydns/net/tacodata/services/internal/cadvisor-4194/fo2:cadvisor:4194
{"host":"10.1.88.3","port":4194}
you get everything you need to connect to that container internally. This technique really starts to shine when coupled with skydns. Skydns presents a dns service using the information presented by registrator. So, long story short, I can simply make my application use the hostname (the hostname defaults to be the name of the docker image, but it can be changed). So in this example here my application can connect to cadvisor-8080, and dns will give it one of the 3 ip addresses it has (it is on 3 machines). The dns also supports srv records, so, if you aren't using a well know port the srv record can give you the port number.
Using coreos and fleet it is difficult not to get the containers themselves involved in the publish/discovery/wait game. At least that's been my experience.
-g

Resources