I'm running a python application in a containerized environment and created a service file which will start and stop this application (under centos)
The service file that I'm using should send NOHUP signal to the container and kill it immediately
Description=py Container
Requires=docker.service
After=docker.service
[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker exec %n stop
ExecStartPre=-/usr/bin/docker rm %n
ExecStart=/usr/bin/docker run --rm --name %n -t=false -i -e environment=env1 container_name
ExecStop=/usr/bin/docker kill --signal=SIGHUP %n
ExecStopPost=/usr/bin/docker rm -f %n
[Install]
WantedBy=local.target
Systemd should stop the container and remove it in a timely manner, but it takes up tp 2 minutes to kill it.
Could you advise on how to accelerate the termination of the container ?
if that signal SIGHUP is not so important for you , you may try to kill the contaier without it /usr/bin/docker kill %n or simply trying to stop it using /usr/bin/docker stop %n
Related
I am trying to create my own docker container, and custom service which I created for my work, this is my service file
[1/1] /etc/systemd/system/qsinavAI.service
[Unit]
Description=uWSGI instance to serve Qsinav AI
After=network.target
[Service]
User=www-data
Group=www-data
WorkingDirectory=/root/AI/
Environment="PATH=/root/AI/bin"
ExecStart=/root/AI/bin/uwsgi --ini ai.ini
[Install]
WantedBy=multi-user.target
and when I am trying to run this service I get this error
System has not been booted with systemd as init system (PID 1). Can't
operate. Failed to connect to bus: Host is down
I searched a lot to find a solution but I could not, how can I enable the systemctl in docker.
this is the command that I am using to run the container
docker run -dt -p 5000:5000 --name AIPython2 --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro --cap-add SYS_ADMIN last_python_image
If your application is only ever run inside a container then you should create a docker-entrypoint.sh script with an "exec" at the end so that your application is run as a remapped PID 1 in the container. That way cloud systems can see if the application is alive and they can send a SIGTERM to stop the application.
#! /bin/bash
cd /root/AI
PATH=/root/AI/bin
exec /root/AI/bin/uwsgi --ini ai.ini
If your application shall be able to run in systemd environment outside of a container then you can choose to reuse the systemd descriptor. It requires an init-daemon on PID 1 and a service manager to check the "enbabled" services. One example would be the systemctl-docker-replacement script.
Docker containers should have an "entrypoint" command that runs in foreground to keep the container running. The basic idea behind a container is that it runs as long as the root process that started it, keeps running. Since you will issue a systemctl start qsinavAI.service, the command will succeed but once this command exits, the container will stop.
By design, containers started in detached mode exit when the root process used to run the container exits, ...
See some reference about this and starting nginx service in the official documentation.
So instead of trying to run your application as a service, you should have an entrypoint statement at the end of your Dockerfile. Then when you start this container with docker run, you can specify -d to run it in "detached" mode.
Example, taking the command from ExecStart and assuming it runs in foreground:
ENTRYPOINT ["/root/AI/bin/uwsgi", "--ini", "ai.ini"]
Exemple how to create image with systemd and boot like a real environment. A Dockerfile is required.
FROM ubuntu:22.04
RUN echo 'root:root' | chpasswd
RUN printf '#!/bin/sh\nexit 0' > /usr/sbin/policy-rc.d
RUN apt-get update
RUN apt-get install -y systemd systemd-sysv dbus dbus-user-session
ENTRYPOINT ["/sbin/init"]
/sbin/init is important to init systemd and enable systemctl.
Then build the system.
docker build -t testimage -f Dockerfile .
docker run -it --privileged --cap-add=ALL testimage
I'm trying to understand why my Docker container does not stop gracefully and just times out. The container is running crond:
FROM alpine:latest
ADD crontab /etc/crontabs/root
RUN chmod 0644 /etc/crontabs/root
CMD ["crond", "-f"]
And the crontab file is:
* * * * * echo 'Working'
# this empty line required by cron
Built with docker build . -t periodic:latest
And run with docker run --rm --name periodic periodic:latest
This is all good, but when I try to docker stop periodic from another terminal, it doesn't stop gracefully, the time out kicks in and is killed abruptly. It's like crond isn't responding to the SIGTERM.
crond is definitely PID 1
/ # ps
PID USER TIME COMMAND
1 root 0:00 crond -f
6 root 0:00 ash
11 root 0:00 ps
However, if I do this:
docker run -it --rm --name shell alpine:latest ash and
docker exec -it shell crond -f in another terminal, I can kill crond from the first shell with SIGTERM so I know it can be stopped with SIGTERM.
Thanks for any help.
Adding an init process to the container (init: true in docker-compose.yml) solved the problem.
EDIT: I read this https://blog.thesparktree.com/cron-in-docker to understand the issues and solutions around running cron in Docker. From this article:
"Finally, as you’ve been playing around, you may have noticed that it’s difficult to kill the container running cron. You may have had to use docker kill or docker-compose kill to terminate the container, rather than using ctrl + C or docker stop.
Unfortunately, it seems like SIGINT is not always correctly handled by cron implementations when running in the foreground.
After researching a couple of alternatives, the only solution that seemed to work was using a process supervisor (like tini or s6-overlay). Since tini was merged into Docker 1.13, technically, you can use it transparently by passing --init to your docker run command. In practice you often can’t because your cluster manager doesn’t support it."
Since my original post and answer, I've migrated to Kubernetes, so init in docker-compose.yml won't work. My container is based on Debian Buster, so I've now installed tini in the Dockerfile, and changed the ENTRYPOINT to ["/usr/bin/tini", "--", "/usr/local/bin/entrypoint.sh"] (my entrypoint.sh finally does exec cron -f)
The key is that you cannot stop a pid=1 process in docker. It supposes that docker stops (or kills if it was launched with --rm).
That's why if you run -it ... ash, shell has pid 1 and you can kill other processes.
If you want your cron is killable without stopping/killing docker, just launch another process as entrypoint:
Launch cron after docker entrypoint (For example, run as cmd tail -F /dev/null and then launch cron docker run -d yourdocker service cron start)
I've looked at : Docker and systemd - service stopping after 10 seconds
and Docker containers shut down after systemd start , but still can't figure out how to get my docker container to start using systemd and not close out.
My psim.service file:
[Unit]
Description=My process
After=docker.service
Requires=docker.service
[Service]
ExecStart=docker run --net=host --name psim -it psim
ExecStop=/bin/docker stop psim
ExecStopPost=/bin/docker rm psim
[Install]
WantedBy=local.target
When I use
"docker run --net=host --name psim -dit psim"
with the detached flag, the container will start but immediately stop and remove itself after a few seconds. If I use either of the following without the detached '-d' flag:
docker run --net=host --name psim -it psim
docker run --net=host --name psim -a STDOUT -it psim
the container won't start at all using systemd, but will start normally if I run that command without systemd. Am I missing something?
Just realized it works if I remove the "-i" flag.
I have a very very simple Go app listening on port 8080
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
w.Header().Set("Content-Type", "text-plain")
w.Write([]byte("Hello World!"))
})
log.Fatal(http.ListenAndServe(":8080", http.DefaultServeMux))
I install it in a Docker container and start it like so:
FROM golang:alpine
ADD . /go/src/github.com/myuser/myapp
RUN go install github.com/myuser/myapp
ENTRYPOINT ["/go/bin/myapp"]
EXPOSE 8080
I then run the container using docker run:
docker run --publish 8080:8080 first-app
I expect that, like most programs, I can send a SIGTERM to the process running docker run and this will cause the container to stop running. I observe that sending SIGTERM has no effect, and instead I need to use a command like docker kill or docker stop.
Is this intended behavior? I've asked in the forums and on IRC and gotten no answer.
Any process run with docker must handle signals itself.
Alternatively use the --init flag to run the tini init as PID 1
The sh shell can become the PID 1 process depending on how you specify a command (CMD).
Detail
A SIGTERM is propagated by the docker run command to the Docker daemon by default but it will not take effect unless the signal is specifically handled in main process being run by Docker.
The first process you run in a container will have PID 1 in that containers context. This is treated as a special process by the linux kernel. It will not be sent a signal unless the process has a handler installed for that signal. It is also PID 1's job to forward signals onto other child processes.
docker run and other commands are API clients for the Remote API hosted by the docker daemon. The docker daemon runs as a seperate process and is the parent for the commands you run inside a container context. This means that there is no direct sending of signals between run and the daemon, in the standard unix manner.
The docker run and docker attach command have a --sig-proxy flag that defaults signal proxying to true. You can turn this off if you want.
docker exec does not proxy signals.
In a Dockerfile, be careful to use the "exec form" when specifying CMD and ENTRYPOINT defaults if you don't want sh to become the PID 1 process (Kevin Burke):
CMD [ "executable", "param1", "param2" ]
Signal Handling Go Example
Using the sample Go code here: https://gobyexample.com/signals
Run both a regular process that doesn't handle signals and the Go daemon that traps signals and put them in the background. I'm using sleep as it's easy and doesn't handle "daemon" signals.
$ docker run busybox sleep 6000 &
$ docker run gosignal &
With a ps tool that has a "tree" view, you can see the two distinct process trees. One for the docker run process under sshd. The other for the actual container processes, under docker daemon.
$ pstree -p
init(1)-+-VBoxService(1287)
|-docker(1356)---docker-containe(1369)-+-docker-containe(1511)---gitlab-ci-multi(1520)
| |-docker-containe(4069)---sleep(4078)
| `-docker-containe(4638)---main(4649)
`-sshd(1307)---sshd(1565)---sshd(1567)---sh(1568)-+-docker(4060)
|-docker(4632)
`-pstree(4671)
The details of docker hosts processes:
$ ps -ef | grep "docker r\|sleep\|main"
docker 4060 1568 0 02:57 pts/0 00:00:00 docker run busybox sleep 6000
root 4078 4069 0 02:58 ? 00:00:00 sleep 6000
docker 4632 1568 0 03:10 pts/0 00:00:00 docker run gosignal
root 4649 4638 0 03:10 ? 00:00:00 /main
Killing
I can't kill the docker run busybox sleep command:
$ kill 4060
$ ps -ef | grep 4060
docker 4060 1568 0 02:57 pts/0 00:00:00 docker run busybox sleep 6000
I can kill the docker run gosignal command that has the trap handler:
$ kill 4632
$
terminated
exiting
[2]+ Done docker run gosignal
Signals via docker exec
If I docker exec a new sleep process in the already running sleep container, I can send an ctrl-c and interrupt the docker exec itself, but that doesn't forward to the actual process:
$ docker exec 30b6652cfc04 sleep 600
^C
$ docker exec 30b6652cfc04 ps -ef
PID USER TIME COMMAND
1 root 0:00 sleep 6000 <- original
97 root 0:00 sleep 600 <- execed still running
102 root 0:00 ps -ef
So there are two factors at play here:
1) If you specify a string for an entrypoint, like this:
ENTRYPOINT /go/bin/myapp
Docker runs the script with /bin/sh -c 'command'. This intermediate script gets the SIGTERM, but doesn't send it to the running server app.
To avoid the intermediate layer, specify your entrypoint as an array of strings.
ENTRYPOINT ["/go/bin/myapp"]
2) I built the app I was trying to run with the following string:
docker build -t first-app .
This tagged the container with the name first-app. Unfortunately when I tried to rebuild/rerun the container I ran:
docker build .
Which didn't overwrite the tag, so my changes weren't being applied.
Once I did both of those things, I was able to kill the process with ctrl+c, and bring down the running container.
A very comprehensive description of this problem and the solutions can be found here:
https://vsupalov.com/docker-compose-stop-slow
In my case, my app expects to receive SIGTERM signal for graceful shutdown didn't receive it because the process started by a bash script which called from a dockerfile in this form: ENTRYPOINT ["/path/to/script.sh"]
so the script didn't propagate the SIGTERM to the app.
The solution was to use exec from the script run the command that starts the app:
e.g. exec java -jar ...
I have been trying to use FleetCtl to launch docker images one is a Data Volume Image and one is a Nginx Image launched with the --volumes-from option. The Nginx Image will not continue to run on the CoreOs server, but if I go to the server and type the command docker start the image starts and runs. Is there an image with launching Docker images that use a data volume with Fleet?
Docker File for Volume:
FROM busybox
MAINTAINER Zombie Possum
VOLUME ["/usr/share/nginx/html", "/usr/share/nginx/conf"]
COPY dist /usr/share/nginx/html
COPY dist_nginx.conf /usr/share/nginx/conf/dist_nginx.conf
CMD ["/usr/bin/true"]
Fleet File For Volume nginxData.service:
[Unit]
Description=Data Container
Requires=docker.service
After=docker.service
[Service]
TimeoutStartSec=0
KillMode=none
User=core
WorkingDirectory=/home/core
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/bin/docker kill DATA_NGINX
ExecStartPre=-/usr/bin/docker rm DATA_NGINX
ExecStartPre=-/usr/bin/docker pull private_repo/data_nginx:latest
ExecStart=/usr/bin/docker run --name DATA_NGINX private_repo/data_nginx:latest
ExecStop=/usr/bin/docker stop DATA_NGINX
Fleet File for nginx.service:
[Unit]
Description=Nginx Container
Requires=docker.service
After=docker.service
[Service]
TimeoutStartSec=0
KillMode=none
User=core
WorkingDirectory=/home/core
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/bin/docker kill NGINX
ExecStartPre=-/usr/bin/docker rm NGINX
ExecStartPre=-/usr/bin/docker pull private_repo/nginx:latest
ExecStart=/usr/bin/docker run -rm -p 80:80 --name NGINX --volumes-from DATA_NGINX private_repo/nginx:latest
ExecStop=/usr/bin/docker stop NGINX
[X-Fleet]
MachineOf=nginxData.service
Fleet Commands:
fleetctl submit nginxData.service
fleetctl submit nginx.service
fleetctl start nginxData.service
fleetctl start nginx.service
The Dockerfile you provided runs with an error on my Docker host (without using fleet); maybe when fleet does detect that error it removes the container for you, while on a Docker host the container still exists in stopped state despite the error.
Here's the error:
$ docker build --force-rm -t so-26469566 .
$ docker run --name DATA_NGINX so-26469566
exec: "/usr/bin/true": stat /usr/bin/true: no such file or directory2014/10/20 16:59:54 Error response from daemon: Cannot start container 767562758b9f30097a5ed16b98fe818d9c9574bb82b1cfd502bc3403e97d5b0
e: exec: "/usr/bin/true": stat /usr/bin/true: no such file or directory
make: *** [run] Error 1
Try the following CMD statement in your Dockerfile and see if it does changes the behavior of fleet.
CMD ["/bin/true"]
If your nginx.service start and run runing the docker run command directly in CoreOS server, mayby your problem is not in docker image but nginx.service.
Notice you configure your service with:
ExecStartPre=-/usr/bin/docker kill NGINX
ExecStartPre=-/usr/bin/docker rm NGINX
trying to kill and remove NGINX container but you run the container with --rm which remove it automatically when it fail or exit.
Maybe your service not start because its ExecStartPre is failing
Try include
Requires=nginxData.service
After=nginxData.service
too.