I have have been trying to make it so that I am able to shutdown a docker container from inside. I have read that using tini is the best way to do this.
I have added init: true to my docker-compose.yml and I can see that docker-init is running as PID 1. However the only command that lets me shutdown the container from my shell script is using kill 1, but I want to gracefully shutdown my container so that it can do some cleanup.
I have tried using commands like kill -SIGINT 1 which results in the error kill: Illegal option -S
or kill -INT 1 and kill -2 1 which both seem to do nothing at all.
I can't seem to figure out the command that I can use. If there is an alternative to init that would also be an option.
The application inside the container doesn't need any special setup in order to shut down; it can just run its own shutdown sequence and exit, and when it does exit, the container will exit as well. If you're trying to do this from a debugging shell you launched with docker exec, you can just use docker stop to send SIGTERM and then SIGKILL. (...and reserve the docker exec shell for debugging; it should not be the primary way you interact with your container.)
If you need to send a container a non-default signal, docker kill has that option:
docker kill --signal SIGINT container_name
In terms of using kill(1) in a debugging shell, the man page for the underlying kill(2) function notes:
The only signals that can be sent to process ID 1, the init process, are those for which init has explicitly installed signal handlers. This is done to assure the system is not brought down accidentally.
It looks like tini collects and forwards a pretty broad range of signals, everything except SIGFPE, SIGILL, SIGSEGV, SIGBUS, SIGABRT, SIGTRAP, SIGSYS, and the uncatchable signals (notably SIGKILL). Since it does register a signal handler, kill -INT 1 should forward that signal on to the actual container process. (Its pid is probably 2, so kill -INT 2 should also tell the process to stop.)
Related
Having an issue with Docker at the moment; I'm using it to run an image that launches an ipython notebook on startup. I'm looking to make some edits to ipython notebook itself, so I need to close it after launch.
However, hitting CTRL+C in the terminal just inputs "^C" as a string. There seems to be no real way of using CTRL+C to actually close the ipython notebook instance.
Would anyone have any clues as to what can cause this, or know of any solutions for it?
Most likely the container image you use is not handling process signals properly.
If you are authoring the image then change it as Roland Webers' answer suggests.
Otherwise try to run it with --init.
docker run -it --init ....
This fixes Ctrl+C for me.
Source: https://docs.docker.com/v17.09/engine/reference/run/#specify-an-init-process
The problem is that Ctrl-C sends a signal to the top-level process inside the container, but that process doesn't necessarily react as you would expect. The top-level process has ID 1 inside the container, which means that it doesn't get the default signal handlers that processes usually have. If the top-level process is a shell, then it can receive the signal through its own handler, but doesn't forward it to the command that is executed within the shell. Details are explained here. In both cases, the docker container acts as if it simply ignores Ctrl-C.
If you're building your own images, the solution is to run a minimal init process, such as tini or dumb-init, as the top-level process inside the container.
This post proposes CTRL-Z as a workaround for sending the process to background and then killing the process by its process id:
Cannot kill Python script with Ctrl-C
Possible problems:
The program catches ctrl-c and does nothing, very unlikely.
There are background processes that are not managed correctly. Only the main process receives the signal and sub-processes hang. Very likely what's happening.
Proposed Solution:
Check the programs documentation on how it's properly started and stopped. ctrl-c seems not to be the proper way.
Wrap the program with a docker-entrypoint.sh bash script that blocks the container process and is able to catch ctrl-c. This bash example should help: https://rimuhosting.com/knowledgebase/linux/misc/trapping-ctrl-c-in-bash
After catching ctrl-c invoke the proper shutdown method for ipython notebook.
From this post on the Docker message boards:
Open a new shell and execute
$ docker ps # get the id of the running container
$ docker stop <container> # kill it (gracefully)
This worked well for me. CTRL-Z, CTRL-\, etc. only came up as strings, but this killed the Docker container and returned the tab to terminal input.
#maybeg's answer already explains very well why this might be happening.
Regarding stopping the unresponsive container, another solution is to simply issue a docker stop <container-id> in another terminal. As opposed to CTRL-C, docker stop does not send a SIGINT but a SIGTERM signal, to which the process might react differently.
Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop a running container by sending SIGTERM and then SIGKILL after a grace period
If that fails, use docker kill <container-id> which sends a SIGKILL immediately.
I execute:
docker run --stop-signal=9 --stop-timeout=30 --name=test test
but this does not kill the container, it does not do anything!
from the Manual Docker Run:
Stop container with signal (--stop-signal)
The --stop-signal flag sets the system call signal that will be sent to the container to exit. This signal can be a valid unsigned number that matches a position in the kernel’s syscall table, for instance 9, or a signal name in the format SIGNAME, for instance SIGKILL.
Stop container with timeout (--stop-timeout)
The --stop-timeout flag sets the timeout (in seconds) that a pre-defined (see --stop-signal) system call signal that will be sent to the container to exit. After timeout elapses the container will be killed with SIGKILL.
How is this used correctly?
AFAIK there aren't any docker flags to run a container for a specified time.
As you quoted the command you execute sets the signal sent to the main process of the container when docker stops (in your case SIGKILL).
If you want to run the container for some time and then stop it you could use the timeout utility that runs a command with a time limit:
timeout --signal=SIGTERM 5 docker run --rm test
The command, on timeout (5 seconds), sends the SIGTERM signal to the docker process which propagates it to the running service. Additionally, you could use the --kill-after duration flag to force the container to stop if it's still running after the initial signal was sent. Hope it helps.
I am attempting to send SIGSTOP, and then later, SIGKILL to a container. This line leads me to believe that it will behave as I expect: https://github.com/docker/docker/issues/5948#issuecomment-43684471
However, it is going ahead and actually removing the containers. The commands are:
docker kill -s STOP container
docker kill -s CONT container
(Equivalent through the dockerode API I am using, but I just went to command line when that wasn't working). Is there some missing options I'm missing?
I think you're actually looking for the commands docker pause and docker unpause. Using the STOP signal is likely to be error-prone and dependent on how the process handles the signal.
I guess what's happening in this case is that Docker thinks the process has terminated and stops the container (it shouldn't be removed however, you can restart it with docker start).
I'm having a docker compose setup of a database container, an application container and one container which pre-loads the database with necessary data.
I want to start all of the containers together with docker-compose up while the pre-loading container terminates after it has completed it work with exit 0.
But terminating this one container takes down the complete setup with the message:
composesetup_load_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)
Stopping composesetup_app_1...
Stopping composesetup_db_1...
Is there any way of having multiple containers with different life-time in one docker-compose setup? If yes, how?
My workaround for now is to keep the pre-loading container running by adding tail -f /dev/null to the end of the entrypoint script. This keeps the process running, while nothing actual happens.
Using -d option at docker-compose up -d will run the process in detached mode. This avoids the need to kill the service with Ctrl+C and therefore stop the containers.
Note: I am asumming you killed the process with Ctrl+C from the message "Gracefully stopping... (press Ctrl+C again to force)" you shared.
Will docker stop fail if processes running inside the container fail to stop?? If i use docker kill, can unsaved data inside the container be preserved. Is docker stop time consuming compared to docker kill?? I want to do a shutdown of the container but without loosing any data(without high latency to complete kill or stop process).
Line reference:
docker stop: Stop a running container (send SIGTERM, and then SIGKILL
after grace period) [...] The main process inside the container will
receive SIGTERM, and after a grace period, SIGKILL. [emphasis mine]
docker kill: Kill a running container (send SIGKILL, or specified
signal) [...] The main process inside the container will be sent
SIGKILL, or any signal specified with option --signal. [emphasis mine]
You can get more info from this post: https://superuser.com/questions/756999/whats-the-difference-between-docker-stop-and-docker-kill
Docker stop:
When you issue a docker stop command a hardware signal is sent to the process inside of that container. In the case of docker stop we send a sig term message which is short for terminate signal its a message that's going to be received by the process telling it essentially to shut down on its own time.
SIGTERM is used any time that you want to stop a process inside of your container and shut the container down, and you want to give that process inside there a little bit of time to shut itself down and do a little bit of clean up.
A lot of different programming languages have the ability for you to listen for these signals inside of your code base, and as soon as you get that signal you could attempt to do a little bit of cleanup or maybe save some file or emit some message or something like that.
On the other hand the docker kill command issue is a sig kill or kills signal to the primary running process inside the container, so kill it essentially means you have to shut down right now and you do not get to do any additional work.
So ideally we always stop a container with the docker stop command in order to get the running process inside of it a little bit of time to shut itself down, otherwise if it feels like the container has locked up and it's not responding to the docker stop command then we could issue docker kill instead.
One kind of little oddity or interesting thing about docker stop, when issue docker stop to a container and if the container dose not automatically stop in 10 seconds then docker is going to automatically fall back to issuing the docker kill command.
So essentially at docker stop is us being nice but it's only got 10 seconds to actually shut down.
A good example could be ping command.
sudo docker run busybox ping google.com
now if you want to stop the container if you use docker stop container_id, you will see it takes 10 seconds before getting shut down because ping command dose not properly respond to a SIGTERM message. In other words the ping command doesn't really have the ability to say oh yeah I understand you want me to shut down.
So after we waited those 10 seconds eventually the kill signal was sent to it telling it hey ping you are done and shut yourself down.
But if you use docker kill container_id you are going to see that's it instantly dead.
You should use docker stop since it stops the container gracefully - like shutting down your laptop, instead of killing them - like forcibly turn off the laptop from it's battery.
But, Docker will force shut down (kill the processes) by the time it takes 10 seconds to stop them gracefully.
docker stop will send SIGTERM (terminate signal) to the process and docker will have 10 seconds to clean up like saving files or emitting some messages.
Use docker kill when container is locked up, if it is not responding.