I am using docker container stop to stop a container. It sends a SIGTERM signal to the child processes and the child processes might take some time to finish before exiting.
So, my docker container stop is waiting for the child processes to finish. But I have no way of knowing why it is waiting for. Is there a way of running docker container stop in an "interactive" mode or some other docker-solution, which will tell me what is the main process waiting for to exit?
Additional information: Using Hangfire to kick off these jobs and monitor these jobs.
Docker container stop documentation doesn't list any way of doing it.
One potential solution I thought of?:
I have a way to know the process id of the main application which is running inside the docker. Can we somehow pipe that information to a file, which I can read simultaneously while the docker stop is working.
you cannot follow logs and stop the container with same command. You can follow the logs in other terminal tab while closing the container in another tab.Thus it will show you the details log of that container.Here is an example:
Here i have two window of terminal in one i am stopping the container with command
docker stop f76
in another one i am following the log with
docker logs --follow f76
While stopping the container the logs printing these lines:
1:signal-handler (1671521182) Received SIGTERM scheduling shutdown...
1:M 20 Dec 2022 07:26:22.689 # User requested shutdown...
1:M 20 Dec 2022 07:26:22.689 * Saving the final RDB snapshot before exiting.
1:M 20 Dec 2022 07:26:22.704 * DB saved on disk
1:M 20 Dec 2022 07:26:22.704 # Redis is now ready to exit, bye bye...
Thus, i can know what happening in my container or what it is waiting for while stopping.
docker stop only manages the process with pid 1 inside the container and nothing else; there's nothing more that Docker could display while it's waiting.
A Docker container normally only runs a single process. In this case the mechanics are clear: Docker sends SIGTERM to this process, waits for it to exit, and eventually sends SIGKILL.
If you do somehow have your container to set up to run multiple processes, there will be one process at the root of the process tree with process ID 1. Depending on your setup this could be supervisord, a shell, or something else. docker stop only sends SIGTERM to this single root process, and after its timeout, if this single root process hasn't exited yet, it will forcibly terminate that one process with SIGKILL. The cleanup sequence will also terminate any child processes there may happen to be, I believe immediately and less politely.
As such, there's not much that could be written from an "interactive docker stop". It sends SIGTERM immediately, and if it doesn't come back immediately, it means the process has handled the signal and not exited yet. Docker itself isn't doing any more than waiting at this point in a way that could be monitored.
Related
I've read that docker stop sends SIGTERM then SIGKILL to the main process, presumably pid 1 (https://www.edureka.co/community/50826/what-the-process-stopping-and-restarting-docker-container), which as a process leader, therefore kills all the processes running beneath it--i.e., "the container".
However, when I do a docker-compose stop, and then docker-compose start, I see 2 unexpected things:
the uptime of a particular container as unchanged--i.e., over 10 days.
the /tmp contents were NOT deleted (it's Debian GNU/Linux 11 (bullseye))
Two questions:
Is it possible to stop a container but keep its uptime unchanged?
Shouldn't the /tmp contents be deleted, as happens on bare metal restarts?
I have have been trying to make it so that I am able to shutdown a docker container from inside. I have read that using tini is the best way to do this.
I have added init: true to my docker-compose.yml and I can see that docker-init is running as PID 1. However the only command that lets me shutdown the container from my shell script is using kill 1, but I want to gracefully shutdown my container so that it can do some cleanup.
I have tried using commands like kill -SIGINT 1 which results in the error kill: Illegal option -S
or kill -INT 1 and kill -2 1 which both seem to do nothing at all.
I can't seem to figure out the command that I can use. If there is an alternative to init that would also be an option.
The application inside the container doesn't need any special setup in order to shut down; it can just run its own shutdown sequence and exit, and when it does exit, the container will exit as well. If you're trying to do this from a debugging shell you launched with docker exec, you can just use docker stop to send SIGTERM and then SIGKILL. (...and reserve the docker exec shell for debugging; it should not be the primary way you interact with your container.)
If you need to send a container a non-default signal, docker kill has that option:
docker kill --signal SIGINT container_name
In terms of using kill(1) in a debugging shell, the man page for the underlying kill(2) function notes:
The only signals that can be sent to process ID 1, the init process, are those for which init has explicitly installed signal handlers. This is done to assure the system is not brought down accidentally.
It looks like tini collects and forwards a pretty broad range of signals, everything except SIGFPE, SIGILL, SIGSEGV, SIGBUS, SIGABRT, SIGTRAP, SIGSYS, and the uncatchable signals (notably SIGKILL). Since it does register a signal handler, kill -INT 1 should forward that signal on to the actual container process. (Its pid is probably 2, so kill -INT 2 should also tell the process to stop.)
I am running Docker in swarm mode with several nodes in the cluster.
According to the documentation written here: https://docs.docker.com/engine/reference/commandline/service_update/ and here: https://docs.docker.com/engine/reference/commandline/service_create/, --stop-grace-period command sets the time to wait before force killing a container.
Expected behavior -
My expectation was that Docker would wait this period of time until it tries to stop a running container, during a rolling update.
Actual behavior -
Docker sends the termination signal after several seconds the new container with the new version of the image starts.
Steps to reproduce the behavior
docker service create --replicas 1 --stop-grace-period 60s --update-delay 60s --update-monitor 5s --update-order start-first --name nginx nginx:1.15.8
Wait for the service to start up the container (aprox. 2 minutes)
docker service update --image nginx:1.15.9 nginx
docker ps -a
As you can see, the new container started and after a second, the
old one was killed by Docker.
Any idea why?
I also opened an issue on Github, here: https://github.com/docker/for-linux/issues/615
The --stop-grace-period value is the amount of time that Docker will wait after sending a sigterm and give up waiting for the container to exit gracefully. Once the grace period is complete, it will kill the container with a sigkill.
The sequence of events seem to happen as is designed based on your description of your setup. Your container exits cleanly and quickly when it gets its sigterm so Docker never needs to send a sigkill.
I see you also specified --update-delay 60 but that won't take effect since you only have one replica. The update delay will tell docker to wait 60 seconds after cycling the first task, so it is only helpful for 2 or more replicas.
It seems like you want your single-replica service to run a new task and an old task concurrently for 60 seconds, but swarm mode is happy to get rid of old containers with sigterm as soon as the new container is up.
I think you can close the issue on GitHub.
stop-grace-period this is the period between stop (SIGTERM) and kill (SIGKILL).
Of course, you can change SIGTERM to another signal by using --stop-signal switch. The behavior of application into a container, when a stop signal is received, is your responsibility.
Here good article explaining this kitchen.
I'm having a docker compose setup of a database container, an application container and one container which pre-loads the database with necessary data.
I want to start all of the containers together with docker-compose up while the pre-loading container terminates after it has completed it work with exit 0.
But terminating this one container takes down the complete setup with the message:
composesetup_load_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)
Stopping composesetup_app_1...
Stopping composesetup_db_1...
Is there any way of having multiple containers with different life-time in one docker-compose setup? If yes, how?
My workaround for now is to keep the pre-loading container running by adding tail -f /dev/null to the end of the entrypoint script. This keeps the process running, while nothing actual happens.
Using -d option at docker-compose up -d will run the process in detached mode. This avoids the need to kill the service with Ctrl+C and therefore stop the containers.
Note: I am asumming you killed the process with Ctrl+C from the message "Gracefully stopping... (press Ctrl+C again to force)" you shared.
Will docker stop fail if processes running inside the container fail to stop?? If i use docker kill, can unsaved data inside the container be preserved. Is docker stop time consuming compared to docker kill?? I want to do a shutdown of the container but without loosing any data(without high latency to complete kill or stop process).
Line reference:
docker stop: Stop a running container (send SIGTERM, and then SIGKILL
after grace period) [...] The main process inside the container will
receive SIGTERM, and after a grace period, SIGKILL. [emphasis mine]
docker kill: Kill a running container (send SIGKILL, or specified
signal) [...] The main process inside the container will be sent
SIGKILL, or any signal specified with option --signal. [emphasis mine]
You can get more info from this post: https://superuser.com/questions/756999/whats-the-difference-between-docker-stop-and-docker-kill
Docker stop:
When you issue a docker stop command a hardware signal is sent to the process inside of that container. In the case of docker stop we send a sig term message which is short for terminate signal its a message that's going to be received by the process telling it essentially to shut down on its own time.
SIGTERM is used any time that you want to stop a process inside of your container and shut the container down, and you want to give that process inside there a little bit of time to shut itself down and do a little bit of clean up.
A lot of different programming languages have the ability for you to listen for these signals inside of your code base, and as soon as you get that signal you could attempt to do a little bit of cleanup or maybe save some file or emit some message or something like that.
On the other hand the docker kill command issue is a sig kill or kills signal to the primary running process inside the container, so kill it essentially means you have to shut down right now and you do not get to do any additional work.
So ideally we always stop a container with the docker stop command in order to get the running process inside of it a little bit of time to shut itself down, otherwise if it feels like the container has locked up and it's not responding to the docker stop command then we could issue docker kill instead.
One kind of little oddity or interesting thing about docker stop, when issue docker stop to a container and if the container dose not automatically stop in 10 seconds then docker is going to automatically fall back to issuing the docker kill command.
So essentially at docker stop is us being nice but it's only got 10 seconds to actually shut down.
A good example could be ping command.
sudo docker run busybox ping google.com
now if you want to stop the container if you use docker stop container_id, you will see it takes 10 seconds before getting shut down because ping command dose not properly respond to a SIGTERM message. In other words the ping command doesn't really have the ability to say oh yeah I understand you want me to shut down.
So after we waited those 10 seconds eventually the kill signal was sent to it telling it hey ping you are done and shut yourself down.
But if you use docker kill container_id you are going to see that's it instantly dead.
You should use docker stop since it stops the container gracefully - like shutting down your laptop, instead of killing them - like forcibly turn off the laptop from it's battery.
But, Docker will force shut down (kill the processes) by the time it takes 10 seconds to stop them gracefully.
docker stop will send SIGTERM (terminate signal) to the process and docker will have 10 seconds to clean up like saving files or emitting some messages.
Use docker kill when container is locked up, if it is not responding.