I am running prometheus inside a docker container on centos. I wanted to know if there is a way to stop prometheus gracefully (without data corruption). Will running docker stop work? I could not find any docs on this and I am new to linux, docker and prometheus.
Yes, this should work.
docker stop sends a SIGTERM signal to the process at PID1, which is prometheus if you're using the official image. If no answer is received in a period of time (default 10s), docker will then send the SIGKILL signal and kill the process.
Prometheus is expected to shut down cleanly when receiving a SIGTERM however this may take longer than 10s. See
https://prometheus.io/docs/introduction/faq/#troubleshooting
You might have to extend the time docker waits before sending a SIGKILL.
e.g.: docker stop --time 50
You'll have to find the correct setting here depending on your Prometheus setup.
Related
I have set live-restore for most Docker hosts to support smooth minor version upgrades, but the documentation states that this feature is not suitable for major version upgrades. So the question is how to shut down dockerd and all containers, as if live-restore was not set?
Of course I can loop over all containers to shut them down one-by-one, but I would guess that dockerd uses a different procedure. Surely it can avoid starting new containers once it has received the signal to shutdown. The external loop cannot. Not to mention that the next Docker version might introduce new features/integrations that have to be taken into account. There has to be some "docker-style" way to do this.
I guess I figured it out myself:
edit /etc/docker/daemon.json to set live-restore to false
run "systemctl reload docker" or send a SIGHUP to dockerd
run "systemctl stop docker docker.socket" or similar to shutdown docker as usual
Correct me if I am wrong.
I would like to have some sort of systemctl stopall docker that stopped the daemon and the containers when live-restore is active. It certainly would be useful in some situations. Unfortunately there does not appear to be a way to opt-in to non-live-restore behavior temporarily. Instead I use:
docker ps -q | xargs docker kill && systemctl stop docker
There is a very small window of time between killing all the containers and stopping docker that a container can be started, so its not perfect.
When I delete a GCE VM I need my docker container to get stopped gracefully before the VM shuts down.
I am using Compute Engine Container Optimized OS (COS) and would expect my containers to be managed properly, but this is not what I am experiencing.
I tried a shutdown-script calling docker stop $(docker ps -a -q) but it doesn't make a difference at all. I can see it runs, but it seems the container is already gone by then.
I've tried trapping SIGTERM in my application. In the VM it's not trapping the signal, but on my local machine it does.
I am a bit lost and don't know what else to try. Any idea?
Take a look at Stopping Docker Containers Gracefully and also Gracefully Stopping Docker Containers
Docker swarm does have a health check options since API 1.25+
I'm looking for a way to update docker swarm services without affect service availability.
A simple solution would be just set the health check to a unhealthy state and wait for docker swarm update with the new image version, but I'm afraid it can kill containers with current active threads.
Since I haven't found a way to disable incoming requests for specific containers I wonder what is the kill signal sent to container.
If a 15)SIGTERM, there is a 9)SIGKILL after some timeout ? and what is the timeout ? There is a way to customize it?
With live restore enabled, docker containers are supposed to continue running even when the docker daemon does down.
During this daemon downtime, does the docker logging driver keep forwarding stdout/stderr from the running containers?
I'm asking b/c I'm unsure if the logging driver depends on the docker daemon process or not.
The docs clued me in, and I was able to verify this behavior.
From the docs:
If the daemon is down for a long time, running containers may fill up
the FIFO log the daemon normally reads. A full log blocks containers
from logging more data. The default buffer size is 64K. If the buffers
fill, you must restart the Docker daemon to flush them.
AFAIK, each container process pipes its logs its corresponding docker-containerd-shim process, which then pipes the logs to dockerd. If dockerd is down, the logs will stay in a buffer and get written to dockerd when it is back up. However as the docs specify, if the buffers fill up before the docker daemon is back up, you will start losing any additional logs.
We have Docker container which is running systemd as main process (PID 1). We have also started our worker processes in Docker container through systemd unit. Our Docker containers uses CentOS 7.2.
We have configured the Docker stop timeout so that we can handle the graceful shutdown of worker processes running inside Docker container.
When we are stopping Docker container, we can see SIGTERM is received to systemd process which is running with PID 1 inside container. Also container waits for stop timeout which we have configured.
But systemd process is not forwarding this SIGTERM to our worker processes which we have started using systemd unit.
From logs it looks like when systemd receives SIGTERM it tries to re-executes itself. We have tried with adding KillMode=mixed in our systemd unit file but it didn’t worked for us.
Is there any way to forward SIGTERM from systemd process to the child processes ?
Instead of running systemd itself you could also try with a replacement that provides a similar functionality. If you put the docker-systemctl-replacement as the entrypoint of the container then it will look for the systemd unit files you have provided - upon container start it runs ExecStart from unit descriptors and upon receiving SIGSTOP it will run ExecStop from each of the unit descriptors. It sounds like that's the functionality you want anyway.
Paste your docker run command, please.
I guess you have to add these 2 options.
--stop-signal=SIGRTMIN+3 --env container=docker