I have some containers running with docker server up. Now if the daemon crashes for some reason (I killed it using kill -9 $Pid_of_daemon to reproduce this behavior), and if I start the docker server again, why does it kill the old running container. The behavior I want is it should go ahead even if there are already running containers. The only reason I found out is, when daemon crashes, it looses its stdin, stdout pipes with the container, so it can no more attach to the old running containers. But if my container does not want stdout, stdin or stderr, then why would the daemon is killing it during startup. Please help
Related
When I delete a GCE VM I need my docker container to get stopped gracefully before the VM shuts down.
I am using Compute Engine Container Optimized OS (COS) and would expect my containers to be managed properly, but this is not what I am experiencing.
I tried a shutdown-script calling docker stop $(docker ps -a -q) but it doesn't make a difference at all. I can see it runs, but it seems the container is already gone by then.
I've tried trapping SIGTERM in my application. In the VM it's not trapping the signal, but on my local machine it does.
I am a bit lost and don't know what else to try. Any idea?
Take a look at Stopping Docker Containers Gracefully and also Gracefully Stopping Docker Containers
I want to restart the docker container only when container crashed due to error. And don't want to restart the container if host reboots.
Which restart_policy will work for the above case?
Start containers automatically
on-failure[:max-retries]
Restart the container if it exits due to an error, which manifests as a non-zero exit code. Optionally, limit the number of times the Docker daemon attempts to restart the container using the :max-retries option.
docker run -d --restart on-failure[:max-retries] CONTAINER
UPDATE
A Docker host is a physical computer system or virtual machine running Linux. This can be your laptop, server or virtual machine in your data center, or computing resource provided by a cloud provider. The component on the host that does the work of building and running containers is the Docker Daemon.
Keep containers alive during daemon downtime
By default, when the Docker daemon terminates, it shuts down running containers. You can configure the daemon so that containers remain running if the daemon becomes unavailable. This functionality is called live restore. The live restore option helps reduce container downtime due to daemon crashes, planned outages, or upgrades.
Enable live restore
There are two ways to enable the live restore setting to keep containers alive when the daemon becomes unavailable. Only do one of the following.
Add the configuration to the daemon configuration file. On Linux, this defaults to /etc/docker/daemon.json. On Docker Desktop for Mac or Docker Desktop for Windows, select the Docker icon from the task bar, then click Preferences -> Daemon -> Advanced.
Use the following JSON to enable live-restore.
{
"live-restore": true
}
Restart the Docker daemon. On Linux, you can avoid a restart (and avoid any downtime for your containers) by reloading the Docker daemon. If you use systemd, then use the command systemctl reload docker. Otherwise, send a SIGHUP signal to the dockerd process.
If you prefer, you can start the dockerd process manually with the --live-restore flag. This approach is not recommended because it does not set up the environment that systemd or another process manager would use when starting the Docker process. This can cause unexpected behavior.
I'm having trouble with docker where docker ps won't return and is stuck.
I found that doinng docker service restart something like
sudo service docker restart (https://forums.docker.com/t/what-to-do-when-all-docker-commands-hang/28103/4)
However I'm worried if it will kill all the running containers? (I guess the service do provide service so that docker containers can run?)
In the default configuration, your assumption is correct: If the docker daemon is stopped, all running containers are shut down.. But, as outlined on the link, this behaviour can be changed on docker >= 1.12 by adding
{
"live-restore": true
}
to /etc/docker/daemon.json. Crux: the daemon must be restarted for this change to take effect. Please take note of the limitations of live reload, e.g. only patch version upgrades are supported, not major version upgrades.
Another possibility is to define a restart policy when starting a container. To do so, pass one of the following values as value for the command line argument --restart when starting the container via docker run:
no Do not automatically restart the container. (the default)
on-failure Restart the container if it exits due to an error, which manifests
as a non-zero exit code.
always Always restart the container if it stops. If it is manually stopped,
it is restarted only when Docker daemon restarts or the container
itself is manually restarted.
(See the second bullet listed in restart policy details)
unless-stopped Similar to always, except that when the container is stopped
(manually or otherwise), it is not restarted even after Docker
daemon restarts.
For your specific situation, this would mean that you could:
Restart all containers with --restart always (more on that further below)
Re-configure the docker daemon to allow for live reload
Restart the docker daemon (which is not yet configured for live reload, but will be after this restart)
This restart would shut down and then restart all your containers once. But from then on, you should be free to stop the docker daemon without your containers terminating.
Handling major version upgrades
As mentioned above, live reload cannot handle major version upgrades. For a major version upgrade, one has to tear down all running containers. With a restart policy of always, however, the containers will be restarted after the docker daemon is restarted after the upgrade.
With live restore enabled, docker containers are supposed to continue running even when the docker daemon does down.
During this daemon downtime, does the docker logging driver keep forwarding stdout/stderr from the running containers?
I'm asking b/c I'm unsure if the logging driver depends on the docker daemon process or not.
The docs clued me in, and I was able to verify this behavior.
From the docs:
If the daemon is down for a long time, running containers may fill up
the FIFO log the daemon normally reads. A full log blocks containers
from logging more data. The default buffer size is 64K. If the buffers
fill, you must restart the Docker daemon to flush them.
AFAIK, each container process pipes its logs its corresponding docker-containerd-shim process, which then pipes the logs to dockerd. If dockerd is down, the logs will stay in a buffer and get written to dockerd when it is back up. However as the docs specify, if the buffers fill up before the docker daemon is back up, you will start losing any additional logs.
I am running a docker container which contains a node server. I want to attach to the container, kill the running server, and restart it (for development). However, when I kill the node server it kills the entire container (presumably because I am killing the process the container was started with).
Is this possible? This answer helped, but it doesn't explain how to kill the container's default process without killing the container (if possible).
If what I am trying to do isn't possible, what is the best way around this problem? Adding command: bash -c "while true; do echo 'Hit CTRL+C'; sleep 1; done" to each image in my docker-compose, as suggested in the comments of the linked answer, doesn't seem like the ideal solution, since it forces me to attach to my containers after they are up and run the command manually.
This is by design by Docker. Each container is supposed to be a stateless instance of a service. If that service is interrupted, the container is destroyed. If that service is requested/started, it is created. If you're using an orchestration platform like k8s, swarm, mesos, cattle, etc at least.
There are applications that exist to represent PID 1 rather than the service itself. But this goes against the design philosophy of microservices and containers. Here is an example of an init system that can run as PID 1 instead and allow you to kill and spawn processes within your container at will: https://github.com/Yelp/dumb-init
Why do you want to reboot the node server? To apply changes from a config file or something? If so, you're looking for a solution in the wrong direction. You should instead define a persistent volume so that when the container respawns the service would reread said config file.
https://docs.docker.com/engine/admin/volumes/volumes/
If you need to restart the process that's running the container, then simply run a:
docker restart $container_name_or_id
Exec'ing into a container shouldn't be needed for normal operations, consider that a debugging tool.
Rather than changing the script that gets run to automatically restart, I'd move that out to the docker engine so it's visible if your container is crashing:
docker run --restart=unless-stopped ...
When a container is run with the above option, docker will restart it for you, unless you intentionally run a docker stop on the container.
As for why killing pid 1 in the container shuts it down, it's the same as killing pid 1 on a linux server. If you kill init/systemd, the box will go down. Inside the namespace of the container, similar rules apply and cannot be changed.