Docker swarm does have a health check options since API 1.25+
I'm looking for a way to update docker swarm services without affect service availability.
A simple solution would be just set the health check to a unhealthy state and wait for docker swarm update with the new image version, but I'm afraid it can kill containers with current active threads.
Since I haven't found a way to disable incoming requests for specific containers I wonder what is the kill signal sent to container.
If a 15)SIGTERM, there is a 9)SIGKILL after some timeout ? and what is the timeout ? There is a way to customize it?
Related
I need some clarification in regards to using HEALTHCHECK on a docker service.
Context:
We are experimenting with a multi-node mariadb cluster and by utilizing HEALTHCHECK we would like the bootstrapping containers to remain unhealthy until bootstrapping is complete. We want this so that front-end users don’t access that particular container in the service until it is fully online and sync’d with the cluster. The issue is that bootstrapping relies on the network between containers in order to do a state transfer and it won’t work when a container isn’t accessible on the network.
Question:
When a container’s status is either starting or unhealthy does HEALTHCHECK completely kill network access to and from the container?
As an example, when a container is healthy I can run the command getent hosts tasks.<service_name>
inside the container which returns the IP address of other containers in a service. However, when the same container is unhealthy that command does not return anything… Hence my suspicion that HEALTHCHECK kills the network at the container level (as opposed to at the service/load balancer level) if the container isn’t healthy.
Thanks in advance
I ran some more tests and found my own answer. Basically docker does not kill container networking when it is either in the started or unhealthy phase. The reason getent hosts tasks.<service_name> command does not work during those phases is that that command goes back to get the container IP address through the service which does not have the unhealthy container(s) assigned to it.
I've been trying to devise a strategy for using Docker Swarm for managing a bunch of headless containers - don't need load balancer, exposing any ports, or auto scaling.
The only thing I want is the ability to update all of the containers (on all nodes), if any of the images are updated. Each container running will need to have a specific --hostname.
Is running docker service even viable for this? Or should I just do a normal docker run targeting specific nodes to specify the --hostname i want? The reason I'm even asking about docker service is because it allows you to do an update (forcing an update for all containers if there are updated images).
Was also thinking that Docker Swarm would make it a bit easier to keep an eye on all the containers (i.e. manage them from a central location).
The other option I was looking at was watchtower, to run on each server that is running one of the containers, as an alternative to swarm. My only issue with this is that it doesn't provide any orchestration, for centralized management.
Anyone have any ideas of what would be a better option given the scenario?
Docker swarm does not give you any advantage regarding rolling updates apart from the docker service command, swarm only provides the user horizontal scaling and places a load balancer in front of those replicas called "service", as well as some other goodies such as replicating the docker events across the swarm nodes.
docker service --force would work as expected.
However, you should probably use both, docker swarm for orchestration and watchtower for rolling updates.
I have a bunch of docker containers running in swarm mode (services). If the whole server restarts then containers start to run one by one after server reboot. Is there a way to set an order of container creation and run?
P.S. I can't user docker-compose as these services were created dynamically through Docker Remote API.
You can try to set a shorter restart delay (with --restart-delay) to the services you want to start firstly and a bigger to next one etc..
But I am not sure that working.
Is it possible to have a docker swarm node leave the swarm, but keep running services started while being a member of the swarm?
Short answer: don't try to do that.
This would be against the design of swarm mode. For the same reason that docker disables the live-restore functionality when you have swarm mode, you shouldn't be able to keep services running when a node leaves the swarm cluster. The logic behind both decisions is when swarm mode detects that the target state of the service doesn't match the current state, it will do what it can to achieve that target state.
With the live-restore, docker would normally leave containers running when the daemon is stopped, and restore those containers to the docker daemon when the daemon restarts. And similarly, if the containers continued to run when the node leaves the swarm, from the view of the swarm manager, the result would be the same, containers running that the manager has no way to track the current state.
Since the current state cannot be tracked in those scenarios, docker errors on the side of stopping the container when it gracefully stops or leaves the swarm. The scenario where containers will continue to run is during an ungraceful disconnection from the swarm manager(s). In that scenario, the worker doesn't know if only it is down and the workload has been rescheduled elsewhere, or if only the manager is down, and stopping the containers would turn a small outage into a big one, so docker errors on the side of leaving the containers running when the disconnect is uncontrolled.
is there any way to disable/leave the swarm mode of docker when starting the daemon manually, e.g. dockerd --leave-swarm, instead of starting the daemon and leave the swarm mode afterwards, e.g. using docker swarm leave?
Many thanks in advance,
Aljoscha
I don't think this is anticipated by docker developers. When node leaves swarm, it needs to notify swarm managers, that it will not be available anymore.
Leaving swarm is a one time action and passing this as an configuration option to the daemon is weird. You may try to suggest that on docker's github, but I don't think it will have much supporters.
Perhaps more intuitive option would be to have ability to start dockerd in a way that communication to docker swarm manager would be suspended - so your dockerd is running only locally, but if you start without that flag (--local?) it would reconnect to swarm that it was attached before.