How can i run a docker stack (from a docker-compose.yml) on a single docker swarm node which automatically starts on system reboots?
I am using docker-compose to compose my application of multiple services and then use docker stack deploy to deploy it on my server to a single instance docker swarm instance.
In my docker-compose.yml i have defined my services with a restart policy:
deploy:
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 120s
placement:
constraints: [node.role == manager]
which imho should force the service to always run/restart. But when the server/docker daemon is restarted the services are not started. Is there some easy way to do this?
docker service list would show:
ID NAME MODE REPLICAS IMAGE PORTS
s9gg88ul584t finalyzer-prod_backend replicated 0/1 registry.gitlab.com/hpoul/finalyzer/finalyzer-backend:latest *:8081->8080/tcp
vb9iwg7zcwxd finalyzer-prod_mongoadmin replicated 0/1 mrvautin/adminmongo:latest *:8082->1234/tcp
qtasgtqi7m0l finalyzer-prod_mongodb replicated 0/1 mongo#sha256:232dfea36769772372e1784c2248bba53c6bdf0893d82375a3b66c09962b5af9
wdnrtlbe8jpw finalyzer-prod_pgdb replicated 0/1 postgres#sha256:73d065c344b419ce97bba953c7887c7deead75e0998053518938231cd7beb22c
so it recognizes that it should run 1 node, but it does not scale it up. What is the right way to force docker swarm, service or docker stack to scale all configured services up to their configured values upon a server restart, (or docker daemon restart)?
Related
I deploy a docker swarm on 5 nodes and I have 5 microservices. The docker swarm assigns the services only in one node. Is there is any way to tell to docker swarm which node to use for every service in order to assign 1 service in every node.
Yes you can do this with the "deploy" configuration option in your compose file. For example:
deploy:
placement:
constraints:
- "node.hostname == desired_machine_hostname"
I am working on a project which uses Raspberry Pis as worker nodes and my laptop as the master node. I hope to control the deployment of my containers from my laptop, but I hope the containers run on the worker nodes only(which means no container on the master node). How can I do it with Docker Swarm?
I am going to presume you are using a stack.yml file to describe your deployment using desired-state, but docker service create does have flags for this too.
There are a number of values that docker defines that can be tested under a placement-constraints node:
version: "3.9"
service:
worker:
image: nginx
deploy:
placement:
constraints:
- node.role==worker
I have 3 container for single service that created with --scale option on docker-compose. when I tried to recreate them, all of containers stop and remove, and after that, docker start to recreate one by one.
how can I to complete this process one by one, for example, stop first container and recreate them , after container up complete, go next one.
Unfortunately docker-compose don't have this feature, but Docker Swarm does!
Just init your docker machine to swarm cluster with
docker swarm init
and then reconfigure your compose file and add rolling updates like so:
deploy:
replicas: 2
update_config:
parallelism: 2
delay: 10s
order: stop-first
I have Docker CE running on windows server 2016 with 2 images.
when I run these in containers, everything is fine.
docker run --detach --name Website1 --publish 94:94 webimage1
docker run --detach --name Website2 --publish 95:95 webimage2
I can access through browser on other PCs:
http://host:94/page1.aspx
http://host:95/page1.aspx
Now I want to run them in swarm.
I've gone through docker tutorial and 've set up docker-compose file, with services, port mapping. The setup has Website 1 of 1 replica, Website 2 of 2 replicas.
On docker stack services websites port numbers show up as follows.
Website1: *:94->94/tcp
Website2: *:95->95/tcp
but i can't access any of them with following url's:
http://host:94/page1.aspx
http://host:95/page1.aspx
I get - This site can't be reached
If I go back to one of my running containers, I see that the port number has a different format.
0.0.0.0:94->94/tcp (WORKING) VS *:94->94/tcp (NOT WORKING)
For initilizing docker swarm I used docker swarm init with IP address of the host on port 2377.
Here is how I deployed docker stack using compose file
docker stack deploy --compose-file docker-stack.yml websites
docker-stack.yml file for reference.
version: "3"
services:
website1:
image: website1:latest
ports:
- 94:94
depends_on:
- website2
deploy:
replicas: 1
restart_policy:
condition: on-failure
delay: 10s
max_attempts: 3
window: 120s
website2:
image: website2:latest
ports:
- 95:95
deploy:
mode: replicated
replicas: 2
update_config:
parallelism: 2
delay: 10s
restart_policy:
condition: on-failure
delay: 10s
max_attempts: 3
window: 120s
any guidance would be greatly appreciated.
many thanks
From the conversations in the comments the problem is with the stack.
Docker stack supports only images for deployment and replicas. I believe your images are custom ones which are neither in dockerhub nor in any private repo. So when you try to deploy it in stack the service which is getting deployed in the worker node doesn't find such image and is unable to download it from repo. Hence the service won't start in worker node. It works perfectly in manager node because the image already exists there.
So, you have to either set up a local/private registry or push the images to docker registry or else you can even copy the images from manager node to worker node using docker save and docker load and then try using swarm and deploy in stack it will work.
Please note when working with swarm and registries, While deploying stack using docker stack deploy -c composefile.yml test you have to pass --with-registry-auth if you are using authentication for registries as docker stack deploy -c composefile.yml test --with-registry-auth else other nodes may not authenticate with the registry which will result in failure to download images if not found.
Also please note if you set up a local private repo without self signed certificate or with self signed certificate you may need to configure insecure registry. I've given reference of the same.
I recommend setting up a local repo without any authentication and certificates and access it by adding insecure registry in daemon.json file for testing purposes.
Now as per the last comment where you removed swarm and tried running docker service using
docker service create --replicas 2 --name contentlinksapi --publish mode=host,target=94,published=94,protocol=tcp contentlinksapi
It throwed port already in use because it tries to create 2 replicas in the same machine. Where the first replica binds to port 94 because of which, The second replica throws port already in use error.
For your reference.
Deploy a registry server
Test an insecure registry
Docker service mode (check to know why services with two replicas deployed in same host on docker service create)
Docker save
Docker load
I have docker swarm cluster with 2 nodes on AWS. I stopped the both instances and initially started swarm manager and then worker. Before stopped the instances i had a service running with 4 replicas distributed among manager and worker.
When i started swarm manager node first all replica containers started on manager itself and not moving to worker at all.
Please tell me how to do load balance?
Is swarm manager not responsible to do when worker started?
Swarm currently (18.03) does not move or replace containers when new nodes are started, if services are in the default "replicated mode". This is by design. If I were to add a new node, I don't necessarily want a bunch of other containers stopped, and new ones created on my new node. Swarm only stops containers to "move" replicas when it has to (in replicated mode).
docker service update --force <servicename> will rebalance a service across all nodes that match its requirements and constraints.
Further advice: Like other container orchestrators, you need to give capacity on your nodes in order to handle the workloads of any service replicas that move during outages. You're spare capacity should match the level of redundancy you plan to support. If you want to handle capacity for 2 nodes failing at once, for instance, you'd need a minimum percentage of resources on all nodes for those workloads to shift to other nodes.
Here's a bash script I use to rebalance:
#!/usr/bin/env bash
set -e
EXCLUDE_LIST="(_db|portainer|broker|traefik|prune|logspout|NAME)"
for service in $(docker service ls | egrep -v $EXCLUDE_LIST |
awk '{print $2}'); do
docker service update --force $service
done
Swarm doesn't do auto-balancing once containers are created. You can scale up/down once all your workers are up and it will distribute containers per your config requirements/roles/etc.
see: https://github.com/moby/moby/issues/24103
There are problems with new nodes getting "mugged" as they are added.
We also avoid pre-emption of healthy tasks. Rebalancing is done over
time, rather than killing working processes. Pre-emption is being
considered for the future.
As a workaround, scaling a service up and down should rebalance the
tasks. You can also trigger a rolling update, as that will reschedule
new tasks.
In docker-compose.yml, you can define:
version: "3"
services:
app:
image: repository/user/app:latest
networks:
- net
ports:
- 80
deploy:
restart_policy:
condition: any
mode: replicated
replicas: 5
placement:
constraints: [node.role == worker]
update_config:
delay: 2s
Remark: the constraint is node.role == worker
Using the flag “ — replicas” implies we don’t care on which node they are put on, if we want one service per node we can use “ — mode=global” instead.
In Docker 1.13 and higher, you can use the --force or -f flag with the docker service update command to force the service to redistribute its tasks across the available worker nodes.