Use docker-compose with docker swarm - jenkins

I'm using docker 1.12.1
I have an easy docker-compose script.
version: '2'
services:
jenkins-slave:
build: ./slave
image: jenkins-slave:1.0
restart: always
ports:
- "22"
environment:
- "constraint:NODE==master1"
jenkins-master:
image: jenkins:2.7.1
container_name: jenkins-master
restart: always
ports:
- "8080:8080"
- "50000"
environment:
- "constraint:NODE==node1"
I run this script with docker-compose -p jenkins up -d.
This Creates my 2 containers but only on my master (from where I execute my command). I would expect that one would be created on the master and one on the node.
I also tried to add
networks:
jenkins_swarm:
driver: overlay
and
networks:
- jenkins_swarm
After every service but this is failing with:
Cannot create container for service jenkins-master: network jenkins_jenkins_swarm not found
While the network is created when I perform docker network ls
Someone who can help me to deploy 2 containers on my 2 nodes with docker-compose. Swarm is defenitly working on my "cluster". I followed this tutorial to verify.

Compose doesn't support Swarm Mode at the moment.
When you run docker compose up on the master node, Compose issues docker run commands for the services in the Compose file, rather than docker service create - which is why the containers all run on the master. See this answer for options.
On the second point, networks are scoped in 1.12. If you inspect your network you'll find it's been created at swarm-level, but Compose is running engine-level containers which can't see the swarm network.

We can do this with docker compose v3 now.
https://docs.docker.com/engine/swarm/#feature-highlights
https://docs.docker.com/compose/compose-file/
You have to initialize the swarm cluster using command
$ docker swarm init
You can add more nodes as worker or manager -
https://docs.docker.com/engine/swarm/join-nodes/
Once you have your both nodes added to the cluster, pass your compose v3 i.e deployment file to create a stack. Compose file should just contain predefined images, you can't give a Dockerfile for deployment in Swarm mode.
$ docker stack deploy -c dev-compose-deploy.yml --with-registry-auth PL
View your stack services status -
$ docker stack services PL
Try to use Labels & Placement constraints to put services on different nodes.
Example "dev-compose-deploy.yml" file for your reference
version: "3"
services:
nginx:
image: nexus.example.com/pl/nginx-dev:latest
extra_hosts:
- "dev-pldocker-01:10.2.0.42”
- "int-pldocker-01:10.2.100.62”
- "prd-plwebassets-01:10.2.0.62”
ports:
- "80:8003"
- "443:443"
volumes:
- logs:/app/out/
networks:
- pl
deploy:
replicas: 3
labels:
feature.description: “Frontend”
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: any
placement:
constraints: [node.role == worker]
command: "/usr/sbin/nginx"
viz:
image: dockersamples/visualizer
ports:
- "8085:8080"
networks:
- pl
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
replicas: 1
labels:
feature.description: "Visualizer"
restart_policy:
condition: any
placement:
constraints: [node.role == manager]
networks:
pl:
volumes:
logs:

Related

Control distribution of docker swarm services across different computers?

Is there a way to control the distribution of services across different computers? I have one master with two workers and 5 services:
web server
database
redis
celery
s3 storage connection
I only want to outsource the celery workers and run everything else on the master. Is there a way to control that with docker swarm? I have not created a registry yet, because I am not sure if that is still necessary.
Here is my current experimental docker-compose file.
version: "3.8"
volumes:
s3data:
driver: local
services:
web:
image: localhost:5000/web
build: .
env_file:
- ./.env
environment:
- ENVIRONMENT=develop
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./app/:/app/
- ./lib/lrg_omics/:/lrg-omics/
- s3data:/datalake/
- /data/media/:/appmedia/
- /data/static/:/static/
ports:
- "8000:8000"
depends_on:
- db
- redis
- s3vol
links:
- redis:redis
restart: always
db:
image: postgres
volumes:
- /data/db/:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
redis:
restart: always
image: redis:alpine
ports:
- "6379:6379"
celery:
restart: on-failure
image: pp-celery-worker
build:
context: .
dockerfile: Dockerfile
command: bash -c "celery -A main worker -l info --concurrency 8"
env_file:
- ./.env
volumes:
- ./app/:/app/
- ./lib/lrg_omics/:/lrg-omics/
- s3data:/datalake/
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=postgres
depends_on:
- db
- redis
- web
- s3vol
deploy:
replicas: 2
placement:
max_replicas_per_node: 1
s3vol:
image: elementar/s3-volume
command: /data s3://PQC
environment:
- BACKUP_INTERVAL=2
- AWS_ACCESS_KEY_ID=...
- AWS_SECRET_ACCESS_KEY=...
- ENDPOINT_URL=https://example.com
volumes:
- s3data:/data
When I deploy this with sudo docker stack deploy --compose-file docker-compose-distributed.yml QC
And then look at the services I get something like this:
sudo docker stack services QC
>>>
ID NAME MODE REPLICAS IMAGE PORTS
xx5hkbswipoz QC_celery replicated 0/2 (max 1 per node) celery-worker:latest
natb3trv9ngi QC_db replicated 0/1 postgres:latest
1bxpkb18ojay QC_redis replicated 1/1 redis:alpine *:6379->6379/tcp
6rsl5gfpd0oa QC_s3vol replicated 1/1 elementar/s3-volume:latest
aszkle6msmqr QC_web replicated 0/1 localhost:5000/web:latest *:8000->8000/tcp
For some reason only redis and the S3 containers run. And both of them on the master. Nothing runs on the workers.
I am quite new to docker swarm so there is probably more than one thing wrong here. Any comments on best practices are welcome.
To determine why the services are not starting
docker service ps QC_celery --no-trunc will show the state of the service and a message from docker.
To control placement consult the Compose file version 3 reference on placement constraints. Basically it entails adding to the deploy: node:
deploy:
replicas: 2
placement:
max_replicas_per_node: 1
constraints:
- node.role==worker
While, nominally, compose.yml and stack.yml files share a format, they support different feature subsets and for complex deployments it becomes helpful to split the deployment into discreet compose.yml files for docker compose and stack.yml files for swarm deployments.
docker stack deploy -c docker-compose.yml -c docker-stack.yml QC can merge a docker-compose.yml base file with stack specific settings, and you can keep docker compose artifacts in your docker-compose.override.yml. these artifacts include:
build: - docker swarm needs the image to be built and available in a registry, either local(swarm hosted?) or docker-hub.
depends_on:, links: - not supported by swarm, which assumes services can be restarted at any time, and will find each other using docker networks.
restart: controlled by restart_policy: under deploy:

How to bind volume in docker-compose for portainer docker swarm deployment?

I am new to docker / docker swarm and I just setup a cluster with 4 pi nodes as a docker swarm. I also have on all 4 pis a shared volume taped with the same path /mnt/docker.
Now I want to deploy portainer with docker compose. I use the default docker compose file from portioners documentation. It works great but now I would like to change the compose file to use /mnt/docker/portainer/data as a persistent storage for all portainer data. So in case of a rebuild or failure I could just start from there again.
But I can't seem to get it working. There are multiple mount points under volumes, but none of them seems to work as the portainer data never gets written to the nfs share path.
What do I need to add to get a persistent data volume bind in docker compose?
This is my compose file so far:
services:
agent:
image: portainer/agent
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
networks:
- agent_network
deploy:
mode: global
placement:
constraints: [node.platform.os == linux]
portainer:
image: portainer/portainer-ce
command: -H tcp://tasks.agent:9001 --tlsskipverify
ports:
- "9000:9000"
- "8000:8000"
volumes:
- portainer_data:/data
networks:
- agent_network
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
networks:
agent_network:
driver: overlay
attachable: true
volumes:
portainer_data:```
Thanks a lot for your help!

Services don't start on docker swarm nodes

I want to deploy HA Postgresql with Failover Patroni and HAProxy (like single entrypoint) in docker swarm.
I have docker-compose.yml -
version: "3.7"
services:
etcd1:
image: patroni
networks:
- test
env_file:
- docker/etcd.env
container_name: test-etcd1
hostname: etcd1
command: etcd -name etcd1 -initial-advertise-peer-urls http://etcd1:2380
etcd2:
image: patroni
networks:
- test
env_file:
- docker/etcd.env
container_name: test-etcd2
hostname: etcd2
command: etcd -name etcd2 -initial-advertise-peer-urls http://etcd2:2380
etcd3:
image: patroni
networks:
- test
env_file:
- docker/etcd.env
container_name: test-etcd3
hostname: etcd3
command: etcd -name etcd3 -initial-advertise-peer-urls http://etcd3:2380
patroni1:
image: patroni
networks:
- test
env_file:
- docker/patroni.env
hostname: patroni1
container_name: test-patroni1
environment:
PATRONI_NAME: patroni1
deploy:
placement:
constraints: [node.role == manager]
# - node.labels.type == primary
# - node.role == manager
patroni2:
image: patroni
networks:
- test
env_file:
- docker/patroni.env
hostname: patroni2
container_name: test-patroni2
environment:
PATRONI_NAME: patroni2
deploy:
placement:
constraints: [node.role == worker]
# - node.labels.type != primary
# - node.role == worker
patroni3:
image: patroni
networks:
- test
env_file:
- docker/patroni.env
hostname: patroni3
container_name: test-patroni3
environment:
PATRONI_NAME: patroni3
deploy:
placement:
constraints: [node.role == worker]
# - node.labels.type != primary
# - node.role == worker
haproxy:
image: patroni
networks:
- test
env_file:
- docker/patroni.env
hostname: haproxy
container_name: test-haproxy
ports:
- "5000:5000"
- "5001:5001"
command: haproxy
networks:
test:
driver: overlay
attachable: true
And deploy this services in docker swarm with this command:
docker stack deploy --compose-file docker-compose.yml test
When i use this command, my services is creating, but service patroni2 and patroni3 don't start on other nodes, which roles are worker. They don't start at all!
I want to see my services deploy on all nodes (3 - one manager and two workers) which existing in docker swarm
But if i delete constraints, all my services start on one node, when i deploy docker-compose.yml in swarm.
May be this services can't see my network, though i deploy it using docker official documentation.
With different service names, docker will not attempt to spread containers across multiple nodes, and will fall back to the least used node that satisfies the requirements, where least used is measured by the number of scheduled containers.
You could attempt to solve this by using the same service name and 3 replicas. This would require that they be defined identically. To make this work, you can leverage a few features, the first being that etcd.tasks will resolve to the individual ip addresses of each etcd service container. And the second are service templates which can be used to inject values like {{.Task.Slot}} into the settings for hostname, volume mounts, and env variables. The challenge is the list at the end will likely not give you what you want, which is a way to uniquely address each replica from the other replicas. Hostname seems like it would work, but it unfortunately does not resolve in docker's DNS implementation (and wouldn't be easy to implement since it's possible to create a container with the capabilities to change the hostname after docker has deployed it).
The option you are left with is configuring constraints on each service to run on specific nodes. That's less than ideal, and reduces the fault tolerance of these services. If you have lots of nodes that can be separated into 3 groups then using node labels would solve the issue.

Docker: Option for automatic redeployment in docker-compose.yml for new image pushed

docker-compose.yml
version: "3"
services:
daggr:
image: "docker.pvt.com/test/daggr:stable"
hostname: '{{.Node.Hostname}}'
deploy:
mode: global
resources:
limits:
cpus: "2"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "4000:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
networks:
- webnet
networks:
webnet:
Right now, I am deploying docker containers with the below command:
docker stack deploy -c docker-compose.yml daggrstack
Is there way to specify in docker-compose.yml file that if the image, docker.pvt.com/test/daggr:stable, is updated (ie. docker build, docker tag, and docker push :stable), then the running containers are automatically re-deployed with the updated image?
So, i dont have to re-run docker stack deploy every time i pushed a new docker image
Is there way to specify in docker-compose.yml file that if the image, docker.pvt.com/test/daggr:stable,
is updated (ie. docker build, docker tag, and docker push :stable), then the running containers are automatically re-deployed with the updated image?
The answer is No. Docker swarm does not auto-update the service when a new image is available. This should handled as part a continuous deployment system.
However, Docker does make it easy to update the images of already running services.
As described in Apply rolling updates to a service, you can update the image of a services, via:
docker service update --image docker.pvt.com/test/daggr:stable daggr

how to set the service mode when using docker compose?

I need to set service mode to global while using compose files .
Any chance we can use this in compose file ?
I have a requirement where for a service there should be exactly one container on every node/host .
This doesn't happen with "spread strategy" of swarm if a node goes down & comes up , it just attains the equal number of containers on each host irrespective of services .
https://github.com/docker/compose/issues/3743
We can do this easily now with docker compose v3 (version 3) under the deploy(mode) section.
Prerequisites -
docker compose version should be 1.10.0+
docker engine version should be 1.13.0+
Example compose file -
version: "3"
services:
nginx:
image: nexus3.example.com/prd-nginx-sm:v1
ports:
- "80:80"
networks:
- cheers
volumes:
- logs:/rest/out/
deploy:
mode: global
labels:
feature.description: "Frontend"
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: any
command: "/usr/sbin/nginx"
networks:
cheers:
volumes:
logs:
data:
Deploy the compose file -
$ docker stack deploy -c sm-deploy-compose.yml --with-registry-auth CHEERS
This will deploy nginx container on all the nodes participating in the cluster .

Resources