I am trying to deploy my working docker-compose set up to a docker-swarm, everything seems ok, except that the only service that got replicate and generate a running container is the redis one, the 3 others got stuck and never generate any running container, they don't even download their respective images.
I can't find any debug feature, all the logs are empty, I'm completely helpless.
Let me show you the current state of my installation.
docker node ls print =>
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
oapl4et92vjp6mv67h2vw8boq boot2docker Ready Active
x2fal9iwj6aqt1vgobyp1smv1 * manager1 Ready Active Leader
lmtuojednaiqmfmm2izl7npf0 worker1 Ready Active
The docker compose =>
version: '3'
services:
mongo:
image: mongo
container_name: mongo
restart: always
volumes:
- /data/db:/data/db
deploy:
placement:
constraints: [node.role == manager]
ports:
- "27017:27017"
redis:
image: redis
container_name: redis
restart: always
bnbkeeper:
image: registry.example.tld/keepers:0.10
container_name: bnbkeeper
deploy:
replicas: 5
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
depends_on:
- mongo
- redis
ports:
- "8080:8080"
links:
- mongo
- redis
environment:
- REDIS_HOST=redis
- MONGO_HOST=mongo
bnbkeeper-ws:
image: registry.example.tld/keepers:0.10
container_name: bnbkeeper-ws
restart: unless-stopped
depends_on:
- mongo
- redis
ports:
- "3800:3800"
links:
- mongo
- redis
environment:
- REDIS_HOST=redis
command: npm run start:subscription
The current state of my services
ID NAME MODE REPLICAS IMAGE PORTS
tbwfswsxx23f stack_bnbkeeper replicated 0/5 registry.example.tld/keepers:0.10
khrqtx28qoia stack_bnbkeeper-ws replicated 0/1 registry.example.tld/keepers:0.10
lipa8nvncpxb stack_mongo replicated 0/1 mongo:latest
xtz2411htcg7 stack_redis replicated 1/1 redis:latest
My redis successful service (docker service ps stack_redis)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
cqv0njcgsw6f stack_redis.1 redis:latest worker1 Running Running 25 minutes ago
my mongo unsuccessful service (docker service ps stack_mongo)
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
yipokxxiftqq stack_mongo.1 mongo:latest Running New 25 minutes ago
I'm completely new to docker swarm, and probably made a silly mistake here, but I couldn't find much documentation on how to setup such a simple stack.
To monitor, try this:
journalctl -f -n10
Then run the docker stack deploy command in a separate session and see what it shows
try removing port publish and add --endpoint-mode dnsrr to your service.
Related
Is there a way to control the distribution of services across different computers? I have one master with two workers and 5 services:
web server
database
redis
celery
s3 storage connection
I only want to outsource the celery workers and run everything else on the master. Is there a way to control that with docker swarm? I have not created a registry yet, because I am not sure if that is still necessary.
Here is my current experimental docker-compose file.
version: "3.8"
volumes:
s3data:
driver: local
services:
web:
image: localhost:5000/web
build: .
env_file:
- ./.env
environment:
- ENVIRONMENT=develop
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./app/:/app/
- ./lib/lrg_omics/:/lrg-omics/
- s3data:/datalake/
- /data/media/:/appmedia/
- /data/static/:/static/
ports:
- "8000:8000"
depends_on:
- db
- redis
- s3vol
links:
- redis:redis
restart: always
db:
image: postgres
volumes:
- /data/db/:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
redis:
restart: always
image: redis:alpine
ports:
- "6379:6379"
celery:
restart: on-failure
image: pp-celery-worker
build:
context: .
dockerfile: Dockerfile
command: bash -c "celery -A main worker -l info --concurrency 8"
env_file:
- ./.env
volumes:
- ./app/:/app/
- ./lib/lrg_omics/:/lrg-omics/
- s3data:/datalake/
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=postgres
depends_on:
- db
- redis
- web
- s3vol
deploy:
replicas: 2
placement:
max_replicas_per_node: 1
s3vol:
image: elementar/s3-volume
command: /data s3://PQC
environment:
- BACKUP_INTERVAL=2
- AWS_ACCESS_KEY_ID=...
- AWS_SECRET_ACCESS_KEY=...
- ENDPOINT_URL=https://example.com
volumes:
- s3data:/data
When I deploy this with sudo docker stack deploy --compose-file docker-compose-distributed.yml QC
And then look at the services I get something like this:
sudo docker stack services QC
>>>
ID NAME MODE REPLICAS IMAGE PORTS
xx5hkbswipoz QC_celery replicated 0/2 (max 1 per node) celery-worker:latest
natb3trv9ngi QC_db replicated 0/1 postgres:latest
1bxpkb18ojay QC_redis replicated 1/1 redis:alpine *:6379->6379/tcp
6rsl5gfpd0oa QC_s3vol replicated 1/1 elementar/s3-volume:latest
aszkle6msmqr QC_web replicated 0/1 localhost:5000/web:latest *:8000->8000/tcp
For some reason only redis and the S3 containers run. And both of them on the master. Nothing runs on the workers.
I am quite new to docker swarm so there is probably more than one thing wrong here. Any comments on best practices are welcome.
To determine why the services are not starting
docker service ps QC_celery --no-trunc will show the state of the service and a message from docker.
To control placement consult the Compose file version 3 reference on placement constraints. Basically it entails adding to the deploy: node:
deploy:
replicas: 2
placement:
max_replicas_per_node: 1
constraints:
- node.role==worker
While, nominally, compose.yml and stack.yml files share a format, they support different feature subsets and for complex deployments it becomes helpful to split the deployment into discreet compose.yml files for docker compose and stack.yml files for swarm deployments.
docker stack deploy -c docker-compose.yml -c docker-stack.yml QC can merge a docker-compose.yml base file with stack specific settings, and you can keep docker compose artifacts in your docker-compose.override.yml. these artifacts include:
build: - docker swarm needs the image to be built and available in a registry, either local(swarm hosted?) or docker-hub.
depends_on:, links: - not supported by swarm, which assumes services can be restarted at any time, and will find each other using docker networks.
restart: controlled by restart_policy: under deploy:
docker-compose.yml
version: "3"
services:
daggr:
image: "docker.pvt.com/test/daggr:stable"
hostname: '{{.Node.Hostname}}'
deploy:
mode: global
resources:
limits:
cpus: "2"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "4000:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
networks:
- webnet
networks:
webnet:
Right now, I am deploying docker containers with the below command:
docker stack deploy -c docker-compose.yml daggrstack
Is there way to specify in docker-compose.yml file that if the image, docker.pvt.com/test/daggr:stable, is updated (ie. docker build, docker tag, and docker push :stable), then the running containers are automatically re-deployed with the updated image?
So, i dont have to re-run docker stack deploy every time i pushed a new docker image
Is there way to specify in docker-compose.yml file that if the image, docker.pvt.com/test/daggr:stable,
is updated (ie. docker build, docker tag, and docker push :stable), then the running containers are automatically re-deployed with the updated image?
The answer is No. Docker swarm does not auto-update the service when a new image is available. This should handled as part a continuous deployment system.
However, Docker does make it easy to update the images of already running services.
As described in Apply rolling updates to a service, you can update the image of a services, via:
docker service update --image docker.pvt.com/test/daggr:stable daggr
Right now, I have two-node in swarm cluster
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
yey1njv9uz8adf33m7oz0h80f * redis2 Ready Active Leader
lbo91v2l15h24isfd5jegoxu1 redis3 Ready Active
this is docker-compose.yml file
version: "3"
services:
daggr:
# replace username/repo:tag with your name and image details
image: daggr
hostname: examplehostname
deploy:
replicas: 1
resources:
limits:
cpus: "0.1"
memory: 50M
restart_policy:
condition: on-failure
ports:
- "4000:80"
networks:
- webnet
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
redis:
image: redis
networks:
- webnet
networks:
webnet:
as you see, i am explictly setting hostname for daggr service
daggr container basically runs a python tornado web server
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write('Hello from daggr running on %s\n' % socket.gethostname())
I've tested out as below
$ curl redis2:4000
Hello from daggr running on examplehostname
Now, instead of statically setting the hostname, i want it to be dynamic matching to hostname of where the container is running. ie. if daggr container is running on redis2 it should say redis2 and on redis3, it should say redis3.
How can i specify that from docker-compose.yml file?
If you are running at least Docker 17.10 then you can use something like this:
services:
daggr:
hostname: '{{.Node.Hostname}}'
See this for more information.
The selected answer above did not work for me (presumably since I am not running docker/docker-compose in swarm mode).
I was able to set the container hostname for my reverse proxy to match the docker host FQDN by doing the following:
version: '3'
reverse-proxy:
hostname: $HOSTNAME
This allowed me to easily configure Nginx to pick the correct server cert / key pair prior to startup.
I tried Constantin Galbenu's solution only work on a swarm setup and brandonsimpkins's lacks the HOSTNAME definition.
A workaround is to set an environment variable to your hostname:
export HOSTNAME="$(cat /etc/hostname)"
my-container:
hostname: ${HOSTNAME}
If like me, your hostname is the same as your username, skip step 1 and only do
my-container:
hostname: ${USER}
I have an application where we use the subdomain string to specify which team a customer belongs to.
For example:
customer1.ourdomain.com
customer2.ourdomain.com
... etc
Behind the scenes in our application we are parsing the string customerX from the origin URL to retrieve the customer's appropriate data.
When I run a docker container from the cli without compose, like the below, I am able to connect to my container and have the expected behavior:
docker run -d -p 7000:7000 MyApplicationImage:latest
However, when I try to access this same behavior through the means of the docker-compose.yaml:
docker stack deploy -c docker-compose.yaml MyApplication`
My browser will not connect and times out/hangs.
Essentially:
localhost:7070 -> works
teamName.localhost:7070 -> Does not connect
This is what my docker-compose.yaml file looks like as of right now:
version: "3"
services:
sone:
image: MyApplicationImage:latest
deploy:
replicas: 1
restart_policy:
condition: on-failure
ports:
- "7000:7000"
networks:
- webnet
stwo:
image: myimagetwo:latest
deploy:
replicas: 1
restart_policy:
condition: on-failure
ports:
- "7001:7001"
networks:
- webnet
networks:
webnet:
I'm using docker 1.12.1
I have an easy docker-compose script.
version: '2'
services:
jenkins-slave:
build: ./slave
image: jenkins-slave:1.0
restart: always
ports:
- "22"
environment:
- "constraint:NODE==master1"
jenkins-master:
image: jenkins:2.7.1
container_name: jenkins-master
restart: always
ports:
- "8080:8080"
- "50000"
environment:
- "constraint:NODE==node1"
I run this script with docker-compose -p jenkins up -d.
This Creates my 2 containers but only on my master (from where I execute my command). I would expect that one would be created on the master and one on the node.
I also tried to add
networks:
jenkins_swarm:
driver: overlay
and
networks:
- jenkins_swarm
After every service but this is failing with:
Cannot create container for service jenkins-master: network jenkins_jenkins_swarm not found
While the network is created when I perform docker network ls
Someone who can help me to deploy 2 containers on my 2 nodes with docker-compose. Swarm is defenitly working on my "cluster". I followed this tutorial to verify.
Compose doesn't support Swarm Mode at the moment.
When you run docker compose up on the master node, Compose issues docker run commands for the services in the Compose file, rather than docker service create - which is why the containers all run on the master. See this answer for options.
On the second point, networks are scoped in 1.12. If you inspect your network you'll find it's been created at swarm-level, but Compose is running engine-level containers which can't see the swarm network.
We can do this with docker compose v3 now.
https://docs.docker.com/engine/swarm/#feature-highlights
https://docs.docker.com/compose/compose-file/
You have to initialize the swarm cluster using command
$ docker swarm init
You can add more nodes as worker or manager -
https://docs.docker.com/engine/swarm/join-nodes/
Once you have your both nodes added to the cluster, pass your compose v3 i.e deployment file to create a stack. Compose file should just contain predefined images, you can't give a Dockerfile for deployment in Swarm mode.
$ docker stack deploy -c dev-compose-deploy.yml --with-registry-auth PL
View your stack services status -
$ docker stack services PL
Try to use Labels & Placement constraints to put services on different nodes.
Example "dev-compose-deploy.yml" file for your reference
version: "3"
services:
nginx:
image: nexus.example.com/pl/nginx-dev:latest
extra_hosts:
- "dev-pldocker-01:10.2.0.42”
- "int-pldocker-01:10.2.100.62”
- "prd-plwebassets-01:10.2.0.62”
ports:
- "80:8003"
- "443:443"
volumes:
- logs:/app/out/
networks:
- pl
deploy:
replicas: 3
labels:
feature.description: “Frontend”
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: any
placement:
constraints: [node.role == worker]
command: "/usr/sbin/nginx"
viz:
image: dockersamples/visualizer
ports:
- "8085:8080"
networks:
- pl
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
replicas: 1
labels:
feature.description: "Visualizer"
restart_policy:
condition: any
placement:
constraints: [node.role == manager]
networks:
pl:
volumes:
logs: