I'm new to docker swarm and looking to set containers to run on a specific node in the swarm.
For example, I have the following nodes:
Manager
Worker1
Worker2
And I have a couple services listed in a compose yml similar to:
services:
my_service:
image: my_image
container_name: my_container_name
networks:
- my_network
my_service2:
image: my_image2
container_name: my_container_name2
networks:
- my_network
How can I make it so that my_service only runs on Worker1 and my_service2 only runs on Worker2?
UPDATE:
I managed to find the solution. Can specify deployment constraints as shown below.
my_service:
image: my_image
container_name: my_container_name
networks:
- my_network
deploy:
placement:
constraints:
- node.hostname == Worker1
my_service2:
image: my_image2
container_name: my_container_name2
networks:
- my_network
deploy:
placement:
constraints:
- node.hostname == Worker2
Related
Following this question, I edited my gateway container to use the host network mode:
services:
gateway:
...
network_mode: "host"
and then the docker compose up -d gives me this:
Error response from daemon: failed to add interface veth701c890 to
sandbox: error setting interface "veth701c890" IP to 172.26.0.11/16:
cannot program address 172.26.0.11/16 in sandbox interface because it
conflicts with existing route {Ifindex: 4 Dst: 172.26.0.0/16 Src:
172.26.0.1 Gw: Flags: [] Table: 254
I restarted the docker and even the server. No luck.
The docker-compose.yml looks like this (only the gateway container has published ports):
version: '3.4'
services:
gateway:
image: <ms-yarp>
environment:
- ASPNETCORE_URLS=https://+:443;http://+:80
ports:
- "80:80"
- "443:443"
volumes:
- ./tls/:/tls/
networks:
- mynet
restart: on-failure
orders:
image: <registry>/orders
environment:
- ASPNETCORE_URLS=http://+:80
networks:
- mynet
restart: on-failure
users:
image: <registry>/users
environment:
- ASPNETCORE_URLS=http://+:80
networks:
- mynet
restart: on-failure
smssender:
image: <registry>/smssender
environment:
- ASPNETCORE_URLS=http://+:80
networks:
- mynet
restart: on-failure
logger:
image: <registry>/logger
environment:
- ASPNETCORE_URLS=http://+:80
networks:
- mynet
restart: on-failure
notifications:
image: <registry>/notifications
environment:
- ASPNETCORE_URLS=http://+:80
networks:
- mynet
restart: on-failure
cacheserver:
image: <registry>/redis
networks:
- mynet
restart: on-failure
...
networks:
mynet:
You can't combine host networking with any other Docker networking option. At least some versions of Compose have given warnings if you combine network_mode: host with other networks: or ports: options.
The other thing host networking means in this particular setup is that the one container that's using it is "outside Docker" for purposes of connecting to other containers. It works exactly the same way a non-container process would. That means the other containers need to publish ports: to be reachable from the gateway, and in turn the gateway configuration needs to use localhost and the published port numbers to reach the other containers.
version: '3.8'
services:
gateway:
image: <ms-yarp>
network_mode: host
orders:
image: <registry>/orders
ports:
- '8001:80'
networks:
- mynet
{
"ReverseProxy": {
"Clusters": {
"cluster": {
"Destinations": {
"orders": {
"Address": "http://localhost:8001"
}
}
}
}
}
}
Something like this:
(doesn't work with Docker Desktop on windows WSL2, at least I couldn't even run the nginx example that is here in the docs)
version: '3.4'
services:
gateway:
image: <ms-yarp>
environment:
- ASPNETCORE_URLS=https://+:443;http://+:80
network_mode: host
volumes:
- ./tls/:/tls/
restart: on-failure
orders:
image: <registry>/orders
environment:
- ASPNETCORE_URLS=http://+:80
ports:
- 8080:80
networks:
- mynet
restart: on-failure
users:
image: <registry>/users
environment:
- ASPNETCORE_URLS=http://+:80
ports:
- 8081:80
networks:
- mynet
restart: on-failure
smssender:
image: <registry>/smssender
environment:
- ASPNETCORE_URLS=http://+:80
ports:
- 8082:80
networks:
- mynet
restart: on-failure
logger:
image: <registry>/logger
environment:
- ASPNETCORE_URLS=http://+:80
ports:
- 8082:80
networks:
- mynet
restart: on-failure
notifications:
image: <registry>/notifications
environment:
- ASPNETCORE_URLS=http://+:80
ports:
- 8083:80
networks:
- mynet
restart: on-failure
cacheserver:
image: <registry>/redis
restart: on-failure
networks:
- mynet
Also in your gateway service configuration you will need to change the
http://orders:80 to http://localhost:8080
http://users:80 to http://localhost:8081
and so on
Also restrict ports on the docker host of 8080 to 8083 to be accessible only from localhost and not from the internet.
You could even put all the containers (except the gateway) to a different docker host that is accessible only from the docker host where the gateway container is running and change the config in gateway from http://orders:80 to http://otherdockerhost:80 and so on.
But for this docker compose will not be viable you will need to "manually" create the containers with docker run commands (or have 2 separate compose project one for the gateway and one for the rest of the services) so this is where more serious container orchestration tools are required like kubernetes (you could try docker swarm or nomad or any other container orchestrator, but these are not so popular so if you are new to both kubernetes and docker swarm or all the other you are better off with starting with kubernetes, you will reap the benefits in the long run for both this project and your personal carrier too)
I'm trying to deploy a compose project to a swarm but after I deploy it I have the problem of not all the services start and some of them keep restarting.
I have the following compose file
version: "3.3"
volumes:
jenkins_home:
external: false
driver: local
driver_opts:
type: none
o: 'bind'
device: '/var/jenkins_home'
docker_certs:
external: false
driver: local
driver_opts:
type: none
o: 'bind'
device: '/etc/certs'
services:
docker:
image: docker:dind
restart: unless-stopped
privileged: true
volumes:
- jenkins_home:/var/jenkins_home
- docker_certs:/certs/client
ports:
- "2376:2376"
environment:
DOCKER_TLS_CERTDIR: /certs
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
jenkins:
image: git.example.com:8444/devops/docker-services/jenkins
build:
context: ./
dockerfile: services/jenkins.dockerfile
restart: unless-stopped
depends_on:
- "docker"
volumes:
- jenkins_home:/var/jenkins_home
- docker_certs:/certs/client
ports:
- "636:636"
- "8443:8443"
- "3268:3268"
- "50000:50000"
environment:
DOCKER_HOST: tcp://docker:2376
DOCKER_CERT_PATH: /certs/client
DOCKER_TLS_VERIFY: 1
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
icecc-scheduler:
image: git.example.com:8444/devops/docker-services/icecc-scheduler
build:
context: ./
dockerfile: services/icecc-scheduler.dockerfile
restart: unless-stopped
ports:
- "8765:8765"
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
icecc-daemon:
image: git.example.com:8444/devops/docker-services/icecc-daemon
build:
context: ./
dockerfile: services/icecc-daemon.dockerfile
restart: unless-stopped
ports:
- "8766:8766"
- "10245:10245"
depends_on:
- "icecc-scheduler"
deploy:
mode: global
and a swarm with two nodes docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
i6edk9ny6z38krv6m5738uzwu st12873 Ready Active 20.10.12
phnvvy2139wft9innou0uermq * st12874 Ready Active Leader 20.10.12
I have all the images built and pushed to the docker registry
When I run docker stack deploy -c docker-compose.yml build-farm it says it deploys sucessfully though I then list the services
docker stack services build-farm
ID NAME MODE REPLICAS IMAGE PORTS
4z6w98jmswav build-farm_docker replicated 0/1 docker:dind *:2376->2376/tcp
r7xuq4vgc92i build-farm_icecc-daemon global 0/2 git.example.com:8444/devops/docker-services/icecc-daemon:latest *:8766->8766/tcp, *:10245->10245/tcp
20ukipii7wli build-farm_icecc-scheduler replicated 0/1 git.example.com:8444/devops/docker-services/icecc-scheduler:latest *:8765->8765/tcp
37r4pm7jgku5 build-farm_jenkins replicated 1/1 git.example.com:8444/devops/docker-services/jenkins:latest *:636->636/tcp, *:3268->3268/tcp, *:8443->8443/tcp, *:50000->50000/tcp
The icecc scheduler and daemon never start on and the docker:dind service keeps starting and stopping
I have a Swarm cluster with a Manager and a Worker node.
All the containers running on the manager are accessible through Traefik and working fine.
I just deployed a new Worker node and joined my swarm on the node.
Now I start scaling some services and realized they were timing out on the worker node.
So I setup a simple example using the whoami container, and cannot figure out why I cannot access it. Here are my configs (all deployed on the MANAGER node):
version: '3.6'
networks:
traefik-net:
driver: overlay
attachable: true
external: true
services:
whoami:
image: jwilder/whoami
networks:
- traefik-net
deploy:
labels:
- "traefik.port=8000"
- "traefik.frontend.rule=Host:whoami.myhost.com"
- "traefik.docker.network=traefik-net"
replicas: 2
placement:
constraints: [node.role != manager]
My traefik:
version: '3.6'
networks:
traefik-net:
driver: overlay
attachable: true
external: true
services:
reverse-proxy:
image: traefik # The official Traefik docker image
command: --docker --docker.swarmmode --docker.domain=myhost.com --docker.watch --api
ports:
- "80:80" # The HTTP port
# - "8080:8080" # The Web UI (enabled by --api)
- "443:443"
networks:
- traefik-net
volumes:
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen
- /home/ubuntu/docker-configs/traefik/traefik.toml:/traefik.toml
- /home/ubuntu/docker-configs/traefik/acme.json:/acme.json
deploy:
labels:
traefik.port: 8080
traefik.frontend.rule: "Host:traefik.myhost.com"
traefik.docker.network: traefik-net
replicas: 1
placement:
constraints: [node.role == manager]
My worker docker ps output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b825f95b0366 jwilder/whoami:latest "/app/http" 4 hours ago Up 4 hours 8000/tcp whoami_whoami.2.tqbh4csbqxvsu6z5i7vizc312
50cc04b7f0f4 jwilder/whoami:latest "/app/http" 4 hours ago Up 4 hours 8000/tcp whoami_whoami.1.rapnozs650mxtyu970isda3y4
I tried opening firewall ports, disabling it completely, nothing seems to work. Any help is appreciated
I had to use --advertise-addr y.y.y.y to make it work
simplified swarm:
manager1 node
- consul-agent
worker1 node
- consul-client1
- web-app:80
- web-network:9000
database1 node
- consul-client2
- redis:6379
- mongo:27017
The web-app and web-network services can connect to redis and mongo through their service names correctly, e.g redis.createClient('6379', 'redis') and mongoose.connect('mongodb://mongo').
However, container web-app cannot connect to web-network, I'm trying to make a request like so:
request('http://web-network:9000')
But get the error:
errorno: ECONNREFUSED
address: 10.0.1.9
port: 9000
Request to web-network using a private IP does work:
request('http://11.22.33.44:9000')
What am I missing? Why can they connect to redis and mongo but not between each container? When moving redis/mongo to the same node as web-app, it will still work, so I don't think the issue comes because the services cannot talk to a service on the same server node.
Can we make docker network use private IP instead of the pre-configured subnet?
docker stack deploy file
version: '3'
services:
web-app:
image: private-repo/private-image
networks:
- swarm-network
ports:
- "80:8080"
deploy:
placement:
constraints:
- node.role==worker
web-network:
image: private-repo/private-image2
networks:
- swarm-network
ports:
- "9000:8080"
deploy:
placement:
constraints:
- node.role==worker
redis:
image: redis:latest
networks:
- swarm-network
ports:
- "6739:6739"
deploy:
placement:
constraints:
- engine.labels.purpose==database
mongo:
image: mongo:latest
networks:
- swarm-network
ports:
- "27017:27017"
deploy:
placement:
constraints:
- engine.labels.purpose==database
networks:
swarm-network:
driver: overlay
docker stack deploy app -c docker-compose.yml
I'm using docker 1.12.1
I have an easy docker-compose script.
version: '2'
services:
jenkins-slave:
build: ./slave
image: jenkins-slave:1.0
restart: always
ports:
- "22"
environment:
- "constraint:NODE==master1"
jenkins-master:
image: jenkins:2.7.1
container_name: jenkins-master
restart: always
ports:
- "8080:8080"
- "50000"
environment:
- "constraint:NODE==node1"
I run this script with docker-compose -p jenkins up -d.
This Creates my 2 containers but only on my master (from where I execute my command). I would expect that one would be created on the master and one on the node.
I also tried to add
networks:
jenkins_swarm:
driver: overlay
and
networks:
- jenkins_swarm
After every service but this is failing with:
Cannot create container for service jenkins-master: network jenkins_jenkins_swarm not found
While the network is created when I perform docker network ls
Someone who can help me to deploy 2 containers on my 2 nodes with docker-compose. Swarm is defenitly working on my "cluster". I followed this tutorial to verify.
Compose doesn't support Swarm Mode at the moment.
When you run docker compose up on the master node, Compose issues docker run commands for the services in the Compose file, rather than docker service create - which is why the containers all run on the master. See this answer for options.
On the second point, networks are scoped in 1.12. If you inspect your network you'll find it's been created at swarm-level, but Compose is running engine-level containers which can't see the swarm network.
We can do this with docker compose v3 now.
https://docs.docker.com/engine/swarm/#feature-highlights
https://docs.docker.com/compose/compose-file/
You have to initialize the swarm cluster using command
$ docker swarm init
You can add more nodes as worker or manager -
https://docs.docker.com/engine/swarm/join-nodes/
Once you have your both nodes added to the cluster, pass your compose v3 i.e deployment file to create a stack. Compose file should just contain predefined images, you can't give a Dockerfile for deployment in Swarm mode.
$ docker stack deploy -c dev-compose-deploy.yml --with-registry-auth PL
View your stack services status -
$ docker stack services PL
Try to use Labels & Placement constraints to put services on different nodes.
Example "dev-compose-deploy.yml" file for your reference
version: "3"
services:
nginx:
image: nexus.example.com/pl/nginx-dev:latest
extra_hosts:
- "dev-pldocker-01:10.2.0.42”
- "int-pldocker-01:10.2.100.62”
- "prd-plwebassets-01:10.2.0.62”
ports:
- "80:8003"
- "443:443"
volumes:
- logs:/app/out/
networks:
- pl
deploy:
replicas: 3
labels:
feature.description: “Frontend”
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: any
placement:
constraints: [node.role == worker]
command: "/usr/sbin/nginx"
viz:
image: dockersamples/visualizer
ports:
- "8085:8080"
networks:
- pl
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
replicas: 1
labels:
feature.description: "Visualizer"
restart_policy:
condition: any
placement:
constraints: [node.role == manager]
networks:
pl:
volumes:
logs: