I'm a bit confused as I'm used to use docker-compose in a single-server environment. Now I have the idea to use a Docker Swarm cluster with docker-compose (as it's what I know better) but I'm a bit confused on how to make it work against my app's needs. For instance:
My app is made up by a manager app and multiple workers. My idea is to have the manager app run in the Docker Swarm manager's server (is that possible?) and then use docker-compose to replicate the workers only through the rest of the Swarm cluster nodes.
A small map would be something like:
Server A -> manager
Server B -> worker1, worker2, worker3
Server C -> worker4, worker5
The workers connect to the manager through a defined IP & port in the environment section in the docker-compose.yml file.
My question is: How do I start up the manager only on a single server, and how do I replicate the workers only in the other nodes, without having a manager per cluster node? (as I don't want/need that). Thanks in advance!
You can to define by constraints
version: '3.8'
services:
manager:
hostname: 'manager'
image: traefik
deploy:
placement:
max_replicas_per_node: 1
constraints: [node.role == manager]
service:
image: service
deploy:
mode: replicated
replicas: 5
placement:
constraints: [node.role == worker]
Related
Using docker swarm, I am trying to deploy N instances of my app on N nodes in a way that each app is deployed on the node with the corresponding index. E.g.: app1 must be deployed on node1, app2 on node2, ...
The bellow is not working as it complains Error response from daemon: rpc error: code = Unknown desc = value 'node{{.Task.Slot}}' is invalid.
Any suggestion how to achieve this ?
I also have the impression, in a long shot, to use something with labels but I cannot wrap my head over it yet. Anyhow please advise.
version: "3.8"
services:
app:
image: app:latest
hostname: "app{{.Task.Slot}}"
networks:
- app-net
volumes:
- "/data/shared/app{{.Task.Slot}}/config:/app/config"
deploy:
replicas: 5
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: any
placement:
constraints:
- "node.hostname==node{{.Task.Slot}}" <========= ERROR
Service template parameters are documented as only resolving in:
the hostname: directive
for volume definitions
in labels.
environment variables.
Placement preference / constraints is not supported, but would be brilliant as it would allow simple deployments of Minio, etcd, consul and other clustered services where you need to pin replicas to nodes.
I have a small network of computers, bundled into a docker swarm. I'm running a service that needs host networking, which is fine when it runs on a dedicated node.
But I want to let it run on any node - yet maintaining the ability to access its web UI by connecting to a fixed hostname/IP address, regardless of which node the service is actually running on.
This is normally handled by docker's ingress network, which allows me to connect to a published port on any node's IP address, and routes the connection to the proper node. However, apparently this doesn't work with host networking, and if I specify the ingress network explicitely, it gets rejected.
So, is there a way to both have host networking, while keeping ingress routing? Or what would be the recommended way to let me connect to the service without worrying about which node it's running on at any given moment?
EDIT:
My stack file is the following:
version: '3'
services:
app:
image: ghcr.io/home-assistant/home-assistant:stable
volumes:
- ...
privileged: true
deploy:
replicas: 1
restart_policy:
condition: any
placement:
constraints:
- node.hostname==nas
networks:
- host
networks:
host:
external: true
I'm trying to deploy a Docker Swarm of three host nodes with a single replicated service and put an HAProxy in front of it. I want the clients to be able to connect via SSL.
My docker-compose.yml:
version: '3.9'
services:
proxy:
image: haproxy
ports:
- 443:8080
volumes:
- haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
deploy:
placement:
constraints: [node.role == manager]
networks:
- servers-network
node-server:
image: glusk/hackathon-2021:latest
ports:
- 8080:8080
command: npm run server
deploy:
mode: replicated
replicas: 2
networks:
- servers-network
networks:
servers-network:
driver: overlay
My haproxy.cfg (based on the official example):
# Simple configuration for an HTTP proxy listening on port 80 on all
# interfaces and forwarding requests to a single backend "servers" with a
# single server "server1" listening on 127.0.0.1:8000
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend servers
backend servers
server server1 127.0.0.1:8000 maxconn 32
My hosts are Lightsail VPS Ubuntu instances and share the same private network.
node-service runs each https server task inside its own container on: 0.0.0.0:8080.
The way I'm trying to make this work at the moment is to ssh into the manager node (which also has a static and public IP), copy over my configuration files from above, and run:
docker stack deploy --compose-file=docker-compose.yml hackathon-2021
but it doesn't work.
Well, first of all and regarding SSL (since it's the first thing that you mention) you need to configure it using the certificate and listen on the port 443, not port 80.
With that modification, your Proxy configuration would already change to:
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend servers
frontend https-in
bind *:443 ssl crt /etc/ssl/certs/hackaton2021.pem
default_backend servers
That would be a really simplified configuration for allowing SSL connection.
Now, let's go for the access to the different services.
First of all, you cannot access to the service on localhost, actually you shouldn't even expose the ports of the services you have to the host. The reason? That you already have those applications in the same network than the haproxy, so the ideal would be to take advantage of the Docker DNS to access directly to them
In order to do this, first we need to be able to resolve the service names. For that you need to add the following section to your configuration:
resolvers docker
nameserver dns1 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 10s
hold refused 10s
hold nx 10s
hold timeout 10s
hold valid 10s
hold obsolete 10s
The Docker Swarm DNS service is always available at 127.0.0.11.
Now to your previous existent configuration, we would have to add the server but using the service-name discovery:
backend servers
balance roundrobin
server-template node- 2 node-server:8080 check resolvers docker init-addr libc,none
If you check what we are doing, we are creating a server for each one of the discovered containers in the Swarm within the node-server service (so the replicas) and we will create those adding the prefix node- to each one of them.
Basically, that would be the equivalent to get the actual IPs of each of the replicas and add them stacked as a basic server configuration.
For deployment, you also have some errors, since we aren't interested into actually expose the node-server ports to the host, but to create the two replicas and use HAProxy for the networking.
For that, we should use the following Docker Compose:
version: '3.9'
services:
proxy:
image: haproxy
ports:
- 80:80
- 443:443
volumes:
- hackaton2021.pem:/etc/ssl/certs/hackaton2021.pem
- haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
deploy:
placement:
constraints: [node.role == manager]
node-server:
image: glusk/hackathon-2021:latest
command: npm run server
deploy:
mode: replicated
replicas: 2
Remember to copy your haproxy.cfg and the self-signed (or real) certificate for your application to the instance before deploying the Stack.
Also, when you create that stack it will automatically create a network with the name <STACK_NAME>-default, so you don't need to define a network just for connecting both services.
I’m setting up a distributed Airflow cluster where everything else except the celery workers are run on one host and processing is done on several hosts. The airflow2.0 setup is configured using the yaml file given at the Airflow documentation https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml . In my initial tests I got the architecture to work nicely when I run everything at the same host. The question is, how to start the celery workers at the remote hosts?
So far, I tried to create a trimmed version of the above docker-compose where I only start the celery workers at the worker host and nothing else. But I run into some issues with db connection. In the trimmed version I changed the URL so that they point to the host that runs the db and redis.
dags, logs, plugins and the postgresql db are located on a shared drive that is visible to all hosts.
How should I do the configuration? Any ideas what to check? Connections etc.?
Celery worker docker-compose configuration:
---
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}
environment:
&airflow-common-env
AIRFLOW_UID: 50000
AIRFLOW_GID: 50000
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN:
postgresql+psycopg2://airflow:airflow#airflowhost.example.com:8080/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow#airflow#airflowhost.example.com:8080/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#airflow#airflowhost.example.com:6380/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
REDIS_PORT: 6380
volumes:
- /airflow/dev/dags:/opt/airflow/dags
- /airflow/dev/logs:/opt/airflow/logs
- /airflow/dev/plugins:/opt/airflow/plugins
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
services:
airflow-remote-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- "CMD-SHELL"
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery#$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
EDIT 1:
I'm Still having some difficulties with the log files. It appears that sharing the log directory doesn't solve the issue of missing log files. I added the extra_host definition on main like suggested and opened the port 8793 on the worker machine.
The worker tasks fail with log:
*** Log file does not exist:
/opt/airflow/logs/tutorial/print_date/2021-07-
01T13:57:11.087882+00:00/1.log
*** Fetching from: http://:8793/log/tutorial/print_date/2021-07-01T13:57:11.087882+00:00/1.log
*** Failed to fetch log file from worker. Unsupported URL protocol ''
Far from being the "ultimate set-up", these are some settings that worked for me using the docker-compose from Airflow in the core node and the workers:
Main node:
The worker nodes have to be reachable from the main node where the Webserver runs. I found this diagram of the CeleryExecutor architecture to be very helpful to sort things out.
When trying to read the logs, if they are not found locally, it will try to retrieve them from the remote worker. Thus your main node may not know the hostname of your workers, so you either change how the hostnames are being resolved (hostname_callable setting, which defaults to socket.getfqdn ) or you just simply add name resolution capability to the Webserver. This could be done by adding the extra_hosts config key in the x-airflow-common definition:
---
version: "3"
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}
environment: &airflow-common-env
...# env vars
extra_hosts:
- "worker-01-hostname:worker-01-ip-address" # "worker-01-hostname:192.168.0.11"
- "worker-02-hostname:worker-02-ip-address"
*Note that in your specific case where you have a shared drive, so I think the logs will be found locally.
Define parallelism, DAG concurrency, and scheduler parsing processes. Could be done by using env vars:
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}
environment: &airflow-common-env
AIRFLOW__CORE__PARALLELISM: 64
AIRFLOW__CORE__DAG_CONCURRENCY: 32
AIRFLOW__SCHEDULER__PARSING_PROCESSES: 4
Of course, the values to be set depend on your specific case and available resources. This article has a good overview of the subject. DAG settings could also be overridden at DAG definition.
Worker nodes:
Define worker CELERY__WORKER_CONCURRENCY, default could be the numbers of CPUs available on the machine (docs).
Define how to reach the services running in the main node. Set an IP or hostname and watch out for matching exposed ports in the main node:
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CELERY__WORKER_CONCURRENCY: 8
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow#main_node_ip_or_hostname:5432/airflow # 5432 is default postgres port
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow#main_node_ip_or_hostname:5432/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#main_node_ip_or_hostname:6379/0
Share the same Fernet Key and Secret Key reading them from an ".env" file:
environment: &airflow-common-env
AIRFLOW__CORE__FERNET_KEY: ${FERNET_KEY}
AIRFLOW__WEBSERVER__SECRET_KEY: ${SECRET_KEY}
env_file:
- .env
.env file: FERNET_KEY=jvYUaxxxxxxxxxxxxx=
It's critical that every node in the cluster (main and workers) has the same settings applied.
Define a hostname to the worker service to avoid autogenerated matching the container id.
Expose port 8793, which is the default port used to fetch the logs from the worker (docs):
services:
airflow-worker:
<<: *airflow-common
hostname: ${HOSTNAME}
ports:
- 8793:8793
command: celery worker
restart: always
Make sure every worker node host is running with the same time configuration, a few minutes difference could cause serious execution errors which may not be so easy to find. Consider enabling NTP service on host OS.
If you have heavy workloads and high concurrency in general, you may need to tune Postgres settings such as max_connections and shared_buffers. The same applies to the host OS network settings such as ip_local_port_range or somaxconn.
In any issues I encountered during the initial cluster setup, Flower and the worker execution logs always provided helpful details and error messages, both the task-level logs and the Docker-Compose service log i.e: docker-compose logs --tail=10000 airflow-worker > worker_logs.log.
Hope that works for you!
The following considerations build on the accepted answer, as I think they might be relevant to any new Airflow Celery setup:
Enabling remote logging usually comes in handy in a distributed setup as a way to centralize logs. Airflow supports remote logging natively, see e.g. this or this
Defining worker_autoscale instead of concurrency will allow to dynamically start/stop new processes when the workload increases/decreases
Setting the environment variable DUMB_INIT_SETSID to 0 in the worker's environment allows for warm shutdowns (see the docs)
Adding volumes to the worker in the Docker Compose pointing at Airflow's base_log_folder allows to safely persist the worker logs locally. Example:
# docker-compose.yml
services:
airflow-worker:
...
volumes:
- worker_logs:/airflow/logs
...
...
volumes:
worker_logs:
I am running a service on Docker Swarm. This is what I did to deploy the service:
docker swarm init
docker stack deploy -c docker-compose.yml MyApplication
Content of docker-compose.yml:
version: "3"
services:
web:
image: myimage:1.0
ports:
- "9000:80"
- "9001:443"
deploy:
replicas: 3
resources:
limits:
cpus: "0.5"
memory: 256M
restart_policy:
condition: on-failure
Let't say that I update the application and build a new image myimage:2.0. What is a proper way to deploy the new version of image to the service without the downtime?
A way to achieve this is:
provide a healthcheck. That way docker will know if your new deployment has succeeded.
https://docs.docker.com/engine/reference/builder/#healthcheck
https://docs.docker.com/compose/compose-file/#healthcheck]
control how docker will update your service with update_config
https://docs.docker.com/compose/compose-file/#update_config
pay attention to order and parallelism, for example if you choose order: stop-first + parallelism: 2 and your replicas are the same amount as parallelism, your app will stop completely when updating
if your update doesn't succeed you probably want to rollback
https://docs.docker.com/compose/compose-file/#rollback_config
don't forget the restart_policy too
I have some examples on that subject:
Docker Swarm Mode Replicated Example with Flask and Caddy
https://github.com/douglasmiranda/lab/tree/master/caddy-healthcheck-of-caddy-itself
With this you can simply run docker stack deploy... again. If there was changes in the service, it will be updated.
you can use the command docker service update --image but it will start a new container with a implicit scale 0/1.
The downtime depends of your application.