How to add airflow variables in docker compose file? - docker

Question

I have an docker compose file which spins up the local airflow instance as below:
version: '3.7'
services:
postgres:
image: postgres:9.6
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
logging:
options:
max-size: 10m
max-file: "3"
webserver:
image: puckel/docker-airflow:1.10.6
restart: always
depends_on:
- postgres
environment:
- LOAD_EX=n
- EXECUTOR=Local
- FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
logging:
options:
max-size: 10m
max-file: "3"
volumes:
- ./dags:/usr/local/airflow/dags
- ${HOME}/.aws:/usr/local/airflow/.aws
ports:
- "8080:8080"
command: webserver
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
I want to add some Airflow variables which the underlying dag uses eg: CONFIG_BUCKET.
I have added them as AIRFLOW_VAR_CONFIG_BUCKET=s3://foo-bucket
in environment section of web server but it does not seem to work. Any ideas how can I achieve this ?

Answer 1

You should not add variables to the webserver, but to scheduler. If you are using LocalExecutor, the tasks are run in the context of Scheduler.
Actually what tou should really do is to set all env variables to be the same for all the containers (this is explained here https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html)
Use the same configuration across all the Airflow components. While each component does not require all, some configurations need to be same otherwise they would not work as expected. A good example for that is secret_key which should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker.
There are a number of ways you can do it - just read the docker-compose documntation on that https://docs.docker.com/compose/environment-variables . You can also see the "Quick start" docker compose from Airflow docs where we used anchors - which is bit more sphisticated way https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html
Just note that the "quick start" should be just inspiration, it is nowhere near production setup and if you want to make your own docker compose you need to really get a deeper understanding of the docker compose - as warned in the note in our docs.

Answer 2

If you add an environment variable named AIRFLOW_VAR_CONFIG_BUCKET to the list under environment:, it should be accessible by Airflow. Sounds like you're doing that correctly.
Two things to note:
Variables (& connections) set via environment variables are not visible in the Airflow UI. You can test if they exist by executing Variable.get("config_bucket") in code.
The Airflow scheduler/worker (depending on Airflow executor) require access to the variable while running a task. Adding a variable to the webserver is not required.