Changing default Airflow configuration options (run on Docker) - docker

Question

I've been trying to set up an Airflow instance on a set of docker containers for the sake of learning. I know just the core basics of both Docker and Airflow.
I've been following this guide on how to set up Airflow using Docker.
When following the guide I can start up the containers and everything work as expected.
My next step was to start changing different configuration alternatives in the docker-compose.yml-file. I started by trying to set different usernames and passwords.
I created an .env-file in the same directory as the docker-compose.yml-file. See below.
AIRFLOW_UID=50000
USER='user'
PASSWORD='password'
DATABASE='airflow'
_AIRFLOW_WWW_USER_USERNAME='user'
_AIRFLOW_WWW_USER_PASSWORD='password'
In order to use these environments variables, I made some changes to the connection strings in the docker-compose.yml aswell. Original lines has been commented out. Se below.
x-airflow-common:
&airflow-common
# In order to add custom dependencies or upgrade provider packages you can use your extended image.
# Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
# and uncomment the "build" line below, Then run `docker-compose build` to build the images.
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.4.2}
# build: .
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${USER}:${PASSWORD}#postgres/${DATABASE}
# For backward compatibility, with Airflow <2.3
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${USER}:${PASSWORD}#postgres/${DATABASE}
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://${USER}:${PASSWORD}#postgres/${DATABASE}
# -- Original settings --
#AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:postgresql+psycopg2://airflow:airflow#postgres/airflow
#AIRFLOW__CORE__SQL_ALCHEMY_CONN:postgresql+psycopg2://airflow:airflow#postgres/airflow
#AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
user: "${AIRFLOW_UID:-50000}:0"
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: ${USER}
POSTGRES_PASSWORD: ${PASSWORD}
POSTGRES_DB: ${DATABASE}
# -- Original settings --
#POSTGRES_USER: airflow
#POSTGRES_PASSWORD: airflow
#POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", '${USER}']
interval: 5s
retries: 5
restart: always
First I run docker-compose down --volumes --rmi all in order to clean up the environment. I then run docker-compose up airflow-init. Below are the last lines displayed in the terminal.
data_collection_proj-airflow-init-1 | User "user" created with role "Admin"
data_collection_proj-airflow-init-1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:367: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
data_collection_proj-airflow-init-1 | FutureWarning,
data_collection_proj-airflow-init-1 | 2.4.2
data_collection_proj-airflow-init-1 exited with code 0
No errors or warnings were displayed so it seems to work fine.
I then run docker-compose up and get the following error.
data_collection_proj-postgres-1 | 2022-11-12 17:47:08.939 UTC [165] FATAL: database "user" does not exist
My changes seem to destroy the set-up but I don't understand why.

Answer 1

You can try this test syntax, it works fine for me:
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: ${USER}
POSTGRES_PASSWORD: ${PASSWORD}
POSTGRES_DB: ${DATABASE}
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "sh -c 'pg_isready -U ${USER} -d ${DATABASE}'"]
interval: 5s
retries: 5
restart: always