Docker container unhealthy but no error in the logs - docker

Question

I am using the official docker-compose file of airflow to spin it up.
Some of my containers seem unhealthy:
34d8698d67e7 apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 28 minutes (unhealthy) 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp airflow_flower_1
a291cf238b9f apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 29 minutes 8080/tcp airflow_airflow-init_1
fdb20e9152f3 apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 29 minutes (unhealthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp airflow_airflow-webserver_1
abf5a16aa846 apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 29 minutes 8080/tcp airflow_airflow-worker_1
f6dc352f407b apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 28 minutes 8080/tcp airflow_airflow-scheduler_1
12dfc71e518f redis:latest "docker-entrypoint.s…" 31 minutes ago Up 29 minutes (healthy) 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp airflow_redis_1
However the logs of one of them for example do not seem very informative.
# docker logs -f fdb20e9152f3
WARNING! You should run the image with GID (Group ID) set to 0
even if you use 'airflow' user (UID=50000)
You started the image with UID=50000 and GID=50000
This is to make sure you can run the image with an arbitrary UID in the future.
See more about it in the Airflow's docker image documentation
http://airflow.apache.org/docs/docker-stack/entrypoint
BACKEND=postgresql+psycopg2
DB_HOST=my-db-endpoint
DB_PORT=5432
WARNING! You should run the image with GID (Group ID) set to 0
even if you use 'airflow' user (UID=50000)
You started the image with UID=50000 and GID=50000
This is to make sure you can run the image with an arbitrary UID in the future.
See more about it in the Airflow's docker image documentation
http://airflow.apache.org/docs/docker-stack/entrypoint
BACKEND=postgresql+psycopg2
DB_HOST=my-db-endpoint
DB_PORT=5432
Regardless of any airflow - specific issues, how can I check docker - wise what's going on?
Docker seems to be aware of a couple of containers not being healty.
edit: both failing containers have the healtcheck condition
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
and
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/"]
that seems to be failing by looking into their inspect output
Failed to connect to localhost port 8080: Connection refused
but I cannot pinpoint what is causing the failure.
edit: I have tried following the instructions to start the init service first as well
# docker-compose up airflow-init
Starting airflow_redis_1 ... done
Starting airflow_airflow-init_1 ... done
Attaching to airflow_airflow-init_1
airflow-init_1 | BACKEND=postgresql+psycopg2
airflow-init_1 | DB_HOST=my-db-endpoint
airflow-init_1 | DB_PORT=5432
but it never exits, it prints the above message and that's it...

Answer 1

I ran into similar issue and it was docker volume causing the issue. As I was running lots of containers on my mac, there wasn;t enough disk space. I managed to fixed this issue my pruning the docker volume.
docker volume prune
This will remove any unused volume on your mac book. Before running this command please check if you got any useful data.

Answer 2

For docker-compose, from the entrypoint, the default value of group id is 0.
"${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}"
Edit your docker-compose.yaml file or ad ass env.sh file in your Airflow project repository.

Answer 3

It does seems to be an error due to less memory allocated to docker for running this image, try to increase the resources available to docker and see the magic

Answer 4

I ran into similar issue and the healthchecks were "causing" this. I was running them using default container's user.
Just to give a try, I changed the healthckeck command to start using airflow user instead, as follow:
$ runuser -u airflow -- <healthckeck command>
And it solved. I'm gonna change the user whom runs docker compose up to airflow from now on.