HEALTHCHECK of a Docker container running Celery tasks? - docker

Question

I know one of the ways to check health for Docker container is using the commmand
HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1
But in case of workers there is no such URL to hit , How to check the container's health in that case ?

Answer 1

The celery inspect ping command comes in handy, as it does a whole trip: it sends a "ping" task on the broker, workers respond and celery fetches the responses.
Assuming your app is named tasks.add, you may ping all your workers:
/app $ celery inspect ping -A tasks.add
-> celery#aa7c21dd0e96: OK
pong
-> celery#57615db15d80: OK
pong
With aa7c21dd0e96 being the Docker hostname, and thus available in $HOSTNAME.
To ping a single node, you would have to run:
celery inspect ping -A tasks.add -d celery#$HOSTNAME
Here, d stands for destination.
The line to add to your Dockerfile:
HEALTHCHECK CMD celery inspect ping -A tasks.add -d celery#$HOSTNAME
Sample outputs:
/app $ celery inspect ping -A tasks.add -d fake_node
Error: No nodes replied within time constraint.
/app $ echo $?
69
Unhealthy if the node does not exist or does not reply
/app $ celery inspect ping -A tasks.add -d celery#$HOSTNAME
-> celery#d39b3d31cc13: OK
pong
/app $ echo $?
0
Healthy when the node replies pong.
/app $ celery inspect ping -d celery#$HOSTNAME
Traceback (most recent call last):
...
raise socket.error(last_err)
OSError: [Errno 111] Connection refused
/app $ echo $?
1
Unhealthy when the broker is not available - I removed the app, so it tries to connect to a local AMPQ and fails
This might not suit your needs, the broker is unhealthy, not the worker.

Answer 2

The below example snippet, derived from that posted by #PunKeel, is applicable for those looking to implement health check in docker-compose.yml which could be used through docker-compose or docker stack deploy.
worker:
build:
context: .
dockerfile: Dockerfile
image: myimage
links:
- rabbitmq
restart: always
command: celery worker --hostname=%h --broker=amqp://rabbitmq:5672
healthcheck:
test: celery -b amqp://rabbitmq:5672 inspect ping -d celery#$$HOSTNAME
interval: 30s
timeout: 10s
retries: 3
Notice the extra $ in the command, so that $HOSTNAME actually gets passed into the container. I also didn't use the -A flag.
Ideally, rabbitmq should also have its own health check, perhaps with curl guest:guest#localhost:15672/api/overview, since docker wouldn't be able to discern if worker is down or the broker is down with celery inspect ping.

Answer 3

For celery 5.2.3 I used celery -A [celery app name] status for the health check. This is how my docker-compose file looks like
worker:
build: .
healthcheck:
test: celery -A app.celery_app status
interval: 10s
timeout: 10s
retries: 10
volumes:
- ./app:/app
depends_on:
- broker
- redis
- database

Answer 4

Landed on this question looking for a health check for Celery workers as part of an Airflow setup (Airflow 2.3.4, Celery 5.2.7), which I eventually figured out. This is a very specific use case of the original question, but might still be useful for some:
# docker-compose.yml
worker:
image: ...
hostname: local-worker
entrypoint: airflow celery worker
...
healthcheck:
test: [ "CMD-SHELL", 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery#$${HOSTNAME}"' ]
interval: 5s
timeout: 10s
retries: 10
restart: always
...
I got inspiration from Airflow's quick-start Docker Compose.