Docker-Compose: how to wait for other service to be ready? - docker

I have the following docker-compose, where I need to wait for the service jhipster-registry to be up and accepting connections before starting myprogram-app.
I tried the healtcheck way, following the official doc https://docs.docker.com/compose/compose-file/compose-file-v2/
version: '2.1'
services:
myprogram-app:
image: myprogram
mem_limit: 1024m
environment:
- SPRING_PROFILES_ACTIVE=prod,swagger
- EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}#jhipster-registry:8761/eureka
- SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}#jhipster-registry:8761/config
- SPRING_DATASOURCE_URL=jdbc:postgresql://myprogram-postgresql:5432/myprogram
- JHIPSTER_SLEEP=0
- SPRING_DATA_ELASTICSEARCH_CLUSTER_NODES=myprogram-elasticsearch:9300
- JHIPSTER_REGISTRY_PASSWORD=53bqDrurQAthqrXG
- EMAIL_USERNAME
- EMAIL_PASSWORD
ports:
- 8080:8080
networks:
- backend
depends_on:
- jhipster-registry:
"condition": service_started
- myprogram-postgresql
- myprogram-elasticsearch
myprogram-postgresql:
image: postgres:9.6.5
mem_limit: 256m
environment:
- POSTGRES_USER=myprogram
- POSTGRES_PASSWORD=myprogram
networks:
- backend
myprogram-elasticsearch:
image: elasticsearch:2.4.6
mem_limit: 512m
networks:
- backend
jhipster-registry:
extends:
file: jhipster-registry.yml
service: jhipster-registry
mem_limit: 512m
ports:
- 8761:8761
networks:
- backend
healthcheck:
test: "exit 0"
networks:
backend:
driver: "bridge"
but I get the following error when running docker-compose up:
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.myprogram-app.depends_on contains {"jhipster-registry": {"condition": "service_started"}}, which is an invalid type, it should be a string
Am I doing something wrong, or this feature is no more supported? How to achieve this sync between services?
Updated version
version: '2.1'
services:
myprogram-app:
image: myprogram
mem_limit: 1024m
environment:
- SPRING_PROFILES_ACTIVE=prod,swagger
- EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}#jhipster-registry:8761/eureka
- SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}#jhipster-registry:8761/config
- SPRING_DATASOURCE_URL=jdbc:postgresql://myprogram-postgresql:5432/myprogram
- JHIPSTER_SLEEP=0
- SPRING_DATA_ELASTICSEARCH_CLUSTER_NODES=myprogram-elasticsearch:9300
- JHIPSTER_REGISTRY_PASSWORD=53bqDrurQAthqrXG
- EMAIL_USERNAME
- EMAIL_PASSWORD
ports:
- 8080:8080
networks:
- backend
depends_on:
jhipster-registry:
condition: service_healthy
myprogram-postgresql:
condition: service_started
myprogram-elasticsearch:
condition: service_started
#restart: on-failure
myprogram-postgresql:
image: postgres:9.6.5
mem_limit: 256m
environment:
- POSTGRES_USER=myprogram
- POSTGRES_PASSWORD=tuenemreh
networks:
- backend
myprogram-elasticsearch:
image: elasticsearch:2.4.6
mem_limit: 512m
networks:
- backend
jhipster-registry:
extends:
file: jhipster-registry.yml
service: jhipster-registry
mem_limit: 512m
ports:
- 8761:8761
networks:
- backend
healthcheck:
test: ["CMD", "curl", "-f", "http://jhipster-registry:8761", "|| exit 1"]
interval: 30s
retries: 20
#start_period: 30s
networks:
backend:
driver: "bridge"
The updated version gives me a different error,
ERROR: for myprogram-app Container "8ebca614590c" is unhealthy.
ERROR: Encountered errors while bringing up the project.
saying that the container of jhipster-registry is unhealthy, but it's reachable via browser. How can I fix the command in the healthcheck to make it work?

Best Approach - Resilient App Starts
While docker does support startup dependencies, they officially recommend updating your app start logic to test for the availability of external dependencies and retry. This has lots of benefits for robust applications that may restart in the wild on the fly in addition to circumventing the race condition in docker compose up
depends_on & service_healthy - Compose 1.27.0+
depends_on is back in docker compose v1.27.0+ (was deprecated in v3) in the Compose Specification
Each service should also implement a service_healthy check to be able to report if it's fully setup and ready for downstream dependencies.
version: '3.0'
services:
php:
build:
context: .
dockerfile: tests/Docker/Dockerfile-PHP
depends_on:
redis:
condition: service_healthy
redis:
image: redis
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 30
wait-for-it.sh
The recommended approach from docker according to their docs on Control startup and shutdown order in Compose is to download wait-for-it.sh which takes in the domain:port to poll and then executes the next set of commands if successful.
version: "2"
services:
web:
build: .
ports:
- "80:8000"
depends_on:
- "db"
command: ["./wait-for-it.sh", "db:5432", "--", "python", "app.py"]
db:
image: postgres
Note: This requires overriding the startup command of the image, so make sure you know what wanted to pass to maintain parity of the default startup.
Further Reading
Docker Compose wait for container X before starting Y
Difference between links and depends_on in docker_compose.yml
How can I wait for a docker container to be up and running?
Docker Compose Wait til dependency container is fully up before launching
depends_on doesn't wait for another service in docker-compose 1.22.0

The documentation suggests that, in Docker Compose version 2 files specifically, depends_on: can be a list of strings, or a mapping where the keys are service names and the values are conditions. For the services where you don't have (or need) health checks, there is a service_started condition.
depends_on:
# notice: these lines don't start with "-"
jhipster-registry:
condition: service_healthy
myprogram-postgresql:
condition: service_started
myprogram-elasticsearch:
condition: service_started
Depending on how much control you have over your program and its libraries, it's better still if you can arrange for the service to be able to start without its dependencies necessarily being available (equivalently, to function if its dependencies die while the service is running), and not use the depends_on: option. You might return an HTTP 503 Service Unavailable error if the database is down, for instance. Another strategy that often is helpful is to immediately exit if your dependencies aren't available but use a setting like restart: on-error to ask the orchestrator to restart the service.

Update to version 3+.
Please follow the documents from version 3:
There are several things to be aware of when using depends_on:
depends_on does not wait for db and redis to be “ready” before
starting web - only until they have been started. If you need to wait
for a service to be ready, see Controlling startup order for more on
this problem and strategies for solving it. Version 3 no longer
supports the condition form of depends_on.
The depends_on option is
ignored when deploying a stack in swarm mode with a version 3 Compose
file.
I would consider using the restart_policy option for configuring your myprogram-app to restart until the jhipster-registry is up and accepting connections:
restart_policy:
condition: on-failure
delay: 3s
max_attempts: 5
window: 60s

With the new docker compose API, we can now use the new --wait option:
docker compose up --wait
If your service has a healthcheck, Docker waits until it has the "healthy" status; otherwise, it waits for the service to be started. That's why it is crucial to have relevant healthchecks for all your services.
Note that this option automatically activate the --detach option.
Check out the documentation here.

The best approach I found is to check for the desired port in the entrypoint. There are different ways to do that e.g. wait-for-it but I like to use this solution that is cross-platform between apline and bash images and doesn't download custom scripts from GitHub:
Install netcat-openbsd (works with apt and apk). Then in the entrypoint (works with both #!/bin/bash and #!/bin/sh):
#!/bin/bash
wait_for()
{
echo "Waiting $1 seconds for $2:$3"
timeout $1 sh -c 'until nc -z $0 $1; do sleep 0.1; done' $2 $3 || return 1
echo "$2:$3 available"
}
wait_for 10 db 5432
wait_for 10 redis 6379
You can also make this into a 1-liner if you don't want to print anything.

Although you already got an answer, it should be mentioned that what you are trying to achieve have some nasty risks.
Ideally a service should be self sufficient and smart enough to retry and await for dependencies to be available (before a going down). Otherwise you will be more exposed to one failure propagating to other services. Also consider that a system reboot, unlike a manual start might ignore the dependencies order.
If one service crash causes all your system to go down, you might have a tool to restart everything again, but it would be better having services that resist that case.

After trying several approaches, IMO the simplest and most elegant option is using the jwilder/dockerize (dockerize) utility image with its -wait flag. Here is a simple example where I need a PostgreSQL database to be ready before starting my app:
version: "3.8"
services:
# Start Postgres.
db:
image: postgres
# Wait for Postgres to be joinable.
check-db-started:
image: jwilder/dockerize:0.6.1
depends_on:
- db
command: 'dockerize -wait=tcp://db:5432'
# Only start myapp once Postgres is joinable.
myapp:
image: myapp:latest
depends_on:
- check-db-started

Related

docker-compose down wait for container X finish before stop container Y

I have to deploy a web app with a Jetty Server. This app need a database, running on MariaDB. Here the docker-compose file used to deploy the app:
version: '2.1'
services:
jetty:
build:
context: .
dockerfile: docker/jetty/Dockerfile
container_name: app-jetty
ports:
- "8080:8080"
depends_on:
mariadb:
condition: service_healthy
networks:
- app
links:
- "mariadb:mariadb"
mariadb:
image: mariadb:10.7
container_name: app-mariadb
restart: always
environment:
MARIADB_ROOT_PASSWORD: myPassword
MARIADB_DATABASE: APPDB
ports:
- "3307:3306"
healthcheck:
test: [ "CMD", "mariadb-admin", "--protocol", "tcp", "ping", "-u root", "-pmyPassword" ]
interval: 10s
timeout: 3m
retries: 10
volumes:
- datavolume:/var/lib/mysql
networks:
- app
networks:
app:
driver: bridge
volumes:
datavolume:
I use a volume to keep the data of mariaDB even if I use docker-compose down. On my Jetty app, the data is store into the database when the contextDestroyed function is load (when the container is stopped).
But I have an another problem: when I execute docker-compose down, all the containers are stopped and deleted. Although the mariaDB is the last stopped container (that's what the terminal is saying), the save on the contexDestroyed is "interrupt" and I lost some informations because mariaDB container is stopped when the Jetty container still saving data. I tested to stop every container but the mariaDB and my data is succefully saved without loss, so the problem is obviously the mariaDB container.
How can I indicate to the mariadb container to wait for all containers for stopping before stop itself?
According to the depends_on documentation - your dependency will force the following order of shutdown:
web
mariadb
You might want to look into what's happening during the shutdown of these containers and add some custom logic that will guarantee your data is consistent.
You can influence what happens during the shutdown of a container in 2 main ways:
adding a custom script as an entrypoint
handling the SIGTERM signal yourself
here's the relevant documentation on this.
Maybe the simplest - not necessarily the smartest - way would be to add a sleep(5) to the db shutdown, so your app has enough time to flush its writes.

Docker compose stop depending service when done [duplicate]

This question already has answers here:
How to stop all containers when one container stops with docker-compose?
(4 answers)
Closed 1 year ago.
I have this docker-compose.yml which runs a node script which depends on Redis.
version: "3.9"
services:
redis:
image: "redis:alpine"
# restart: always
ports:
- "127.0.0.1:6379:6379"
volumes:
- ./docker/redis:/data
node:
image: "node:17-alpine"
user: "node"
depends_on:
- redis
environment:
- NODE_ENV=production
- REDIS_HOST_ENV=redis
volumes:
- ./docker/node/src:/home/node/app
- ./docker/node/log:/home/node/log
expose:
- "8081"
working_dir: /home/node/app
command: "npm start"
When starting this script with docker compose up both services will start. However when the node service is finished, the redis service keeps running. Is there a way to define that the redis service can stop when the node service is done?
I have examined the documentation for Compose Spec but I have not found anything that allows you to immediately stop containers based on the state of another one. Perhaps there really is a way, but you can always control the behaviour of the redis service by using an healthcheck:
services:
redis:
image: "redis:alpine"
# restart: always
ports:
- "127.0.0.1:6379:6379"
volumes:
- ./docker/redis:/data
healthcheck:
test: ping -c 2 mynode || kill 1
interval: 5s
retries: 1
start_period: 20s
node:
image: "node:17-alpine"
container_name: mynode
user: "node"
depends_on:
- redis
environment:
- NODE_ENV=production
- REDIS_HOST_ENV=redis
volumes:
- ./docker/node/src:/home/node/app
- ./docker/node/log:/home/node/log
expose:
- "8081"
working_dir: /home/node/app
command: "npm start"
As for the node service, I have added a container_name: mynode, necessary by the redis service in order to contact it. The container name becomes also the hostname, if not specified with the hostname property.
The redis service has an healthcheck that ping the node container every 5 seconds, starting after 30 seconds from the container start. If the ping is successful, the container is labeled as healthy, otherwise it is killed.
This solution might work in your case but has some downsides:
The healthcheck feature is abused here, besides what if you had another healthcheck?
You cannot always kill the init process, because protected by default. There are some discussions about this and it seems the most popular decision is to use tini as the init process. Fortunately, in the image you are using, it is possible.
redis service contacts the node service via the hostname, which means that they are supposed to be in the same network in your case. The current network is the default bridge network that should be avoided most of the times. I suggest you to declare a custom bridge network.
This solution is based on polling the node container, which is not very elegant, firstly because you have to hope that the time-based parameters in the healthcheck section are "good-enough".

docker compose network not serving reqeusts from host

[docker-compose question]
hello all! I've been stuck on this for a while so hopefully we can debug together.
I'm using docker compose to bring up three separate services.
Everything builds and comes up great. Health check for the app passes, the services make contact with each other but I can't seem to my curl my app from the host.
I've tried the following values for app.ports:
"127.0.0.1:3000:3000"
"3000:3000"
"0.0.0.0:3000:3000"
I've also tried to run this with a "host" network, but that also didn't seem to work and I don't prefer it because apparently that is not supported on Mac and my local developer environment is Macosx. The prod server is ubuntu.
And I've tried defining the default bridge netowrk explicitly:
networks:
default:
driver: bridge
Here is my docker-compose.yml
version: "2.4"
services:
rabbitmq:
image: rabbitmq
volumes:
- ${ML_FILE_PATH}/taskqueue/config/:/etc/rabbitmq/
environment:
LC_ALL: "C.UTF-8"
LANG: "C.UTF-8"
celery-worker:
image: ${ML_IMAGE_NAME}
entrypoint: "celery --broker='amqp://<user>:<password>#rabbitmq:5672//' -A taskqueue.celeryapp worker --uid 1111"
runtime: ${RUNTIME} ## either "runc" if running locally on debug mode or "nvidia" on production with multi processors
volumes:
- ${ML_FILE_PATH}:/host
depends_on:
- rabbitmq
- app
environment:
LC_ALL: "C.UTF-8"
LANG: "C.UTF-8"
MPLCONFIGDIR: /host/tmp
volumes:
- ${ML_FILE_PATH}:/host
celery-beat:
image: ${ML_IMAGE_NAME}
entrypoint: "celery --broker='amqp://<user>:<password>#rabbitmq:5672//' -A taskqueue.celeryapp beat --uid 1111"
runtime: ${RUNTIME} ## either "runc" if running locally on debug mode or "nvidia" on production with multi processors
depends_on:
- rabbitmq
- app
environment:
LC_ALL: "C.UTF-8"
LANG: "C.UTF-8"
MPLCONFIGDIR: /host/tmp
volumes:
- ${ML_FILE_PATH}:/host
app:
build: .
entrypoint: ${ML_ENTRYPOINT} # just starts a flask app
image: ${ML_IMAGE_NAME}
ports:
- "3000:3000"
expose:
- "3000"
volumes:
- ${ML_FILE_PATH}:/host
restart: always
runtime: ${RUNTIME}
healthcheck:
test: ["CMD", "curl", "http:/localhost:3000/?requestType=health-check"]
start_period: 30s
interval: 30s
timeout: 5s
environment:
SCHEDULER: "off"
TZ: "UTC"
LC_ALL: "C.UTF-8"
LANG: "C.UTF-8"
I can hit the service from within the container as expected.
I'm not sure what I'm missing. Thanks so much for any help!
I'm not sure but i think you dont can route traffic from the host to containers on mac osx.
https://docs.docker.com/desktop/mac/networking/
This ended up being mostly unrelated to docker-compose.
My flask app was starting up on 127.0.0.1. I needed to be starting it as an
externally visible server.
I just had to add --host=0.0.0.0 to my start script.

What is the alternative to condition form of depends_on in docker-compose Version 3?

docker-compose 2.1 offers the nice feature to specify a condition with depends_on. The current docker-compose documentation states:
Version 3 no longer supports the condition form of depends_on.
Unfortunately the documentation does not explain, why the condition form was removed and is lacking any specific recommondation on how to implement that behaviour using V3 upwards.
There's been a move away from specifying container dependencies in compose. They're only valid at startup time and don't work when dependent containers are restarted at run time. Instead, each container should include mechanism to retry to reconnect to dependent services when the connection is dropped. Many libraries to connect to databases or REST API services have configurable built-in retries. I'd look into that. It is needed for production code anyway.
From 1.27.0, 2.x and 3.x are merged with COMPOSE_SPEC schema.
version is now optional. So, you can just remove it and specify a condition as before:
services:
web:
build: .
depends_on:
redis:
condition: service_healthy
redis:
image: redis
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 30
There are some external tools that let you mimic this behaviour. For example, with the dockerize tool you can wrap your CMD or ENTRYPOINT with dockerize -wait and that will prevent running your application until specified services are ready.
If your docker-compose file used to look like this:
version: '2.1'
services:
kafka:
image: spotify/kafka
healthcheck:
test: nc -z localhost 9092
webapp:
image: foo/bar # your image
healthcheck:
test: curl -f http://localhost:8080
tests:
image: bar/foo # your image
command: YOUR_TEST_COMMAND
depends_on:
kafka:
condition: service_healthy
webapp:
condition: service_healthy
then you can use dockerize in your v3 compose file like this:
version: '3.0'
services:
kafka:
image: spotify/kafka
webapp:
image: foo/bar # your image
tests:
image: bar/foo # your image
command: dockerize -wait tcp://kafka:9092 -wait web://webapp:8080 YOUR_TEST_COMMAND
Just thought I'd add my solution for when running postgres and an application via docker-compose where I need the application to wait for the init sql script to complete before starting.
dockerize seems to wait for the db port to be available (port 5432) which is the equivilant of depends_on which can be used in docker 3:
version: '3'
services:
app:
container_name: back-end
depends_on:
- postgres
postgres:
image: postgres:10-alpine
container_name: postgres
ports:
- "5432:5432"
volumes:
- ./docker-init:/docker-entrypoint-initdb.d/
The Problem:
If you have a large init script the app will start before that completes as the depends_on only waits for the db port.
Although I do agree that the solution should be implemented in the application logic, the problem we have is only for when we want to run tests and prepopulate the database with test data so it made more sense to implement a solution outside the code as I tend not like introducing code "to make tests work"
The Solution:
Implement a healthcheck on the postgres container.
For me that meant checking the command of pid 1 is postgres as it will be running a different command on pid 1 while the init db scripts are running
Write a script on the application side which will wait for postgres to become healthy. The script looks like this:
#!/bin/bash
function check {
STATUS=\`curl -s --unix-socket /var/run/docker.sock http:/v1.24/containers/postgres/json | python -c 'import sys, json; print json.load('sys.stdin')["State"]["Health"]["Status"]'\`
if [ "$STATUS" = "healthy" ]; then
return 0
fi
return 1
}
until check; do
echo "Waiting for postgres to be ready"
sleep 5
done
echo "Postgres ready"
Then the docker-compose should mount the directories of the scripts so that we don't edit the Dockerfile for the application and if we're using a custom postgres image, this way we can continue to use the docker files for your published images.
We're also overriding the entry point defined in the docker file of the app so that we can run the wait script before the app starts
version: '3'
services:
app:
container_name: back-end
entrypoint: ["/bin/sh","-c","/opt/app/wait/wait-for-postgres.sh && <YOUR_APP_START_SCRIPT>"]
depends_on:
- postgres
volumes:
- //var/run/docker.sock:/var/run/docker.sock
- ./docker-scripts/wait-for-postgres:/opt/app/wait
postgres:
image: postgres:10-alpine
container_name: postgres
ports:
- "5432:5432"
volumes:
- ./docker-init:/docker-entrypoint-initdb.d/
- ./docker-scripts/postgres-healthcheck:/var/lib
healthcheck:
test: /var/lib/healthcheck.sh
interval: 5s
timeout: 5s
retries: 10
I reached this page because one container would not wait for the one depending upon and I had to run a docker system prune to get it working. There was an orphaned container error that prompted me to run the prune.

Docker healthcheck in composer file

I try to integrate the new healthcheck into my docker system, but I don't really know how to do it in the right way :/
The problem is, my database container needs more time to start up and initialize the database then the container who starts my main application.
As a result: the main container wont start correct, cause of the missing database connection.
I wrote an healthcheck.sh script to check the database container for connectivity, so the main container starts booting after the connectivity is available. But I dont know how to integrate it correctly in the Dockerfile and my docker-compose.yml
healthcheck.sh is like:
#!bin/bash
COUNTER=0
while [[ $COUNTER = 0 ]]; do
mysql --host=HOST --user="user" --password="password" --database="databasename" --execute="SELECT 1";
if [[ $? == 1 ]]; then
sleep 1
echo "Let's sleep again"
else
COUNTER=1
echo "OK, lets go!"
fi
done
mysql container Dockerfile:
FROM repository/mysql-5.6:latest
MAINTAINER Me
... some copies, chmod and so on
VOLUME ["/..."]
EXPOSE 3306
CMD [".../run.sh"]
HEALTHCHECK --interval=1s --timeout=3s CMD ./healthcheck.sh
docker-compose.yml like:
version: '2'
services:
db:
image: db image
restart: always
dns:
- 10.
ports:
- "${MYSQL_EXTERNAL_PORT}:${MYSQL_INTERNAL_PORT}"
environment:
TZ: Europe/Berlin
data:
image: data image
main application:
image: application image
restart: always
dns:
- 10.
ports:
- "${..._EXTERNAL_PORT}:${..._INTERNAL_PORT}"
environment:
TZ: Europe/Berlin
volumes:
- ${HOST_BACKUP_DIR}:/...
volumes_from:
- data
- db
What do I have to do to integrate this healthcheck into my docker-compose.yml file to work?
Or is there any other chance to delay the container startup of my main container?
Thx Markus
I believe this is similar to Docker Compose wait for container X before starting Y
Your db_image needs to support curl.
To do that, create your own db_image as:
FROM base_image:latest
RUN apt-get update
RUN apt-get install -y curl
EXPOSE 3306
Then all you should need is a docker-compose.yml that looks like this:
version: '2'
services:
db:
image: db_image
restart: always
dns:
- 10.
ports:
- "${MYSQL_EXTERNAL_PORT}:${MYSQL_INTERNAL_PORT}"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:${MYSQL_INTERNAL_PORT}"]
interval: 30s
timeout: 10s
retries: 5
environment:
TZ: Europe/Berlin
main_application:
image: application_image
restart: always
depends_on:
db:
condition: service_healthy
links:
- db
dns:
- 10.
ports:
- "${..._EXTERNAL_PORT}:${..._INTERNAL_PORT}"
environment:
TZ: Europe/Berlin
volumes:
- ${HOST_BACKUP_DIR}:/...
volumes_from:
- data
- db
In general your application should be able to cope with unavailable resources, but there are also some cases when starting up where it is pretty convenient to have one container waiting for another to be "fully available". Docker itself doesn't handle that for you, but there are ways to handle the startup in the resource-using container by delaying the actual command with some script.
There is a good example for a postgresql startup check that can be used in any container that needs to wait for the database to be "fully started". Please see the sample code in the docker docs: https://docs.docker.com/compose/startup-order/
Since docker-compose 1.10.0 you can specify healthchecks in your compose file: https://github.com/docker/docker.github.io/blob/master/compose/compose-file.md#healthcheck
It makes use of https://docs.docker.com/engine/reference/builder/#/healthcheck which has been introducded with Docker 1.12

Resources