I am trying to take a very difficult, multi-step docker setup and make it into an easy docker-compose. I haven't really used docker-compose before. I am used to using a dockerfile to build and image, then using something like
docker run --name mysql -v ${PWD}/sql-files:/docker-entrypoint-initdb.d ... -h mysql -d mariadb:10.4
Then running the web app in the same manner after building the dockerfile that is simple. Trying to combine these into a docker-compose.yml file seems to be quite difficult. I'll post up my docker-compose.yml file, edited to remove passwords and such and the error I am getting, hopefully someone can figure out why it's failing, because I have no idea...
docker-compose.yml
version: "3.7"
services:
mysql:
image: mariadb:10.4
container_name: mysql
environment:
MYSQL_ROOT_PASSWORD: passwordd1
MYSQL_ALLOW_EMPTY_PASSWORD: "true"
volumes:
- ./sql-files:/docker-entrypoint-initdb.d
- ./ssl:/var/lib/mysql/ssl
- ./tls.cnf:/etc/mysql/conf.d/tls.cnf
healthcheck:
test: ["CMD", "mysqladmin ping"]
interval: 10s
timeout: 2s
retries: 10
web:
build: ./base/newdockerfile
container_name: web
hostname: dev.website.com
volumes:
- ./ssl:/etc/httpd/ssl
- ./webfiles:/var/www
depends_on:
mysql:
condition: service_healthy
ports:
- "8443:443"
- "8888:80"
entrypoint:
- "/sbin/httpd"
- "-D"
- "FOREGROUND"
The error I get when running docker-compose up in the terminal window is...
Service 'web' depends on service 'mysql' which is undefined.
Why would mysql be undefined. It's the first definition in the yml file and has a health check associated with it. Also, it fails very quickly, like within a few seconds, so there's no way all the healthchecks ran and failed, and I'm not getting any other errors in the terminal window. I do docker-compose up and within a couple seconds I get that error. Any help would be greatly appreciated. Thank you.
according to this documentation
depends_on does not wait for db and redis to be “ready” before
starting web - only until they have been started. If you need to wait
for a service to be ready, see Controlling startup order for more on
this problem and strategies for solving it.
Designing your application so it's resilient when database is not available or set up yet is what we all have to deal with. Healthcheck doesn't guarantee you database is ready before the next stage. The best way is probably write a wait-for-it script or wait-for and run it after depends-on:
depends_on:
- "db"
command: ["./wait-for-it.sh"]
Related
I have to deploy a web app with a Jetty Server. This app need a database, running on MariaDB. Here the docker-compose file used to deploy the app:
version: '2.1'
services:
jetty:
build:
context: .
dockerfile: docker/jetty/Dockerfile
container_name: app-jetty
ports:
- "8080:8080"
depends_on:
mariadb:
condition: service_healthy
networks:
- app
links:
- "mariadb:mariadb"
mariadb:
image: mariadb:10.7
container_name: app-mariadb
restart: always
environment:
MARIADB_ROOT_PASSWORD: myPassword
MARIADB_DATABASE: APPDB
ports:
- "3307:3306"
healthcheck:
test: [ "CMD", "mariadb-admin", "--protocol", "tcp", "ping", "-u root", "-pmyPassword" ]
interval: 10s
timeout: 3m
retries: 10
volumes:
- datavolume:/var/lib/mysql
networks:
- app
networks:
app:
driver: bridge
volumes:
datavolume:
I use a volume to keep the data of mariaDB even if I use docker-compose down. On my Jetty app, the data is store into the database when the contextDestroyed function is load (when the container is stopped).
But I have an another problem: when I execute docker-compose down, all the containers are stopped and deleted. Although the mariaDB is the last stopped container (that's what the terminal is saying), the save on the contexDestroyed is "interrupt" and I lost some informations because mariaDB container is stopped when the Jetty container still saving data. I tested to stop every container but the mariaDB and my data is succefully saved without loss, so the problem is obviously the mariaDB container.
How can I indicate to the mariadb container to wait for all containers for stopping before stop itself?
According to the depends_on documentation - your dependency will force the following order of shutdown:
web
mariadb
You might want to look into what's happening during the shutdown of these containers and add some custom logic that will guarantee your data is consistent.
You can influence what happens during the shutdown of a container in 2 main ways:
adding a custom script as an entrypoint
handling the SIGTERM signal yourself
here's the relevant documentation on this.
Maybe the simplest - not necessarily the smartest - way would be to add a sleep(5) to the db shutdown, so your app has enough time to flush its writes.
I have a docker container which is brought up with the help of a docker-compose file along with a database container. I want to do this:
Keep the database container running
Schedule the container with my python program to run daily, generate results and stop again
This is my configuration file:
version: '3.7'
services:
database:
container_name: sql_database
image: mysql:latest
command: --init-file /docker-entrypoint-initdb.d/init.sql
ports:
- 13306:3306
environment:
MYSQL_ROOT_PASSWORD: root
volumes:
- ./backup.sql:/docker-entrypoint-initdb.d/init.sql
python-container:
container_name: python-container
build: ./python_project
command: python main.py
depends_on:
- database
volumes:
- myvol:/python_project/data
volumes:
myvol:
Can someone please help me with this? Thanks!
I was just about to ask the same thing. Seems silly to keep a container going 24/7 just to run one job a day (in my case, certbot renew).
I think there may be a way to fake this using the RESTART_POLICY option with a long delay and no maximum retries, but I haven't tried it yet.
EDIT: Apparently restart_policy only works for swarms. Pity.
If the underlying container has a bash shell, you set the command to run a loop with a delay, like this:
while true; do python main.py; sleep 1; done
I'm somewhat inexperienced with both Docker and Airflow, so this might be a silly question. I have a Dockerfile that uses the apache/airflow image together with some of my own DAGs. I would like to launch the airflow web server together with the scheduler and I'm having trouble with this. I can get it working, but I feel that I'm approaching this incorrectly.
Here is what my Dockerfile looks like:
FROM apache/airflow
COPY airflow/dags/ /opt/airflow/dags/
RUN airflow initdb
Then I run docker build -t learning/airflow .. Here is the tough part: I then run docker run --rm -tp 8080:8080 learning/airflow:latest webserver and in a separate terminal I run docker exec `docker ps -q` airflow scheduler. The trouble is, that in practice this generally happens on a VM somewhere, so opening up a second terminal is just not an option and multiple machines will probably not have access to the same docker container. Running webserver && scheduler does not seem to work, the server appears to be blocking and I'm still seeing the message "The scheduler does not appear to be running" in Airflow UI.
Any ideas on what the right way to run server and scheduler should be?
Many thanks!
First, thanks to #Alex and #abestrad for suggesting docker-compose here -- I think this is the best solution. I finally managed to get it working by referring to this great post. So here is my solution:
First, my Dockerfile looks like this now:
FROM apache/airflow
RUN pip install --upgrade pip
RUN pip install --user psycopg2-binary
COPY airflow/airflow.cfg /opt/airflow/
Note that I'm no longer copying dags to the VM, this information is going to be passed through volumes. I then build the docker file via docker build -t learning/airflow .. My docker-compose.yaml looks like this:
version: "3"
services:
postgres:
image: "postgres:9.6"
container_name: "postgres"
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
ports:
- "5432:5432"
volumes:
- ./data/postgres:/var/lib/postgresql/data
initdb:
image: learning/airflow
entrypoint: airflow initdb
depends_on:
- postgres
webserver:
image: learning/airflow
restart: always
entrypoint: airflow webserver
healthcheck:
test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
ports:
- "8080:8080"
depends_on:
- postgres
volumes:
- ./airflow/dags:/opt/airflow/dags
- ./airflow/plugins:/opt/airflow/plugins
- ./data/logs:/opt/airflow/logs
scheduler:
image: learning/airflow
restart: always
entrypoint: airflow scheduler
healthcheck:
test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-scheduler.pid ]"]
interval: 30s
timeout: 30s
retries: 3
depends_on:
- postgres
volumes:
- ./airflow/dags:/opt/airflow/dags
- ./airflow/plugins:/opt/airflow/plugins
- ./data/logs:/opt/airflow/logs
To use it, first run docker-compose up postgres, then docker-compose up initdb and then docker-compose up webserver scheduler. That's it!
spinning up two docker containers alone may not achieve your goal, as you would need communications between containers. You can manually set up a docker network between your containers, although I haven't tried this approach personally.
An easier way is to use docker-compose, which you can define your resources in a yml file, and let docker-compose create them for you.
version: '2.1'
services:
webserver:
image: puckel/docker-airflow:1.10.4
restart: always
...
scheduler:
image: puckel/docker-airflow:1.10.4
restart: always
depends_on:
- webserver
...
You can find the complete file here
Note: your question applies to any processes, not only Airflow
It's not recommended, of course, but you can find Docker documentation on supervisor which monitors and runs multiple processes under a single supervisord daemon
https://docs.docker.com/config/containers/multi-service_container/
This question already has an answer here:
Docker-compose depends on not waiting until depended on service isn't fully started
(1 answer)
Closed 3 years ago.
I am dealing with a docker composer project. Here is the compose file :
version: '3.3'
services:
tomcatserver:
build: ./mavenServer
depends_on:
- db
ports:
- "8010:8080"
db:
image: mariadb
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: "true"
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "tpformation"
MYSQL_USER: "tomcat"
MYSQL_PASSWORD: "tomcat"
expose:
- "3306"
when I start my stack with docker-composer up, it is always the maven containers that starts first. However, it should be the db one. Could you help please.
Docker Compose used to have a condition configuration under depends_on. This is no longer the case.
From this page:
depends_on does not wait for db and redis to be “ready” before starting web - only until
they have been started. If you need to wait for a service to be ready, see Controlling
startup order for more on this problem and strategies for solving it.
Version 3 no longer supports the condition form of depends_on.
The simplest way to handle this issue, is to simply avoid it. Do docker-compose up -d db and then you can start and restart anything that depends on it (docker-compose up tomcatserver).
In most cases, dependencies normally "become healthy" before any other container needs them, so you can just docker-compose up your-app and never notice the issue.
Of course, all the above statements are valid assuming you are not using the same docker-compose in production.
If however, you are still interested in hardening the startup sequence, you can take a look at the Control startup and shutdown order docker manual page, for some possibilities.
Relevant snippet from this page:
version: "2"
services:
web:
build: .
ports:
- "80:8000"
depends_on:
- "db"
command: ["./wait-for-it.sh", "db:5432", "--", "python", "app.py"]
db:
image: postgres
Personally, I recommend against such tricks wherever possible.
docker-compose 2.1 offers the nice feature to specify a condition with depends_on. The current docker-compose documentation states:
Version 3 no longer supports the condition form of depends_on.
Unfortunately the documentation does not explain, why the condition form was removed and is lacking any specific recommondation on how to implement that behaviour using V3 upwards.
There's been a move away from specifying container dependencies in compose. They're only valid at startup time and don't work when dependent containers are restarted at run time. Instead, each container should include mechanism to retry to reconnect to dependent services when the connection is dropped. Many libraries to connect to databases or REST API services have configurable built-in retries. I'd look into that. It is needed for production code anyway.
From 1.27.0, 2.x and 3.x are merged with COMPOSE_SPEC schema.
version is now optional. So, you can just remove it and specify a condition as before:
services:
web:
build: .
depends_on:
redis:
condition: service_healthy
redis:
image: redis
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 30
There are some external tools that let you mimic this behaviour. For example, with the dockerize tool you can wrap your CMD or ENTRYPOINT with dockerize -wait and that will prevent running your application until specified services are ready.
If your docker-compose file used to look like this:
version: '2.1'
services:
kafka:
image: spotify/kafka
healthcheck:
test: nc -z localhost 9092
webapp:
image: foo/bar # your image
healthcheck:
test: curl -f http://localhost:8080
tests:
image: bar/foo # your image
command: YOUR_TEST_COMMAND
depends_on:
kafka:
condition: service_healthy
webapp:
condition: service_healthy
then you can use dockerize in your v3 compose file like this:
version: '3.0'
services:
kafka:
image: spotify/kafka
webapp:
image: foo/bar # your image
tests:
image: bar/foo # your image
command: dockerize -wait tcp://kafka:9092 -wait web://webapp:8080 YOUR_TEST_COMMAND
Just thought I'd add my solution for when running postgres and an application via docker-compose where I need the application to wait for the init sql script to complete before starting.
dockerize seems to wait for the db port to be available (port 5432) which is the equivilant of depends_on which can be used in docker 3:
version: '3'
services:
app:
container_name: back-end
depends_on:
- postgres
postgres:
image: postgres:10-alpine
container_name: postgres
ports:
- "5432:5432"
volumes:
- ./docker-init:/docker-entrypoint-initdb.d/
The Problem:
If you have a large init script the app will start before that completes as the depends_on only waits for the db port.
Although I do agree that the solution should be implemented in the application logic, the problem we have is only for when we want to run tests and prepopulate the database with test data so it made more sense to implement a solution outside the code as I tend not like introducing code "to make tests work"
The Solution:
Implement a healthcheck on the postgres container.
For me that meant checking the command of pid 1 is postgres as it will be running a different command on pid 1 while the init db scripts are running
Write a script on the application side which will wait for postgres to become healthy. The script looks like this:
#!/bin/bash
function check {
STATUS=\`curl -s --unix-socket /var/run/docker.sock http:/v1.24/containers/postgres/json | python -c 'import sys, json; print json.load('sys.stdin')["State"]["Health"]["Status"]'\`
if [ "$STATUS" = "healthy" ]; then
return 0
fi
return 1
}
until check; do
echo "Waiting for postgres to be ready"
sleep 5
done
echo "Postgres ready"
Then the docker-compose should mount the directories of the scripts so that we don't edit the Dockerfile for the application and if we're using a custom postgres image, this way we can continue to use the docker files for your published images.
We're also overriding the entry point defined in the docker file of the app so that we can run the wait script before the app starts
version: '3'
services:
app:
container_name: back-end
entrypoint: ["/bin/sh","-c","/opt/app/wait/wait-for-postgres.sh && <YOUR_APP_START_SCRIPT>"]
depends_on:
- postgres
volumes:
- //var/run/docker.sock:/var/run/docker.sock
- ./docker-scripts/wait-for-postgres:/opt/app/wait
postgres:
image: postgres:10-alpine
container_name: postgres
ports:
- "5432:5432"
volumes:
- ./docker-init:/docker-entrypoint-initdb.d/
- ./docker-scripts/postgres-healthcheck:/var/lib
healthcheck:
test: /var/lib/healthcheck.sh
interval: 5s
timeout: 5s
retries: 10
I reached this page because one container would not wait for the one depending upon and I had to run a docker system prune to get it working. There was an orphaned container error that prompted me to run the prune.