Docker compose getting mysql db fully started before flask app starts - docker

Question

I just started to dockerize my app. I've built my Dockerfile and docker-compose.yml and everything seems to work fine except one thing. There are times my flask app will start too quick and throw a connection refused error (because the MySQL db is not fully up). I am using healthcheck to check if the db is up but this seems to not be reliable (I'm even making sure I can see show databases, but mysql apparently initializes more things after the healthcheck passes? not sure what the healthcheck is for then). In my output, I see that the db does get created first but it is still initializing when the flask app starts up. Ideally, when I run docker-compose up I want to be able to see this line first,
db_1_eae741771281 | 2018-11-10T00:50:21.473098Z 0 [Note] mysqld: ready for connections.
and then start my flask app entry point. Currently, it doesn't do this.
Is there a more reliable way to ensure the MySQL is fully up before starting my start.sh?
Dockerfile:
FROM python:3.5-alpine
RUN apk update && apk upgrade
RUN apk add --no-cache curl python build-base openldap-dev python2-dev python3-dev pkgconfig python-dev libffi-dev musl-dev make gcc
RUN pip install --upgrade pip
RUN adduser -D user
WORKDIR /home/user
COPY requirements.txt requirements.txt
RUN python -m venv venv
RUN venv/bin/pip install -r requirements.txt
COPY app app
COPY start.sh ./
RUN chmod +x start.sh
RUN chown -R user:user ./
USER user
EXPOSE 5000
ENTRYPOINT ["./start.sh"]
docker-compose.yml:
version: "2.1"
services:
db:
image: mysql:5.7
ports:
- "32000:3306"
environment:
- MYSQL_DATABASE=mydb
- MYSQL_USER=user
- MYSQL_PASSWORD=user123
- MYSQL_ROOT_PASSWORD=user123
volumes:
- ./db:/docker-entrypoint-initdb.d/:ro
healthcheck:
test: "mysql --user=user --password=user123 --execute \"SHOW DATABASES;\""
timeout: 20s
retries: 20
app:
build: ./
ports:
- "5000:5000"
depends_on:
db:
condition: service_healthy
start.sh
#!/bin/sh
source venv/bin/activate
# Start Gunicorn processes
echo Starting Gunicorn.
exec gunicorn -b 0.0.0.0:5000 wsgi --chdir my_app --timeout 9999 --workers 3 --access-logfile - --error-logfile - --capture-output --log-level debug

Answer 1

OK I also had problems with health_check...
Maybe not the most optimal, but the most reliable solution is to use a MySQL client (mysqladmin) to ping your MySQL server before starting your application.
1 - Create a wait.sh script (db is your MySQL service name here) :
#!/bin/sh
# Wait until MySQL is ready
while ! mysqladmin ping -h"db" -P"3306" --silent; do
echo "Waiting for MySQL to be up..."
sleep 1
done
2 - Get a MySQL client from your app Dockerfile :
# install mysql client, will be used to ping mysql
apt-get -y install mysql-client
3 - In your docker-compose.yml file, just add scripts to your container (I used volumes but you can keep using COPY) and run wait.sh before start.sh :
app:
build: ./
ports:
- "5000:5000"
depends_on:
db:
command: bash -c "/usr/local/bin/wait.sh && /usr/local/bin/start.sh"
volumes:
- ./start.sh:/usr/local/bin/start.sh
- ./wait.sh:/usr/local/bin/wait.sh
This should work.
If you really don't want having to download a MySQL client, try this (again db is your MySQL service name here). It has worked in most of my project but not in all (may depend of the distribution?) :
#!/bin/sh
# Wait until MySQL is ready
while ! exec 6<>/dev/tcp/db/3306; do
echo "Trying to connect to MySQL at 3306..."
sleep 5
done
PS : avoid naming your services "app" or "db", you may have problems later if you have other containers with those same service names (even in different networks).

Answer 2

While using a health check is easier, it totally depends on how reliable the check is.
Another approach would be to rely on projects like wait-for-it or wait-for, in your app container.
Since you are getting a connection refused, these scripts could return only once the connection is possible and your app can start only after.
Also, in case that doesn't work too, you could have a separate script (python in your case) to check until the DB is ready and you can call this script in your start.sh before starting the flask app.

Answer 3

This is a common problem with multi containers. It is difficult to control the speed at which different containers start. Container orchestration solution like Kubernetes might be able to help you in such cases.
Kubernetes has the concept of init containers which run to completion before your dependent container can start. You can find sample of init container here
https://www.handsonarchitect.com/2018/08/understand-kubernetes-object-init.html
This youtube vide might be helpful for you as well
https://www.youtube.com/watch?v=n2FPsunhuFc