Debugging a docker compose mariadb healthcheck - docker

Question

I'm trying to add an healthcheck in a mariadb service but it then prevents the client connection to the server, showing the message:
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (0)
The connection can be established and everything runs fine without the following healthcheck:
healthcheck:
test: /usr/bin/learnintouch/db-health-check.sh
interval: 30s
timeout: 10s
retries: 3
... with the healthcheck behaving as expected:
root#428b1e6c7c1a:/usr/bin/learnintouch/www/folkuniversitet# /usr/bin/learnintouch/db-health-check.sh
+ /usr/bin/mysql/install/bin/mysql --protocol=tcp -h mysql -u root -pxxxxxx -e 'show databases;'
Warning: Using a password on the command line interface can be insecure.
+--------------------+
| Database |
+--------------------+
| db_engine |
| db_learnintouch |
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
The initial working service configuration is:
mysql:
image: localhost:5000/mariadb:10.1.24
environment:
- MYSQL_ROOT_PASSWORD=xxxxxx
networks:
- learnintouch
volumes:
- "~/dev/docker/projects/learnintouch/volumes/database/data:/usr/bin/mariadb/install/data"
- "~/dev/docker/projects/learnintouch/volumes/logs:/usr/bin/mariadb/install/logs"
deploy:
replicas: 1
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 10s
The failing service configuration is:
mysql:
image: localhost:5000/mariadb:10.1.24
environment:
- MYSQL_ROOT_PASSWORD=xxxxxx
networks:
- learnintouch
volumes:
- "~/dev/docker/projects/learnintouch/volumes/database/data:/usr/bin/mariadb/install/data"
- "~/dev/docker/projects/learnintouch/volumes/logs:/usr/bin/mariadb/install/logs"
deploy:
replicas: 1
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 10s
healthcheck:
test: /usr/bin/learnintouch/db-health-check.sh
interval: 30s
timeout: 10s
retries: 3
At server startup, the mariadb log is exactly the same, with or without the healthcheck, and ends with:
2017-12-29 10:27:48 139873749194560 [Note] /usr/bin/mariadb/install/bin/mysqld: ready for connections.
Version: '10.1.24-MariaDB' socket: '/usr/bin/mariadb/install/tmp/mariadb.sock' port: 3306 Source distribution
Here is the db-health-check.sh bash script:
#!/bin/bash -x
/usr/bin/mysql/install/bin/mysql --protocol=tcp -h mysql -u root -pxxxxxx -e "show databases;" || exit 1
I'm on Docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Experimental: false
UPDATE:
There were a few issues in fact, which together made the whole thing puzzling.
The first one was that the db-health-check.sh file was sitting in the client application container, and thus could not be found by the service database container, when this latter needed to perform its healthcheck.
The second one was that the db-health-check.sh file used a wrong path for its mysql client /usr/bin/mariadb/install/bin/mysql.
The third one was that the db-health-check.sh file used a wrong hostname in its connection, wrongly using mysql instead of localhost.
The fourth one was that the root user could not connect on the 1045 (28000): Access denied for user 'root'#'localhost' as it should not have any password as in /usr/bin/mariadb/install/bin/mysql --protocol=tcp -h localhost -u root -e "show databases;" || exit 1.
After correcting these points, the application can start and run fine even with the healthcheck in place.
The one thing that blurred the lines even further, was that the service is not available until the first healthcheck has passed successfully. Indeed I configured an heatlcheck as:
healthcheck:
test: exit 0
interval: 60s
timeout: 10s
retries: 3
with the container state showing this at first:
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": []
}
and the application was stillnot able to connect to the service until after the first interval of 60s expired and the healthcheck being successful.
I should therefore have a short interval time so as to have the application not wait for too long to be available.