Rails container cannot connect to mysql container with gitlab ci - ruby-on-rails

Question

I am setting up a simple gitlab ci for a Rails app with build, test and release stages:
build:
stage: build
script:
- docker build --pull -t $TEST_IMAGE .
- docker push $TEST_IMAGE
test:
stage: test
services:
- docker:dind
script:
- docker pull $TEST_IMAGE
- docker run -d --name mysql -e MYSQL_ROOT_PASSWORD=mysql_strong_password mysql:5.7
- docker run -e RAILS_ENV=test --link mysql:db $TEST_IMAGE bundle exec rake db:setup
build succeeds building the docker image and pushing to registry
test launches another mysql container which I use as my host db, but fails when establishing connection to mysql.
Couldn't create database for {"host"=>"db", "adapter"=>"mysql2", "pool"=>5, "username"=>"root", "encoding"=>"utf8", "timeout"=>5000, "password"=>"mysql_strong_password", "database"=>"my_tests"}, {:charset=>"utf8"}
(If you set the charset manually, make sure you have a matching collation)
rails aborted!
Mysql2::Error: Can't connect to MySQL server on 'db' (111 "Connection refused")
I also tried creating seperate docker network using --network instead of link approach, did not help.
That happens only on Gitlab runner instance. When I perform those steps on local machine it works fine.
After much reading I get to think it is a bug with docker executor. Am I missing something?

Answer 1

Connection refused indicates that the containers know how to reach each other, but the target container does not have anything accepting connections on the selected port. This most likely means you are starting your application up before the database has finished initializing. My recommendation is to update/create your application or create an entrypoint in your application container that polls the database for it to be up and running, and fail after a few minutes if it doesn't start up. I'd also recommend using networks and not links since links are deprecated and do not gracefully handle containers being recreated.
The behavior you're seeing is documented in the mysql image:
No connections until MySQL init completes
If there is no database initialized when the container starts, then a default database will be created. While this is the expected behavior, this means that it will not accept incoming connections until such initialization completes. This may cause issues when using automation tools, such as docker-compose, which start several containers simultaneously.
If the application you're trying to connect to MySQL does not handle MySQL downtime or waiting for MySQL to start gracefully, then a putting a connect-retry loop before the service starts might be necessary. For an example of such an implementation in the official images, see WordPress or Bonita.
From the linked wordpress example, you can see their retry code:
$maxTries = 10;
do {
$mysql = new mysqli($host, $user, $pass, '', $port, $socket);
if ($mysql->connect_error) {
fwrite($stderr, "\n" . 'MySQL Connection Error: (' . $mysql->connect_errno . ') ' . $mysql->connect_error . "\n");
--$maxTries;
if ($maxTries <= 0) {
exit(1);
}
sleep(3);
}
} while ($mysql->connect_error);
A sample entrypoint script to wait for mysql without changing your application itself could look like:
#!/bin/sh
wait-for-it.sh mysql:3306 -t 300
exec "$#"
The wait-for-it.sh comes from vishnubob/wait-for-it, and the exec "$#" at the end replaces pid 1 with the command you passed (e.g. bundle exec rake db:setup). The downside of this approach is that the database could potentially be listening on a port before it is really ready to accept connections, so I still recommend doing a full login with your application in a retry loop.