I have my own CI server with gitlab and I'm trying to run docker runner (version 10.6) with this configuration:
image: php:7.1
services:
- mysql:latest
- redis:latest
- elasticsearch:latest
before_script:
- bash ci/install.sh > /dev/null
- php composer install -a
stages:
- test
test:
stage: test
variables:
API_ENVIRONMENT: 'test'
script:
- echo "Running tests"
- php composer app:tests
But everytime when I pull docker container with elastic, I've got error message:
*** WARNING: Service runner-1de473ae-project-225-concurrent-0-elasticsearch-2 probably didn't start properly.
Error response from daemon: Conflict. The container name "/runner-1de473ae-project-225-concurrent-0-elasticsearch-2-wait-for-service" is already in use by container "f26f56b2905e8c3da1977bc7c48e7eba00e943532146b7a8711f91fe67b67c3b". You have to remove (or rename) that container to be able to reuse that name.
*********
I also tried to log into this server and list all containers, but there is only redis one:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5cec961e03b2 811c03fb36bc "gitlab-runner-ser..." 39 hours ago Up 39 hours runner-1de473ae-project-247-concurrent-1-redis-1-wait-for-service
After googling this problem I found this issue: https://gitlab.com/gitlab-org/gitlab-runner/issues/2667 then I update runner to 10.6, but problem persists.
After all, there is no running elastic on my server, then my tests fails on:
FAILED: Battle/BattleDataElasticProviderTest.php method=testGetLocalBattles
Exited with error code 255 (expected 0)
Elasticsearch\Common\Exceptions\NoNodesAvailableException: No alive nodes found in your cluster
Is there any way, how to start ES or at least put ES into more verbosive mode?
Thanks!
When a container is stopped, it still exists even though it's now in an exited state. Using command docker ps -a shows you all the running and exited containers.
To start a new container with an already existing name, you need to first manually remove the old container occupying this name by using docker rm.
A convenient way is to use the --rm argument when starting a container, the container will be automatically removed once it stops.
Related
I've got a gitlab-ci.yml job that brings up two Docker containers. The first container is a standard Redis container, the second runs a custom process that populates the container. The goal of this job is to get the Redis container's dump.rdb file and pass that along to other jobs. Therefore, my job looks like this:
script:
- cd subdir
- docker run -d --name redis -v $PWD/redis-data:/data redis:latest
- docker run my_custom_image
- docker stop redis
artifacts:
paths:
- subdir/redis-data/dump.rdb
expire_in: 5 minutes
So far so good. my_custom_image connects to the redis container and reports that it loaded everything correctly, and both containers shutdown cleanly. However, subdir/redis-data is empty. No dump.rdb, no nothing. Adding a ls -la subdir/redis-data after the docker stop call confirms this. I can run on my machine and everything is working correctly, it's only on gitlab that this breaks.
It looks to me like the gitlab-runner isn't running every step in the same directory, but that doesn't make much sense. Can anyone tell me why the mounted volume isn't getting the dump.rdb?
I found an old bug report here from someone who was having the same problem. It didn't look like anyone was able to fix the issue, but they suggested just doing a docker cp to pull the data out of the running container. So I did:
script:
- cd subdir
- docker run -d --name redis -v $PWD/redis-data:/data redis:latest
- docker run my_custom_image
- echo SAVE | docker exec -i redis redis-cli
- docker cp redis:/data/dump.rdb redis-data/dump.rdb
- docker stop redis
artifacts:
paths:
- subdir/redis-data/dump.rdb
expire_in: 5 minutes
The SAVE command was to ensure redis had flushed everything to disk before running docker cp to pull the dump file. This is working fine for me, wish it was more elegant but there you go.
I tested a gitlab-runner on a virtual machine, it worked perfectly. I followed this tutorial at part Use docker-in-docker executor :
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
When i register a runner with exactly the same configuration on my dev server, the runner is called when there is a commit but i got alot of errors :
*** WARNING: Service runner-XXX-project-XX-concurrent-X-docker-X probably didn't start properly.
ContainerStart: Error response from daemon: Cannot link to a non running container: /runner-XXX-project-XX-concurrent-X-docker-X AS /runner-XXX-project-XX-concurrent-X-docker-X-wait-for-service/service (executor_docker.go:1337:1s)
DEPRECATION: this GitLab server doesn't support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab
$ docker info
error during connect: Get http://docker:2375/v1.39/info: dial tcp: lookup docker on MY.DNS.IP:53: no such host
ERROR: Job failed: exit code 1
I believe all these error are due to the first warning. I tried to :
Add a second DNS with 8.8.8.8 IP to my machine, same error
Add privileged=true manually in /etc/gitlab-runner/config.toml, same error, so it's not due to the privileged = true parameter
Replace tcp://docker:2375 by tcp://localhost:2375, can't find docker daemon on the machine when docker info
gitlab-ci.yml content :
image: docker:stable
stages :
- build
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
services:
- docker:dind
before_script:
- docker info
build-folder1:
stage: build
script:
- docker build -t image1 folder1/
- docker run --name docker1 -p 3001:5000 -d image1
only:
refs:
- dev
changes:
- folder1/**/*
build-folder2:
stage: build
script:
- docker build -t image2 folder2/
- docker run --name docker2 -p 3000:3000 -d image2
only:
refs:
- dev
changes:
- folder2/**/*
If folder1 of branch dev is modified, we build and run the docker1
If folder2 of branch dev is modified, we build and run the docker2
docker version on dev server :
docker -v
Docker version 17.03.0-ce, build 3a232c8
gitlab-runner version on dev server :
gitlab-runner -v
Version: 11.10.1
I will try to provide an answer for you, as I come to fix this same problem when trying yo run DinD.
This message:
*** WARNING: Service runner-XXX-project-XX-concurrent-X-docker-X probably didn't start properly.
Means that either you have not properly configured your runner, or it is not linked by the gitlab-ci.yml file. You should be able to ckeck the ID of the runner used in the log page at Gitlab.
To start with, verify that you entered the gitlab-runner register command right, with the proper registration token.
Second, since you are setting a specific runner manually, verify that you have set some unique tag to it (eg. build_docker), and call it from your gitlab-ci.yml file. For example:
...
build-folder1:
stage: build
script:
- docker build -t image1 folder1/
- docker run --name docker1 -p 3001:5000 -d image1
tags:
- build_docker
...
That way it should work.
I am setting up a simple gitlab ci for a Rails app with build, test and release stages:
build:
stage: build
script:
- docker build --pull -t $TEST_IMAGE .
- docker push $TEST_IMAGE
test:
stage: test
services:
- docker:dind
script:
- docker pull $TEST_IMAGE
- docker run -d --name mysql -e MYSQL_ROOT_PASSWORD=mysql_strong_password mysql:5.7
- docker run -e RAILS_ENV=test --link mysql:db $TEST_IMAGE bundle exec rake db:setup
build succeeds building the docker image and pushing to registry
test launches another mysql container which I use as my host db, but fails when establishing connection to mysql.
Couldn't create database for {"host"=>"db", "adapter"=>"mysql2", "pool"=>5, "username"=>"root", "encoding"=>"utf8", "timeout"=>5000, "password"=>"mysql_strong_password", "database"=>"my_tests"}, {:charset=>"utf8"}
(If you set the charset manually, make sure you have a matching collation)
rails aborted!
Mysql2::Error: Can't connect to MySQL server on 'db' (111 "Connection refused")
I also tried creating seperate docker network using --network instead of link approach, did not help.
That happens only on Gitlab runner instance. When I perform those steps on local machine it works fine.
After much reading I get to think it is a bug with docker executor. Am I missing something?
Connection refused indicates that the containers know how to reach each other, but the target container does not have anything accepting connections on the selected port. This most likely means you are starting your application up before the database has finished initializing. My recommendation is to update/create your application or create an entrypoint in your application container that polls the database for it to be up and running, and fail after a few minutes if it doesn't start up. I'd also recommend using networks and not links since links are deprecated and do not gracefully handle containers being recreated.
The behavior you're seeing is documented in the mysql image:
No connections until MySQL init completes
If there is no database initialized when the container starts, then a default database will be created. While this is the expected behavior, this means that it will not accept incoming connections until such initialization completes. This may cause issues when using automation tools, such as docker-compose, which start several containers simultaneously.
If the application you're trying to connect to MySQL does not handle MySQL downtime or waiting for MySQL to start gracefully, then a putting a connect-retry loop before the service starts might be necessary. For an example of such an implementation in the official images, see WordPress or Bonita.
From the linked wordpress example, you can see their retry code:
$maxTries = 10;
do {
$mysql = new mysqli($host, $user, $pass, '', $port, $socket);
if ($mysql->connect_error) {
fwrite($stderr, "\n" . 'MySQL Connection Error: (' . $mysql->connect_errno . ') ' . $mysql->connect_error . "\n");
--$maxTries;
if ($maxTries <= 0) {
exit(1);
}
sleep(3);
}
} while ($mysql->connect_error);
A sample entrypoint script to wait for mysql without changing your application itself could look like:
#!/bin/sh
wait-for-it.sh mysql:3306 -t 300
exec "$#"
The wait-for-it.sh comes from vishnubob/wait-for-it, and the exec "$#" at the end replaces pid 1 with the command you passed (e.g. bundle exec rake db:setup). The downside of this approach is that the database could potentially be listening on a port before it is really ready to accept connections, so I still recommend doing a full login with your application in a retry loop.
I have gitlab-runner installed locally.
km#Karls-MBP ~ $ gitlab-runner --version
Version: 10.4.0
Git revision: 857480b6
Git branch: 10-4-stable
GO version: go1.8.5
Built: Mon, 22 Jan 2018 09:47:12 +0000
OS/Arch: darwin/amd64
Docker:
km#Karls-MBP ~ $ docker --version
Docker version 17.12.0-ce, build c97c6d6
.gitlab-ci.yml:
image: docker/compose:1.19.0
before_script:
- echo wtf
test:
script:
- echo test
Results:
km#Karls-MBP ~ $ sudo gitlab-runner exec docker --docker-privileged test
WARNING: Since GitLab Runner 10.0 this command is marked as DEPRECATED and will be removed in one of upcoming releases
WARNING: You most probably have uncommitted changes.
WARNING: These changes will not be tested.
Running with gitlab-runner 10.4.0 (857480b6)
on ()
Using Docker executor with image docker/compose:1.19.0 ...
Using docker image sha256:be4b46f2adbc8534c7f6738279ebedd6106969695f5e596079e89e815d375d9c for predefined container...
Pulling docker image docker/compose:1.19.0 ...
Using docker image docker/compose:1.19.0 ID=sha256:e06b58ce9de2ea3f11634e022ec814984601ea3a5180440c2c28d9217b713b30 for build container...
Running on runner--project-0-concurrent-0 via x.x.x...
Cloning repository...
Cloning into '/builds/project-0'...
done.
Checking out b5a262c9 as km/ref...
Skipping Git submodules setup
No such command: sh
Commands:
build Build or rebuild services
bundle Generate a Docker bundle from the Compose file
config Validate and view the Compose file
create Create services
down Stop and remove containers, networks, images, and volumes
events Receive real time events from containers
exec Execute a command in a running container
help Get help on a command
images List images
kill Kill containers
logs View output from containers
pause Pause services
port Print the public port for a port binding
ps List containers
pull Pull service images
push Push service images
restart Restart services
rm Remove stopped containers
run Run a one-off command
scale Set number of containers for a service
start Start services
stop Stop services
top Display the running processes
unpause Unpause services
up Create and start containers
version Show the Docker-Compose version information
Don't really know what the issue is.
It seems that the docker/compose image is configured with docker-compose as an entrypoint.
You can override the default entrypoint of the docker/compose image in your .gitlab-ci.yml file :
image:
name: docker/compose:1.19.0
entrypoint: [""]
before_script:
- echo wtf
test:
script:
- echo test
The docker/compose image has the command docker-compose as its entrypoint (until version 1.24.x), which enables a usage similar to this (assuming a compatible volume mount):
docker run --rm -t docker/compose -f some-dir/compose-file.yml up
Unfortunately that same feature makes it incompatible with usage within GitLab CI’s Docker Runner. Theoretically you could have a construct like this:
job-name:
image: docker/compose:1.24.1
script:
- up
- --build
- --force-recreate
But the GitLab Docker Runner assumes the entrypoint is /bin/bash - or at least functions likewise (many Docker images thoughtfully use a shell script with "$#" as its final line for the entrypoint) - and from the array elements that you specify for the script, it creates its own temporary shell script on the fly. It starts with statements like set -e and set -o pipeline and will be used in a statement like sh temporary-script.sh as the container command. That’s what causes the unexpected error message you got.
This behaviour was recently documented more clearly:
The Docker executor doesn’t overwrite the ENTRYPOINT of a Docker image.
That means that if your image defines the ENTRYPOINT and doesn’t allow to run scripts with CMD, the image will not work with the Docker executor.
Overriding the entrypoint with [""] will allow usage of docker/docker-compose (before version 1.25.x) with the Docker Runner, but the script that GitLab will create on the fly is not going to run as process 1 and because of that the container will not stop at the end of the script. Example:
job-name:
image:
name: docker/docker-compose
entrypoint: [""]
script:
- docker-compose
- up
- --build
- --force-recreate
At the time I write this the latest version of docker/docker-compose is 1.25.0-rc2. Your mileage may vary, but it suffices for my purposes and entirely resolves both problems.
I tried to use docker-gc for automatically collecting unused docker images and containers. I tried this config in docker-compose for running:
gc:
container_name: docker-gc
build: ./docker/docker-gc
dockerfile: Dockerfile
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /etc:/etc
When I first run, all unused docker images and containers are removed automatically. But after that, this container exit. I want this container runs and check periodically.
Just another approach:
Use cron to run docker-compose periodically.
About exiting container:
Containers run a script/service until its finished/killed.
For example library/nginx runs nginx service. The container will be in running state until the nginx service stopped/killed. Then it will show up as exited with appropriate exit code.
entrypoint/cmd directive will specify what script/service container will execute when run with no user override.