How to run several gitlab-runners in one docker-compose-setup - docker

I'm running 10 runners on one server with docker-compose. Sometimes the pipeline failes with errors like this:
fatal: Unable to create '/builds/xxx.de/xxx.de/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
or
Reinitialized existing Git repository in /builds/xxx.de/xxx.de/.git/
fatal: shallow file has changed since we read it
Each runner has a configuration like this, only the name, id and tokens are different:
aconcurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "runner-1-farm-1"
url = "https://git.xyz.com/"
id = 12
token = "-94xSx9-blablabla"
token_obtained_at = 2022-09-16T07:52:10Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
host = "tcp://dind:2375"
tls_verify = false
image = "registry.xyz.com/docker/docker:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
docker-compose config:
version: "3.9"
services:
dind:
image: docker:20-dind
restart: always
privileged: true
environment:
DOCKER_TLS_CERTDIR: ""
command:
- --storage-driver=overlay2
runner_1:
restart: always
image: registry.gitlab.com/gitlab-org/gitlab-runner:alpine
depends_on:
- dind
environment:
- DOCKER_HOST=tcp://dind:2375
- REGISTRATION_TOKEN=${REGISTRATION_TOKEN}
volumes:
- ./config:/etc/gitlab-runner:rw
So far, the errors only occur in one project currently.
How can I fix it?

Related

Gitlab runner: run mvn test with need of Docker on tests

How to configure gitlab-runner config.toml and .gitlab-ci.yml to run 'mvn test' with the need of Docker daemon running in the same stage ? I need this to make testcontainer work in CI.
So far I have solved this by creating an image called 'ci-server' with both maven and Docker. I want to solve this following the approach described in Testcontainers website, in this link, but none of these examples worked for me.
gitlab-ci.yml (example)
variables:
DOCKER_HOST: "tcp://docker:2375"
DOCKER_TLS_CERTDIR: ""
services:
- docker:dind
stages:
- clean-build-test
maven-build-test:
image: ci-server:1.0
stage: clean-build-test
tags:
- xxx
script:
- mvn $MAVEN_CLI_OPTS clean package -Dspring.profiles.active=test
gitlab-runner configuration:
[[runners]]
name = "xxx"
url = "xxx"
id = 99
token = "xxx"
token_obtained_at = 2023-01-18T21:24:45Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "ci-server:1.0"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
pull_policy = "if-not-present"

Gitlab Runner Docker login not working error during connect: Post http://docker:2375/v1.40/auth: dial tcp: lookup docker on 67.207.67.3:53

I am using docker executor on gitlab runner as an image I am using docker:stable
I am getting this error:
$ docker login gitlab.mydomain.com:5050 -u myusername-p mytoken
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post http://docker:2375/v1.40/auth: dial tcp: lookup docker on 67.207.67.3:53: no such host
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
Here is my config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "dev-env"
url = "https://gitlab.mydomain.com/"
token = "REDACTED"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Here is my ci file
deploy-dev:
stage: deploy
before_script:
- apk add make
script:
- docker login gitlab.mydomain.com:5050 -u myusnerma -p mytoken
only:
- mybranch
tags:
- dev
You must add the docker:dind service to your job configuration:
deploy-dev:
variables:
# these values may need to be different if using TLS, k8s, etc.
# You can alternatively set defaults in your runner config
DOCKER_TLS_CERTDIR: ""
DOCKER_HOST: "tcp://docker:2375"
services:
- docker:dind
# ...
For the dind container to work, your runner must allow privileged containers:
[runners.docker]
privileged = true
# ...
Beside privileged = true you should also map /var/run/docker.sock inside docker container to /var/run/docker.sock on host. something like this:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "my-runner"
url = "https://gitlab.mydomain.com/"
token = "GITLAB_RUNNER_TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
More info from techoverflow

gitlab-runner docker exeuctor (dind) - Error https://docker:2375/v1.40/info dial tcp: lookup docker on

Ive an issue with gitlab-runner executor docker. After I ran my gitlab-ci.yml file , pipeline fail on step docker info during before_script with:
Running with gitlab-runner 13.10.0 (54944146)
on docker-runner N2_yEgUD
Preparing the "docker" executor 00:07
Using Docker executor with image docker:19.03.0 ...
Starting service docker:19.03.0-dind ...
Pulling docker image docker:19.03.0-dind ...
Using docker image sha256:fd0c64832f7e46b63a180e6000dbba7ad7a63542c5764841cba73429ba74a39e for docker:19.03.0-dind with digest docker#sha256:442ac4b31375cbe617f31759b5199d240f11d5f430e54946575b274b2fb6f096 ...
Waiting for services to be up and running...
.............................................................................................
$ docker info
Client:
Debug Mode: false
Server:
ERROR: error during connect: Get https://docker:2375/v1.40/info: dial tcp: lookup docker on 127.0.0.53:53: server misbehaving
errors pretty printing info
Cleaning up file based variables 00:01
ERROR: Job failed: exit code 1
I did a research on stack and official gitlab forum but none of the answers fix my issue:
add to .toml -> volume: ['/certs/client']
run against old: docker:18.x.x / docker:18.x.x -dind | docker:stable / docker:dind
run with: DOCKER_TLS_CERTDIR:""
run with/without:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
add endpoint to service:
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
Content of gitlab-runner toml
concurrent = 1
check_interval = 0
log_level = "debug"
[session_server]
session_timeout = 1800
[[runners]]
name = "docker-runner"
url = "xxxxxxxx"
token = "xxxxxxx"
executor = "docker"
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
privileged = true
image = "docker:19.03.12"
disable_cache = false
volumes = ["/cache", "/certs/client"]
network_mode = "host"
Content of gitlab-ci.yml
image: docker:19.03.0
services:
- docker:19.03.0-dind
stages:
- build
- test_framework
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
CONTAINER_TEST_IMAGE: xxxx
CONTAINER_RELEASE_IMAGE: xxxx
before_script:
- docker info
- docker login -u xxxx -p $CI_JOB_TOKEN xxxx
build:
stage: build
tags:
- adm-docker
script:
- docker pull $CONTAINER_RELEASE_IMAGE || true
- docker build -t $CONTAINER_TEST_IMAGE --cache-from $CONTAINER_RELEASE_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
timeout: 1 hours
.test_commit: &test_commit
stage: test_framework
image: $CONTAINER_TEST_IMAGE
tags:
- adm-docker
timeout: 1 hours
artifacts:
reports:
junit: 'results/xunit.xml'
expire_in: 1 day
except:
- master
test-unit:
<<: *test_commit
script:
- python3 -m pytest --junitxml=results/xunit.xml test_unit/
Only one thing fix issue (workaround issue). When I add to .toml
volume: ["/var/run/docker.sock:/var/run/docker.sock"]
But after that Iam loosing DIND possibility to run my gitlab-ci.yml with different image for test stage (without using under script: -docker run MY_IMAGE python3....).
Which is not what I want
gitlab-runner under Ubuntu20 / Docker version 20.10.5, build 55c4c88
Ive worked with very similar gitlab-ci.yml around 1Yr ago and there was no issue with docker executor
Any ideas/suggestions ?
I was able to fix issue by changing flow of my gitlab-ci.yml
image: docker:19.03.5
services:
- docker:19.03.5-dind
stages:
- build
- test_framework
- release
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
CONTAINER_TEST_IMAGE: xxxxx
CONTAINER_RELEASE_IMAGE: xxxxx
build:
stage: build
tags:
- adm-docker
before_script:
- docker info
- docker login -u xxxxx -p $CI_JOB_TOKEN xxxxx
script:
- docker pull $CONTAINER_RELEASE_IMAGE || true
- docker build -t $CONTAINER_TEST_IMAGE --cache-from $CONTAINER_RELEASE_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
timeout: 1 hours
.test_commit: &test_commit
stage: test_framework
tags:
- adm-docker
timeout: 1 hours
artifacts:
reports:
junit: 'results/xunit.xml'
expire_in: 1 day
except:
- master
test-unit:
<<: *test_commit
image: $CONTAINER_TEST_IMAGE
script:
- python3 -m pytest --junitxml=results/xunit.xml test_unit/
and toml
[[runners]]
name = "docker-runner"
url = xxxxx
token = xxxxx
executor = "docker"
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
privileged = true
image = "docker:19.03.12"
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
network_mode = "host"
issue was fixed by volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
and issue with test stage was cased by:
before_script:
- docker info
- docker login -u xxxxx -p $CI_JOB_TOKEN xxxxx
in root structure of .yml file. I had to move it to build stage
I hope that will help ppl in the future

How to run docker in docker in Gitlab ci?

I have /etc/gitlab-runner/config.toml
[[runners]]
name = "My Docker Runner"
url = "###"
token = "###"
executor = "docker"
pre_clone_script = "git config --global http.sslVerify false"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "tiangolo/docker-with-compose"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
[runners.cache]
Insecure = false
[runners.cache.s3]
[runners.cache.gcs]
.gitlab-ci.yml
stages:
- test
before_script:
- apk update
- apk upgrade
- apk add make
test:
stage: test
tags: [test]
script:
- make prod-up # docker-compose up -d --build
- make composer-init-dev
- make test
When I run the script in gitlab, I get the error:
$make prod-up
docker-compose -f docker-compose.prod.yml up -d --build
Couldn't connect to Docker daemon at http://docker:2375 - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
make: *** [Makefile:16: docker-prod-up] Error 1
ERROR: Job failed: exit code 2
Please help, what am I doing wrong?
You're missing Docker service (and it's best to specify image explicitly):
stages:
- test
before_script:
- apk update
- apk upgrade
- apk add make
test:
stage: test
image: tiangolo/docker-with-compose:latest
services:
- docker:dind
tags: [test]
script:
- make prod-up # docker-compose up -d --build
- make composer-init-dev
- make test
Try mounting docker.sock:
test:
stage: test
tags: [test]
script:
- make prod-up # docker-compose up -d --build
- make composer-init-dev
- make test
volumes:
- /var/run/docker.sock:/var/run/docker.sock

Gitlab CI push to docker-registry fails

I have GitLab server and a separate docker-registry. I have created a testuser with a testpassword.
My .gitlab-ci.yml
image: docker:latest
services:
- docker:18.09.7-dind
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
# See https://github.com/docker-library/docker/pull/166
DOCKER_TLS_CERTDIR: ""
REGISTRY: myregistry:5000/golang-test
stages:
- build
# - test
before_script:
- docker login -u testuser -p testpassword myregistry:5000
build_project:
stage: build
tags:
- docker-ci
script:
- docker ps
- docker build -t $REGISTRY .
- docker push $REGISTRY
When CI pipeline runs, I get following error.
WARNING! Using --password via the CLI is insecure. Use
--password-stdin. Error response from daemon: Get https://myregistry:5000/v2/: dial tcp: lookup worker1 on
192.168.123.456:53: no such host ERROR: Job failed: exit code 1
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "docker-ci"
url = "https://gitlab.sample.com:8000/"
token = "89MfxRTby83ny3eTkTf2"
tls-ca-file = "/etc/gitlab-runner/certs/gitlab.sample.com.crt"
executor = "docker"
pre_build_script = "export DOCKER_HOST=tcp://docker:2375"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "ubuntu:18.04"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
This was after suggested modifications here.
I can successfully login to my docker-registry using docker login -u testuser -p testpassword https://myregistry:5000 from gitlab running server. refered
After uwe-plonuscomment, I have added following part, but still same error.
before_script:
- echo "192.168.10.219 worker1" >> /etc/hosts
- docker login -u testuser -p testpassword myregistry:5000

Resources