Why fails my gitlab-runner (in Docker Container) since a few days, I didn't change anything - docker

I have a Ubuntu 18.04 server. On this server is running an gitlab-ce 12.7.5 instance in a docker container and a gitlab-runner 12.8.0 also in a docker container. Since a few days all the pipelines ar failing with:
*Running with gitlab-runner 12.8.0 (1b659122)
on ddddd kbuTsA6j
Using Docker executor with image ubuntu:18.04 ...
00:02
Pulling docker image ubuntu:18.04 ...
Using docker image sha256:2c047404e52d7f17bdac4121a13cd844447b74e13063f8cb8f8b314467feed06 for ubuntu:18.04 ...
ERROR: Job failed (system failure): Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: resulting devices cgroup doesn't match target mode: unknown (executor_docker.go:810:0s)*
The config.toml file of the gitlab-runner container looks like this:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "ddddd"
url = "https://myreplacedname.stratoserver.net:8929/"
token = "kbuTsA6jgE_GrM"
tls-ca-file = "/etc/gitlab-runner/certs/myreplacedname.stratoserver.net.crt"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "docker:19.03.1"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/srv/gitlab-runner/config:/etc/gitlab-runner", "/cache", "/builds:/builds"]
shm_size = 0
Until a few days ago, everything worked fine. I would be very happy if you could help me. Thank you!

I spent several days trying to find a solution to the same problem. I found out that assigning the value false to the privileged field of your [runners.docker] section solves this problem.
So, in your case, the resulting configuration will look like this:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "ddddd"
url = "https://myreplacedname.stratoserver.net:8929/"
token = "kbuTsA6jgE_GrM"
tls-ca-file = "/etc/gitlab-runner/certs/myreplacedname.stratoserver.net.crt"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "docker:19.03.1"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/srv/gitlab-runner/config:/etc/gitlab-runner", "/cache", "/builds:/builds"]
shm_size = 0
Of course, to apply these changes, you must restart your gitlab-runner instance after performing these actions. I hope it helps you, too.

Related

Error in gitlab CI/CD while docker login invalid certificate

Hey i got GitLab and gitlab runner runned in docker in gitlab i got working container registry.
when in CI/CD im trying to do
stages: # List of stages for jobs, and their order of execution
- build
docker-build:
# Use the official docker image.
image: docker:latest
services:
- docker:dind
stage: build
script:
- echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
I got error like
Get "https://git.domain.com:5050/v2/": x509: certificate signed by unknown authority
my runner config.toml looks like this
concurrent = 1
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Runner"
url = "http://git.domain.com/"
id = 3
token = "xxxxxxxxxcccccc"
token_obtained_at = 2023-02-07T22:14:54Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
clone_url = "http://git.domain.com/"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "node:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
What can i do with this?
When i do login from my pc its works perfect
edit:
My cert is not selfsigned
I added my CA certificate to certificate, and now its works

Private Gitlab Runner for code quality without Docker-in-Docker

I want to run the code quality widget in our repo. Actually i want to register an gitlab runner on my kubernetes cluster to execute docker builds. I am trying to install the gitlab runner via Kubernetes Helm Charts.
this is my helm chart: `
gitlabUrl: https://blabla/
runnerRegistrationToken: blabla
runners:
config: |
[[runners]]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock", "/tmp/builds:/tmp/builds"]
shm_size = 0
[runners.cache]
Type = "s3"
Path = "runner"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = "blabla"
BucketLocation = "us-west-1"
Insecure = false
imagePullPolicy: IfNotPresent
unfortunately this does not work. i get the output Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
Can anyone please help with the correct configuration for this problem?
my sources are:
https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html#set-up-a-private-runner-for-code-quality-without-docker-in-docker
https://docs.gitlab.com/runner/install/kubernetes.html
https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/master/values.yaml
Thanks in advance. You are my last hope
docker daemon error can be resolved by using the following code:
privileged = true

The difference between running a docker image form a gitlab-runner or directly

I need a customised maven docker image to execute my tests.
It requires me to tweak the time zone and add true type fonts.
I created my own maven image FROM maven:3.6.3-openjdk-8
I am using the gitlab/gitlab-ce:latest locally on my Mac and executing my tests in two ways
1.
From the container shell
docker run -it -v /Users/myuser/.m2:/root/.m2 -v \
/local/config:/local/ config -v \
/Users/ myuser /workspace/:/build \
--network=mynetwrok \
maven- tweak:1.0.1 bash
2.
from the local gitlab with the following runner config
oncurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "maven- tweak "
url = "http://gitlab"
token = "xyz"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "maven- tweak:1.0.0"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache","/Users/myuser/.m2:/root/.m2",“/Users/ myuser/workspace/:/build "]
network_mode = "mynetwork"
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
I am trying to figure it out for a while now.
What are the deferences between the two executions so I could figure out the problems.

Docker-ssh non-root path/getsockopt: connection refused

I’m trying to use the gitlab-runner with docker-ssh. Here is how my config.toml looks like:
[[runners]]
name = “CI/CD docker-ssh alfa”
url = “https://gitlab.com/”
token = “<SOME_TOKEN>“
executor = “docker-ssh”
[runners.ssh]
user = “myuser”
password = “my password”
[runners.docker]
tls_verify = false
image = “ubuntu:latest”
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
But I got this error:
Running with gitlab-runner 11.3.0 (d78e9e67)
on CI/CD docker-ssh alfa 1f147b76
Using Docker executor with image ubuntu:latest …
ERROR: Preparation failed: build directory needs to be absolute and non-root path
Will be retried in 3s …
Using Docker executor with image ubuntu:latest …
ERROR: Preparation failed: build directory needs to be absolute and non-root path
So I tried to change the build directory and here hows my config.toml file looks like now:
[[runners]]
name = “CI/CD docker-ssh alfa”
url = “https://gitlab.com/”
token = “<SOME_TOKEN>“
executor = “docker-ssh”
builds_dir = “/home/myuser/“
[runners.ssh]
user = “myuser”
password = “my password”
[runners.docker]
tls_verify = false
image = “ubuntu:latest”
privileged = false
disable_cache = false
volumes = [”/cache"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
But I got this new error:
Running with gitlab-runner 11.3.0 (d78e9e67)
on CI/CD docker-ssh alfa 1f147b76
Using Docker executor with image ubuntu:latest …
WARNING: Since GitLab Runner 10.0 docker-ssh and docker-ssh+machine executors are marked as DEPRECATED and will be removed in one of the upcoming releases
Pulling docker image ubuntu:latest …
Using docker image sha256:cd6d8154f1e16e38493c3c2798977c5e142be5e5d41403ca89883840c6d51762 for ubuntu:latest …
ERROR: Preparation failed: dial tcp 172.17.0.2:22: getsockopt: connection refused
Will be retried in 3s …
Any idea what am I doing wrong?
Stick with an HTTPS URL, and try fixing instead the error:
build directory needs to be absolute and non-root path
See this thread
I was running my CI on an old gitlab-ci-multi-runner 9.5.1.
I update to gitlab-runner 10.8.0 and now it’s ok.
Or this thread:
Set build_dir="C:\\gitlab-runner\\builds" in the config.toml.

Running docker gitlab-runner with net=host option

I've also been having enormous problem setting up docker gitlab-runner containers with net="host" option. Even though in version 1.1 option "docker-network-mode" was introduced I cannot make it work. My config.toml in gitlab-runner container is:
[[runners]]
name = "name"
url = "giturl"
token = "token"
executor = "docker"
[runners.docker]
tls_verify = false
image = "maven:3.3.9-jdk-8"
network_mode = "host"
privileged = false
disable_cache = false
volumes = ["/cache"]
and runner does not work with net=host mode. Anyone made it work?

Resources