How to make GitLab Runner in Docker see a custom CA Root certificate - docker

Question

I have installed and configured:
an on-premises GitLab Omnibus on ServerA running on HTTPS
an on-premises GitLab-Runner installed as Docker Service in ServerB
ServerA certificate is generated by a custom CA Root
The Configuration
I've have put the CA Root Certificate on ServerB:
/srv/gitlab-runner/config/certs/ca.crt
Installed the Runner on ServerB as described in Run GitLab Runner in a container - Docker image installation and configuration:
docker run -d --name gitlab-runner --restart always \
-v /srv/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest
Registered the Runner as described in Registering Runners - One-line registration command:
docker run --rm -t -i
-v /srv/gitlab-runner/config:/etc/gitlab-runner
--name gitlab-docker-runner gitlab/gitlab-runner register \
--non-interactive \
--executor "docker" \
--docker-image alpine:latest \
--url "https://MY_PRIVATE_REPO_URL_HERE/" \
--registration-token "MY_PRIVATE_TOKEN_HERE" \
--description "MyDockerServer-Runner" \
--tag-list "TAG_1,TAG_2,TAG_3" \
--run-untagged \
--locked="false"
This command gave the following output:
Updating CA certificates...
Runtime platform arch=amd64 os=linux pid=5 revision=cf91d5e1 version=11.4.2
Running in system-mode.
Registering runner... succeeded runner=8UtcUXCY
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
I checked with
$ docker exec -it gitlab-runner bash
and once in the container with
$ awk -v cmd='openssl x509 -noout -subject' '
/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
and the custom CA root is correctly there.
The Problem
When running Gitlab-Runner from GitLab-CI, the pipeline fails miserably telling me that:
$ git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}#ServerA/foo/bar/My-Project.wiki.git
Cloning into 'My-Project.wiki'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#ServerA/foo/bar/My-Project.wiki.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
ERROR: Job failed: exit code 1
It does not recognize the Issuer (my custom CA Root), but according to The self-signed certificates or custom Certification Authorities, point n.1, it should out-of-the-box:
Default: GitLab Runner reads system certificate store and verifies the GitLab server against the CA’s stored in system.
I've then tried the solution from point n.3, editing
/srv/gitlab-runner/config/config.toml:
and adding:
[[runners]]
tls-ca-file = "/srv/gitlab-runner/config/certs/ca.crt"
But it still doesn't work.
How can I make Gitlab Runner read the CA Root certificate?

Answer 1

You have two options:
Ignore SSL verification
Put this at the top of your .gitlab-ci.yml:
variables:
GIT_SSL_NO_VERIFY: "1"
Point GitLab-Runner to the proper certificate
As outlined in the official documentation, you can use the tls-*-file options to setup your certificate, e.g.:
[[runners]]
...
tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
[runners.docker]
...
As the documentation states, "this file will be read every time when runner tries to access the GitLab server."
Other options include tls-cert-file to define the certificate to be used if needed.

Answer 2

While I've still not got why it doesn't work out-of-the-box, I've found the Egg of Columbus:
Gitlab-Runner configuration:
[[runners]]
name = "MyDockerServer-Runner"
url = "https://MY_PRIVATE_REPO_URL_HERE/"
token = "MY_TOKEN_HERE"
executor = "docker"
...
[runners.docker]
image = "ubuntu:latest"
# The trick is the following:
volumes = ["/cache","/srv/gitlab-runner/config:/etc/gitlab-runner"]
...
Gitlab-ci.yml pipeline:
MyJob:
image: ubuntu:latest
script:
- awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
- git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}#ServerA/foo/bar/My-Project.wiki.git
- wget -O foo.png https://ServerA/foo/bar/foo.png
before_script:
- apt-get update -y >/dev/null
- apt-get install -y apt-utils dialog >/dev/null
- apt-get install -y git >/dev/null
- apt-get install -y wget >/dev/null
# The trick is the following:
- cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
- update-ca-certificates
That's it:
Mount the volume once (per Docker executor)
Update the CA certificates once (per job)
And everything will work as expected: git clone, wget https, etc...
A great workaround, until someone at GitLab will fix it or explain me where I'm wrong (be my guest!)

Answer 3

Not sure it's the best approach, but at least it worked for me. You can create a customized gitlab runner image and add your root CA inside:
├── Dockerfile
└── myca.crt
# Dockerfile
FROM gitlab/gitlab-runner:latest
COPY myca.crt /usr/local/share/ca-certificates
RUN update-ca-certificates
Build it:
docker build -t custom-gitlab-runner .
And rerun all your commands, just remember to use this new image name.
Off-topic, but related and might be useful
Dockerized gitlab-runner seem to also ignore entries in your /etc/hosts, so if you have launched Gitlab on a custom domain, e.g. https://gitlab.local.net, you need to pass the values from /etc/hosts when launching/registering gitlab runner:
docker run -d --name gitlab-runner --restart always \
--add-host="gitlab.local.net:192.168.1.100" \
...
If you want to launch docker:dind (docker in docker service) container to build docker images, you also need to set these values inside /srv/gitlab-runner/config/config.toml:
[[runners]]
url = "https://gitlab.local.net/"
executor = "docker"
pre_clone_script = "echo '192.168.1.100 gitlab.local.net registry.local.net' >> /etc/hosts"
...

Answer 4

From the output you provided i think that the certificate might be OK but you are lacking the CRL file : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
The CRL file is used to verify that even if the certificate is valid is hasn't been revoked by the CA owner. You shoudl then need to :
1) Generate a CRL file based on your CA:
openssl ca -gencrl -keyfile ca.key -cert ca.crt -out crl.pem
source: https://blog.didierstevens.com/2013/05/08/howto-make-your-own-cert-and-revocation-list-with-openssl/
2) Instruct the runner to use it :
[[runners]]
...
tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
crl-file = "/etc/gitlab-runner/ssl/ca.crl"
3) Of course setting GIT_SSL_NO_VERIFY will work but you will be more sensitive to man-in-the-middle attacks