GitLab Runner not using Registry Mirror - docker

Since docker introduced the pull rate limit i'm running a pull-through registry.
I have done the setup with the documentation on dockers site (https://docs.docker.com/registry/recipes/mirror/#configure-the-cache).
In my runners config.toml two volumes will be mounted, one for the docker socket and another one for the daemon.json.
But everytime i want to build something i get the following error message that i have reached my pull rate limit:
Running with gitlab-runner 13.11.0 (7f7a4bb0)
on srv-gitlab-ba5 vXWs_kze
Preparing the "docker" executor
00:25
Using Docker executor with image alpine:3.12 ...
WARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrate-helper-image-to-registrygitlabcom
Pulling docker image gitlab/gitlab-runner-helper:x86_64-7f7a4bb0 ...
Using docker image sha256:25f27e06750273451a8211bbe80dc14f5d2bb389ff912ae0adffb477fa26058f for gitlab/gitlab-runner-helper:x86_64-7f7a4bb0 with digest gitlab/gitlab-runner-helper#sha256:20ff486a25738d42d2bca770092d2c4b1ae28881de3b599877ab954d3654a08a ...
Pulling docker image alpine:3.12 ...
WARNING: Failed to pull image with policy "always": toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (manager.go:205:2s)
ERROR: Preparation failed: failed to pull image "alpine:3.12" with specified policies [always]: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (manager.go:205:2s)
Will be retried in 3s ...
My config.toml for the runner looks like:
concurrent = 5
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "srv-gitlab-ba5"
url = "***"
token = "***"
executor = "docker"
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "***"
AccessKey = "***"
SecretKey = "***"
BucketName = "gitlab-cache"
Insecure = false
[runners.docker]
tls_verify = false
image = "docker:19.03.8"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/etc/docker:/etc/docker:ro", "/cache"]
shm_size = 0
docker-compose.yml for the runner:
version: "3.7"
services:
gitlab_runner:
image: {self build image with config.toml above}
restart: always
container_name: gitlab-runner
privileged: true
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /etc/docker:/etc/docker:ro
Can someone tell me if my configuration is correct or is it possible that i forgot something?
I have googled a lot but didn't find the correct solution for my problem.

If you do not switch off instance runner very often, try to avoid using option "always"
WARNING: Failed to pull image with policy "always"
To switch this value, set up gitlab-runner registry with --docker-pull-policy "if-not-present"
then docker will keep layers of images and will not pull it every time from docker registry, no clue then to create additional registry

Related

How to build docker image in gitlab-runner and push to local registry? And why --insecure-registry command make --tls=false not working

I'm trying to construct gitlab CI/CD with docker.
The goal is building docker image on runner and push to local insecure registry, but docker build keep throwing this error.
error during connect: Post "http://docker:2375/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=test_project&target=&ulimits=null&version=1": dial tcp: lookup docker on xxx.xxx.xxx.xxx:53: no such host
This is my gitlab-ci file.
...
package:
image: docker:latest
stage: package
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- name: docker:20-dind
alias: docker
command: ["--tls=false", "--insecure-registry=$DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT"] # OPTIONS
script:
- docker build -t $DOCKER_IMAGE_NAME . # Error occurred
- docker tag $DOCKER_IMAGE_NAME $DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT/$DOCKER_IMAGE_NAME
- docker push $DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT/$DOCKER_IMAGE_NAME
cache:
key: "$DOCKER_IMAGE_NAME"
policy: pull
paths:
- build
...
I set priviliged=true to solve first error.
[[runners]]
name = "docker-runner"
url = "http://192.165.100.2/"
id = 10
token = "47e5LRRhgBPcnhfYdDxB"
token_obtained_at = 2022-12-28T06:50:33Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true #set to true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Then first error as solved, but second error has occurred.
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
To solve second error, I added "--tls=false" command to docker:dind service according to this link.
services:
- name: docker:20-dind
alias: docker
command: ["--tls=false"]
Then no such host error has solved, But third error has occurred.
Get "https://xxx.xxx.xxx.xxx:5000/v2/": http: server gave HTTP response to HTTPS client
To solve third error, I added insecure-registry command according to this link.
services:
- name: docker:20-dind
alias: docker
command: ["--tls=false", "--insecure-registry=$DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT"]
I expected to solve https problem, but error went back to the beginning.
error during connect: Post "http://docker:2375/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=test_project&target=&ulimits=null&version=1": dial tcp: lookup docker on xxx.xxx.xxx.xxx:53: no such host
Why --insecure-registry command make --tls=false not working?
How to build docker image in gitlab-runner and push to local registry?
Thanks.

Trouble connecting to Docker daemon in GitLab CI

I'm trying to create a fairly simple GitLab CI file to build out Docker images. Whenever I run the pipeline, I end up getting a Docker daemon connection issue. What can I do to properly build my image? Thanks!
GitLab CI:
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_HOST: tcp://docker:2375
iac-build:
stage: build
extends: .iac
rules:
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
when: always
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
when: never
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
script:
- docker build -t testfirstimage .
allow_failure: false
Error:
$ docker build -t testfirstimage .
failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused
Cleaning up project directory and file based variables
ERROR: Job failed: command terminated with exit code 1
First thing I would check is if you already have something running on that local host - I've literally tried running a server on a local host port for hours and kept having it refused, only to find out that I had forgotten to terminate my connection to that port.
If that isn't the issue, I had this issue before and had to run this command to get it to work:
concurrent = 1
check_interval = 0
[[runners]]
name = "#####"
url = "#####"
token = "#####"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = false
disable_cache = false
cache_dir = "cache"
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
[runners.cache]
Insecure = false
I spent forever trying to figure it out and couldn't get anything to work until I found out to add
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
I didn't figure it out by magic though - props to this issues page: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1986
Hopefully that works.
In order to get Docker-in-Docker working with GitLab CI, you will first need to decide if you want to use Docker-in-Docker with or without TLS. Then, change /etc/gitlab-runner/config.toml settings, and assign the DOCKER_TLS_CERTDIR in your .gitlab-ci.yml file. See the Docker-in-docker section of the GitLab docs.
Docker-in-docker with TLS:
# /etc/gitlab-runner/config.toml
[[runners]]
url = "https://gitlab.com/"
token = TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:20.10.16"
privileged = true
disable_cache = false
volumes = ["/certs/client", "/cache"]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
# .gitlab-ci.yml
image: docker:20.10.16
variables:
DOCKER_TLS_CERTDIR: "/certs"
services:
- docker:20.10.16-dind
before_script:
- docker info
# rest of .gitlab-ci.yml

DinD configuration for Gitlab CI with private Docker registry in Sonatype Nexus 3

I have done a setup of my own Gitlab(-p 7022:22, 7080:9080), Gitlab Runner(-p 7093:8093), Sonatype Nexus3(Maven, Docker, Helm)(-p 10081:8081, 10082:10082, 10083:10083, 10084:10084). All are running as docker images of their own. And till docker build everything runs great(below code snippets doesn't have docker build related code). Problem is, I want to upload the final docker image to my Nexus3 Docker Registry which I am unable to do so.
My Gitlab Runner config is below
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "testing dind runner"
url = "http://192.168.0.250:7080/" ----> Gitlab git repo external url
token = "SOME TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
My Docker registry in Nexus3 is hosted as below
Web UI = http://192.168.0.250:10081/ (container internal port is 8081 and exposed to host on 10081)
Docker(Group) = 10084 (exposed as the same port through docker)
Docker(Hosted) = 10082 (exposed as the same port through docker)
Docker(Proxy) = 10083 (exposed as the same port through docker)
My project CI config is below
image: docker:19.03.12
services:
- name: docker:19.03.12-dind
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
before_script:
- docker info
- docker login -u nx-uploader -p 1234 192.168.0.250:10082
stages:
- test docker reg
test-docker:
stage: test docker reg
script:
- docker images
- docker search httpd
I am constantly getting errors as
$ docker login -u nx-uploader -p 1234 192.168.0.250:10082
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post http://docker:2376/v1.40/auth: dial tcp: lookup docker on 192.168.0.1:53: no such host
ERROR: Job failed: exit code 1
Need help/pointers to fix this.
Thanks in advance.

GitLab with Docker runner on localhost: how to expose host to container?

I'm learning to use GitLab CI.
Just now I'm using GitLab on localhost (external_url "http://localhost"). And I've registered a Docker runner with vanilla ubuntu:20.04 image and tried to run some test job on it.
Alas, it tries to clone my repo from localhost repository in the container, but cannot do it, because my localhost's port 80 is not visible from container.
Running with gitlab-runner 13.5.0 (ece86343)
on docker0 x8pHJPn7
Preparing the "docker" executor
Using Docker executor with image ubuntu:20.04 ...
Pulling docker image ubuntu:20.04 ...
Using docker image sha256:d70eaf7277eada08fca944de400e7e4dd97b1262c06ed2b1011500caa4decaf1 for ubuntu:20.04 with digest ubuntu#sha256:fff16eea1a8ae92867721d90c59a75652ea66d29c05294e6e2f898704bdb8cf1 ...
Preparing environment
Running on runner-x8phjpn7-project-6-concurrent-0 via gigant...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/root/ci_fuss/.git/
fatal: unable to access 'http://localhost:80/root/ci_fuss.git/': Failed to connect to localhost port 80: Connection refused
Uploading artifacts for failed job
Uploading artifacts...
WARNING: report.xml: no matching files
ERROR: No files to upload
Cleaning up file based variables
ERROR: Job failed: exit code 1
How can I can my Docker runner to expose host's localhost:80 as container's localhost:80?
Well, i have coped with this stuff.
I have added network_mode = "host"to my runner configuration in /etc/gitlab-runner/config.toml to make my docker use host network connections.
Also I've added --pull_policy="if-not-present" to first search for container image locally, then in remote repo.
[[runners]]
name = "docker0"
url = "http://localhost/"
token = "TTBRFis_W_yJJpN1LLzV"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "exposed_ctr:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
network_mode = "host"
pull_policy = "if-not-present"

Docker-ssh non-root path/getsockopt: connection refused

I’m trying to use the gitlab-runner with docker-ssh. Here is how my config.toml looks like:
[[runners]]
name = “CI/CD docker-ssh alfa”
url = “https://gitlab.com/”
token = “<SOME_TOKEN>“
executor = “docker-ssh”
[runners.ssh]
user = “myuser”
password = “my password”
[runners.docker]
tls_verify = false
image = “ubuntu:latest”
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
But I got this error:
Running with gitlab-runner 11.3.0 (d78e9e67)
on CI/CD docker-ssh alfa 1f147b76
Using Docker executor with image ubuntu:latest …
ERROR: Preparation failed: build directory needs to be absolute and non-root path
Will be retried in 3s …
Using Docker executor with image ubuntu:latest …
ERROR: Preparation failed: build directory needs to be absolute and non-root path
So I tried to change the build directory and here hows my config.toml file looks like now:
[[runners]]
name = “CI/CD docker-ssh alfa”
url = “https://gitlab.com/”
token = “<SOME_TOKEN>“
executor = “docker-ssh”
builds_dir = “/home/myuser/“
[runners.ssh]
user = “myuser”
password = “my password”
[runners.docker]
tls_verify = false
image = “ubuntu:latest”
privileged = false
disable_cache = false
volumes = [”/cache"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
But I got this new error:
Running with gitlab-runner 11.3.0 (d78e9e67)
on CI/CD docker-ssh alfa 1f147b76
Using Docker executor with image ubuntu:latest …
WARNING: Since GitLab Runner 10.0 docker-ssh and docker-ssh+machine executors are marked as DEPRECATED and will be removed in one of the upcoming releases
Pulling docker image ubuntu:latest …
Using docker image sha256:cd6d8154f1e16e38493c3c2798977c5e142be5e5d41403ca89883840c6d51762 for ubuntu:latest …
ERROR: Preparation failed: dial tcp 172.17.0.2:22: getsockopt: connection refused
Will be retried in 3s …
Any idea what am I doing wrong?
Stick with an HTTPS URL, and try fixing instead the error:
build directory needs to be absolute and non-root path
See this thread
I was running my CI on an old gitlab-ci-multi-runner 9.5.1.
I update to gitlab-runner 10.8.0 and now it’s ok.
Or this thread:
Set build_dir="C:\\gitlab-runner\\builds" in the config.toml.

Resources