I'm trying to create a fairly simple GitLab CI file to build out Docker images. Whenever I run the pipeline, I end up getting a Docker daemon connection issue. What can I do to properly build my image? Thanks!
GitLab CI:
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_HOST: tcp://docker:2375
iac-build:
stage: build
extends: .iac
rules:
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
when: always
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
when: never
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
script:
- docker build -t testfirstimage .
allow_failure: false
Error:
$ docker build -t testfirstimage .
failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused
Cleaning up project directory and file based variables
ERROR: Job failed: command terminated with exit code 1
First thing I would check is if you already have something running on that local host - I've literally tried running a server on a local host port for hours and kept having it refused, only to find out that I had forgotten to terminate my connection to that port.
If that isn't the issue, I had this issue before and had to run this command to get it to work:
concurrent = 1
check_interval = 0
[[runners]]
name = "#####"
url = "#####"
token = "#####"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = false
disable_cache = false
cache_dir = "cache"
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
[runners.cache]
Insecure = false
I spent forever trying to figure it out and couldn't get anything to work until I found out to add
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
I didn't figure it out by magic though - props to this issues page: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1986
Hopefully that works.
In order to get Docker-in-Docker working with GitLab CI, you will first need to decide if you want to use Docker-in-Docker with or without TLS. Then, change /etc/gitlab-runner/config.toml settings, and assign the DOCKER_TLS_CERTDIR in your .gitlab-ci.yml file. See the Docker-in-docker section of the GitLab docs.
Docker-in-docker with TLS:
# /etc/gitlab-runner/config.toml
[[runners]]
url = "https://gitlab.com/"
token = TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:20.10.16"
privileged = true
disable_cache = false
volumes = ["/certs/client", "/cache"]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
# .gitlab-ci.yml
image: docker:20.10.16
variables:
DOCKER_TLS_CERTDIR: "/certs"
services:
- docker:20.10.16-dind
before_script:
- docker info
# rest of .gitlab-ci.yml
Related
I'm trying to construct gitlab CI/CD with docker.
The goal is building docker image on runner and push to local insecure registry, but docker build keep throwing this error.
error during connect: Post "http://docker:2375/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=test_project&target=&ulimits=null&version=1": dial tcp: lookup docker on xxx.xxx.xxx.xxx:53: no such host
This is my gitlab-ci file.
...
package:
image: docker:latest
stage: package
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- name: docker:20-dind
alias: docker
command: ["--tls=false", "--insecure-registry=$DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT"] # OPTIONS
script:
- docker build -t $DOCKER_IMAGE_NAME . # Error occurred
- docker tag $DOCKER_IMAGE_NAME $DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT/$DOCKER_IMAGE_NAME
- docker push $DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT/$DOCKER_IMAGE_NAME
cache:
key: "$DOCKER_IMAGE_NAME"
policy: pull
paths:
- build
...
I set priviliged=true to solve first error.
[[runners]]
name = "docker-runner"
url = "http://192.165.100.2/"
id = 10
token = "47e5LRRhgBPcnhfYdDxB"
token_obtained_at = 2022-12-28T06:50:33Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true #set to true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Then first error as solved, but second error has occurred.
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
To solve second error, I added "--tls=false" command to docker:dind service according to this link.
services:
- name: docker:20-dind
alias: docker
command: ["--tls=false"]
Then no such host error has solved, But third error has occurred.
Get "https://xxx.xxx.xxx.xxx:5000/v2/": http: server gave HTTP response to HTTPS client
To solve third error, I added insecure-registry command according to this link.
services:
- name: docker:20-dind
alias: docker
command: ["--tls=false", "--insecure-registry=$DOCKER_REGISTRY_IP:$DOCKER_REGISTRY_PORT"]
I expected to solve https problem, but error went back to the beginning.
error during connect: Post "http://docker:2375/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=test_project&target=&ulimits=null&version=1": dial tcp: lookup docker on xxx.xxx.xxx.xxx:53: no such host
Why --insecure-registry command make --tls=false not working?
How to build docker image in gitlab-runner and push to local registry?
Thanks.
I am using testcontainers to start a Postgresql DB for my junit test. Locally everything works fine. This is my small test project:
https://gitlab.com/janning/tpj-testcontainer
I can run the test inside my IDE and on CLI with ./gradlew test
Now I want to run it in my Gitlab-CI Pipeline but without docker-in-docker (dind). So I need to mount the docker socket, which is documented here: https://www.testcontainers.org/supported_docker_environment/continuous_integration/gitlab_ci/
so I configred my gitlab-runner like this:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "kt103-tpj"
url = "https://gitlab.com"
token = "XXXXXXXXXXXXXXXXX"
executor = "docker"
[runners.docker]
tls_verify = false
image = "registry.gitlab.com/janning/tpj-testcontainer/debian:latest"
privileged = false
disable_entrypoint_overwrite = false
extra_hosts = ["host.docker.internal:host-gateway"]
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
First my gitlab-ci.yml builds a Dockerimage as I need openjdk-17 and some docker commands.
variables:
TESTCONTAINERS_HOST_OVERRIDE: "host.docker.internal"
stages:
- dockerimage
- test
dockerimage:
image: docker:latest
stage: dockerimage
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- tag=":latest"
- docker build --pull -t "$CI_REGISTRY_IMAGE/debian${tag}" .
- docker push "$CI_REGISTRY_IMAGE/debian${tag}"
test:
stage: test
image: $CI_REGISTRY_IMAGE/debian:latest
script:
- echo $DOCKER_HOST
- echo $DOCKER_TLS_VERIFY
- echo $DOCKER_CERT_PATH
- ./gradlew test
artifacts:
when: always
reports:
junit: build/test-results/test/**/TEST-*.xml
The job "test" failed (complete log here)
Task :test HalloTest > test() FAILED
java.lang.IllegalStateException at RyukResourceReaper.java:129 1 test completed, 1 failed
Task :test FAILED
If i dig deeper into the junit result I can see the following stacktrace:
java.lang.IllegalStateException: Could not connect to Ryuk at host.docker.internal:49158
at org.testcontainers.utility.RyukResourceReaper.maybeStart(RyukResourceReaper.java:129)
at org.testcontainers.utility.RyukResourceReaper.init(RyukResourceReaper.java:42)
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:203)
at ...
I guess it has something to do with the docker image, which I build. But I am stuck as I don't know what the problem is. I don't how to solve or debug this situation.
The ryuk container is started in my host which is running gitlab-runner. But it can't connect:
$ docker container logs -f ba62a173aafd
2022/06/24 09:42:51 Pinging Docker...
2022/06/24 09:42:51 Docker daemon is available!
2022/06/24 09:42:51 Starting on port 8080...
2022/06/24 09:42:51 Started!
panic: Timed out waiting for the first connection
goroutine 1 [running]:
main.main()
/go/src/github.com/testcontainers/moby-ryuk/main.go:50 +0x449
The problem was just a firewall problem blocking traffic between the docker containers. As I could reach any docker container from external, it took my quite a while until I realized it.
Since docker introduced the pull rate limit i'm running a pull-through registry.
I have done the setup with the documentation on dockers site (https://docs.docker.com/registry/recipes/mirror/#configure-the-cache).
In my runners config.toml two volumes will be mounted, one for the docker socket and another one for the daemon.json.
But everytime i want to build something i get the following error message that i have reached my pull rate limit:
Running with gitlab-runner 13.11.0 (7f7a4bb0)
on srv-gitlab-ba5 vXWs_kze
Preparing the "docker" executor
00:25
Using Docker executor with image alpine:3.12 ...
WARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrate-helper-image-to-registrygitlabcom
Pulling docker image gitlab/gitlab-runner-helper:x86_64-7f7a4bb0 ...
Using docker image sha256:25f27e06750273451a8211bbe80dc14f5d2bb389ff912ae0adffb477fa26058f for gitlab/gitlab-runner-helper:x86_64-7f7a4bb0 with digest gitlab/gitlab-runner-helper#sha256:20ff486a25738d42d2bca770092d2c4b1ae28881de3b599877ab954d3654a08a ...
Pulling docker image alpine:3.12 ...
WARNING: Failed to pull image with policy "always": toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (manager.go:205:2s)
ERROR: Preparation failed: failed to pull image "alpine:3.12" with specified policies [always]: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (manager.go:205:2s)
Will be retried in 3s ...
My config.toml for the runner looks like:
concurrent = 5
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "srv-gitlab-ba5"
url = "***"
token = "***"
executor = "docker"
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "***"
AccessKey = "***"
SecretKey = "***"
BucketName = "gitlab-cache"
Insecure = false
[runners.docker]
tls_verify = false
image = "docker:19.03.8"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/etc/docker:/etc/docker:ro", "/cache"]
shm_size = 0
docker-compose.yml for the runner:
version: "3.7"
services:
gitlab_runner:
image: {self build image with config.toml above}
restart: always
container_name: gitlab-runner
privileged: true
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /etc/docker:/etc/docker:ro
Can someone tell me if my configuration is correct or is it possible that i forgot something?
I have googled a lot but didn't find the correct solution for my problem.
If you do not switch off instance runner very often, try to avoid using option "always"
WARNING: Failed to pull image with policy "always"
To switch this value, set up gitlab-runner registry with --docker-pull-policy "if-not-present"
then docker will keep layers of images and will not pull it every time from docker registry, no clue then to create additional registry
I have a local Gitlab setup and trying to build a pipeline that runs a SAST scan using MobSF. Upon trying to pull the image of MobSF in order to run it I get the following error:
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
The error comes up on any script line referencing a Docker command.
The whole output of the pipeline is:
Running with gitlab-runner 14.0.0 (3b6f852e)
on pipeline 5qvFbM4s
Preparing the "docker" executor 00:04
Preparing environment 00:01
Running on runner-5qvfbm4s-project-2-concurrent-0 via TheOneWhoKnocks...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/root/sast-dast-security-testing/.git/
Checking out e71038e1 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:25a1e57c774167d28c44d88fa296f3e1122c6d79e99b98653c899b170393bbd6 for docker:18.09.7-dind with digest docker#sha256:a490c83561c1cef49b6fe12aba2c31f908391ec3efe4eb173225809c981e50c3 ...
$ export DOCKER_HOST=tcp://docker:2375
$ docker pull opensecurity/mobile-security-framework-mobsf
Using default tag: latest
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
ERROR: Job failed: exit code 1
This is my .gitlab-ci.yaml:
stages:
- build
- mobsf
build:
image: docker:18.09.7-dind
stage: build
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
script:
- docker pull opensecurity/mobile-security-framework-mobsf
- docker run -i --env-file ./env.list -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest
mobsf:
image: owasp/glue:raw-latest
stage: mobsf
script:
- ./scan.sh
- docker run -it -v $(pwd):/app owasp/glue:raw-latest ruby bin/glue -t Dynamic -T /app/report.json --mapping-file mobsf --finding-file-path /app/android.json -z 2
And this is my runner's config.toml:
[[runners]]
name = "pipeline"
url = "http://192.168.179.129/"
token = "XXXXX"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Any help would be appreciated!
It's fairly obvious that Google's public DNS servers won't resolve your local DNS requests. "docker"
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
Try this answer, i was facing similar one when registering local gitlab-runner to local domain name (gitlab.local).
Docker cannot resolve dns on private network
I have done a setup of my own Gitlab(-p 7022:22, 7080:9080), Gitlab Runner(-p 7093:8093), Sonatype Nexus3(Maven, Docker, Helm)(-p 10081:8081, 10082:10082, 10083:10083, 10084:10084). All are running as docker images of their own. And till docker build everything runs great(below code snippets doesn't have docker build related code). Problem is, I want to upload the final docker image to my Nexus3 Docker Registry which I am unable to do so.
My Gitlab Runner config is below
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "testing dind runner"
url = "http://192.168.0.250:7080/" ----> Gitlab git repo external url
token = "SOME TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
My Docker registry in Nexus3 is hosted as below
Web UI = http://192.168.0.250:10081/ (container internal port is 8081 and exposed to host on 10081)
Docker(Group) = 10084 (exposed as the same port through docker)
Docker(Hosted) = 10082 (exposed as the same port through docker)
Docker(Proxy) = 10083 (exposed as the same port through docker)
My project CI config is below
image: docker:19.03.12
services:
- name: docker:19.03.12-dind
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
before_script:
- docker info
- docker login -u nx-uploader -p 1234 192.168.0.250:10082
stages:
- test docker reg
test-docker:
stage: test docker reg
script:
- docker images
- docker search httpd
I am constantly getting errors as
$ docker login -u nx-uploader -p 1234 192.168.0.250:10082
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post http://docker:2376/v1.40/auth: dial tcp: lookup docker on 192.168.0.1:53: no such host
ERROR: Job failed: exit code 1
Need help/pointers to fix this.
Thanks in advance.