Gitlab pipeline, kubernetes runner - How to configure - docker

I have problem with running docker in kubernetes runner.
I've installed kubernetes runner with helm and set privileged mode to true
runners:
config: |
[[runners]]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "ubuntu:20.04"
privileged = true
allow_privilege_escalation = true
I've created simple .gitlab-ci.yaml for test
stages:
- docker_test
services:
- docker:dind
docker_test:
stage: docker_test
image: docker:latest
variables:
DOCKER_HOST: "tcp://docker:2375"
script:
- docker version
But when I fire this pipeline I'm gettint error
Running with gitlab-runner 14.6.0 (5316d4ac)
on gitlab-runner-gitlab-runner-5cc654bdf7-gjfvm augRojS5
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image docker:latest ...
Using attach strategy to execute scripts...
Preparing environment
00:06
Waiting for pod gitlab-runner/runner-augrojs5-project-30333904-concurrent-0k66kk to be running, status is Pending
Waiting for pod gitlab-runner/runner-augrojs5-project-30333904-concurrent-0k66kk to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper svc-0]"
ContainersNotReady: "containers with unready status: [build helper svc-0]"
Running on runner-augrojs5-project-30333904-concurrent-0k66kk via gitlab-runner-gitlab-runner-5cc654bdf7-gjfvm...
Getting source from Git repository
00:03
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/gurita/gurita-core/.git/
Created fresh repository.
Checking out fe720f2f as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ docker version
Client:
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:40:57 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1
I tried to set without variable but at this case there is no /var/run/docker.sock.

You need to mount the host's docker socket:
[runners.kubernetes]
image = "ubuntu:18.04"
privileged=true
[[runners.kubernetes.volumes.host_path]]
name = "docker-socket"
mount_path = "/var/run/docker.sock"
read_only = false
host_path = "/var/run/docker.sock"
(NOTE: This is from one of my old gitlab installations, I haven't tested this against the latest release)
Here's my full Runner block. You can swapping my config in for yours (make a backup of your old config first) and see if it works. Obviously change things as needed -- for example I use a specific node pool, hence the node_selector and node_tolerations sections
## Installation & configuration of gitlab/gitlab-runner
## See requirements.yaml for current version
gitlab-runner:
install: true
rbac:
create: true
runners:
locked: false
privileged: true
cache:
secretName: google-application-credentials
config: |
[[runners]]
[runners.feature_flags]
FF_GITLAB_REGISTRY_HELPER_IMAGE = true
FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE = true
[runners.kubernetes]
image = "ubuntu:18.04"
privileged=true
[[runners.kubernetes.volumes.host_path]]
name = "docker-socket"
mount_path = "/var/run/docker.sock"
read_only = false
host_path = "/var/run/docker.sock"
[runners.kubernetes.node_selector]
"cloud.google.com/gke-nodepool" = "gitlab-runners"
[runners.kubernetes.node_tolerations]
"appName=gitlab" = "NoExecute"
{{- if .Values.global.minio.enabled }}
[runners.cache]
Type = "gcs"
Path = "gitlab-runner"
Shared = true
[runners.cache.gcs]
BucketName = "runner-cache"
{{ end }}
podAnnotations:
gitlab.com/prometheus_scrape: "true"
gitlab.com/prometheus_port: 9252

Thank you for your hint about mounting docker.sock.
this worked for me
runners:
config: |
[[runners]]
[runners.kubernetes]
image = "ubuntu:20.04"
privileged = true
[[runners.kubernetes.volumes.empty_dir]]
name = "docker-emptydir"
mount_path = "/var/run"
medium = "Memory"
Thanks again

Related

Trouble connecting to Docker daemon in GitLab CI

I'm trying to create a fairly simple GitLab CI file to build out Docker images. Whenever I run the pipeline, I end up getting a Docker daemon connection issue. What can I do to properly build my image? Thanks!
GitLab CI:
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_HOST: tcp://docker:2375
iac-build:
stage: build
extends: .iac
rules:
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
when: always
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
when: never
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
script:
- docker build -t testfirstimage .
allow_failure: false
Error:
$ docker build -t testfirstimage .
failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused
Cleaning up project directory and file based variables
ERROR: Job failed: command terminated with exit code 1
First thing I would check is if you already have something running on that local host - I've literally tried running a server on a local host port for hours and kept having it refused, only to find out that I had forgotten to terminate my connection to that port.
If that isn't the issue, I had this issue before and had to run this command to get it to work:
concurrent = 1
check_interval = 0
[[runners]]
name = "#####"
url = "#####"
token = "#####"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = false
disable_cache = false
cache_dir = "cache"
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
[runners.cache]
Insecure = false
I spent forever trying to figure it out and couldn't get anything to work until I found out to add
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
I didn't figure it out by magic though - props to this issues page: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1986
Hopefully that works.
In order to get Docker-in-Docker working with GitLab CI, you will first need to decide if you want to use Docker-in-Docker with or without TLS. Then, change /etc/gitlab-runner/config.toml settings, and assign the DOCKER_TLS_CERTDIR in your .gitlab-ci.yml file. See the Docker-in-docker section of the GitLab docs.
Docker-in-docker with TLS:
# /etc/gitlab-runner/config.toml
[[runners]]
url = "https://gitlab.com/"
token = TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:20.10.16"
privileged = true
disable_cache = false
volumes = ["/certs/client", "/cache"]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
# .gitlab-ci.yml
image: docker:20.10.16
variables:
DOCKER_TLS_CERTDIR: "/certs"
services:
- docker:20.10.16-dind
before_script:
- docker info
# rest of .gitlab-ci.yml

Local gitlab pipeline error during connect

I have a local Gitlab setup and trying to build a pipeline that runs a SAST scan using MobSF. Upon trying to pull the image of MobSF in order to run it I get the following error:
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
The error comes up on any script line referencing a Docker command.
The whole output of the pipeline is:
Running with gitlab-runner 14.0.0 (3b6f852e)
on pipeline 5qvFbM4s
Preparing the "docker" executor 00:04
Preparing environment 00:01
Running on runner-5qvfbm4s-project-2-concurrent-0 via TheOneWhoKnocks...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/root/sast-dast-security-testing/.git/
Checking out e71038e1 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:25a1e57c774167d28c44d88fa296f3e1122c6d79e99b98653c899b170393bbd6 for docker:18.09.7-dind with digest docker#sha256:a490c83561c1cef49b6fe12aba2c31f908391ec3efe4eb173225809c981e50c3 ...
$ export DOCKER_HOST=tcp://docker:2375
$ docker pull opensecurity/mobile-security-framework-mobsf
Using default tag: latest
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
ERROR: Job failed: exit code 1
This is my .gitlab-ci.yaml:
stages:
- build
- mobsf
build:
image: docker:18.09.7-dind
stage: build
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
script:
- docker pull opensecurity/mobile-security-framework-mobsf
- docker run -i --env-file ./env.list -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest
mobsf:
image: owasp/glue:raw-latest
stage: mobsf
script:
- ./scan.sh
- docker run -it -v $(pwd):/app owasp/glue:raw-latest ruby bin/glue -t Dynamic -T /app/report.json --mapping-file mobsf --finding-file-path /app/android.json -z 2
And this is my runner's config.toml:
[[runners]]
name = "pipeline"
url = "http://192.168.179.129/"
token = "XXXXX"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Any help would be appreciated!
It's fairly obvious that Google's public DNS servers won't resolve your local DNS requests. "docker"
error during connect: Post http://docker:2375/v1.39/images/create?fromImage=opensecurity%2Fmobile-security-framework-mobsf&tag=latest: dial tcp: lookup docker on 8.8.8.8:53: no such host
Try this answer, i was facing similar one when registering local gitlab-runner to local domain name (gitlab.local).
Docker cannot resolve dns on private network

WARNING: Checking for jobs... failed in docker executer in gitlab-runner

I am trying to run gitlab-ci on a local running using docker executer
This is the config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
listen_address = "0.0.0.0:8093"
[[runners]]
url = "https://gitlab.com/<ACCOUNT>/my-static-website"
token = XXXXXX
executor = "docker"
builds_dir = ""
clone_url = "https://gitlab.com/<ACCOUNT>/my-static-website.git"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = true
disable_cache = false
volumes = ["/cache"]
[runners.cache]
Insecure = false
My .gitlab-ci.yml:
image: node
stages:
- build
- test
build website:
stage: build
script:
- npm install
- npm install -g gatsby-cli
- gatsby build
artifacts:
paths:
- ./public
tags:
- trials
test artifacts:
image: alpine
stage: test
script:
- grep -q "Gatsby" ./public/index.html
Here is the error I am getting:
Runtime platform arch=amd64 os=linux
pid=28815 revision=4c96e5ad version=12.9.0
Starting multi-runner from ./config.toml... builds=0
Running in system-mode.
Configuration loaded builds=0
listen_address not defined, metrics & debug endpoints disabled builds=0
Session server listening address=0.0.0.0:8093
builds=0
WARNING: Checking for jobs... failed runner=kYtFEV-i
status=404 Not Found
WARNING: Checking for jobs... failed runner=kYtFEV-i
status=404 Not Found
WARNING: Checking for jobs... failed runner=kYtFEV-i
status=404 Not Found
I am using gitlab-runner version 12.9 and gitlab server: 12.10.0-pre
I have my runner on the server as follows:
I am running the command: gitlab-runner run -c ./config.toml
What did I miss here?
Your runner is not able to check for jobs. Can you double check the endpoint URL?
If your repository is on gitlab.com, you should be using the endpoint https://gitlab.com/
In your GitLab Web UI, go to Settings -> CI/CD -> Runners -> Set up a specific Runner manually
You'll see the endpoint URL and the token you'll need to register your runner.
This is covered in my GitLab CI tutorial at https://gitpitch.com/atsaloli/cicd/master?grs=gitlab#/41 (it takes a few seconds to load)
Let me know if that helps?

Gitlab - Job with "docker in docker" service results in "Cannot connect to the Docker daemon"

I've added:
services:
- docker:dind
to my .gitlab-ci.yaml file, however it still does not seem to have the docker daemon running. Is additional configuration required?
I'm using the Gitlab Runner on Kubernetes - installed directly via the Gitlab web interface / admin.
image: docker:latest
services:
- docker:dind
stages:
- build
- deploy
build-web:
stage: build
script:
- docker info
only:
- master
deploy-web:
stage: deploy
script:
- docker info
- exit 1
only:
- master
Build output:
Running with gitlab-runner 12.2.0 (a987417a)
on runner-gitlab-runner-857c466884-kzql9 BB7a6A8L
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image docker:latest ...
Waiting for pod gitlab-managed-apps/runner-bb7a6a8l-project-1-concurrent-0prm8w to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-bb7a6a8l-project-1-concurrent-0prm8w to be running, status is Pending
Running on runner-bb7a6a8l-project-1-concurrent-0prm8w via runner-gitlab-runner-857c466884-kzql9...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/root/cinema/.git/
Created fresh repository.
From http://mygitlab.ddns.net/root/cinema
* [new branch] master -> origin/master
Checking out e138a25e as master...
Skipping Git submodules setup
$ docker info
Client:
Debug Mode: false
Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
ERROR: Job failed: command terminated with exit code 1
I see the following config in the gitlab runner pod:
bash-4.4$ cat .gitlab-runner/config.toml
listen_address = "[::]:9252"
concurrent = 4
check_interval = 3
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "runner-gitlab-runner-857c466884-kzql9"
request_concurrency = 1
url = "http://mygitlab.ddns.net/"
token = "BB7a6A8LRvZ4Y-9KsLvj"
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "ubuntu:16.04"
namespace = "gitlab-managed-apps"
namespace_overwrite_allowed = ""
privileged = true
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.pod_security_context]
[runners.kubernetes.volumes]
Warning: The below might not be 100% secure
Seems to be the issue mentioned here: https://gitlab.com/gitlab-org/gitlab-runner/issues/4501
A workaround is provided here: https://gitlab.com/gitlab-org/gitlab-runner/issues/4501#note_194648457
Essentially set the following:
# .gitlab-ci.yml
variables:
DOCKER_TLS_CERTDIR: ""

Gitlab-CI multi runner start docker container, scripts now work

runner had been started docker container, the config.toml is:
concurrent = 1
check_interval = 0
[[runners]]
name = "spring-boot-scripts"
url = "http://xxxxx.com/ci"
token = "xxxxxx"
executor = "docker"
builds_dir = "/tmp/builds"
[runners.docker]
tls_verify = false
image = "spring-boot-demo:ci"
privileged = false
disable_cache = true
volumes = ["/cache"]
[runners.cache]
Insecure = false
and .gitlab-ci.yml is:
image: spring-boot-demo:ci
stages:
- build
before_scipts:
- mkdir /tmp/before_scripts
- echo "============before_scripts========="
job1:
stage: build
script:
- sh /home/admin/spring-boot-demo-application/bin/entrypoint.sh
after_scipts:
- mkdir /tmp/after_scripts
- echo "============after_scripts========="
gitlab's output like this,and building task cannot been stopped until cancelled:
Running with gitlab-ci-multi-runner 1.4.2 (bcc1794)
Using Docker executor with image spring-boot-demo:ci ...
Pulling docker image spring-boot-demo:ci ...
WARNING: Cannot pull the latest version of image spring-boot-demo:ci : Error: image library/spring-boot-demo:ci not found
WARNING: Locally found image will be used instead.
Running on runner-278e2660-project-114610-concurrent-0 via 6ca6af37d681...
Cloning repository...
Cloning into '/tmp/builds/spring-boot/startup-scripts'...
Checking out b58711bc as debug...
Now questions is:
if the before_scripts/script/after_scripts would been exec in container,why i can not find the directory /tmp/before_scripts,/tmp/after_scripts and jobs's script not work in container
You can't find the folders because the jobs are not running. You have misspelled "script" in both the before_script and after_script jobs. Also, note that the correct job is not plural and does not have the 's' at the end.
GitLab docs

Resources