I have /etc/gitlab-runner/config.toml
[[runners]]
name = "My Docker Runner"
url = "###"
token = "###"
executor = "docker"
pre_clone_script = "git config --global http.sslVerify false"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "tiangolo/docker-with-compose"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
[runners.cache]
Insecure = false
[runners.cache.s3]
[runners.cache.gcs]
.gitlab-ci.yml
stages:
- test
before_script:
- apk update
- apk upgrade
- apk add make
test:
stage: test
tags: [test]
script:
- make prod-up # docker-compose up -d --build
- make composer-init-dev
- make test
When I run the script in gitlab, I get the error:
$make prod-up
docker-compose -f docker-compose.prod.yml up -d --build
Couldn't connect to Docker daemon at http://docker:2375 - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
make: *** [Makefile:16: docker-prod-up] Error 1
ERROR: Job failed: exit code 2
Please help, what am I doing wrong?
You're missing Docker service (and it's best to specify image explicitly):
stages:
- test
before_script:
- apk update
- apk upgrade
- apk add make
test:
stage: test
image: tiangolo/docker-with-compose:latest
services:
- docker:dind
tags: [test]
script:
- make prod-up # docker-compose up -d --build
- make composer-init-dev
- make test
Try mounting docker.sock:
test:
stage: test
tags: [test]
script:
- make prod-up # docker-compose up -d --build
- make composer-init-dev
- make test
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Related
How to configure gitlab-runner config.toml and .gitlab-ci.yml to run 'mvn test' with the need of Docker daemon running in the same stage ? I need this to make testcontainer work in CI.
So far I have solved this by creating an image called 'ci-server' with both maven and Docker. I want to solve this following the approach described in Testcontainers website, in this link, but none of these examples worked for me.
gitlab-ci.yml (example)
variables:
DOCKER_HOST: "tcp://docker:2375"
DOCKER_TLS_CERTDIR: ""
services:
- docker:dind
stages:
- clean-build-test
maven-build-test:
image: ci-server:1.0
stage: clean-build-test
tags:
- xxx
script:
- mvn $MAVEN_CLI_OPTS clean package -Dspring.profiles.active=test
gitlab-runner configuration:
[[runners]]
name = "xxx"
url = "xxx"
id = 99
token = "xxx"
token_obtained_at = 2023-01-18T21:24:45Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "ci-server:1.0"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
pull_policy = "if-not-present"
Been trying to build a simple gitlab CI pipeline which builds an image and pushes it to Google container repository. I am running through this error -
ERROR: error during connect: Get "http://docker:2375/v1.24/info": dial
tcp: lookup docker on 169.254.169.254:53: no such host
I have tried all the solutions posted across gitlab issues threads but no help. I am using public runners, it's a pretty simple ci script.
image: docker:latest
variables:
GCR_IMAGE: <GCR_IMAGE>
services:
- docker:dind
build:
stage: build
before_script:
- docker info
- echo $GOOGLE_CLOUD_ACCOUNT | docker login -u _json_key --password-stdin https://us.gcr.io
script:
- docker build -t $GCR_IMAGE:latest .
- docker push $GCR_IMAGE:$CI_COMMIT_SHA
Relevant issue thread: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4794
Using gitlab-runner 15.7.1
A few weeks ago I encountered this problem and was able to solve it with this method:
image:
name: docker:20.10.16
services:
- name: docker:20.10.16-dind
variables:
DOCKER_HOST: tcp://docker:2376/
DOCKER_TLS_CERTDIR: "/certs"
DOCKER_TLS_VERIFY: 1
DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
before_script:
- until docker info; do sleep 1; done
- echo $GOOGLE_CLOUD_ACCOUNT | docker login -u _json_key --password-stdin https://us.gcr.io
script:
- docker build -t $GCR_IMAGE:latest .
- docker push $GCR_IMAGE:$CI_COMMIT_SHA
Also add this configuration to runner
[[runners]]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "ubuntu:20.04"
[[runners.kubernetes.volumes.empty_dir]]
name = "docker-certs"
mount_path = "/certs/client"
medium = "Memory"
Ive an issue with gitlab-runner executor docker. After I ran my gitlab-ci.yml file , pipeline fail on step docker info during before_script with:
Running with gitlab-runner 13.10.0 (54944146)
on docker-runner N2_yEgUD
Preparing the "docker" executor 00:07
Using Docker executor with image docker:19.03.0 ...
Starting service docker:19.03.0-dind ...
Pulling docker image docker:19.03.0-dind ...
Using docker image sha256:fd0c64832f7e46b63a180e6000dbba7ad7a63542c5764841cba73429ba74a39e for docker:19.03.0-dind with digest docker#sha256:442ac4b31375cbe617f31759b5199d240f11d5f430e54946575b274b2fb6f096 ...
Waiting for services to be up and running...
.............................................................................................
$ docker info
Client:
Debug Mode: false
Server:
ERROR: error during connect: Get https://docker:2375/v1.40/info: dial tcp: lookup docker on 127.0.0.53:53: server misbehaving
errors pretty printing info
Cleaning up file based variables 00:01
ERROR: Job failed: exit code 1
I did a research on stack and official gitlab forum but none of the answers fix my issue:
add to .toml -> volume: ['/certs/client']
run against old: docker:18.x.x / docker:18.x.x -dind | docker:stable / docker:dind
run with: DOCKER_TLS_CERTDIR:""
run with/without:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
add endpoint to service:
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
Content of gitlab-runner toml
concurrent = 1
check_interval = 0
log_level = "debug"
[session_server]
session_timeout = 1800
[[runners]]
name = "docker-runner"
url = "xxxxxxxx"
token = "xxxxxxx"
executor = "docker"
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
privileged = true
image = "docker:19.03.12"
disable_cache = false
volumes = ["/cache", "/certs/client"]
network_mode = "host"
Content of gitlab-ci.yml
image: docker:19.03.0
services:
- docker:19.03.0-dind
stages:
- build
- test_framework
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
CONTAINER_TEST_IMAGE: xxxx
CONTAINER_RELEASE_IMAGE: xxxx
before_script:
- docker info
- docker login -u xxxx -p $CI_JOB_TOKEN xxxx
build:
stage: build
tags:
- adm-docker
script:
- docker pull $CONTAINER_RELEASE_IMAGE || true
- docker build -t $CONTAINER_TEST_IMAGE --cache-from $CONTAINER_RELEASE_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
timeout: 1 hours
.test_commit: &test_commit
stage: test_framework
image: $CONTAINER_TEST_IMAGE
tags:
- adm-docker
timeout: 1 hours
artifacts:
reports:
junit: 'results/xunit.xml'
expire_in: 1 day
except:
- master
test-unit:
<<: *test_commit
script:
- python3 -m pytest --junitxml=results/xunit.xml test_unit/
Only one thing fix issue (workaround issue). When I add to .toml
volume: ["/var/run/docker.sock:/var/run/docker.sock"]
But after that Iam loosing DIND possibility to run my gitlab-ci.yml with different image for test stage (without using under script: -docker run MY_IMAGE python3....).
Which is not what I want
gitlab-runner under Ubuntu20 / Docker version 20.10.5, build 55c4c88
Ive worked with very similar gitlab-ci.yml around 1Yr ago and there was no issue with docker executor
Any ideas/suggestions ?
I was able to fix issue by changing flow of my gitlab-ci.yml
image: docker:19.03.5
services:
- docker:19.03.5-dind
stages:
- build
- test_framework
- release
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
CONTAINER_TEST_IMAGE: xxxxx
CONTAINER_RELEASE_IMAGE: xxxxx
build:
stage: build
tags:
- adm-docker
before_script:
- docker info
- docker login -u xxxxx -p $CI_JOB_TOKEN xxxxx
script:
- docker pull $CONTAINER_RELEASE_IMAGE || true
- docker build -t $CONTAINER_TEST_IMAGE --cache-from $CONTAINER_RELEASE_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
timeout: 1 hours
.test_commit: &test_commit
stage: test_framework
tags:
- adm-docker
timeout: 1 hours
artifacts:
reports:
junit: 'results/xunit.xml'
expire_in: 1 day
except:
- master
test-unit:
<<: *test_commit
image: $CONTAINER_TEST_IMAGE
script:
- python3 -m pytest --junitxml=results/xunit.xml test_unit/
and toml
[[runners]]
name = "docker-runner"
url = xxxxx
token = xxxxx
executor = "docker"
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
privileged = true
image = "docker:19.03.12"
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
network_mode = "host"
issue was fixed by volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
and issue with test stage was cased by:
before_script:
- docker info
- docker login -u xxxxx -p $CI_JOB_TOKEN xxxxx
in root structure of .yml file. I had to move it to build stage
I hope that will help ppl in the future
I am trying to build a CI/CD pipeline in GitLab. The goal is to build a docker image from a Dockerfile, run tests on the running container, push the image to DockerHub, then deploy it to a Kubernetes cluster. This is what I currently have for my gitlab-ci.yml.
variables:
DOCKER_TLS_CERTDIR: "/certs"
DOCKER_DRIVER: overlay2
CONTAINER_IMAGE: ${DOCKER_USER}/my_app
services:
- docker:19.03.12-dind
build:
image: docker:19.03.12
stage: build
script:
- echo ${DOCKER_PASSWORD} | docker login --username ${DOCKER_USER} --password-stdin
- docker pull ${CONTAINER_IMAGE}:latest || true
- docker build --cache-from ${CONTAINER_IMAGE}:latest --tag ${CONTAINER_IMAGE}:$CI_COMMIT_SHA --tag ${CONTAINER_IMAGE}:latest .
- docker push ${CONTAINER_IMAGE}:$CI_COMMIT_SHA
- docker push ${CONTAINER_IMAGE}:latest
deploy:
image:
name: bitnami/kubectl:1.16.15
entrypoint: [""]
stage: deploy
variables:
GIT_STRATEGY: none
script:
- kubectl get pods -A # <- Won't work until I pass a Kubeconfig file with cluster details
I have a few main questions:
How can I deploy this image? I know I need to pass a KUBECONFIG file to bitnami/kubectl, but not sure how to do that with GitLab CI/CD
Can I pass the built image to a test stage before pushing to DockerHub
---
stages:
- test app
- build
- test
- deploy
test app:
stage: test_app
image: node:latest
script:
- git clone (path to code)
- npm install
- lint
- audit fix
rules:
- if: '$CI_COMMIT_REF_NAME != "master"
build image:
stage: build
script:
- docker build your_image:$CI_COMMIT_REF_NAME
- deploy push your_image:$CI_COMMIT_REF_NAME
rules:
- if: '$CI_COMMIT_REF_NAME != "master"
test image:
stage: test
image: anchor:latest (one you have built yourself or use another testing suite)
script:
- anchore-cli image add user/image:v1
- anchore-cli image wait user/image:v1
- anchore-cli image content user/image:v1
- image vuln user/image:v1 all
- anchore-cli evaluate check user/image:v1 > result .txt
- if ( grep -ci "fail" result.txt >= 1); then exit 1 fi
rules:
- if: '$CI_COMMIT_REF_NAME != "master"
deploy image:
image:
name: kubectl:latest (build your own image that installed kubectl)
entrypoint: [""]
stage: deploy
tags:
- privileged
# Optional: Manual gate
when: manual
dependencies:
- build-docker
script:
- kubectl config set-cluster k8s --server="$CLUSTER_ADDRESS"
- kubectl config set clusters.k8s.certificate-authority-data $CA_AUTH_DATA
- kubectl config set-credentials gitlab-service-account --token=$K8S_TOKEN
- kubectl config set-context default --cluster=k8s --user=gitlab-service-account --namespace=my-service
- kubectl config use-context default
- kubectl set image $K8S_DEPLOYMENT_NAME $CI_PROJECT_NAME=$IMAGE_TAG
- kubectl rollout restart $K8S_DEPLOYMENT_NAME
1. have variables passed in for cluter address, cert data, and token stuff... so you can target other clusters, pre-prod, prod, qa...
2. you can't test an image that isn't on the repo, as the testing suite needs to pull the image from somewhere... You should have a clean up script running to cleanup old image in your repo anyway, so the initial push should be a (test location)
like: docker push untrusted/image:v1
You should also have before scripts and after scripts... before calls docker login
after calls docker logout...
I do not have an answer for deploying to Kubernetes, but I do recommend publishing a test/construction image to Dockerhub when working a merge request/development branch of building the image. Then only deploy the latest image when you merge the branch to master .
---
stages:
- build
- test
- deploy
build image:
stage: build
script:
- docker build your_iamge:test
- deploy push your_image:test
rules:
- if: '$CI_COMMIT_REF_NAME != "master"
test image:
stage: test
image: your_image:test
script:
- commands to test image
rules:
- if: '$CI_COMMIT_REF_NAME != "master"
deploy image:
stage: deploy
script:
- docker build your_image:latest
- docker push your_image:latest
rules:
- if: '$CI_COMMIT_REF_NAME == "master"
---
stages:
- build
- test
- deploy
build image:
stage: build
script:
- docker build your_image:$CI_COMMIT_REF_NAME
- deploy push your_image:$CI_COMMIT_REF_NAME
rules:
- if: '$CI_COMMIT_REF_NAME != "master"
test image:
stage: test
image: your_image:test
script:
- commands to test image
rules:
- if: '$CI_COMMIT_REF_NAME != "master"
deploy image:
stage: deploy
script:
- docker build your_image:latest
- docker push your_image:latest
- export BRANCH=${CI_COMMIT_TITLE#*\'}; export BRANCH=${BRANCH%\' into*}
- docker delete your_image:$BRANCH
rules:
- if: '$CI_COMMIT_REF_NAME == "master"
I have GitLab server and a separate docker-registry. I have created a testuser with a testpassword.
My .gitlab-ci.yml
image: docker:latest
services:
- docker:18.09.7-dind
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
# See https://github.com/docker-library/docker/pull/166
DOCKER_TLS_CERTDIR: ""
REGISTRY: myregistry:5000/golang-test
stages:
- build
# - test
before_script:
- docker login -u testuser -p testpassword myregistry:5000
build_project:
stage: build
tags:
- docker-ci
script:
- docker ps
- docker build -t $REGISTRY .
- docker push $REGISTRY
When CI pipeline runs, I get following error.
WARNING! Using --password via the CLI is insecure. Use
--password-stdin. Error response from daemon: Get https://myregistry:5000/v2/: dial tcp: lookup worker1 on
192.168.123.456:53: no such host ERROR: Job failed: exit code 1
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "docker-ci"
url = "https://gitlab.sample.com:8000/"
token = "89MfxRTby83ny3eTkTf2"
tls-ca-file = "/etc/gitlab-runner/certs/gitlab.sample.com.crt"
executor = "docker"
pre_build_script = "export DOCKER_HOST=tcp://docker:2375"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "ubuntu:18.04"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
This was after suggested modifications here.
I can successfully login to my docker-registry using docker login -u testuser -p testpassword https://myregistry:5000 from gitlab running server. refered
After uwe-plonuscomment, I have added following part, but still same error.
before_script:
- echo "192.168.10.219 worker1" >> /etc/hosts
- docker login -u testuser -p testpassword myregistry:5000