These are the steps I followed to setup a custom Gitlab runner:
Install Gitlab runner by following the instructions here: https://docs.gitlab.com/runner/install/linux-repository.html
Register a runner with docker exectuor and docker:19 image
Write gitlab-ci.yml as follow:
image: docker:19.03.1
services:
- name: docker:19.03.1-dind
alias: docker
stages:
- build
build:
stage: build
variables:
IMAGE_TAG: repo.azurecr.io/some-repo
DOCKER_HOST: tcp://172.17.0.1:2375
DOCKER_TLS_CERTDIR: ''
script:
- docker login someacr.azurecr.io -u "$SERVICE_PRINCIPAL_USER" -p "$SERVICE_PRINCIPAL_PASSWORD"
- if [[ "$CI_COMMIT_REF_NAME" == "develop" ]]; then docker build -t $IMAGE_TAG .; fi
Running the pipeline throws this error:
Login succeeded
if [[ "$CI_COMMIT_REF_NAME" == "develop" ]]; then docker build -t $IMAGE_TAG .; fi
Cannot connect to the Docker daemon at tcp://172.17.0.1:2375. Is the docker daemon running?
Cleaning up project directory and file based variables
00:02
ERROR: Job failed: exit code 1
This is what I have tried so far:
privileged: true in /etc/gitlab-runner/config.toml
dind:19 docker image in gitlab-ci.yml
adding volumes = ["/var/run/docker.sock /var/run/docker.sock" "/cache"] in /etc/gitlab-runner/config.toml
used DOCKER_TLS_CERTDIR: '' in gitlab-ci.yml as variable
populating DOCKER_HOST variable with DOCKER_HOST: tcp://172.17.0.1:2375
much more...
All these changes result in either of the 2 issues described in Gitlab Runner Troubleshooting here: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#troubleshooting
docker: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
Docker no such host error.
Your DOCKER_HOST variable value is not correct.
Use DOCKER_HOST: "tcp://docker:2375"
adding volumes = ["/var/run/docker.sock /var/run/docker.sock" "/cache"] in /etc/gitlab-runner/config.toml
I would recommend removing the volume mount of /var/run/docker.sock from the runner configuration here. This is not necessary when leveraging the docker:dind service container method. Though, it technically won't cause issues.
privileged: true in /etc/gitlab-runner/config.toml
[...]
used DOCKER_TLS_CERTDIR: '' in gitlab-ci.yml as variable
These changes are good, keep these changes.
Related
i have this pipeline to execute :
stages:
- build-gitlab
- deploy-uat
build:
image: node:14-alpine
stage: build-gitlab
services:
- docker
before_script:
- docker login $CI_REGISTRY_URL -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
script:
- docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
- docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $CI_FRONTEND_REGISTRY_URL
- docker push $CI_FRONTEND_REGISTRY_URL
deploy:
image:
name: bitnami/kubectl:latest
stage: deploy-uat
before_script:
- kubectl config set-cluster deploy-cluster --server="$K8S_SERVER" --insecure-skip-tls-verify
- kubectl config set-credentials gitlab --token=$(echo $K8S_TOKEN | base64 -d)
- kubectl config set-context deploy-cluster --cluster=deploy-cluster --namespace=ns-frontend-dev --user=gitlab
- kubectl config use-context deploy-cluster
script:
- envsubst < deploy.tmpl > deploy.yaml
- kubectl apply -f deploy.yaml
Initially i defined a runner for my gitlab with shell executor. Docker is installed in my runner that is why the build stage executed itself successfully. But if i would like to use multiple docker images as you can see in my gitlab-ci.yaml file, the shell executor is not the appropriate one.
I saw this documentation about gitlab executors
but it is not explicit enough.
i register a new runner with docker executor, then i got this result :
Preparing the "docker" executor
Using Docker executor with image node:14-alpine ...
Starting service docker:latest ...
Pulling docker image docker:latest ...
Using docker image sha256:0f8d12a73562adf6588be88e37974abd42168017f375a1e160ba08a7ee3ffaa9 for docker:latest with digest docker#sha256:75026b00c823579421c1850c00def301a6126b3f3f684594e51114c997f76467 ...
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-jdn9pn3z-project-33-concurrent-0-0e760484a3d3cab3-docker-0 probably didn't start properly.
Health check error:
service "runner-jdn9pn3z-project-33-concurrent-0-0e760484a3d3cab3-docker-0-wait-for-service" health check: exit code 1
Health check container logs:
2023-01-18T15:50:31.037166246Z FATAL: No HOST or PORT found
and the deploy part did not succeed. What is the right executor to choose between :
docker, shell, ssh, kubernetes, custom, parallels, virtualbox, docker+machine, docker-ssh+machine, instance, docker-ssh
And how to use it
I am trying to build a super-simple CI/CD pipeline using GitLab CI.
Upon running it I get presented with the error:
Server:
ERROR: Cannot connect to the Docker daemon at tcp://docker:2375.
Is the docker daemon running?
My .gitlab-ci.yml is :
image: docker:latest
variables:
DOCKER_HOST: tcp://docker:2375
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
before_script:
- docker --version
docker_build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker build -t arieltar/hubsec:1.1 .
- docker push arieltar/hubsec:1.1
Based on the error message I would ask, does the gitlab-runner user belong to the docker group?
You will need to decide if you want to use Docker-in-Docker with, or without TLS. This requires changing /etc/gitlab-runner/config.toml settings, and assigning the DOCKER_TLS_CERTDIR in your .gitlab-ci.yml file. See the Docker-in-docker section of the GitLab docs.
Please check below things as prelim.
Whether docker is running or not
Login with gitlab-user if you are running pipeline with gitlab user and check if that user can access or run docker ps without sudo :).
add below entry if pt1. and pt2 satisfied.
services:
name: docker:dind
entrypoint: ["dockerd-entrypoint.sh", "--tls=false"]
script:
export DOCKER_HOST=tcp://127.0.0.1:2375 && docker build -t arieltar/hubsec:1.1 .
I'm trying to implement a docker build stage into my gitlab pipeline. Ideally I'd be able to build the image first and then run later stages against a container. However, when I use the build step in the gitlab-ci.yaml below I sometimes get the error:
error during connect: Get http://docker:2375/v1.37/info: dial tcp: lookup docker on 10.51.240.10:53: no such host
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1
On the occasions when it doesn't work I can only guess that the docker deamon isn't ready for connections. Does anyone have any ideas on how get this going?
gitlab-ci.yaml:
stages:
- build
build:
image: docker:18
services:
- docker:18-dind
variables:
DOCKER_DRIVER: overlay
DOCKER_HOST: tcp://docker:2375/
before_script:
- docker info
script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
- docker pull ${CI_REGISTRY_IMAGE}:latest || echo Could not fetch ${CI_REGISTRY_IMAGE}:latest from registry
- docker pull ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} || echo Could not fetch ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} from registry
- docker build -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} --cache-from ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} .
- docker push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}
- if [[ ${CI_COMMIT_REF_SLUG} == "master" ]] ; then docker tag ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} ${CI_REGISTRY_IMAGE}:latest ; fi
- if [[ ${CI_COMMIT_REF_SLUG} == "master" ]] ; then docker push ${CI_REGISTRY_IMAGE}:latest ; fi
stage: build
UPDATE: I am using a kubernetes executor.
Are you using gitlab-runner to run jobs? If so, are you setting --executor docker registering your runner?
Are you passing /var/run/docker.sock as a volume to the runner?
I have docker images building in GitLab CI working without any problems. I'm not passing any DOCKER_DRIVER or DOCKER_HOST variables.
We've seen this EXACT same error with nearly all the same conditions (we specify localhost instead of docker for DOCKER_HOST, and we use overlay2 instead of plain
overlay). We're seeing it on the Docker executor mainly; for now we're just chalking it up to the executor being flakey, because we retry the job and it succeeds. We've checked and there was no previous contention for the runner, so we couldn't find any explanation.
In the end we self hosted a runner on kubernetes that was deployed with the helm chart and set it going in privileged mode.
We also switched to overlay2
In https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-in-docker-executor they say that kubernetes should have tcp://localhost:2375 as DOCKER_HOST:
# Note that if you're using Kubernetes executor, the variable should be set to
# tcp://localhost:2375 because of how Kubernetes executor connects services
# to the job container
DOCKER_HOST: tcp://docker:2375/
I am trying to automate the deployment process of my project. The environment looks like:
we use GitLab to store our code
we execute a CD/CI pipeline within GitLab to build a Docker image and to store it in Amazon repository
once the build stage is completed, Docker have to run in deployment stage the latest image on the first of two instances and after successful execution to scale the containers on the second instance.
This is how the .gutlab-ci.yml file looks like:
image: docker:latest
services:
- docker:dind
stages:
- build
- deploy
variables:
DOCKER_DRIVER: overlay2
testBuild:
stage: build
script:
- docker login -u AWS -p <password> <link to Amazons' repo>
- docker build -t <repo/image:latest> app/
- docker push <repo/image:latest>
testDeploy:
stage: deploy
variables:
AWS_DEFAULT_REGION: "us-east-2"
AWS_ACCESS_KEY_ID: "access key"
AWS_SECRET_ACCESS_KEY: "ssecretAK"
AWS_CLUSTER: "testCluster"
AWS_SIZE: "2"
before_script:
- apk add --update curl
- curl -o /usr/local/bin/ecs-cli https://s3.amazonaws.com/amazon-ecs-cli/ecs-cli-linux-amd64-latest
- chmod +x /usr/local/bin/ecs-cli
script:
- docker login -u AWS -p <password> <repo_link>
- docker run --rm --name <name-ofcontainer> -p 80:8000 -i <repo/image:latest>
- ecs-cli configure --region $AWS_DEFAULT_REGION --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --cluster $AWS_CLUSTER
- ecs-cli scale --capability-iam --size $AWS_SIZE
only:
- development
Now when the script is successfully executed I SSH the instances and and enter docker ps -a it does not list a running container also it does not find the image with docker image.
If I enter manually the commands on one of the instances the website is available.
My questions is how to make the container available?
EDIT 1:
We use shared runner, if this is what you asks. The reason we use docker:dind is because when we do not use it the following error occurs and we cannot go further:
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
GitLab's running in kubernetes cluster. Runner can't build docker image with build artifacts. I've already tried several approaches to fix this, but no luck. Here are some configs snippets:
.gitlab-ci.yml
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
stages:
- build
- package
- deploy
maven-build:
image: maven:3-jdk-8
stage: build
script: "mvn package -B --settings settings.xml"
artifacts:
paths:
- target/*.jar
docker-build:
stage: package
script:
- docker build -t gitlab.my.com/group/app .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
- docker push gitlab.my.com/group/app
config.toml
concurrent = 1
check_interval = 0
[[runners]]
name = "app"
url = "https://gitlab.my.com/ci"
token = "xxxxxxxx"
executor = "kubernetes"
[runners.kubernetes]
privileged = true
disable_cache = true
Package stage log:
running with gitlab-ci-multi-runner 1.11.1 (a67a225)
on app runner (6265c5)
Using Kubernetes namespace: default
Using Kubernetes executor with image docker:latest ...
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Running on runner-6265c5-project-4-concurrent-0h9lg9 via gitlab-runner-3748496643-k31tf...
Cloning repository...
Cloning into '/group/app'...
Checking out 10d5a680 as master...
Skipping Git submodules setup
Downloading artifacts for maven-build (61)...
Downloading artifacts from coordinator... ok id=61 responseStatus=200 OK token=ciihgfd3W
$ docker build -t gitlab.my.com/group/app .
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1
What am I doing wrong?
Don't need to use this:
DOCKER_DRIVER: overlay
cause it seems like OVERLAY isn't supported, so svc-0 container is unable to start with it:
$ kubectl logs -f `kubectl get pod |awk '/^runner/{print $1}'` -c svc-0
time="2017-03-20T11:19:01.954769661Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2017-03-20T11:19:01.955720778Z" level=info msg="libcontainerd: new containerd process, pid: 20"
time="2017-03-20T11:19:02.958659668Z" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."
Also, add export DOCKER_HOST="tcp://localhost:2375" to the docker-build:
docker-build:
stage: package
script:
- export DOCKER_HOST="tcp://localhost:2375"
- docker build -t gitlab.my.com/group/app .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
- docker push gitlab.my.com/group/app
When using Kubernetes, you have to adjust your Build image to connect with the Docker engine.
Add to your build image:
DOCKER_HOST=tcp://localhost:2375
Quote from the docs:
Running the docker:dind also known as the docker-in-docker image is also
possible but sadly needs the containers to be run in privileged mode.
If you're willing to take that risk other problems will arise that might not
seem as straight forward at first glance. Because the docker daemon is started
as a service usually in your .gitlab-ci.yaml it will be run as a separate
container in your pod. Basically containers in pods only share volumes assigned
to them and an IP address by wich they can reach each other using localhost.
/var/run/docker.sock is not shared by the docker:dind container and the docker
binary tries to use it by default. To overwrite this and make the client use tcp
to contact the docker daemon in the other container be sure to include
DOCKER_HOST=tcp://localhost:2375 in your environment variables of the build container.
Gitlab-CI on Kubernetes
based on #Yarik 's comment what worked for me was
- export DOCKER_HOST=$DOCKER_PORT
no other answers worked.
I had the same problem, and I could not get the above workarounds to work for me (I did not try the volumes trick mentioned by #fkpwolf).
Now GitLab has an alternative solution by using Kaniko, which did work for me:
The .gitlab-ci.yaml could then be something like this, in that case:
stages:
- build
- package
- deploy
maven-build:
image: maven:3-jdk-8
stage: build
script: "mvn package -B --settings settings.xml"
artifacts:
paths:
- target/*.jar
docker-kaniko-build:
stage: package
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- echo "{\"auths\":{\"gitlab.my.com\":{\"username\":\"gitlab-ci-token\",\"password\":\"$CI_BUILD_TOKEN\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination gitlab.my.com/group/app
From the GitLab docs it is mentioned that:
kaniko solves two problems with using the docker-in-docker build method:
Docker-in-docker requires privileged mode in order to function, which is a significant security concern.
Docker-in-docker generally incurs a performance penalty and can be quite slow.
See: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html