I have some troubles getting my gitlab-runner to execute docker buildx command.
I have a gitlab-runner which is configured like this:
[[runners]]
name = "Name"
url = "https://gitlab.mypage.com/"
token = "token"
executor = "shell"
shell = "powershell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
And the pipeline which is triggered:
stages:
- test
- build
test_backend:
stage: test
script:
- exit 0
only:
- merge_request
- master
build:
stage: build
script:
- docker login someregistry -u xxxx -p yyyy
- docker buildx ls
only:
- merge_request
- master
- dev
I obfuscated the code a bit.
The problem I have is, that the docker login command is executed correctly but the docker buildx command not.
I already tested the command manually on the machine and it was successfull.
Can somebody help me here?
In my experience with the docker runners the most likly situation here is, that the docker runner doesnt have the experimental features enabled just because the docker base have it... I have experienced things like that in the past: The docker in the runner IS NOT the docker where you hosting the runner at!
You problably have to add the DIND (Docker in Docker) Service for that, because as far as I understand this runner systems, only then the docker from your host is connected with the docker within the runner.
We did it like that:
# gitlab-runner
gitlab-runner:
container_name: vivavis.gitlab-runner
image: gitlab/gitlab-runner:latest
restart: always
volumes:
- gitlab-runner:/etc/gitlab-runner
- /var/run/docker.sock:/var/run/docker.sock // <<<<<< THIS IS THE IMPORTANT LINE
networks:
- swp-infra-code
A little bit of warning here:
When activating that, we experienced a bug / not well cleaned up thing:
Because the GitLab Runner is now connected with the base docker, the images that will be created while running a CI / CD are not deleted properly: GitLab doesnt implemented that feature, it just assumes, that when the gitlab-runner-container dies, all data dies within. But with this connection the data here is not dying becuase it is not created within the container - it is created in the base docker.
I found a solution for my problem.
To use the experimental features you can set environment variables.
$env:DOCKER_CLI_EXPERIMENTAL=enabled
this command can be used in ci pipeline.
It looks like the docker cli in shell executor is not the same as the docker cli you can use on system if you try out.
Very confusing.
Related
I tested a gitlab-runner on a virtual machine, it worked perfectly. I followed this tutorial at part Use docker-in-docker executor :
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
When i register a runner with exactly the same configuration on my dev server, the runner is called when there is a commit but i got alot of errors :
*** WARNING: Service runner-XXX-project-XX-concurrent-X-docker-X probably didn't start properly.
ContainerStart: Error response from daemon: Cannot link to a non running container: /runner-XXX-project-XX-concurrent-X-docker-X AS /runner-XXX-project-XX-concurrent-X-docker-X-wait-for-service/service (executor_docker.go:1337:1s)
DEPRECATION: this GitLab server doesn't support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab
$ docker info
error during connect: Get http://docker:2375/v1.39/info: dial tcp: lookup docker on MY.DNS.IP:53: no such host
ERROR: Job failed: exit code 1
I believe all these error are due to the first warning. I tried to :
Add a second DNS with 8.8.8.8 IP to my machine, same error
Add privileged=true manually in /etc/gitlab-runner/config.toml, same error, so it's not due to the privileged = true parameter
Replace tcp://docker:2375 by tcp://localhost:2375, can't find docker daemon on the machine when docker info
gitlab-ci.yml content :
image: docker:stable
stages :
- build
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
services:
- docker:dind
before_script:
- docker info
build-folder1:
stage: build
script:
- docker build -t image1 folder1/
- docker run --name docker1 -p 3001:5000 -d image1
only:
refs:
- dev
changes:
- folder1/**/*
build-folder2:
stage: build
script:
- docker build -t image2 folder2/
- docker run --name docker2 -p 3000:3000 -d image2
only:
refs:
- dev
changes:
- folder2/**/*
If folder1 of branch dev is modified, we build and run the docker1
If folder2 of branch dev is modified, we build and run the docker2
docker version on dev server :
docker -v
Docker version 17.03.0-ce, build 3a232c8
gitlab-runner version on dev server :
gitlab-runner -v
Version: 11.10.1
I will try to provide an answer for you, as I come to fix this same problem when trying yo run DinD.
This message:
*** WARNING: Service runner-XXX-project-XX-concurrent-X-docker-X probably didn't start properly.
Means that either you have not properly configured your runner, or it is not linked by the gitlab-ci.yml file. You should be able to ckeck the ID of the runner used in the log page at Gitlab.
To start with, verify that you entered the gitlab-runner register command right, with the proper registration token.
Second, since you are setting a specific runner manually, verify that you have set some unique tag to it (eg. build_docker), and call it from your gitlab-ci.yml file. For example:
...
build-folder1:
stage: build
script:
- docker build -t image1 folder1/
- docker run --name docker1 -p 3001:5000 -d image1
tags:
- build_docker
...
That way it should work.
I looked at any other questions but can't find my own solution! I setting up a CI in gitlab and use the gitlab's shared runner. In build stage I used docker image as base image but when i use docker command it says :
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I looked at this topic but still don't understand what should I do?
.gitlab-ci.yml :
stages:
- test
- build
- deploy
job_1:
image: python:3.6
stage: test
script:
- sh ./sh_script/install.sh
- python manage.py test -k
job_2:
image: docker:stable
stage: build
before_script:
- docker info
script:
- docker build -t my-docker-image .
I know that the gitlab runner must registered to use docker and share /var/run/docker.sock! But how to do this when using the gitlab own runner?
Ahh, that's my lovely topic - using docker for gitlab ci. The problem you are experiencing is better known as docker-in-docker.
Before configuring it, you may want to read this brilliant post: http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
That will give you a bit of understanding what is the problem and which solution best fits you. Generally there are 2 major approaches: actual installation of docker daemon inside docker and sharing host's daemon to containers. Which approach to choose - depends on your needs.
In gitlab you can go in several ways, I will just share our experience.
Way 1 - using docker:dind as a service.
It is pretty simple to setup. Just add docker:dind as a shared service to your gitlab-ci.yml file and use docker:latest image for your jobs.
image: docker:latest # this sets default image for jobs
services:
- docker:dind
Pros:
simple to setup.
simple to run - your source codes are available by default to your job in cwd because they are being pulled directly to your docker runner
Cons: you have to configure docker registry for that service, otherwise you will get your Dockerfiles built from scratch each time your pipeline starts. As for me, it is unacceptable, because can take more than an hour depending on the number of containers you have.
Way 2 - sharing /var/run/docker.sock of host docker daemon
We setup our own docker executor with docker daemon and shared the socket by adding it in /etc/gitlab-runner/config.toml file. Thus we made our machine's docker daemon available to docker cli inside containers. Note - you DONT need privileged mode for executor in this case.
After that we can use both docker and docker-compose in our custom docker images. Moreover, we dont need special docker registry because in this case we share executor's registry among all containers.
Cons
You need to somehow pass sources to your containers in this case, because you get them mounted only to docker executor, but not to containers, launched from it. We've stopped on cloning them with command like git clone $CI_REPOSITORY_URL --branch $CI_COMMIT_REF_NAME --single-branch /project
I have a docker-compose file which builds two containers, a node app and a ngnix server. Now I would like to automate the build and run process on the server with the help of Gitlab runners. I am pretty new to CI-related stuff so please excuse my approach:
I would want to create multiple repositories on gitlab.com and have a Dockerfile for each one of these. Do I now have to associate a gitlab-runner instance with each of these projects in order to build the image, push it to a docker repo and let the server pull it from there? And then I would have to somehow push the docker-compose file on the server and compose everything from there.
So my questions are:
Am I able to run multiple (2 or 3) gitlab-runner for all of my repos on one server?
Do I need a specific or shared runner and what exactly is the difference?
Why are all tutorials using self hosted Gitlab instances instead of just using gitlab repos (Is it not possible to use gitlab-runner with gitlab.com repos?)
Is it possible to use docker-compose in a gitlab-runner pipeline and just build everything at once?
First of all, you can obviously use GitLab CI/CD features on https://gitlab.com as well as on self hosted GitLab instances. It doesn't change anything, except the host on which you will register your runner:
https://gitlab.com/ in case you uses GitLab without hosting it
https://your-custom-domain/ in case you host your own instance of GitLab
You can add as many runners as you want (I think so, and at least I have 5-6 runners per project without problem). You just need to register each of those runners for your project. See Registering Runners for that.
As for shared runners versus specific runner, I think you should stick to share runners if you wish to try GitLab CI/CD.
Shared Runners on GitLab.com run in autoscale mode and are powered by DigitalOcean. Autoscaling means reduced wait times to spin up builds, and isolated VMs for each project, thus maximizing security.
They're free to use for public open source projects and limited to 2000 CI minutes per month per group for private projects. Read about all GitLab.com plans.
You can install your own runners on literraly any machine though, for example your laptotp. You can deploy it with Docker for a quick start.
Finally, yes you can use docker-compose in a gitlab-ci.yml file if you use ssh executor and have docker-compose install on your server.
But I recommend using the docker executor and use docker:dind (Docker in Docker) image
What is Docker in Docker?
Although running Docker inside Docker is generally not recommended, there are > some legitimate use cases, such as development of Docker itself.
Here is an example usage, without docker-compose though:
image: docker:latest
services:
- name: docker:dind
command: ["--experimental"]
before_script:
- apk add --no-cache py-pip # <-- add python package install pip
- pip install docker-compose # <--- add docker-compose
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin # <---- Login to your registry
build-master:
stage: build
script:
- docker build --squash --pull -t "$CI_REGISTRY_USER"/"$CI_REGISTRY_IMAGE":latest .
- docker push "$CI_REGISTRY_USER"/"$CI_REGISTRY_IMAGE":latest
only:
- master
build-dev:
stage: build
script:
- docker build --squash --pull -t "$CI_REGISTRY_USER"/"$CI_REGISTRY_IMAGE":"$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_USER"/"$CI_REGISTRY_IMAGE":"$CI_COMMIT_REF_SLUG"
except:
- master
As you can see, I build the Docker image, tag it, then push it to my Docker registry, but you could push to any registry. And of course you could use docker-compose at any time in a script declaration
My Git repository looks like :
/my_repo
|---- .gitignore
|---- .gitlab-ci.yml
|---- Dockerfile
|---- README.md
And the config.toml of my runner looks like:
[[runners]]
name = "4Gb digital ocean vps"
url = "https://gitlab.com"
token = "efnrong44d77a5d40f74fc2ba84d8"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:dind"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
[runners.cache]
You can take a look at https://docs.gitlab.com/runner/configuration/advanced-configuration.html for more information about Runner configuration.
Note : All the variables used here are secret variables. See https://docs.gitlab.com/ee/ci/variables/ for explanations
I hope it answers your questions
GitLab's running in kubernetes cluster. Runner can't build docker image with build artifacts. I've already tried several approaches to fix this, but no luck. Here are some configs snippets:
.gitlab-ci.yml
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
stages:
- build
- package
- deploy
maven-build:
image: maven:3-jdk-8
stage: build
script: "mvn package -B --settings settings.xml"
artifacts:
paths:
- target/*.jar
docker-build:
stage: package
script:
- docker build -t gitlab.my.com/group/app .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
- docker push gitlab.my.com/group/app
config.toml
concurrent = 1
check_interval = 0
[[runners]]
name = "app"
url = "https://gitlab.my.com/ci"
token = "xxxxxxxx"
executor = "kubernetes"
[runners.kubernetes]
privileged = true
disable_cache = true
Package stage log:
running with gitlab-ci-multi-runner 1.11.1 (a67a225)
on app runner (6265c5)
Using Kubernetes namespace: default
Using Kubernetes executor with image docker:latest ...
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Running on runner-6265c5-project-4-concurrent-0h9lg9 via gitlab-runner-3748496643-k31tf...
Cloning repository...
Cloning into '/group/app'...
Checking out 10d5a680 as master...
Skipping Git submodules setup
Downloading artifacts for maven-build (61)...
Downloading artifacts from coordinator... ok id=61 responseStatus=200 OK token=ciihgfd3W
$ docker build -t gitlab.my.com/group/app .
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1
What am I doing wrong?
Don't need to use this:
DOCKER_DRIVER: overlay
cause it seems like OVERLAY isn't supported, so svc-0 container is unable to start with it:
$ kubectl logs -f `kubectl get pod |awk '/^runner/{print $1}'` -c svc-0
time="2017-03-20T11:19:01.954769661Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2017-03-20T11:19:01.955720778Z" level=info msg="libcontainerd: new containerd process, pid: 20"
time="2017-03-20T11:19:02.958659668Z" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."
Also, add export DOCKER_HOST="tcp://localhost:2375" to the docker-build:
docker-build:
stage: package
script:
- export DOCKER_HOST="tcp://localhost:2375"
- docker build -t gitlab.my.com/group/app .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
- docker push gitlab.my.com/group/app
When using Kubernetes, you have to adjust your Build image to connect with the Docker engine.
Add to your build image:
DOCKER_HOST=tcp://localhost:2375
Quote from the docs:
Running the docker:dind also known as the docker-in-docker image is also
possible but sadly needs the containers to be run in privileged mode.
If you're willing to take that risk other problems will arise that might not
seem as straight forward at first glance. Because the docker daemon is started
as a service usually in your .gitlab-ci.yaml it will be run as a separate
container in your pod. Basically containers in pods only share volumes assigned
to them and an IP address by wich they can reach each other using localhost.
/var/run/docker.sock is not shared by the docker:dind container and the docker
binary tries to use it by default. To overwrite this and make the client use tcp
to contact the docker daemon in the other container be sure to include
DOCKER_HOST=tcp://localhost:2375 in your environment variables of the build container.
Gitlab-CI on Kubernetes
based on #Yarik 's comment what worked for me was
- export DOCKER_HOST=$DOCKER_PORT
no other answers worked.
I had the same problem, and I could not get the above workarounds to work for me (I did not try the volumes trick mentioned by #fkpwolf).
Now GitLab has an alternative solution by using Kaniko, which did work for me:
The .gitlab-ci.yaml could then be something like this, in that case:
stages:
- build
- package
- deploy
maven-build:
image: maven:3-jdk-8
stage: build
script: "mvn package -B --settings settings.xml"
artifacts:
paths:
- target/*.jar
docker-kaniko-build:
stage: package
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- echo "{\"auths\":{\"gitlab.my.com\":{\"username\":\"gitlab-ci-token\",\"password\":\"$CI_BUILD_TOKEN\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination gitlab.my.com/group/app
From the GitLab docs it is mentioned that:
kaniko solves two problems with using the docker-in-docker build method:
Docker-in-docker requires privileged mode in order to function, which is a significant security concern.
Docker-in-docker generally incurs a performance penalty and can be quite slow.
See: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
I am trying to set up automated deployment using a GitLab CI runner to deploy our 4-container app via docker-compose. I can pull the container images down using docker pull commands, but I'm stuck on how to connect to the Google Compute Engine instance in order to run the full docker-compose script.
Typically, from my local machine, I run something like:
eval $(docker-machine env <machine-instance>)
docker-compose up -d
But my .gitlab-ci.yml script doesn't have docker-machine available.
Do I have to install docker-machine via the script section in my
.gitlab-ci.yml file?
How do I provision the instance without
creating a new one every time? Normally, from my local host, I would
run docker-machine create ... once then just use the eval
command above to reconnect to the instance. But how would this work
with CI?
Here's a sample of my .gitlab-ci.yml:
deploy staging:
image: docker:latest
services:
- docker:dind
environment: staging
stage: deploy
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN my-registry.githost.io
script:
- docker pull my-registry.githost.io/group/project1:develop
- docker pull my-registry.githost.io/group/project2:develop
- docker pull my-registry.githost.io/group/project3:develop
- docker pull my-registry.githost.io/group/project4:develop
- docker-machine ls
Not sure what you need docker-machine for in this case. You might want to get rid of it.
But to go back to your question, the docker image you're using does not come with neither docker-machine, nor docker-compose :
https://github.com/docker-library/docker/blob/36e2107fb879d5d5c3dbb5d8d93aeef0a2d45ac8/1.12/Dockerfile
So you will need to create a new image (or find an existing one) that comes with those two installed.
So in the .gitlab-ci.yml, instead of image: docker:latest, it's going to be something like image: mydocker
You maybe have to install docker-machine in the GitLab CI Runner to use it with GCE
https://docs.docker.com/machine/install-machine/
https://docs.docker.com/machine/drivers/gce/