I'm trying to create a custom gitlab-runner to run a docker process, following:
https://github.com/gitlabhq/gitlabhq/blob/master/doc/ci/docker/using_docker_build.md
I tried the second approach in which I registered a runner using:
sudo gitlab-runner register -n \ --url https://gitlab.com/ \
--registration-token xxx \ --executor docker \ --description "My Docker Runner" \ --docker-image "docker:stable" \ --docker-volumes
/var/run/docker.sock:/var/run/docker.sock
However,at gitlab, whenever the pipeline starts I'm facing the following error:
ERROR: Failed to create container volume for /builds/xxx Unable to
load image: gitlab-runner-prebuilt: "open
/var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or
directory"
I can't find much information online, any help appreciated.
For The record, I got it working following this tutorial
https://angristan.xyz/build-push-docker-images-gitlab-ci/
Since the docker image worked, I suspect there's something wrong with the debian gitlab-runner distribution
Related
I'm trying to write-down a Dockerfile to create create and register a new runner to a private gitlab repository. According to gitlab documentation, I wrote down the following Dockerfile:
FROM gitlab/gitlab-runner:latest
RUN gitlab-runner register \
--non-interactive \
--url "https://gitlab.com/" \
--registration-token "GITLAB_REPO_TOKEN" \
--executor "docker" \
--docker-image alpine:latest \
--description "docker-runner" \
--maintenance-note "Free-form maintainer notes about this runner" \
--run-untagged="true" \
--locked="false"
Then build it with:
docker build -t test .
And then run it in a container via:
docker run test:latest
The runner is correctly seen by gitlab (the runner is available under Settings\CI/CD\Runners).
Then, I set up the following CI, for testing:
image: python:3.7-alpine
testci:
stage: test
script:
- python test.py
The job is then pulled by the runner, but I immediately get the following error:
Running with gitlab-runner 15.8.2 (4d1ca121)
on docker-runner yVa1JDny, system ID: xxxxxxxxx
Preparing the "docker" executor
00:09
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:753:0s)
Can anyone please provide support in that? I didn't get what it is missing from the configuration I've made.
I've tried to modify the docker run call trying with the volume mount guide found here, but nothing changes.
I've also found here a similar Dockerfile, but using a gitlab-ci-multi-runner which is not the desired service.
You're attempting to use the docker executor for your runner, but your runner doesn't have access to the docker socket in order to create new containers. Your runner manager (what your docker file is creating) is attempting to start up new docker containers to handle each of your jobs, but failing to connect to docker.
In your docker run command, you will need to do a couple things:
Set your container to use privileged mode: --privileged
Map in the docker socket: -v /var/run/docker.sock:/var/run/docker.sock
With those two things, you can likely connect to the docker daemon and start new containers. If you want to perform docker builds within CI using this runner, note you'll need to configure your runner manager (again, what your docker file is creating) to allow these same two settings on the build containers. You can get information about how to do that here: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-socket-binding
Installed a brand new Gitlab CE 13.9.1 on a Ubuntu Server 20.04.2.0.
This is the pipeline
image: node:latest
before_script:
- apt-get update -qq
stages:
- install
install:
stage: install
script:
- npm install --verbose
To run it I configure my Gitlab Runner using the same procedure as in my previous Gitlab CE 12:
I pull last Gitlab runner image:
docker pull gitlab/gitlab-runner:latest
First try:
Start GitLab Runner container mounting on local volume
docker run -d \
--name gitlab-runner \
--restart always \
-v /srv/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest
And register runner
docker run --rm -t -i \
-v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register
When registering runner, for executor I pick shell
Finally, when I push to Gitlab, on the pipeline, I see this error:
$ apt-get update -qq
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
ERROR: Job failed: exit status 1
Second try:
Start GitLab Runner container mounting on Docker volume
Create volume
docker volume create gitlab-runner-config
Start GitLab Runner container
docker run -d \
--name gitlab-runner \
--restart always \
-v gitlab-runner-config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest
Register runner (picking shell again as executor)
docker run \
--rm -t -i \
-v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner register
Same results.
$ apt-get update -qq
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
ERROR: Job failed: exit status 1
Third try:
Granting permissions to gitlab-runner
I ended up reading In gitlab CI the gitlab runner choose wrong executor and https://docs.gitlab.com/runner/executors/shell.html#running-as-unprivileged-user, which states these solutions:
move to docker
grant user gitlab-runner the permissions he needs to run specified commands. gitlab-runner may run apt-get without sudo, also he will need perms for npm install and npm run.
grant sudo nopasswd to user gitlab-runner. Add gitlab-runner ALL=(ALL) NOPASSWD: ALL (or similar) to /etc/sudoers on the machine gitlab-runner is installed and change the lines apt-get update to sudo apt-get update, which will execute them as privileged user (root).
I need to use shell
I already did that with sudo usermod -aG docker gitlab-runner
Tried as well with sudo nano /etc/sudoers, adding gitlab-runner ALL=(ALL) NOPASSWD: ALL, and using sudo apt-get update -qq in the pipeline, which results in bash: line 106: sudo: command not found
I'm pretty lost here now. Any idea will be welcome.
IMHO, using shell executor on a Docker runner with already mounted Docker socket on it is not a good idea. You'd better use docker executor, which will take care of everything and probably is how it's supposed to be run.
Edit
Alternatively, you can use a customized Docker image to allow using the shell executor with root permissions. First, you'll need to create a Dockerfile:
FROM gitlab/gitlab-runner:latest
# Change user to root
USER root
Then, you'll have to build the image (here, I tagged it as custom-gitlab-runner):
$ docker build -t custom-gitlab-runner .
Finally, you'll need to use this image:
docker run -d \
--name gitlab-runner \
--restart always \
-v /srv/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
custom-gitlab-runner:latest
I had a similar issue trying to use locally installed gitlab-runner on ubuntu with a shell executor (I had other issues using docker executor not being able to communicate between stages)
$ docker build -t myapp .
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=myapp&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied
ERROR: Job failed: exit status 1
I then printed what user was running the docker command within the gitlab-ci.yml file, which was gitlab-runner
...
build:
script:
- echo $USER
- docker build -t myapp .
...
I then added gitlab-runner to the docker group using
sudo usermod -aG docker gitlab-runner
which fixed my issue. No more docker permission errors.
After setting up gitlab-runner as a Docker container with an executor of docker, I fail to run any builds. The displayed log reads like the following:
Running with gitlab-runner 11.4.2 (cf91d5e1)
on <hostname> 9f1c1a0d
Using Docker executor with image docker:stable-git ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:acfec978837639b4230111b35a775a67ccbc2b08b442c1ae2cca4e95c3e6d08a for docker:stable-dind ...
Waiting for services to be up and running...
Pulling docker image docker:stable-git ...
Using docker image sha256:a8a2d0da40bc37344c35ab723d4081a5ef6122d466bf0a0409f742ffc09c43b9 for docker:stable-git ...
Running on runner-9f1c1a0d-project-1-concurrent-0 via a7b6a57c58f8...
Fetching changes...
HEAD is now at 5430a3d <Commit message>
Checking out 5430a3d8 as master...
Skipping Git submodules setup
$ # Auto DevOps variables and functions # collapsed multi-line command
$ setup_docker
$ build
Logging to GitLab Container Registry with CI credentials...
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password
ERROR: Job failed: exit code 1
Note the attempt to ligin to docker-hub (I guess) and the credentials-error. But I do not desire nor configured a username/password to access docker-hub. Any suggestion what is wrong here or how to go on debugging this?
The runner was registered with the following command (which also dictates the contents of the configuration file):
docker run --rm -ti \
-v <config-volume>:/etc/gitlab-runner \
-v $(pwd)/self-signed-server.crt:/etc/ssl/certs/server.crt \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner register \
--tls-ca-file /etc/ssl/certs/server.crt \
--url https://my.server.url/gitlab/ --registration-token <token> \
--name myserver --tag-list "" \
--executor docker --docker-privileged --docker-image debian \
--non-interactive
I used --docker-privileged because I originally had the same problem discussed here (thanks, wendellmva). I just can't configure running the gitlab-runner container itself privileged, but don't see link-failure-problem problem even though I don't.
To get past this point, one needs to overwrite the CI_REGISTRY_USER variable in the projects Settings -> CI / CD -> Variables block. Assigning an empty value will get past this point.
Background: by exporting the project and then parsing the JSON settings with jq, one can get the preconfigured list of commands that run:
jq -r .pipelines[0].stages[0].statuses[0].commands project.json
# ...
function registry_login() {
if [[ -n "$CI_REGISTRY_USER" ]]; then
echo "Logging to GitLab Container Registry with CI credentials..."
docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
echo ""
fi
}
# ...
So there is apparently some non-empty string preloaded to CI_REGISTRY_USER, but with an invalid CI_REGISTRY_PASSWORD.
What I haven't found yet is where to make such settings globally for all projects or how to edit the AutoDevOps pipeline.
thirst of all thanks for spending time reading this..
I am trying to achieve:
installing Puppet on all my instances (Master, agent1, agent2, etc) DONE
from puppet master installing puppetlabs/docker now I got docker on all my instances.. DONE
I put all my instances in docker SWARM-manager MODE! DONE
on Master installing Jenkins docker service create --name jenkins-master -p 50000:50000 -p 80:8080 jenkins and in Jenkins installing self-organizing swarm plugin. DONE
creating docker secret for all instances echo "-master http://35.23... -password admin -username admin" | docker secret create jenkins-v1 - DONE
When trying to create a jenkins node.. FAIL nothing happens
docker service create \
--mode=global \
--name jenkins-swarm-agent \
-e LABELS=docker-test \
--mount
"type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
--mount "type=bind,source=/tmp/,target=/tmp/" \
--secret source=jenkins-v1,target=jenkins \
vipconsult/jenkins-swarm-agent
I read before.. puppet module doesn't work with docker SWARM mode..
Do you know any alternative ways to use.. Puppet>Docker>SWARM>Jenkins>slave-nodes/
please advice!
Done!
echo "-master http://35.23... -password admin -username admin" | docker secret create jenkins-v1 -
pssd and user should be exactly like in jenkins user log in !
I'm in the process of deploying a gitlab-runner to kubernetes on google-cloud-engine so can quickly scale runners / send the configuration off to other people so they can run their own runners. But first, I wanted to try to see if I could get the runner hooked up locally on my laptop.
Setup
I have a project that on gitlab that is public, but I don't think it matters, as the goal here is just to be able to run docker-based tests on the gitlab runner.
With docker, I've proved to myself that it's possible via: https://github.com/NullVoxPopuli/vsts-agent-with-aws-ecr (a project I did for work)
Here is what I have so far for the gitlab runner:
#/bin/bash
docker stop gitlab-runner && docker rm gitlab-runner
docker run -d --name gitlab-runner --restart always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /usr/local/gitlab-ci-runner/config:/etc/gitlab-runner \
-v "$(pwd)/config.toml":/etc/gitlab-runner/config.toml:ro \
gitlab/gitlab-runner:alpine
docker exec -it gitlab-runner gitlab-runner register \
-n \
--url https://gitlab.com/ci \
--tag-list "docker,docker-compose" \
--run-untagged \
--registration-token my-runner-token \
--executor docker \
--description "Docker Runner" \
--docker-image "docker:dind" \
--docker-volumes /var/run/docker.sock:/var/run/docker.sock \
--docker-privileged
and here is my config.toml:
concurrent = 4
[[runners]]
name = "precognition-gitlab-runner"
url = "https://gitlab.example.com/ci"
token = "my-runner-token"
executor = "docker"
run_untagged = true
[runners.docker]
tls_verify = false
image = "alpine"
privileged = true
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
[runners.cache]
Insecure = false
Now, when I run the script that run the gitlab-runner and then registers it, I do see it in gitlab in https://gitlab.com/project_name/settings/ci_cd
But, it says that the runner has not connected yet:
What's the correct way to configure this?
Here is my .gitlab-ci.yml for anyone curious: https://gitlab.com/precognition-llc/aeonvera-ui/blob/registration-rework/.gitlab-ci.yml
it just runs other scripts, which in-turn run docker-compose which then runs the tests.
I had the same problem as you before.
It seems that the docker container quit after he finished registering the runner.
And for some reason I got the "container already exists" on the follow up command even though the docker run -rm flag was set. Maybe the tutorial has an error or we misuderstand some of the steps there.
The way I solved it was to remove the container first with
docker stop gitlab-runner
docker rm gitlab-runner
And then start the already registerd container with:
docker run -d --name gitlab-runner --restart always -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
I hope it helps.