After setting up gitlab-runner as a Docker container with an executor of docker, I fail to run any builds. The displayed log reads like the following:
Running with gitlab-runner 11.4.2 (cf91d5e1)
on <hostname> 9f1c1a0d
Using Docker executor with image docker:stable-git ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:acfec978837639b4230111b35a775a67ccbc2b08b442c1ae2cca4e95c3e6d08a for docker:stable-dind ...
Waiting for services to be up and running...
Pulling docker image docker:stable-git ...
Using docker image sha256:a8a2d0da40bc37344c35ab723d4081a5ef6122d466bf0a0409f742ffc09c43b9 for docker:stable-git ...
Running on runner-9f1c1a0d-project-1-concurrent-0 via a7b6a57c58f8...
Fetching changes...
HEAD is now at 5430a3d <Commit message>
Checking out 5430a3d8 as master...
Skipping Git submodules setup
$ # Auto DevOps variables and functions # collapsed multi-line command
$ setup_docker
$ build
Logging to GitLab Container Registry with CI credentials...
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password
ERROR: Job failed: exit code 1
Note the attempt to ligin to docker-hub (I guess) and the credentials-error. But I do not desire nor configured a username/password to access docker-hub. Any suggestion what is wrong here or how to go on debugging this?
The runner was registered with the following command (which also dictates the contents of the configuration file):
docker run --rm -ti \
-v <config-volume>:/etc/gitlab-runner \
-v $(pwd)/self-signed-server.crt:/etc/ssl/certs/server.crt \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner register \
--tls-ca-file /etc/ssl/certs/server.crt \
--url https://my.server.url/gitlab/ --registration-token <token> \
--name myserver --tag-list "" \
--executor docker --docker-privileged --docker-image debian \
--non-interactive
I used --docker-privileged because I originally had the same problem discussed here (thanks, wendellmva). I just can't configure running the gitlab-runner container itself privileged, but don't see link-failure-problem problem even though I don't.
To get past this point, one needs to overwrite the CI_REGISTRY_USER variable in the projects Settings -> CI / CD -> Variables block. Assigning an empty value will get past this point.
Background: by exporting the project and then parsing the JSON settings with jq, one can get the preconfigured list of commands that run:
jq -r .pipelines[0].stages[0].statuses[0].commands project.json
# ...
function registry_login() {
if [[ -n "$CI_REGISTRY_USER" ]]; then
echo "Logging to GitLab Container Registry with CI credentials..."
docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
echo ""
fi
}
# ...
So there is apparently some non-empty string preloaded to CI_REGISTRY_USER, but with an invalid CI_REGISTRY_PASSWORD.
What I haven't found yet is where to make such settings globally for all projects or how to edit the AutoDevOps pipeline.
Related
I'm trying to write-down a Dockerfile to create create and register a new runner to a private gitlab repository. According to gitlab documentation, I wrote down the following Dockerfile:
FROM gitlab/gitlab-runner:latest
RUN gitlab-runner register \
--non-interactive \
--url "https://gitlab.com/" \
--registration-token "GITLAB_REPO_TOKEN" \
--executor "docker" \
--docker-image alpine:latest \
--description "docker-runner" \
--maintenance-note "Free-form maintainer notes about this runner" \
--run-untagged="true" \
--locked="false"
Then build it with:
docker build -t test .
And then run it in a container via:
docker run test:latest
The runner is correctly seen by gitlab (the runner is available under Settings\CI/CD\Runners).
Then, I set up the following CI, for testing:
image: python:3.7-alpine
testci:
stage: test
script:
- python test.py
The job is then pulled by the runner, but I immediately get the following error:
Running with gitlab-runner 15.8.2 (4d1ca121)
on docker-runner yVa1JDny, system ID: xxxxxxxxx
Preparing the "docker" executor
00:09
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:753:0s)
Can anyone please provide support in that? I didn't get what it is missing from the configuration I've made.
I've tried to modify the docker run call trying with the volume mount guide found here, but nothing changes.
I've also found here a similar Dockerfile, but using a gitlab-ci-multi-runner which is not the desired service.
You're attempting to use the docker executor for your runner, but your runner doesn't have access to the docker socket in order to create new containers. Your runner manager (what your docker file is creating) is attempting to start up new docker containers to handle each of your jobs, but failing to connect to docker.
In your docker run command, you will need to do a couple things:
Set your container to use privileged mode: --privileged
Map in the docker socket: -v /var/run/docker.sock:/var/run/docker.sock
With those two things, you can likely connect to the docker daemon and start new containers. If you want to perform docker builds within CI using this runner, note you'll need to configure your runner manager (again, what your docker file is creating) to allow these same two settings on the build containers. You can get information about how to do that here: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-socket-binding
In Gitlab CI, when I want to push my image, I get:
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
Here is the code:
docker-push:
stage: publish
image: docker:stable
services:
- docker:dind
before_script:
- *inject-gopath
- docker login -u gitlab-ci-token -p $GITLAB_PERSONAL_ACCESS_TOKEN $REGISTRY_URL
# $CI_BUILD_TOKEN
script:
- docker build --build-arg GITLAB_TOKEN=${GITLAB_PERSONAL_ACCESS_TOKEN} --target=prod -t $REGISTRY_PACKAGE_API_NAME:$CI_BUILD_ID .
- docker build --build-arg GITLAB_TOKEN=${GITLAB_PERSONAL_ACCESS_TOKEN} --target=prod -t $REGISTRY_PACKAGE_API_NAME:latest .
- docker push $REGISTRY_PACKAGE_API_NAME:$CI_BUILD_ID
- docker push $REGISTRY_PACKAGE_API_NAME:latest
Here is the output:
Pulling docker image docker:stable ...
Using docker image sha256:23fb2c9b38b59433ea1913eafa12d2e15651ca0d08819dc7067d27d8f92e0428 for docker:stable ...
Running on runner-wmKFtEwx-project-7124308-concurrent-0 via ubuntu...
Fetching changes...
Removing release/
HEAD is now at 9c4894a Merge branch '5-supprimer-les-threads' into 'master'
Checking out 9c4894af as master...
Skipping Git submodules setup
Downloading artifacts for build (324707453)...
Downloading artifacts from coordinator... ok id=324707453 responseStatus=200 OK token=SVLY__Jy
$ mkdir -p $(dirname ${PACKAGE_PATH}) && ln -s ${CI_PROJECT_DIR} ${PACKAGE_PATH} && cd ${PACKAGE_PATH}
$ docker login -u gitlab-ci-token -p $GITLAB_PERSONAL_ACCESS_TOKEN $REGISTRY_URL
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build --build-arg GITLAB_TOKEN=${GITLAB_PERSONAL_ACCESS_TOKEN} --target=prod -t $REGISTRY_PACKAGE_API_NAME:$CI_BUILD_ID .
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
ERROR: Job failed: exit code 1
Weird thing is that I am in the image docker-stable, so I should have docker available, and previous step docker login is working well...
Is it a bug from Gitlab, or something I am doing wrong ?
PD: I am using Gitlab.com
PD2: I can push the image manually without any issue
As per comment it is just a GitLab problem.
There is no guarantee, with a free runner, that it will always work. In this case the docker deamon is not available so your build will keep failing until the runner is restarted when GitLab detects the error.
Notice this is my assumption given that often the error comes and then after a few hours it is gone. As you can see also my Gitlab Forums Thread received no reply from anyone so I am afraid I dont think you will be able to fix this. If mission critical then run your own runner so you can restart it on failure
I want to run sentry-cli inside my docker image like this:
sentry-frontend:
stage: sentry
services:
- docker:18-dind
before_script:
- docker login -u gitlab-ci-token -p "$CI_JOB_TOKEN" registry.xxx.xx
script:
- export SENTRY_AUTH_TOKEN=xxxxxxxxxxxxxxxxxx
- export IMAGE=$CI_REGISTRY_IMAGE/frontend-builder:$CI_COMMIT_REF_NAME
- export RELEASE_VERSION=$CI_COMMIT_REF_NAME
- docker pull getsentry/sentry-cli
- docker run --rm -v $(pwd):/work getsentry/sentry-cli releases -o org -p frontend new $RELEASE_VERSION
tags:
- dind
However the job fails because
error: API request failed
caused by: sentry reported an error: Authentication credentials were not provided. (http status: 401)
I tried:
- docker run --rm -v $(pwd):/work getsentry/sentry-cli --auth-token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
however after that I get the same message as I would if I ran
docker run --rm -v $(pwd):/work sentry-cli --help
and after that it fails as if the command was not correct.I can't seem to find any information on how to do that correctly either. How to provide credentials inside that image?
If you want to pass the SENTRY_AUTH_TOKEN environment variable to the container, you can adapt your docker run command like this:
docker run --rm -v "$PWD:/work" -e SENTRY_AUTH_TOKEN="$SENTRY_AUTH_TOKEN" getsentry/sentry-cli releases -o org -p frontend new $RELEASE_VERSION
or more concisely:
docker run --rm -v "$PWD:/work" -e SENTRY_AUTH_TOKEN getsentry/sentry-cli releases -o org -p frontend new $RELEASE_VERSION
(but note that the latter version won't work if docker is an alias of sudo docker)
The relevant documentation page is:
docs.docker.com/engine/reference/commandline/run/
As an aside, note that -v "$PWD:/work" is more efficient than -v "$(pwd):/work" (one less process to spin).
I'm trying to create a custom gitlab-runner to run a docker process, following:
https://github.com/gitlabhq/gitlabhq/blob/master/doc/ci/docker/using_docker_build.md
I tried the second approach in which I registered a runner using:
sudo gitlab-runner register -n \ --url https://gitlab.com/ \
--registration-token xxx \ --executor docker \ --description "My Docker Runner" \ --docker-image "docker:stable" \ --docker-volumes
/var/run/docker.sock:/var/run/docker.sock
However,at gitlab, whenever the pipeline starts I'm facing the following error:
ERROR: Failed to create container volume for /builds/xxx Unable to
load image: gitlab-runner-prebuilt: "open
/var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or
directory"
I can't find much information online, any help appreciated.
For The record, I got it working following this tutorial
https://angristan.xyz/build-push-docker-images-gitlab-ci/
Since the docker image worked, I suspect there's something wrong with the debian gitlab-runner distribution
I'm having an issue with google container registry and dockerhub where docker pull returns the following errors.
gcr
Error: Status 403 trying to pull repository PROJECT_ID/IMAGE_NAME: "Unable to access the repository: PROJECT_ID/IMAGE_NAME; please verify that it exists and you have permission to access it (no valid credential was supplied)."
dockerhub
Using default tag: latest test-node0: Pulling
k8tems/hello-world:latest... : Error: image k8tems/hello-world not
found Error: image k8tems/hello-world not found
This only happens when docker is pointing to a swarm.
Steps to reproduce:
DOCKER_REGISTRY=asia.gcr.io/$PROJECT_ID
KEY_STORE=test-keystore
NODE_BASE=test-node
echo pushing hello-world image to gcr
docker pull hello-world
docker tag hello-world $DOCKER_REGISTRY/hello-world
docker push $DOCKER_REGISTRY/hello-world
echo setting up key store
docker-machine create \
-d digitalocean \
"$KEY_STORE"
docker $(docker-machine config "$KEY_STORE") run -d \
-p "8500:8500" \
-h "consul" \
progrium/consul -server -bootstrap
eval $(docker-machine env "$KEY_STORE")
docker-machine create \
-d digitalocean \
--swarm \
--swarm-master \
--swarm-discovery="consul://$(docker-machine ip "$KEY_STORE"):8500" \
--engine-opt="cluster-store=consul://$(docker-machine ip "$KEY_STORE"):8500" \
--engine-opt="cluster-advertise=eth0:2376" \
"$NODE_BASE"0
echo this fails
eval $(docker-machine env -swarm "$NODE_BASE"0)
docker pull $DOCKER_REGISTRY/hello-world
echo this succeeds
eval $(docker-machine env "$NODE_BASE"0)
docker pull $DOCKER_REGISTRY/hello-world
Along with the above snippet, I've also tried forcing the remote docker version to 1.10.3 and swarm to 1.1.3 but the error still persists.
ubuntu:~$ docker-machine ls | grep test
test-keystore - digitalocean Running tcp://:2376 v1.10.3
test-node0 * digitalocean Running tcp://:2376 test-node0 (master) v1.10.3
ubuntu:~$ docker exec swarm-agent-master /swarm -v
swarm version 1.1.3 (7e9c6bd)
ubuntu:~$ docker -v
Docker version 1.10.2, build c3959b1
Is there anything I can do to make this work with the -swarm flag or do I have to run the pull command for each node?
Try JSON key file! It is a long-lived credential and much more consistent than access token when you are using clusters like swarm or kubernetes.
Example command:
docker login -e 1234#5678.com -u _json_key -p "$(cat keyfile.json)" https://gcr.io
Here is the page with more details:
https://cloud.google.com/container-registry/docs/advanced-authentication#using_a_json_key_file