Operation not permitted on gitlab-runner - docker

I'm trying to run a gitlab ci on my own server. I registered gitlab-runner in a separated machine using privileges
sudo gitlab-runner -n \
--url https://git.myServer.com/ \
--registration-token TOKEN \
--executor docker \
--description "Docker runner" \
--docker-image "myImage:version" \
--docker-privileged
Then I created a simple .gitlab-ci.yml configuration
stages:
- build
default:
image: myImage:version
build-os:
stage: build
script: ./build
My build script builds some cpp files and triggers some cmake files. However, one of those cmake files fails when trying to execute configure_file command
CMake Error at CMakeLists.txt:80 (configure_file):
Operation not permitted
I think it's a problem of privileges of my gitlab-runner but I registered it with sudo privileges.
Any idea of what I'm missing? thank you!
edit:
Here's my config.toml file
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Description"
url = "https://git.myServer.com/"
token = "TOKEN"
executor = "docker"
environment = [
"DOCKER_AUTH_CONFIG={config}",
"GIT_STRATEGY=clone",
]
clone_url = "https://git.myServer.com"
builds_dir = "/home/gitlab-runner/build"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "myImage:version"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = [
"/tmp/.X11-unix:/tmp/.X11-unix",
"/dev:/dev",
"/run/user/1000/gdm/Xauthority:/home/gitlab-runner/.Xauthority",
]
memory = "8g"
memory_swap = "8g"
ulimit = ["core=0", "memlock=-1", "rtprio=99"]
shm_size = 0
pull_policy = ["if-not-present"]
network_mode = "host"
I have also tried changing the user from gitlab-runner to my host user following this but it didn't work.
This is the line which makes my build fail.

I entered the container from the runner machine while the ci was running and I noticed that the repository was cloned as root but the build directories were created under a user. The cofigure_file command is trying to modify a file from the repository, so it's like user is trying to modify a file created as root (when cloned). I didn't manage to make the gitlab-runner software clone the repository as user. Instead, my workaround was to change the permissions of the folder before building. My .gitlab-ci.yml looks like this now
stages:
- build
default:
image: myImage:version
build-os:
stage: build
script:
- cd ../ && sudo chown -R user:sudo my-repo/ && cd my-repo/
- ./build

Related

Gitlab runner docker compose: not found

I have registered gitlab-runner with the following command:
gitlab-runner register --non-interactive \
--url ${URL} \
--registration-token ${REGISTRATION_TOKEN} \
--description ${RUNNER_NAME} \
--tag-list ${TAGS} \
--executor "docker" \
--docker-image="docker:stable" \
--docker-pull-policy if-not-present \
--locked=false \
--docker-privileged=true \
--docker-volumes=["/var/run/docker.sock:/var/run/docker.sock", "/cache"] \
There is my runner config:
[[runners]]
name = "XXX"
url = "XXX"
id = 19981753
token = "XXX"
token_obtained_at = 2022-12-24T11:43:10Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
pull_policy = ["if-not-present"]
shm_size = 0
Then I am trying to run ci with docker-compose, but there is an error: docker-compose: not found:
There is a part from my .gitlab-ci.yml:
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
stages:
- build
- staging
- release
- deploy
build:
stage: build
script:
- echo "IMAGE_APP_TAG=$STAGE_IMAGE_APP_TAG" >> .env
- docker-compose build
- docker-compose push
only:
- dev
- main
Which image for docker executor should i use to run docker-compose? Should i change .gitlac-ci.yml or gitlab-runner config.toml?
In the buid job, just add the docker compose image
build:
image: docker/compose
Or you can use any other image and install the docker compose by yourself.

Why docker can't connect to host while running Gitlab runner?

I'm getting this error output from gitlab runner I have installed locally:
error during connect: Get "http://docker:2375/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3Dproject-0%22%3Atrue%7D%7D&limit=0": dial tcp: lookup docker on 192.168.1.254:53: no such host
Test failed: users users-lint client
ERROR: Job failed: exit code 1
FATAL: exit code 1
my config.toml file:
concurrent = 1
check_interval = 0
[[runners]]
name = "tdd"
url = "https://gitlab.com/"
token = "redacted_XXXXXXXX"
executor = "docker"
builds_dir = "~/tdd"
[runners.docker]
tls_verify = false
disable_entrypoint_overwrite = false
image = "docker:stable"
privileged = true
oom_kill_disable = false
disable_cache = false
cache_dir = ""
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
taking a look at .gitlab-ci.yml file may help:
image: docker:stable
# services:
# - docker:dind
variables:
DOCKER_DRIVER: overlay
DOCKER_HOST: tcp://docker:2375
POSTGRES_DB: users_dev
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_HOST: postgres
stages:
- build
before_script:
- apk update
- apk add --no-cache --update python3-dev py3-pip curl
- curl -L https://github.com/docker/compose/releases/download/v2.3.3/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- mv docker-compose /usr/local/bin
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
- export SECRET_KEY=hakuna000matata
build:
stage: build
script:
- sh test.sh
I commented out services because it works OK for CI environment but threads on the web suggests that it's better to keep it off when running the build locally.
Any ideas what may help ? I'm on Ubuntu.

Why local Gitlab runner isn't detecting running Docker instance?

I've just installed Gitlab-runner locally on my Ubuntu machine so I can debug my pipeline without using shared runners.
I'm getting this error output:
$ docker-compose up -d --build
Couldn't connect to Docker daemon at http://docker:2375 - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
ERROR: Failed to cleanup volumes
ERROR: Job failed: exit code 1
FATAL: exit code 1
when I run docker --version I get:
Docker version 20.10.12, build e91ed57
and when I run sudo systemctl status docker I get:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-01-01 20:26:25 GMT; 37min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 1404 (dockerd)
Tasks: 20
Memory: 112.0M
CGroup: /system.slice/docker.service
└─1404 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
so it is installed and running hence the error output is confusing.
Here's my pipeline:
image: docker:stable
services:
- docker:dind
stages:
- build
- test
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm
- cache/Cypress
- node_modules
before_script:
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
job:
stage: build
script:
- apk add --update --no-cache gcc g++ make python2 python2-dev py-pip python3-dev docker-compose
- docker-compose up -d --build
- docker logs testdriven_e2e:latest -f
after_script:
- docker-compose down
I start the running executing gitlab-runner exec docker --docker-privileged job
Any suggestion as to why the runner is complaining about Docker not running ?
update: based on opinions from this thread https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1986
image: docker:stable
variables:
DOCKER_HOST: tcp://localhost:2375
DOCKER_DRIVER: overlay2
services:
- docker:dind
stages:
- build
- test
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm
- cache/Cypress
- node_modules
before_script:
- docker info
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
job:
stage: build
script:
- apk add --update --no-cache gcc g++ make python2 python2-dev py-pip python3-dev curl
- curl -L "https://github.com/docker/compose/releases/download/v2.2.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- chmod +x /usr/local/bin/docker-compose
- docker-compose up -d --build
- docker logs testdriven_e2e:latest -f
after_script:
- docker-compose down
config.toml:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "testdriven"
url = "https://gitlab.com/"
token = "yU2yn4eUmFJ-xr3HzzmE"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
insecure = false
[runners.docker]
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
cache_dir = "cache"
tls_verify = false
image = "docker:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
shm_size = 0
error output:
$ docker info
Client:
Debug Mode: false
Server:
ERROR: Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
errors pretty printing info
ERROR: Failed to cleanup volumes
ERROR: Job failed: exit code 1
FATAL: exit code 1
Strangely, what worked for me was pinning down the dind version like this:
services:
- docker:18.09-dind
Which port is being used by docker in your system? It seems that it's running in a non-default port. Try adding this to your .gitlab-ci.yaml file, but change the 2375 port.
variables:
DOCKER_HOST: "tcp://docker:2375"

Access GitLab CI docker container for testing purpose

I am currently trying to deploy an application using GitLab CI and docker.
Spoiler : My pipeline fails everytime so I went a bit into it but I do not understand everything that's happening.
My gitlab runner is a Google Cloud Platform instance. This instance runs the GitLabCI file that call docker builds (so far so good ?). One of the docker build fails for a python/gensim version incompatibility problem.
What I don't understand :
This is my GitLab CI :
image: "ubuntu:bionic"
default:
before_script:
- apt-get update -y
- apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
- apt-key fingerprint *KEY*
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- apt-get update -y
- apt-get install -y docker-ce docker-ce-cli containerd.io
api:
script:
- "pwd"
- "cp scraping/src/variables.env /srv/variables.env"
- "cp scraping/src/crontab /etc/crontab"
- "cat /etc/crontab"
- "docker build -t app-api api/"
- "docker rm -f app-api || :"
- "docker run --restart always -d --env-file=/srv/variables.env --name app-api app-api"
- "docker ps"
As you can see, the runner starts by printing his current working directory and copying some files.
Problem is : When I access to the GCP instance and go into the runner docker container (with docker exec -ti docker-ID /bin/bash), the current working directory isn't the same
builds/username/myapp is printed in pipeline but / is printed whenever I connect with SSH. And I can't manage to find the "builds" directory or any of my app directories/files.
My assumption is that when called, the gitlab runner runs a docker container with my project copied inside. And this container builds the other containers.
Am I right ? If yes, how could I access to this container for testing purpose ?
As you can see, I'm a bit lost about what's happening between GitLabCI runner and dockers container. If someone could explain to me who's calling what from where, that'd be great !
EDIT :
config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gcp"
url = "https://gitlab.com/"
token = *TOKEN*
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "ubuntu:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock","/srv:/srv","/var/spool/cron/crontabs/:/crons"]
shm_size = 0

Using SSH in gitlab job to restart docker-container

I am trying to add some continuous deployment for a typescript API built with node, and mongodb.
I would like to do so via the gitlab instance that I already have :
Runner config (/etc/gitlab-runner/config.toml) :
[[runners]]
name = "runner"
url = "https://git.[DOMAIN].[EXT]"
token = "[ID]"
executor = "docker"
[runners.docker]
tls_verify = false
image = "mhart/alpine-node:6.5"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
So my deploy job looks as follow :
Deployment_preprod:
stage: Deploy
before_script:
# https://docs.gitlab.com/ee/ci/ssh_keys/
- 'which ssh-agent || ( apk add --no-cache --virtual openssh-client )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- chmod 700 ~/.ssh
script:
- scp -r dist user#[IP]:/home/[user]/preprod-back
- ssh -tt user#[IP] cd /home/[user]/preprod-back && yarn run doc && docker-compose restart
environment:
name: preprod
url: https://preprod.api.[DOMAIN].[EXT]
only:
- develop
Question :
this job fail on /bin/sh: eval: line 91: docker-compose: not found which suprise me since running docker-compose [whatever] just works fine server-side when I log in the server via ssh.
The && are tripping you up. You should quote the entire remote command.
script:
- scp -r dist user#[IP]:/home/[user]/preprod-back
- ssh -tt user#[IP] "cd /home/[user]/preprod-back && yarn run doc && docker-compose restart"

Resources