I am currently trying to deploy an application using GitLab CI and docker.
Spoiler : My pipeline fails everytime so I went a bit into it but I do not understand everything that's happening.
My gitlab runner is a Google Cloud Platform instance. This instance runs the GitLabCI file that call docker builds (so far so good ?). One of the docker build fails for a python/gensim version incompatibility problem.
What I don't understand :
This is my GitLab CI :
image: "ubuntu:bionic"
default:
before_script:
- apt-get update -y
- apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
- apt-key fingerprint *KEY*
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- apt-get update -y
- apt-get install -y docker-ce docker-ce-cli containerd.io
api:
script:
- "pwd"
- "cp scraping/src/variables.env /srv/variables.env"
- "cp scraping/src/crontab /etc/crontab"
- "cat /etc/crontab"
- "docker build -t app-api api/"
- "docker rm -f app-api || :"
- "docker run --restart always -d --env-file=/srv/variables.env --name app-api app-api"
- "docker ps"
As you can see, the runner starts by printing his current working directory and copying some files.
Problem is : When I access to the GCP instance and go into the runner docker container (with docker exec -ti docker-ID /bin/bash), the current working directory isn't the same
builds/username/myapp is printed in pipeline but / is printed whenever I connect with SSH. And I can't manage to find the "builds" directory or any of my app directories/files.
My assumption is that when called, the gitlab runner runs a docker container with my project copied inside. And this container builds the other containers.
Am I right ? If yes, how could I access to this container for testing purpose ?
As you can see, I'm a bit lost about what's happening between GitLabCI runner and dockers container. If someone could explain to me who's calling what from where, that'd be great !
EDIT :
config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gcp"
url = "https://gitlab.com/"
token = *TOKEN*
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
tls_verify = false
image = "ubuntu:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock","/srv:/srv","/var/spool/cron/crontabs/:/crons"]
shm_size = 0
Related
I'm trying to run a gitlab ci on my own server. I registered gitlab-runner in a separated machine using privileges
sudo gitlab-runner -n \
--url https://git.myServer.com/ \
--registration-token TOKEN \
--executor docker \
--description "Docker runner" \
--docker-image "myImage:version" \
--docker-privileged
Then I created a simple .gitlab-ci.yml configuration
stages:
- build
default:
image: myImage:version
build-os:
stage: build
script: ./build
My build script builds some cpp files and triggers some cmake files. However, one of those cmake files fails when trying to execute configure_file command
CMake Error at CMakeLists.txt:80 (configure_file):
Operation not permitted
I think it's a problem of privileges of my gitlab-runner but I registered it with sudo privileges.
Any idea of what I'm missing? thank you!
edit:
Here's my config.toml file
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Description"
url = "https://git.myServer.com/"
token = "TOKEN"
executor = "docker"
environment = [
"DOCKER_AUTH_CONFIG={config}",
"GIT_STRATEGY=clone",
]
clone_url = "https://git.myServer.com"
builds_dir = "/home/gitlab-runner/build"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "myImage:version"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = [
"/tmp/.X11-unix:/tmp/.X11-unix",
"/dev:/dev",
"/run/user/1000/gdm/Xauthority:/home/gitlab-runner/.Xauthority",
]
memory = "8g"
memory_swap = "8g"
ulimit = ["core=0", "memlock=-1", "rtprio=99"]
shm_size = 0
pull_policy = ["if-not-present"]
network_mode = "host"
I have also tried changing the user from gitlab-runner to my host user following this but it didn't work.
This is the line which makes my build fail.
I entered the container from the runner machine while the ci was running and I noticed that the repository was cloned as root but the build directories were created under a user. The cofigure_file command is trying to modify a file from the repository, so it's like user is trying to modify a file created as root (when cloned). I didn't manage to make the gitlab-runner software clone the repository as user. Instead, my workaround was to change the permissions of the folder before building. My .gitlab-ci.yml looks like this now
stages:
- build
default:
image: myImage:version
build-os:
stage: build
script:
- cd ../ && sudo chown -R user:sudo my-repo/ && cd my-repo/
- ./build
For personal use (and fun) I'm trying to setup a VM on which I want to host my website (Nginx, Django and Postgres running in docker containers). I'm trying to learn how to setup the server using Terraform and Cloud init in a safe manner.
My current cloud-init code:
#cloud-config
groups:
- docker
users:
- default
# the docker service account
- name: test
shell: /bin/bash
home: /home/test
groups: docker
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_import_id: None
lock_passwd: true
ssh-authorized-keys:
- ssh-rsa my_public_ssh_key
package_update: true
package_upgrade: true
packages:
- git
- sudo
- apt-transport-https
- ca-certificates
- curl
- gnupg-agent
- software-properties-common
runcmd:
# install docker following the guide: https://docs.docker.com/install/linux/docker-ce/ubuntu/
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- sudo apt-get -y update
- sudo apt-get -y install docker-ce docker-ce-cli containerd.io
- sudo systemctl enable docker
# install docker-compose following the guide: https://docs.docker.com/compose/install/
- sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- sudo chmod +x /usr/local/bin/docker-compose
power_state:
mode: reboot
message: Restarting after installing docker & docker-compose
The VM is Ubuntu 20.04
Technically I want the "test" user to be able to pull the latest code from my git repo and (re-)deploy the website (in /home/test/website) using docker-compose. Is it possible that the user does not have sudo permissions (I don't want to have it have elevated permissions). And secondly: how do I create a root account with a separate SSH key (and would this be a safe setup)?
The Terraform code that produces the VM.
resource "scaleway_instance_server" "app_server" {
type = var.instance_type
image = "ubuntu-focal"
name = var.instance_name
enable_ipv6 = true
tags = [ "FocalFossa", "MyUbuntuInstance" ]
root_volume {
size_in_gb = 20
delete_on_termination = true
}
lifecycle {
create_before_destroy = true
}
ip_id = scaleway_instance_ip.public_ip.id
security_group_id = scaleway_instance_security_group.www.id
# cloud init: setup
cloud_init = file("${path.module}/cloud-init.yml")
}
Help is much appreciated.
Is it possible that the user does not have sudo permissions (I don't want to have it have elevated permissions).
Anything run by cloud-init is run as root, including the bootcmd/runcmd commands. To run things as a different user, you can use sudo in your runcmd.
sudo -u test whoami >> /var/tmp/run_cmd
would write test to /var/tmp/run_cmd.
And secondly: how do I create a root account with a separate SSH key (and would this be a safe setup)?
Your users section would something look like this.
users:
- default
# the docker service account
- name: test
shell: /bin/bash
home: /home/test
groups: docker
sudo: ALL=(ALL) NOPASSWD:ALL
lock_passwd: true
ssh-authorized-keys:
- ssh-rsa my-public-key
- name: root
ssh-authorized-keys:
- ssh-rsa root-public-key
disable_root: false
Is it safe? I think that's debatable, but there's a reason root login is disabled by default. It should be possible to ssh into the default user and then sudo su for your root access needs.
Also, just FYI, the ssh_import_id: None in your config was raising an exception in the cloud-init log because it was trying to import an ssh id for user None.
I'm using AWS ECR to host a private Dockerfile image, and I would like to use it in GitLab CI.
Accordingly to the documentation I need to set docker-credential-ecr-login to fetch the private image, but I have no idea how to do that before anything else. That's my .gitlab-ci file:
image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest
tests:
stage: test
before_script:
- echo "before_script"
- apt install amazon-ecr-credential-helper
- apk add --no-cache curl jq python py-pip
- pip install awscli
script:
- echo "script"
- bundle install
- bundle exec rspec
allow_failure: true # for now as we do not have tests
Thank you.
I confirm the feature at stake is not yet available in GitLab CI; however I've recently seen it is possible to implement a generic workaround to run a dedicated CI script within a container taken from a private Docker image.
The template file .gitlab-ci.yml below is adapted from the OP's example, using the Docker-in-Docker approach I suggested in this other SO answer, itself inspired by the GitLab CI doc dealing with dind:
stages:
- test
variables:
IMAGE: "0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest"
REGION: "ap-northeast-1"
tests:
stage: test
image: docker:latest
services:
- docker:dind
variables:
# GIT_STRATEGY: none # uncomment if "git clone" is unneeded for this job
before_script:
- ': before_script'
- apt install amazon-ecr-credential-helper
- apk add --no-cache curl jq python py-pip
- pip install awscli
- $(aws ecr get-login --no-include-email --region "$REGION")
- docker pull "$IMAGE"
script:
- ': script'
- |
docker run --rm -v "$PWD:/build" -w /build "$IMAGE" /bin/bash -c "
export PS4='+ \e[33;1m($CI_JOB_NAME # line \$LINENO) \$\e[0m ' # optional
set -ex
## TODO insert your multi-line shell script here ##
echo \"One comment\" # quotes must be escaped here
: A better comment
echo $PWD # interpolated outside the container
echo \$PWD # interpolated inside the container
bundle install
bundle exec rspec
## (cont'd) ##
"
- ': done'
allow_failure: true # for now as we do not have tests
This example assumes the Docker $IMAGE contains the /bin/bash binary, and relies on the so-called block style of YAML.
The above template already contains comments, but to be self-contained:
You need to escape double quotes if your Bash commands contain them, because the whole code is surrounded by docker run … " and ";
You also need to escape local Bash variables (cf. the \$PWD above), otherwise these variables will be resolved prior running the docker run … "$IMAGE" /bin/bash -c "…" command itself.
I replaced the echo "stuff" or so commands with their more effective colon counterpart:
set -x
: stuff
: note that these three shell commands do nothing
: but printing their args thanks to the -x option.
[Feedback is welcome as I can't directly test this config (I'm not an AWS ECR user), but I'm puzzled by the fact the OP's example contained at the same time some apt and apk commands…]
Related remark on a pitfall of set -e
Beware that the following script is buggy:
set -e
command1 && command2
command3
Namely, write instead:
set -e
command1 ; command2
command3
or:
set -e
( command1 && command2 )
command3
To be convinced about this, you can try running:
bash -e -c 'false && true; echo $?; echo this should not be run'
→ 1
→ this should not be run
bash -e -c 'false; true; echo $?; echo this should not be run'
bash -e -c '( false && true ); echo $?; echo this should not be run'
From GitLab documentation. In order to interact with your AWS account, the GitLab CI/CD pipelines require both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be defined in your GitLab settings under Settings > CI/CD > Variables. Then add to your before script:
image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest
tests:
stage: test
before_script:
- echo "before_script"
- apt install amazon-ecr-credential-helper
- apk add --no-cache curl jq python py-pip
- pip install awscli
- $( aws ecr get-login --no-include-email )
script:
- echo "script"
- bundle install
- bundle exec rspec
allow_failure: true # for now as we do not have tests
Also, you had a typo is awscli, not awsclir.Then add the builds, tests and push accordingly.
I think that you have some sort of logic error in the case. image in the build configuration is a CI scripts runner image, not image you build and deploy.
I think you don't have to use it in any case since it is just an image which has utilities & connections to the GitLab CI & etc. The image shouldn't have any dependencies of your project normally.
Please check examples like this one https://gist.github.com/jlis/4bc528041b9661ae6594c63cd2ef673c to get it more clear how to do it a correct way.
I faced the same problem using docker executor mode of gitlab runner.
SSH into the EC2 instance showed that docker-credential-ecr-login was present in /usr/bin/. To pass it to the container I had to mount this package to the gitlab runner container.
gitlab-runner register -n \
--url '${gitlab_url}' \
--registration-token '${registration_token}' \
--template-config /tmp/gitlab_runner.template.toml \
--executor docker \
--tag-list '${runner_name}' \
--description 'gitlab runner for ${runner_name}' \
--docker-privileged \
--docker-image "alpine" \
--docker-disable-cache=true \
--docker-volumes "/var/run/docker.sock:/var/run/docker.sock" \
--docker-volumes "/cache" \
--docker-volumes "/usr/bin/docker-credential-ecr-login:/usr/bin/docker-credential-ecr-login" \
--docker-volumes "/home/gitlab-runner/.docker:/root/.docker"
More information on this thread as well: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583#note_375018948
We have a similar setup where we need to run CI jobs based off of an Image that is hosted on ECR.
Steps to follow:-
follow this guide here>> https://github.com/awslabs/amazon-ecr-credential-helper
gist of this above link is if you are on "Amazon Linux 2"
sudo amazon-linux-extras enable docker
sudo yum install amazon-ecr-credential-helper
open the ~/.docker/config.json on your gitlab runner in VI editor
Paste this code in the ~/.docker/config.json
{
"credHelpers":
{
"aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login"
}
}
source ~/.bashrc
systemctl restart docker
also remove any references of DOCKER_AUTH_CONFIG from your GitLab>>CI/CD>> Variables
That's it
I am trying to add some continuous deployment for a typescript API built with node, and mongodb.
I would like to do so via the gitlab instance that I already have :
Runner config (/etc/gitlab-runner/config.toml) :
[[runners]]
name = "runner"
url = "https://git.[DOMAIN].[EXT]"
token = "[ID]"
executor = "docker"
[runners.docker]
tls_verify = false
image = "mhart/alpine-node:6.5"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
So my deploy job looks as follow :
Deployment_preprod:
stage: Deploy
before_script:
# https://docs.gitlab.com/ee/ci/ssh_keys/
- 'which ssh-agent || ( apk add --no-cache --virtual openssh-client )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- chmod 700 ~/.ssh
script:
- scp -r dist user#[IP]:/home/[user]/preprod-back
- ssh -tt user#[IP] cd /home/[user]/preprod-back && yarn run doc && docker-compose restart
environment:
name: preprod
url: https://preprod.api.[DOMAIN].[EXT]
only:
- develop
Question :
this job fail on /bin/sh: eval: line 91: docker-compose: not found which suprise me since running docker-compose [whatever] just works fine server-side when I log in the server via ssh.
The && are tripping you up. You should quote the entire remote command.
script:
- scp -r dist user#[IP]:/home/[user]/preprod-back
- ssh -tt user#[IP] "cd /home/[user]/preprod-back && yarn run doc && docker-compose restart"
I have a Dockerfile that starts with installing the texlive-full package, which is huge and takes a long time. If I docker build it locally, the intermedate image created after installation is cached, and subsequent builds are fast.
However, if I push to my own GitLab install and the GitLab-CI build runner starts, this always seems to start from scratch, redownloading the FROM image, and doing the apt-get install again. This seems like a huge waste to me, so I'm trying to figure out how to get the GitLab DinD image to cache the intermediate images between builds, without luck so far.
I have tried using the --cache-dir and --docker-cache-dir for the gitlab-runner register command, to no avail.
Is this even something the gitlab-runner DinD image is supposed to be able to do?
My .gitlab-ci.yml:
build_job:
script:
- docker build --tag=example/foo .
My Dockerfile:
FROM php:5.6-fpm
MAINTAINER Roel Harbers <roel.harbers#example.com>
RUN apt-get update && apt-get install -qq -y --fix-missing --no-install-recommends texlive-full
RUN echo Do other stuff that has to be done every build.
I use GitLab CE 8.4.0 and gitlab/gitlab-runner:latest as runner, started as
docker run -d --name gitlab-runner --restart always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /usr/local/gitlab-ci-runner/config:/etc/gitlab-runner \
gitlab/gitlab-runner:latest \
; \
The runner is registered using:
docker exec -it gitlab-runner gitlab-runner register \
--name foo.example.com \
--url https://gitlab.example.com/ci \
--cache-dir /cache/build/ \
--executor docker \
--docker-image gitlab/dind:latest \
--docker-privileged \
--docker-disable-cache false \
--docker-cache-dir /cache/docker/ \
; \
This creates the following config.toml:
concurrent = 1
[[runners]]
name = "foo.example.com"
url = "https://gitlab.example.com/ci"
token = "foobarsldkflkdsjfkldsj"
tls-ca-file = ""
executor = "docker"
cache_dir = "/cache/build/"
[runners.docker]
image = "gitlab/dind:latest"
privileged = true
disable_cache = false
volumes = ["/cache"]
cache_dir = "/cache/docker/"
(I have experimented with different values for cache_dir, docker_cache_dir and disable_cache, all with the same result: no caching whatsoever)
I suppose there's no simple answer to your question. Before adding some details, I strongly suggest to read this blog article from the maintainer of DinD, which was originally named "do not use Docker in Docker for CI".
What you might try is declaring /var/lib/docker as a volume for your GitLab runner. But be warned, depending on your file-system drivers you may use AUFS in the container on an AUFS filesystem on your host, which is very likely to cause problems.
What I'd suggest to you is creating a separate Docker-VM, only for the runner(s), and bind-mount docker.sock from the VM into your runner-container.
We are using this setup with GitLab with great success (>27.000 builds in about 12 months).
You can take a look at our runner with docker-compose support which is actually based on the shell-executor of GitLab's runner.
Currently you cannot cache intermediate layers in GitLab Docker-in-Docker. Altough there are plans to add that (that are mentioned in the link below). What you can do today to speed up your DinD build is to use the overlay filesystem. To do this you need to be running a liunx kernel >=3.18 and make sure you load the overlay kernel module. Then you set this variable in your gitlab-ci.yml:
variables:
DOCKER_DRIVER: overlay
For more information see this issue and in particular this comment on "The state of optimising Docker Builds!", see the "Using docker executor with dind" section.
https://gitlab.com/gitlab-org/gitlab-ce/issues/17861#note_12991518
For build dependencies that do not change so ofter you can do kinda manual caching with gitlab image registry.
In CI script you do not explicitely call docker build but rather wrap it in a shell script
# cat build_dependencies.sh
registry=registry.example.com
project=group/project
imagebase=$registry/$project/linux
docker pull $imagebase/devbase:1.0
if [ $? -ne 0 ]; then
docker build -f devbase.dockerfile -t $imagebase/devbase:1.0 .
docker push $imagebase/devbase:1.0
fi
...
and call that script in your CI
...
script:
- ./build_dependencies.sh
The downside to this is that when your devbase.dockerfile is updated this would get unnoticed by CI, so you need to force build and push of a new image. So for dynamicly changing images this does not work well, but for your use case this seems like a possible way to go.