I would like to execute an external (on the local machine) bash script from gitlab-ci.yml which uses the docker:stable image. I would like to execute startup.sh located outside the gitlab docker image. Is this possible or are there better options?
gitlab-ci.yaml
image: docker:stable
#Build script
variables:
CI_DEBUG_TRACE: "true"
DOCKER_DRIVER: overlay
before_script:
- docker --version
build:
services:
- docker:dind
script:
- docker build --no-cache -t <tag> .
- docker login -u root -p <pass> <registry>
- docker tag ...
- docker push ...
- echo "build completed"
stage: build
tags:
- <tag>
deploy_staging:
stage: deploy
script:
- ./sh startup.sh
bash script
#!/bin/bash
docker login -u root -p <pass>
docker pull <image>
docker-compose up -d
I am not sure if this is the best practice for your use-case but
the simple way to share files with an image is to add volume and share this code to the image by editing your config.toml file .
add this line to config.toml under [runners.docker]
volumes = ["/cache",path to startup.sh:/root/scripts"]
and then inside your.gilatb.yml
deploy_staging:
stage: deploy
script:
- chmod +x /root/scripts/startup.sh
- ./sh /root/scripts/startup.sh
Related
i have a job that starts android UI-tests on GitLab CI/CD. It somehow runs a container from image android-uitests:1.0 from registry. I don't know where and how Gitlab CI/CD runs that image using command "docker run ...", but i need to extend that command and i want to pass some variables (or arguments) in this command.
Here below example of command that i want Gitlab to do:
docker run -d \
-t \
--privileged \
-e "SNAPSHOT_DISABLED"="true" \
-e "QTWEBENGINE_DISABLE_SANDBOX"=1 \
-e "WINDOW"="true" \
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
-p 5555:5555 -p 5554:5554 -p 5901:5901 \
--name emulator \
android-uitest:1.0
this is a stage and its job with image
ui-tests:
image: registry.myproject:5000/android-uitests:1.0
stage: ui-tests
only:
- merge_requests
- schedules
when: manual
script:
- bash /run-emulator.sh
- adb devices
- adb shell input keyevent 82
- adb shell settings put global window_animation_scale 0 &
- adb shell settings put global transition_animation_scale 0 &
- adb shell settings put global animator_duration_scale 0 &
- ./gradlew mobile:connectedDevDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.package=app.online.tests.uitests
tags:
- androidtest
So another words i want to configure that "under the hood docker run command" that runs my image.
Tell me please how to do that?
Considering you're using a Docker container, I'll assume you're using a Gitlab Runner on Docker executor mode, which means you're essentially running a similar script to this when you don't specify a docker image to run the CI job:
image: docker:19.03.13
variables:
DOCKER_TLS_CERTDIR: "/certs"
services:
- docker:19.03.13-dind
script:
- (your commands)
To understand what's going on, let's break it in multiple steps:
image: docker:19.03.13
(...)
services:
- docker:19.03.13-dind
docker-19.03.13-dind, what's the difference to docker:19.03.13?
why is it a service instead of an image?
DIND means Docker-in-Docker, this part of Gitlab's documentation can explain it in further technical details, but what is important to understand from this part is what is a service on Gitlab's CI context and why they have to specify an additional Docker image when already using a Docker image as a default environment. When you write a service on Gitlab CI, you are able to use its command while you're inside an existing container. e.g. when you want to connect a PostgreSQL (database) container to a backend container you're building, but without having to set up a docker-compose or multiple containers.
Using a docker service to run together with a docker image, it means you can directly use docker run within your job without any additional setup. This previous StackOverflow question explains this question further.
Back to your code, instead of deploying your registry image as a job directly:
ui-tests:
image: registry.myproject:5000/android-uitests:1.0
You may want to first, build your container and upload it to your registry:
image: docker:19.03.12
services:
- docker:19.03.12-dind
variables:
# Use TLS https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled
DOCKER_HOST: tcp://docker:2376
DOCKER_TLS_CERTDIR: "/certs"
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
This snippet will build your Dockerfile in the root folder of your repository and upload it to your Gitlab private (or public, if your project is set as public) image registry. Now you can specify an additional job specifically to do what you want:
Final example
image: docker:19.03.12
stages:
- build
- release
services:
- docker:19.03.12-dind
variables:
# Use TLS https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled
DOCKER_HOST: tcp://docker:2376
DOCKER_TLS_CERTDIR: "/certs"
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
build:
stage: build
script:
- docker pull $CI_REGISTRY_IMAGE:latest || true
- docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
deploy:
stage: release
script:
- docker pull $CI_REGISTRY_IMAGE:latest || true
- docker run -d -t --privileged -e "SNAPSHOT_DISABLED"="true" -e "QTWEBENGINE_DISABLE_SANDBOX"=1 -e "WINDOW"="true" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" -p 5555:5555 -p 5554:5554 -p 5901:5901 --name emulator $CI_REGISTRY_IMAGE:latest
Why are you using $VARIABLES?
In case of an environment variable may sound confusing, here's the list of default environment variables Gitlab generates for every job it creates.
The last example I cited will result in a Docker container running in the same machine your executor is registered, with the environment variables you have specified.
If you need a practical example, you can use this gitlab-ci.yml of a project of mine as reference.
I'm trying to run a mounted shell-script inside a docker container by following these steps:
build stage: build the docker image.
test stage: mount a directory into the container at runtime with a shell-script file inside.
test stage: run the shell-script file from inside the docker.
could someone please explain how this should be done?
see line: #- ?? HERE I SHOULD RUN THE TEST: /test/check.sh ??
services:
- docker:dind
stages:
- build
- test
before_script:
- docker info
# Build the docker image
build:
image: docker:latest
services:
- docker:dind
before_script:
- docker login docker.example.com -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
only:
- master
script:
- docker build -t our-docker .
- docker save our-docker > our-docker.tar
artifacts:
paths:
- our-docker.tar
expire_in: 1 week
stage: build
test:
image: docker:latest
only:
- master
script:
- docker load < our-docker.tar
- docker run --volume source="$(pwd)/test",target="/test" our-docker
#- ?? HERE I SHOULD RUN THE TEST: /test/check.sh ??
stage: test
First, there was an issue with the docker run command itself:
docker run --volume source="$(pwd)/test",target="/test" our-docker # buggy
as the syntax to setup a bind-mount is:
either docker run -v "$PWD/test":"/test" our-docker
(-v being the short form of --volume)
or docker run --mount type=bind,source="$PWD/test",target="/test" our-docker
(Note: I replaced above "$(pwd)" with the special shell variable "$PWD" which avoids spinning yet another process.)
Next, you cannot just append the line /test/check.sh after the docker run line because you precisely need to run that command within the context of docker run. To this aim, you may want to use the pattern I proposed in this other SO thread: How do I set docker-credential-ecr-login in my PATH before anything else in GitLab CI (which contains more details/remarks about set -e, quotes and shell escaping in the context of that pattern).
Wrap-up
More precisely, could you try the following adaptation of your .gitlab-ci.yml? (I've added some ls commands that should help debugging your configuration):
services:
- docker:dind
stages:
- build
- test
before_script:
- docker info
# Build the docker image
build:
image: docker:latest
services:
- docker:dind
before_script:
- docker login docker.example.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
only:
- master
script:
- docker build -t our-docker .
- docker save our-docker > our-docker.tar
artifacts:
paths:
- our-docker.tar
expire_in: 1 week
stage: build
test:
image: docker:latest
# note: use /bin/sh below as this image doesn't provide /bin/bash
only:
- master
script:
- docker load < our-docker.tar
- echo "$PWD"
- ls
- ls -Rhal test
- |
docker run --rm -v "$PWD/test":"/test" our-docker /bin/sh -c "
set -ex
ls -Rhal /test
/test/check.sh
"
stage: test
Is there a way to run two jobs using two different Docker images?
I've tried to run this configuration but without any success:
before_script:
- docker info
build:default:
image: ubuntu:latest
script:
- pip3 install -r requirements.txt
- [..]
build:docker:
image: docker:latest
script:
- docker build -t app .
- docker run -d -p 8000:8000 --rm app:latest
- [..]
As mentioned by others in the comments the config file was fine, the problem was with gitlab runner itself.
So i made the following changes:
Change volume value to volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] into /etc/gitlab-runner/config.toml
Add variables:
DOCKER_DRIVER: overlay
to your Dockerfile.
I want to upload my frontend to sentry, but I need to get the folder using docker commands. However when I use image: getsentry/sentry-cli
docker doesn't works and e.g. in before_script I get error that docker doesn't exist
sentry_job:
stage: sentry_job
image: getsentry/sentry-cli
services:
- docker:18-dind
before_script:
- docker login -u gitlab-ci-token -p "$CI_JOB_TOKEN" registry.gitlab.cz
script:
# script...
. # Get the dist folder from the image
- mkdir frontend_dist
- docker run --rm -v $PWD/frontend_dist:/mounted --entrypoint="" $IMAGE /bin/sh -c "cp /frontend/dist /mounted"
- ls frontend_dist
tags:
- dind
How do I fix that?
To achieve what you want, you need to use a single job (to have the same build context) and specify docker:stable as the job image (along with docker:stable-dind as a service).
This setup is called docker-in-docker and this is the standard way to allow a GitLab CI script to run docker commands (see doc).
Thus, you could slightly adapt your .gitlab-ci.yml code like this:
sentry_job:
stage: sentry_job
image: docker:stable
services:
- docker:stable-dind
variables:
IMAGE: "${CI_REGISTRY_IMAGE}:latest"
before_script:
- docker login -u gitlab-ci-token -p "${CI_JOB_TOKEN}" registry.gitlab.cz
script:
- git pull "$IMAGE"
- mkdir -v frontend_dist
- docker run --rm -v "$PWD/frontend_dist:/mounted" --entrypoint="" "$IMAGE" /bin/sh -c "cp -v /frontend/dist /mounted"
- ls frontend_dist
- git pull getsentry/sentry-cli
- docker run --rm -v "$PWD/frontend_dist:/work" getsentry/sentry-cli
tags:
- dind
Note: the git pull commands are optional (they ensure Docker will use the latest version of the images).
Also, you may need to change the definition of variable IMAGE.
I use Gitlab.com for my CI using their shared docker runners. I have a project which requires PHP and composer installed, while it also needs docker to build a docker image of the project.
I've tried for hours to build a docker image which has PHP, composer and docker installed, but I can't seem to figure it out.
For reference, my gitlab-ci.yml file looks like this;
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
services:
- docker:dind
composer:install:
stage: build
artifacts:
paths:
- /
expire_in: 1 week
script:
- docker run --rm --interactive --tty --volume $PWD:/app composer install
build:image:
stage: build
dependencies:
- composer:install
script:
- docker login registry.gitlab.com -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD
- docker build -t registry.gitlab.com/accountname/projectname/develop .
- docker push registry.gitlab.com/accountname/projectname/develop
Using the sample build script provided by Stefan below, I put together the following build file which appears to work perfectly. It builds the project using the project Dockerfile, and pushed the resulting image to my Gitlab repository.
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
services:
- docker:dind
before_script:
- docker run --rm --volume $PWD:/app composer install
build:image:
stage: build
script:
- docker login registry.gitlab.com -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD
- docker build -t registry.gitlab.com/accountname/projectname/develop .
- docker push registry.gitlab.com/accountname/projectname/develop
You should use a 2 job CI and use the artifact feature provided by Gitlab to pass the composer install result as dependencies between jobs.
Your first job (composer:install) could use something like https://hub.docker.com/r/library/composer/ in your script section to install all composer packages, then pass it to the build:image job that builds your Docker image.
This would roughly look like this:
image: docker:latest
variables:
DOCKER_DRIVER: overlay2
services:
- docker:dind
composer:install:
stage: composer
artifacts:
paths:
- /
expire_in: 1 week
script:
- docker run --rm --interactive --tty --volume $PWD:/app composer install
build:image:
stage: build
dependencies:
- composer:install
script:
- docker build -t myimage:latest .
Where your Dockerfile could be something like this (based on this):
FROM php:7.0-cli
COPY . /usr/src/myapp
WORKDIR /usr/src/myapp
CMD [ "php", "./your-script.php" ]