My school has a personal gitlab setup, but it doesn't have a registry setup for docker images.
What I want to do is run my pipeline with docker, so that I can build, test etc in a docker environment.
Right now i am trying random stuff because I don't know what I am doing. This is what I have now:
Gitlab-ci:
image: docker:latest
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
build-master:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
build:
stage: build
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
My secret variables on gitlab:
My error message in the pipeline:
Something else I tried uses a gitlab repo. This uses the docker image for ros correctly, but in my application I also use opencv, so I want to add more to the docker image. If i know how to do that in the example below, thats also an option. On top of this, in the example below i can't run tests.
Gitlab-ci:
image: ros:kinetic-ros-core
stages:
- build
variables:
ROS_PACKAGES_TO_INSTALL: ""
USE_ROSDEP: "true"
cache:
paths:
- ccache/
before_script:
- git clone https://gitlab.com/VictorLamoine/ros_gitlab_ci.git
- source ros_gitlab_ci/gitlab-ci.bash
catkin_make:
stage: build
script:
- catkin_make
catkin_build:
stage: build
script:
- catkin build --summarize --no-status --force-color
As I said I have tried many things, this is just the latest thing I have tried. How can I run my runners and gitlab-ci with docker without a gitlab registry?
Just use it withouth registry.
You just need to insert this to gitlab runner config file:
pull_policy = "if-not-present"
Thats enough, and remove commands like:
docker push ...
docker pull ...
Or even insert "|| true" at the end of the push pull command if you want to keep push pull in case, like this:
docker pull ... || true;
Which keeps your code to continue if command fail.
Just dont forget that : pull_policy = "if-not-present" , which allow You to run docker image withouth pull and push.
As image is in case if mussing builded, this works.
example:
[[runners]]
name = "Runner name"
url = ...
...
executor = "docker"
[runners.docker]
image = ...
pull_policy = "if-not-present"
...
You can change these secret variables to point to docker-hub registry server.
You have to create your account on that https://hub.docker.com/ and then use that details to configure - gitlab secret variables.
Related
I am generally still relatively new to the GitLab CI topic and unfortunately I cannot test this myself yet, so this is more of a theoretical attempt.
I want to start a Docker container from one of my other projects in Gitlab in the CI pipeline of my main project.
This Container (I now call it Mock-Container) is created and published in the GitLab CI pipeline of the corresponding project and contains various mocked services.
In the project in which I want to run the Mock-Container, it should be able to start that container in the GitLab CI.
I know it is possible to use a build of the project in a different stage in the same pipeline, like here for example:
variables:
DOCKER_HOST: tcp://docker:2376
DOCKER_TLS_CERTDIR: "/certs"
CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest
is it for example possible if the $CI_REGISTRY_IMAGE used in CONTAINER-IMAGE-Variables is like:
registry.gitlab.com/foo/bar/mainproject
to add a variable here like:
MOCK_CONTAINER_IMAGE: registry.gitlab.com/foo/bar/mockproject:latest
so I could for example could use it in the services list in the test stage:
build:
stage: build
image: quay.io/podman/stable
script:
- podman login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY --log-level=debug
- podman build --format docker --pull -t $CONTAINER_TEST_IMAGE .
- podman push $CONTAINER_TEST_IMAGE
test:
stage: test
image:
name: postman/newman
entrypoint: [ "" ]
services:
- name: $CONTAINER_TEST_IMAGE
alias: main-project
- name: $MOCK_CONTAINER_IMAGE
alias: mock-container
...
Is this possible or is there a better way to achieve this.
If you're asking that you want to set a variable in the .gitlab-ci.yml file with the registry URL of the other container like this:
variables:
MOCK_CONTAINER_IMAGE: registry.gitlab.com/foo/bar/mockproject:latest
then yes you can. And you can use the variable in different stages in your file as you please. If you want to pull this image here from the registry, you can do that in a stage as well.
Check this reference for more info: https://docs.gitlab.com/ee/ci/yaml/#variables
i have two project on gitlab with same CI config file and ci variables. When i try to build dockerfile, one project passed, but second say:
Error: Cannot perform an interactive login from a non TTY device
config:
image: docker:latest
services:
- docker:dind
stages:
- build
variables:
CONTAINER_IMAGE: sleezy/go-hello-world:${CI_COMMIT_SHORT_SHA}
build:
stage: build
script:
- docker login -u ${DOCKER_USER} -p ${DOCKER_PASSWORD}
- docker build -t ${CONTAINER_IMAGE} .
- docker tag ${CONTAINER_IMAGE} ${CONTAINER_IMAGE}
- docker tag ${CONTAINER_IMAGE} sleezy/go-hello-world:latest
- docker push ${CONTAINER_IMAGE}
How i said, everything is same, variables, dockerhub account - username, password, config, even gitlab runner version, so i really dont know why? Any help, thanks.
I've been trying to wrap my head around some old CI/CD scripts my company has written previously, to deploy some applications. The gitlab pipeline has several stages, as is seen in the beginning of the .gitlab-ci.yml file:
image: docker:stable
variables:
DOCKER_DRIVER: overlay2
CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_API: $CI_REGISTRY_IMAGE/career_api:latest
CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_APP: $CI_REGISTRY_IMAGE/career_app:latest
CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP: $CI_REGISTRY_IMAGE/career_dev_app:latest
CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_API: $CI_REGISTRY_IMAGE/career_dev_api:latest
# from https://storage.googleapis.com/kubernetes-release/release/stable.txt
K8S_STABLE_VERSION_URL: https://storage.googleapis.com/kubernetes-release/release/v1.18.4/bin/linux/amd64/kubectl
stages:
- prebuild
- test
- transform
- build
- deploy
Then, later on, the file specifies all these stages for a DEV and a MASTER branch. Specifically, I have trouble understanding the script in the prebuild stage of the dev branch:
prebuild_dev:
stage: prebuild
extends: .prebuildreq
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker pull $CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP || true
- docker build -f Dockerfile --pull -t $CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP --cache-from $CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP .
- docker push $CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP
only:
refs:
- dev
tags:
- testcicd
How I see it now is that the gitlab runner is ran as docker container? (Signified by the image:docker and DOCKER_DRIVER:overlay2 in the beginning of the file). Then, in the prebuild stage it does 4 steps:
login to the container registry with predefined vars $CI_REGISTRY_USER, $CI_REGISTRY_PASSWORD, and $CI_REGISTRY.
Pull CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP from this registry. First question: What does || true do here?
Build a dockerfile but also pull $CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP? Second question: What is happening in this line?
push image (The pulled one or the built one?) back to container registry
Some help to understand this would be greatly appreciated.
docker pull $CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP || true
i assume not 100% sure, not to fail command if docker pull image doesn't exist.
Question : 1
docker build -f Dockerfile --pull -t
$CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP --cache-from
$CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP .
Docker build --pull fetch the specified image digest always for the base image. Instead of using the local version.
Consider it like your base image available at your Build Jenkin machine but it won't use and pull again.
note : --pull --no-cache are flags, you wont be passing any values with it.
Like we do with docker -t or docker -p
Question : 2
docker build -f Dockerfile --pull -t
$CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP --cache-from
$CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP .
$CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP is not pulling image, after -t it's tagging the image with name.
Question : 3
push image (The pulled one or the built one?) back to container
registry
Build one since you have tag image with $CONTAINER_RELEASE_IMAGE_CAREER_GROWTH_DEV_APP
What I need is a way to build a Dockerfile within the repository as an image and use this as the image for the next step(s).
I've tried the Bitbucket Pipeline configuration below but in the "Build" step it doesn't seem to have the image (which was built in the previous step) in its cache.
pipelines:
branches:
main:
- step:
name: Docker Image(s)
script:
- docker build -t foo/bar .docker/composer
services:
- docker
caches:
- docker
- step:
name: Build
image: foo/bar
script:
- echo "Hello, World"
- composer --version
services:
- docker
caches:
- docker
I've tried the answer on the StackOverflow question below but the context in that question is pushing the image in the following step. It's not about using the image which was built for the step itself.
Bitbucket pipeline use locally built image from previous step
There's a few conceptual mistakes in your current pipeline. Let me first first run through those before giving you some possible solutions.
Clarifications
Caching
Bitbucket Pipelines uses the cache keyword to persist data across multiple pipelines. Whilst it will also persist across steps, the primary use-case is for the data to be used on separate builds. The cache takes 7 days to expire, and thus will not be updated with new data during those 7 days. You can manually delete the cache on the main Pipelines page. If you want to carry data across steps in the same pipelines, you should use the artifacts keyword.
Docker service
You should only need to use the docker service whenever you want to have a docker daemon available to your build. Most commonly whenever you need to use a docker command in your script. In your second step, you do not need this. So it doesn't need the docker service.
Solution 1 - Combine the steps
Combine the steps, and run composer within the created image by using the docker run command.
pipelines:
branches:
main:
- step:
name: Docker image and build
script:
- docker build -t foo/bar .docker/composer
# Replace <destination> with the working directory of the foo/bar image.
- docker run -v $BITBUCKET_CLONE_DIR:<destination> foo/bar composer --version
services:
- docker
Solution 2 - Using two steps with DockerHub
This example keeps the two step approach. In this scenario, you will push your foo/bar image to a public repository in Dockerhub. Pipelines will then pull it to use in the subsequent step.
pipelines:
branches:
main:
- step:
name: Docker Image(s)
script:
- docker build -t foo/bar .docker/composer
- docker login -u $DOCKERHUB_USER -p $DOCKERHUB_PASSWORD
- docker push foo/bar
services:
- docker
- step:
name: Build
image: foo/bar
script:
- echo "Hello, World. I'm running insider of the previously pushed foo/bar container"
- composer --version
If you'd like to use a private repository instead, you can replace the second step with:
...
- step:
name: Build
image:
name: foo/bar
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
email $DOCKERHUB_EMAIL
script:
- echo "Hello, World. I'm running insider of the previously pushed foo/bar container"
- composer --version
To expand on phod's answer. If you really want two steps, you can transfer the image from one step to another.
pipelines:
branches:
main:
- step:
name: Docker Image(s)
script:
- docker build -t foo/bar .docker/composer
- docker image save foo/bar -o foobar.tar.gz
services:
- docker
caches:
- docker
artifacts:
- foobar.tar.gz
- step:
name: Build
script:
- docker image load -i foobar.tar.gz
- docker run -v $BITBUCKET_CLONE_DIR:<destination> foo/bar composer --version
services:
- docker
Note that this will upload all the layers and dependencies for the image. It can take quite a while to execute and may therefor not be the best solution.
I recently got into CI/CD, and a good starting point for me was GitLab, since they provide an easy interface for that and i got started about what pipelines and stages are, but i have run into some kind of contradictory thought about GitLab CI running on Docker.
My app runs on Docker Compose. It contains (blah blah) that makes it easy to build & run containers. Each service in the Docker Compose creates a single Docker container, excepting the php-fpm one, which is able to do the thing called "horizontal scale", so I can scale it later.
I will use that Docker Compose for production, I am currently using it in development and I want to use it too in CI/CD pipelines.
However the .gitlab-ci.yml provides support for only one image, so I have to build it and push it to either their GitLab Registry or Docker Hub in order to pull it later in the CI/CD process.
How can I build my Docker Compose's service as a single image in order to push it to the Registry/Docker so I can pull it in the CI/CD?
My project contains a docker folder and a docker-compose.yml. In the docker folder, each service has its own separate directory (php-fpm, nginx, mysql, etc.) and each one (prepare yourself) contains a Dockerfile with build details, especially the php-fpm one (deps and libs are strong with this one)
Each service in the docker-compose.yml has a build context in each of their own folder.
If I was unclear, I can provide additonal info.
However the .gitlab-ci.yml provides support for only one image
This is not true. From the official documentation:
Your image will be named after the following scheme:
<registry URL>/<namespace>/<project>/<image>
GitLab supports up to three levels of image repository names.
Following examples of image tags are valid:
registry.example.com/group/project:some-tag
registry.example.com/group/project/image:latest
registry.example.com/group/project/my/image:rc1
So the solution to your problem is simple - just build individual images and push them to GitLab container registry under different image name.
If you would like an example, my pipelines are set up like this:
.template: &build_template
image: docker:stable
services:
- docker:dind
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker pull $CI_REGISTRY_IMAGE/$IMAGE_NAME:latest || true
- if [ -z ${CI_COMMIT_TAG+x} ];
then docker build
--cache-from $CI_REGISTRY_IMAGE/$IMAGE_NAME:latest
--file $DOCKERFILE_NAME
--tag $CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHA
--tag $CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_TAG
--tag $CI_REGISTRY_IMAGE/$IMAGE_NAME:latest . ;
else docker build
--cache-from $CI_REGISTRY_IMAGE/$IMAGE_NAME:latest
--file $DOCKERFILE_NAME
--tag $CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHA
--tag $CI_REGISTRY_IMAGE/$IMAGE_NAME:latest . ;
fi
- docker push $CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHA
- if [ -z ${CI_COMMIT_TAG+x} ]; then
docker push $CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_TAG;
fi
- docker push $CI_REGISTRY_IMAGE/$IMAGE_NAME:latest
build:image1:
<<: *build_template
variables:
IMAGE_NAME: image1
DOCKERFILE_NAME: Dockerfile.1
build:image2:
<<: *build_template
variables:
IMAGE_NAME: image2
DOCKERFILE_NAME: Dockerfile.2
And you should be able to pull the same image using $CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHA in later pipeline jobs or your compose file (provided that the variables are passed to where you run your compose file).
You don't need dind to run a docker-compose stack. You can run multiple docker-compose up commands.
acceptance_testing:
stage: test
before_script:
- docker-compose -p $CI_JOB_ID up -d
script:
- docker-compose -p $CI_JOB_ID exec -T /run/your/test/suite.sh
after_script:
- docker-compose -p $CI_JOB_ID down -v --remove-orphans || true
I think you search something like this
# .gitlab-ci.yml
image: docker
services:
- docker:dind
build:
script:
- apk add --no-cache py-pip
- pip install docker-compose
- docker-compose up -d
Also good to know:
In Docker, what's the difference between a container and an image?
Building Docker images with GitLab CI/CD
I have a project of Drupal which contains two images: one for Drupal source code & another for MySQL database.
I tagged them:
docker build -t registry.mysite.net/drupal/blog/blog_db:v1.3 mysql/db
docker build -t registry.mysite.net/drupal/blog/blog_drupal:v1.3 src/drupal
Where registry.mysite.net is the url of the git site, and can be found under Container registry settings.
drupal is the group name,
blog is the project name,
blog_db is the image for database, mysql/db is the location for the Dockerfile, and likewise for the other image.
And then to push it to gitlab use:
docker push registry.mysite.net/drupal/blog/blog_db:v1.3
docker push registry.mysite.net/drupal/blog/blog_drupal:v1.3
Hope this might help someone.