I'm trying to build an image in one job and push to AWS ECR in another, since the steps are different I'm trying to pass the image as an artifact:
.gitlab-ci.yml:
stages:
- build
- push
build_image:
stage: build
image: docker
services:
- docker:19.03.12-dind
script:
# building docker image....
- mkdir image
- docker save apis_server > image/apis_server.tar
artifacts:
paths:
- image
push_image:
stage: push
image: docker
services:
- docker:19.03.12-dind
before_script:
- apk add --no-cache python3 py3-pip && pip3 install --upgrade pip && pip3 install --no-cache-dir awscli
script:
- ls
- docker load -i image/apis_server.tar
- docker images
# ecr auth and push to repo...
I get the following warning in the pipeline:
Uploading artifacts for successful job
Uploading artifacts...
WARNING: image: no matching files. Ensure that the artifact path is relative to the working directory
The second job fails with the following message:
$ docker load -i image/apis_server.tar
open image/apis_server.tar: no such file or directory
This approach is based on the answer provided here
For your question, use the full directory address for artifacts.
I have some recommendations for you to speed up you pipeline. If you always install some packages in your pipeline, make a docker image based on your requirements, then use that image in your pipeline instead.
If you need to deploy an image in another place, I recommend you to use docker hub, or make a self hosted docker repository. It is more efficient. Because in docker deployment, the changed layers will be downloaded. But the way you are using, you download all the layers.
Related
Summary of Problem
I have a task in gitlab that requires an npm build to run.
This build then generates the static folder that is needed for my docker build for the server which copies the generated files into the build. I think I can use artifacts and depends_on to make the second task wait on the npm build and get the files it needs, but this makes the artifacts downloadable from the UI which is not desirable. I found a gitlab issue that seems stale and unlikely to ever go anywhere. Is there any other method I can use?
Dependency build
build-web:
stage: build
image: node:17.6.0-slim
before_script:
- set -euo pipefail
- set -x
- cd web
- npm install
- npm run check || true
- npm run lint || true
script:
- npm run build
```yml
## Server build
build-server:
stage: build
tags:
- shell
before_script:
- echo Building server image with tag $CI_COMMIT_REF_NAME
script:
- DOCKER_BUILDKIT=1 BUILDKIT_INLINE_CACHE=1 docker build --tag "server:$CI_COMMIT_REF_NAME" -f ./deployment/server/Dockerfile .
Relative Dockerfile Lines
COPY ./api .
COPY ./api/web ./web
Notes/edits
I host my own runners. I use a shell executor for docker build instead of dind
I have built the following CI in gitlab in order to execute unit-tests and integration tests.
stages:
- build
- test
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
cache:
key: "$CI_BUILD_REF"
paths:
- .m2/repository/
unit-tests:
image: maven:latest
stage: test
script:
- cd source_code
- mvn test -P test
intergration-tests:
image: docker
stage: test
services:
- docker:dind
script:
- apk add --no-cache docker-compose
- docker-compose up -d
- docker exec -t account_service_container sh "integration_tests.sh"
- docker-compose down --rmi all
The point is that when I use maven image between stages then I am able to cache maven dependencies in m2 repository. However in case of intergration test I use different image and the container with docker dind creates an isolated set of containers using docker-compose and then there is no acess to previously defined and cached mvn repository. Is there any solution to this case. Should I created customized image including all fetched and required mvn dependencies and keep it on docker hub and then use that image between each stage and in docker-compose.
I have this gitlab-ci file:
services:
- docker:18.09.7-dind
variables:
SONAR_TOKEN: "$PROJECT_SONAR_TOKEN"
GIT_DEPTH: 0
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
DOCKER_HOST: "tcp://docker:2375"
DOCKER_DRIVER: overlay2
sonarqube-check:
image: maven:latest
stage: test
before_script:
- "docker version"
- "mkdir $PWD/.m2"
- "cp -f /cache/settings.xml $PWD/.m2/settings.xml"
script:
- mvn $MAVEN_CLI_OPTS clean verify sonar:sonar -Dsonar.qualitygate.wait=true -Dsonar.login=$SONAR_TOKEN -Dsonar.projectKey="project-key"
after_script:
- "rm -rf $PWD/.m2"
allow_failure: false
only:
- merge_requests
For some reason docker in docker service does not find the binaries for docker (the docker version command, line 16):
/bin/bash: line 111: docker: command not found
I'm wondering if there is a way of doing this inside of the gitlab-ci file because I need to run docker for the tests, if there is an image that contains both maven and docker binaries or if I'll have to create my own docker image.
It has to be all in one stage, I cannot divide it in two stages (or at least I don't know how to compile in maven in one stage and run the tests witha docker image in another stage)
Thank you!
As you correctly pointed out. You need mvn and docker binaries in the image you are using for that GitLab-CI job.
The quickest win is probably to install docker in your maven:latest build image during run time in the before_script section.
before_script:
- apt-get update && apt-get install -y docker.io
- docker version
If that's slowing down your job too much you might want to build your own custom docker image that contains both Maven and Docker.
Also have a look at the article about dind on Gitlab if you end up moving to Docker 19.03+
I just got into the (wonderful) world of CI/CD and have working pipelines. They are not optimal, though.
The application is a dockerized website:
the source needs to be compiled by webpack and end up in dist
this dist directory is copied to a docker container
which is then remotely built and deployed
My current setup is quite naïve (I added some comments to show why I believe the various elements are needed/useful):
# I start with a small image
image: alpine
# before the job I need to have npm and docker
# the problem: I need one in one job, and the second one in the other
# I do not need both on both jobs but do not see how to split them
before_script:
- apk add --update npm
- apk add docker
- npm install
- npm install webpack -g
stages:
- create_dist
- build_container
- stop_container
- deploy_container
# the dist directory is preserved for the other job which will make use of it
create_dist:
stage: create_dist
script: npm run build
artifacts:
paths:
- dist
# the following three jobs are remote and need to be daisy chained
build_container:
stage: build_container
script: docker -H tcp://eu13:51515 build -t widgets-sentinels .
stop_container:
stage: stop_container
script: docker -H tcp://eu13:51515 stop widgets-sentinels
allow_failure: true
deploy_container:
stage: deploy_container
script: docker -H tcp://eu13:51515 run --rm -p 8880:8888 --name widgets-sentinels -d widgets-sentinels
This setups works bit npm and docker are installed in both jobs. This is not needed and slows down the deployment. Is there a way to state that such and such packages need to be added for specific jobs (and not globally to all of them)?
To make it clear: this is not a show stopper (and in reality not likely to be an issue at all) but I fear that my approach to such a job automation is incorrect.
You don't necessarily need to use the same image for all jobs. Let me show you one of our pipelines (partially) which does a similar thing, just with composer for php instead of npm:
cache:
paths:
- vendor/
build:composer:
image: registry.example.com/base-images/php-composer:latest # use our custom base image where only composer is installed on to build the dependencies)
stage: build dependencies
script:
- php composer.phar install --no-scripts
artifacts:
paths:
- vendor/
only:
changes:
- composer.{json,lock,phar} # build vendor folder only, when relevant files change, otherwise use cached folder form s3 bucket (configured in runner config)
build:api:
image: docker:18 # use docker image to build the actual application image
stage: build api
dependencies:
- build:composer # reference dependency dir
script:
- docker login -u gitlab-ci-token -p "$CI_BUILD_TOKEN" "$CI_REGISTRY"
- docker build -t $CI_REGISTRY_IMAGE:latest.
- docker push $CI_REGISTRY_IMAGE:latest
The composer base image contains all necessary packages to run composer, so in your case you'd create a base image for npm:
FROM alpine:latest
RUN apk add --update npm
Then, use this image in your create_dist stage and use image: docker:latest as image in the other stages.
As well as referncing different images for different jobs you may also try gitlab anchors which provides reusable templates for the jobs:
.install-npm-template: &npm-template
before_script:
- apk add --update npm
- npm install
- npm install webpack -g
.install-docker-template: &docker-template
before_script:
- apk add docker
create_dist:
<<: *npm-template
stage: create_dist
script: npm run build
...
deploy_container:
<<: *docker-template
stage: deploy_container
...
Try multistage builder, you can intermediate temporary images and copy generated content final docker image. Also, npm should be part on docker image, create one npm image and use in final docker image as builder image.
I am creating a node.js(angular/cli) application and want to run the tests by using docker container. I successfully created docker image and uploaded to docker hub where it runs automated build after each commit.
The problem is: I don't know what to put in bitbucket-pipelines.yml file.
I want that after each commit it would take a newly built container from dockerhub and run the tests against it.
Also, I want to minimize bitbucket build time.
bitbucket-pipelines.yml
pipelines:
default:
- step:
image: [Enter your docker image here, as on hub.docker.com]
script:
- npm install
- npm install -g #angular/cli
- ng build --prod
- ls -ltr
- pwd