I am working on a gitlab CI/CD project to build an asp.net core application into a docker.
Currently I have 2 possible implementations in mind. The first one have the full logic in the Dockerfile, but I can't visualize the stages in Gitlab this way (build, test, publish). So I thought about moving the main logic to the gitlab-ci.yml file. But what bothers me now is that I have to manage the image docker dotnet versions on 2 places (sdk:3.1, aspnet:3.1.1-alpine3.10). Is it a good idea to deliver the version via build-arg or is there a more elegant solution?
.gitlab-ci.yml
stages:
- build
- test
- docker
build:
stage: build
image: mcr.microsoft.com/dotnet/core/sdk:3.1
only:
- master
script:
- cd src
- dotnet restore --interactive
- dotnet build --configuration Release
- dotnet publish --configuration Release --output ../publish/
artifacts:
paths:
- ./publish/*.*
expire_in: 1 week
tags:
- docker
test:
stage: test
image: mcr.microsoft.com/dotnet/core/sdk:3.1
only:
- master
script:
- cd src
- dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=../../MyProject.xml"
artifacts:
paths:
- ./MyProject.xml
reports:
junit: ./MyProject.xml
tags:
- docker
docker:
stage: docker
image: docker:stable
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
only:
- master
script:
- docker login -u "gitlab-ci-token" -p "$CI_JOB_TOKEN" $CI_REGISTRY
- docker build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" --tag "$CI_REGISTRY_IMAGE:latest" --build-arg EXECUTABLE=Test.WebApi.dll .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
- docker push "$CI_REGISTRY_IMAGE:latest"
tags:
- docker
Dockerfile
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1.1-alpine3.10
ARG EXECUTABLE
WORKDIR /app
COPY /publish .
ENV ASPNETCORE_URLS "http://*:5000"
ENV ASPNETCORE_ENVIRONMENT "Staging"
CMD ["dotnet", "$EXECUTABLE"]
Here is my solution, I have defined the variables above and replace them in the docker file with sed
My Solution have this two Projects
Test.WebApi (WebApi Project)
Test.WebApi.UnitTest (Unit Test Project)
#ThomasBrüggemann thanks for the Inspiration.
.gitlab-ci.yml
variables:
PROJECT_NAME: "Test.WebApi"
BUILD_IMAGE: "mcr.microsoft.com/dotnet/core/sdk:3.1"
RUNTIME_IMAGE: "mcr.microsoft.com/dotnet/core/aspnet:3.1.1-alpine3.10"
stages:
- build
- test
- docker
build:
stage: build
image: $BUILD_IMAGE
only:
- master
script:
- cd src/$PROJECT_NAME
- dotnet restore --interactive
- dotnet build --configuration Release
- dotnet publish --configuration Release --output ../../publish/
artifacts:
paths:
- ./publish/*
expire_in: 1 week
tags:
- docker
test:
stage: test
image: $BUILD_IMAGE
only:
- master
script:
- cd src/$PROJECT_NAME.UnitTest
- dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=../../UnitTestResult.xml"
artifacts:
paths:
- ./UnitTestResult.xml
reports:
junit: ./UnitTestResult.xml
tags:
- docker
docker:
stage: docker
image: docker:stable
services:
- docker:18.09.7-dind
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
only:
- master
script:
# Prepare Dockerfile
- sed -i "s~\$DOCKERIMAGE~$RUNTIME_IMAGE~g" Dockerfile
- sed -i 's/$ENVIRONMENT/Staging/g' Dockerfile
- sed -i "s/\$ENTRYPOINT/$PROJECT_NAME.dll/g" Dockerfile
# Process Dockerfile
- docker login -u "gitlab-ci-token" -p "$CI_JOB_TOKEN" $CI_REGISTRY
- docker build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" --tag "$CI_REGISTRY_IMAGE:latest" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
- docker push "$CI_REGISTRY_IMAGE:latest"
tags:
- docker
Dockerfile
FROM $DOCKERIMAGE
WORKDIR /app
COPY /publish .
EXPOSE 5000/tcp
ENV ASPNETCORE_URLS "http://*:5000"
ENV ASPNETCORE_ENVIRONMENT "$ENVIRONMENT"
CMD ["dotnet", "$ENTRYPOINT"]
Here some Version handling when tagging in GitLab-Pipeline:
script:
- COMMIT_DATE=$(git log -1 --format=%cd --date=iso-strict | grep -o '\([0-9]*\)' | tr -d '\n')
- VERSION_PREFIX=$CI_COMMIT_TAG
- VERSION_SUFFIX="${COMMIT_DATE::-6}"
- echo $VERSION_PREFIX-$VERSION_SUFFIX
- sed -i "s:<VersionPrefix>.*</VersionPrefix>:<VersionPrefix>$VERSION_PREFIX</VersionPrefix>:g" [PROJECT].csproj
- dotnet publish --version-suffix $VERSION_SUFFIX -c Release -o ./out
- docker build --tag "$CI_REGISTRY_IMAGE:$VERSION_PREFIX"
only:
- tags
In Project file must
<!-- Version is set by CI-Script do not modify manually -->
<VersionPrefix>0.0.0</VersionPrefix>
<Deterministic>False</Deterministic>
be set
Maybe this is helpful.
Something similar can be done when build without tagging.
Related
I have tried many ways through searching for a solution.
I think my problem is different.
I am wanting to have a docker image that has the environment installed and then active and ready for shell commands like: flake8, pylint, black, isort, coverage
Dockerfile
FROM continuumio/miniconda3
# Create the environment:
COPY conda_env_unit_tests.yml .
RUN conda env create -f conda_env_unit_tests.yml
RUN echo "conda activate up-and-down-pytorch" >> ~/.bashrc
conda_env_unit_test.yml
name: up-and-down-pytorch
channels:
- defaults
- conda-forge
dependencies:
- python=3.9
- pytest
- pytest-cov
- black
- flake8
- isort
- pylint
.gitlab-ci.yml (slimmed down)
stages:
- docker
- linting
- test
build_unit_test_docker:
stage: docker
tags:
- docker
image: docker:stable
services:
- docker:dind
variables:
IMAGE_NAME: "miniconda3-up-and-down-unit-tests"
script:
- cp /builds/upanddown1/mldl/up_and_down_pytorch/conda_env_unit_tests.yml /builds/upanddown1/mldl/up_and_down_pytorch/docker/unit_tests/
- docker -D login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
- docker -D build -t $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME docker/unit_tests/
- docker -D push $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME
rules:
- changes:
- docker/unit_tests/Dockerfile
- conda_env_unit_tests.yml
unit-test:
stage: test
# image: continuumio/miniconda3:latest
image: $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/miniconda3-up-and-down-unit-tests
script:
# - conda env create --file conda_env.yml
# - source activate up-and-down-pytorch
- coverage run --source=. -m pytest --verbose
- coverage report
- coverage xml
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
The Docker Image gets uploaded to the gitlab registry and the unit test stage uses that image, however:
/bin/bash: line 127: coverage: command not found
(ultimate goal was to not have to create the conda environment every time I wanted to lint or run unit tests)
Figured it out today.
Dropped the duration for the unit tests.
Change was to source the environment in the unit-test job. Didn't need to do that in the Dockerfile.
Dockerfile
FROM continuumio/miniconda3
# Create the environment:
COPY conda_env_unit_tests.yml .
RUN conda env create -f conda_env_unit_tests.yml
conda_env_unit_tests.yml
name: up-and-down-pytorch
channels:
- defaults
- conda-forge
dependencies:
- python=3.9
- pandas
- pytest
- pytest-cov
- black
- flake8
- isort
- pylint
.gitlab-ci.yml (slimmed down)
stages:
- docker
- linting
- test
build_unit_test_docker:
stage: docker
tags:
- docker
image: docker:stable
services:
- docker:dind
variables:
IMAGE_NAME: "miniconda3-up-and-down-unit-tests"
script:
- cp /builds/upanddown1/mldl/up_and_down_pytorch/conda_env_unit_tests.yml /builds/upanddown1/mldl/up_and_down_pytorch/docker/unit_tests/
- docker -D login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
- docker -D build -t $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME docker/unit_tests/
- docker -D push $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/$IMAGE_NAME
rules:
- changes:
- docker/unit_tests/Dockerfile
- conda_env_unit_tests.yml
unit-test:
stage: test
image: $CI_REGISTRY/upanddown1/mldl/up_and_down_pytorch/miniconda3-up-and-down-unit-tests
script:
- source activate up-and-down-pytorch
- coverage run --source=. -m pytest --verbose
- coverage report
- coverage xml
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
Background
in my gitlab-ci file I am trying to build a docker image, however even though I have docker:dind as a service, it is failing.
.gitlab-ci
---
stages:
- build
- docker
build:
stage: build
image: fl4m3ph03n1x/my-app:1.0
variables:
MIX_ENV: prod
script:
- mix deps.get
- mix deps.compile
- mix compile
artifacts:
paths:
- .hex/
- _build/
- deps/
- mix.lock
build_image:
stage: docker
image: fl4m3ph03n1x/my-app:1.0
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
DOCKER_HOST: tcp://docker:2375/
services:
- docker:dind
script:
- echo ${CI_JOB_TOKEN} | docker login --password-stdin -u ${CI_REGISTRY_USER} ${CI_REGISTRY}
- docker build . -t ${CI_REGISTRY_IMAGE}:latest
- docker push ${CI_REGISTRY_IMAGE}:latest
The problematic stage is docker.
As you can see I am trying to:
login into docker
build an image from gitlab's registry
push that image
Error
However, I am getting the following error:
$ echo ${CI_JOB_TOKEN} | docker login --password-stdin -u
${CI_REGISTRY_USER} ${CI_REGISTRY} /bin/bash: line 110: docker:
command not found
Which is confusing, because docker:dind is supposed to actually prevent this from happening:
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#enable-registry-mirror-for-dockerdind-service
Question
So clearly I am missing something here. What am I doing wrong?
EDIT
This is my Dockerfile
FROM elixir:1.10
# Install Hex + Rebar
RUN mix do local.hex --force, local.rebar --force
COPY . /
WORKDIR /
ENV MIX_ENV=prod
RUN mix do deps.get --only $MIX_ENV, deps.compile
RUN mix release
EXPOSE 8080
ENV PORT=8080
ENV SHELL=/bin/bash
CMD ["_build/prod/rel/my_app/bin/my_app", "start"]
image is used to specify the image in which to run the script. You want to run the script in a docker image, to build your image.
The image keyword is the name of the Docker image the Docker executor runs to perform the CI tasks.
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-image-and-services-from-gitlab-ciyml
After all, isn't your application image CI_REGISTRY_IMAGE in this? You don't want to build the image in itself.
- docker build . -t ${CI_REGISTRY_IMAGE}:latest
- docker push ${CI_REGISTRY_IMAGE}:latest
I am trying to deploy my NodeJS repo to a DO droplet via GitLab CI. I have been following this guide to do so. What is odd is that the deployment pipeline seems to succeed but if I SSH into the box, I can see that the app is not running as has failed to find a package.json in /usr/src/app which is the WORKDIR my Dockerfile is pointing to.
gitlab-ci.yml
cache:
key: "${CI_COMMIT_REF_NAME} node:latest"
paths:
- node_modules/
- .yarn
stages:
- build
- release
- deploy
build:
stage: build
image: node:latest
script:
- yarn
artifacts:
paths:
- node_modules/
release:
stage: release
image: docker:latest
only:
- master
services:
- docker:dind
variables:
DOCKER_DRIVER: "overlay"
before_script:
- docker version
- docker info
- docker login -u ${CI_REGISTRY_USER} -p ${CI_BUILD_TOKEN} ${CI_REGISTRY}
script:
- docker build -t ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest --pull .
- docker push ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest
after_script:
- docker logout ${CI_REGISTRY}
deploy:
stage: deploy
image: gitlab/dind:latest
only:
- master
environment: production
when: manual
before_script:
- mkdir -p ~/.ssh
- echo "${DEPLOY_SERVER_PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- eval "$(ssh-agent -s)"
- ssh-add ~/.ssh/id_rsa
- ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
script:
- printf "DB_URL=${DB_URL}\nDB_NAME=${DB_NAME}\nPORT=3000" > .env
- scp -r ./.env ./docker-compose.yml root#${DEPLOYMENT_SERVER_IP}:~/
- ssh root#${DEPLOYMENT_SERVER_IP} "docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}; docker-compose rm -sf scraper; docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest; docker-compose up -d"
Dockerfile
FROM node:10
WORKDIR /usr/src/app
COPY package.json ./
RUN yarn
COPY . .
EXPOSE 3000
CMD [ "yarn", "start" ]
docker-compose.yml
version: "3"
services:
scraper:
build: .
image: registry.gitlab.com/arby-better/scraper:latest
volumes:
- .:/usr/src/app
- /usr/src/app/node_modules
ports:
- 3000:3000
environment:
- NODE_ENV=production
env_file:
- .env
I'm using GitLab Shared Runners for my pipeline. My pipeline looks like it executes completely fine but for this symlink failure at the end:
...which I don't think is anything to worry about. If I SSH into my box & go to where the docker compose was copied & inspect:
Docker has not created /usr/src/app.
Versions:
Docker: 19.03.1
Docker-compose: 1.22.0
My DO box is Docker 1-click btw. Any help appreciated!
EDIT
I have altered my Dockerfile to attempt to force the dir creation so have added RUN mkdir -p /usr/src/app before the line declaring it as the working dir. This still does not create the directory...
When I look at the container status' (docker-compose ps), I can see that the containers are in an exit state & have exited with code either 1 or 254...any idea as to why?
Your compose file is designed for a development environment, where the code directory is replaced by a volume mount to the code on the developers machine. You don't have this persistent directory in production, nor should you be depending on code outside of the image in production, defeating the purpose of copying it into your image.
version: "3"
services:
scraper:
build: .
image: registry.gitlab.com/arby-better/scraper:latest
# Comment out or delete these lines, they do not belong in production
#volumes:
# - .:/usr/src/app
# - /usr/src/app/node_modules
ports:
- 3000:3000
environment:
- NODE_ENV=production
env_file:
- .env
I am trying to deploy my docker images from Docker Hub to Rancher. When I do that, I am getting only updating status and then after some time, I am getting deployment timeout. I am new to Rancher and trying to go through documents but not able to understand it properly. Can anyone assist me to do this?
I already tried to pull the image and after some time it shows "deployment does not have minimum availability".
Below is my .gitlab-ci.yml file configuration:
stages:
- build
- publish
.build: &build_template
stage: build
image: microsoft/dotnet:2.2-sdk-alpine
cache:
key: "$CI_PROJECT_NAMESPACE-$CI_PROJECT_NAME"
paths:
- .nuget/
script:
- dotnet restore -s https://api.nuget.org/v3/index.json --packages ./.nuget/
- dotnet publish --no-restore -c Release -o ./docker/publish/
develop_build:
<<: *build_template
only:
- develop
artifacts:
expire_in: 1 day
paths:
- docker/
.docker: &docker_template
stage: publish
image: docker:stable-dind
script:
- mkdir docker/app/
- mv docker/publish/$ASSEMBLY_NAME.* docker/app/
- mv docker/publish/*.config docker/app/
- mv docker/publish/*.json docker/app/
- cp Dockerfile docker/
- docker build -t $TAG ./docker/
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker push $TAG
develop_docker:
<<: *docker_template
only:
- develop
dependencies:
- develop_build
variables:
ASSEMBLY_NAME: app
TAG: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_PIPELINE_ID
Below is my DockerFile:
# Dockerfile
FROM microsoft/dotnet:sdk AS build-env
WORKDIR /app
# Copy csproj and restore as distinct layers set
COPY ./DataStore/DataStore/*.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY *[^.csproj] ./
RUN dotnet publish -c Release -o out
# Build runtime images
FROM microsoft/dotnet:aspnetcore-runtime
ENTRYPOINT ["dotnet", "DataStore.dll"]
I want to run my C# web APIs after deploying it to Rancher.
I'm practicing with Gitlab CI to understand how to build an application and then use that within a Docker image. For now, my repo consists simply of helloworld.txt, dockerfile, and gitlab-ci.yml.
PROBLEM: During the build stage, I use a shell executor to 'zip helloworld.zip helloworld.txt". Then, I "docker build -t myproject/myapp ." where I expect to COPY helloworld.zip /" but it seems that the zip file I created is not available during the docker build context. Am I not saving the helloworld.zip file to the right location? Or something else? My long term intent is to write a python application, and during the build stage to compile into a single executable and copy into a docker container.
#cat helloworld.txt
hello world
#cat dockerfile
FROM centos:7
COPY helloworld.zip /
CMD ["/bin/bash"]
#cat gitlab-ci.yml
stages:
- build
- test
- release
- deploy
variables:
IMAGE_TEST_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
IMAGE_RELEASE_NAME: $CI_REGISTRY_IMAGE:latest
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
build:
stage: build
script:
- echo "compile the program"
- zip zipfile.zip helloworld.txt
- docker build --pull -t $IMAGE_TEST_NAME .
- docker push $IMAGE_TEST_NAME
test:
stage: test
script:
- docker pull $IMAGE_TEST_NAME
- docker run $IMAGE_TEST_NAME yum install unzip -y && unzip /helloworld.zip && cat /helloworld.txt
release:
stage: release
script:
- docker pull $IMAGE_TEST_NAME
- docker tag $IMAGE_TEST_NAME $IMAGE_RELEASE_NAME
- docker push $IMAGE_RELEASE_NAME
only:
- master
deploy:
stage: deploy
script:
- ./deploy.sh
only:
- master
when: manual
I expect that within the same stage (in this case build), I can run a program such as zip and then COPY that zip file into a given directory within a newly built docker image during the docker build process.
EDIT
After learning that I can't do this, I've created two different stages: build_app and build_container. Also knowing that artifacts are used by default in following stages, I didn't add an artifacts to the first stage or a dependancies to the next stage. This is the gitlab-ci.yml below and is still producing the same error.
stages:
- build_app
- build_container
- test
- release
- deploy
# you can delete this line if you're not using Docker
#image: centos:latest
variables:
IMAGE_TEST_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
IMAGE_RELEASE_NAME: $CI_REGISTRY_IMAGE:latest
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
build_app:
stage: build_app
script:
- echo "compile the program"
- zip zipfile.zip helloworld.txt
build_container:
stage: build_container
script:
- docker build --pull -t $IMAGE_TEST_NAME .
- docker push $IMAGE_TEST_NAME
test:
stage: test
script:
- docker pull $IMAGE_TEST_NAME
- docker run $IMAGE_TEST_NAME yum install unzip -y && unzip /helloworld.zip && cat /helloworld.txt
release:
stage: release
script:
- docker pull $IMAGE_TEST_NAME
- docker tag $IMAGE_TEST_NAME $IMAGE_RELEASE_NAME
- docker push $IMAGE_RELEASE_NAME
only:
- master
deploy:
stage: deploy
script:
- ./deploy.sh
only:
- master
when: manual
Job Status:
Build App: Passed
Build Container: Failed
Running with gitlab-runner 11.6.1 (8d829975)
on gitrunner-shell trtHcQTS
Using Shell executor...
Running on gitrunner.example.com...
Fetching changes...
Removing zipfile.zip
HEAD is now at e0a0a95 Update .gitlab-ci.yml
Checking out e0a0a952 as newFeature...
Skipping Git submodules setup
$ echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build --pull -t $IMAGE_TEST_NAME .
Sending build context to Docker daemon 112.1kB
Step 1/3 : FROM centos:7
7: Pulling from library/centos
Digest: sha256:184e5f35598e333bfa7de10d8fb1cebb5ee4df5bc0f970bf2b1e7c7345136426
Status: Image is up to date for centos:7
---> 1e1148e4cc2c
Step 2/3 : COPY helloworld.zip /
COPY failed: stat /var/lib/docker/tmp/docker-builder312764301/helloworld.zip: no such file or directory
ERROR: Job failed: exit status 1
This is not possible. Gitlab CI's job model assumes that jobs of the same stage are independent.
See the manual for the dependencies keyword in gitlab-ci.yml:
This feature [...] allows you to define the artifacts to pass between different jobs.
Note that artifacts from all previous stages are passed by default.
[...] You can only define jobs from stages that are executed before the current one.