GitLab CI/CD not taking latest code changes - docker

So I have used GitLab CI/CD to deploy changes to private docker hub repo and using Digital Ocean droplet to run the server using docker but the changes are not being reflected in the docker container running on digital ocean. Here's the config file.
variables:
IMAGE_NAME: codelyzer/test-repo
IMAGE_TAG: test-app-1.0
stages:
- test
- build
- deploy
run_tests:
stage: test
image:
node:16
before_script:
- npm install jest
script:
npm run test
build_image:
stage: build
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_TLS_CERTDIR: "/certs"
before_script:
- docker login -u $REGISTRY_USER -p $REGISTRY_PASS
script:
- docker build -t $IMAGE_NAME:$IMAGE_TAG .
- docker push $IMAGE_NAME:$IMAGE_TAG
deploy:
stage: deploy
before_script:
- chmod 400 $SSH_KEY
script:
- ssh -o StrictHostKeyChecking=no -i $SSH_KEY root#159.89.175.212 "
docker login -u $REGISTRY_USER -p $REGISTRY_PASS &&
docker image prune -f &&
docker ps -aq | xargs docker stop | xargs docker rm &&
docker run -d -p 5001:5001 $IMAGE_NAME:$IMAGE_TAG"

The digital ocean server wasn't fetching the latest image from the repository so I added docker prune as additional step to do.
deploy:
stage: deploy
before_script:
- chmod 400 $SSH_KEY
script:
- ssh -o StrictHostKeyChecking=no -i $SSH_KEY root#159.89.175.212 "
docker login -u $REGISTRY_USER -p $REGISTRY_PASS &&
docker ps -aq | (xargs docker stop || true) | (xargs docker rm || true) &&
docker system prune -a -f &&
docker run -d -p 5001:5001 $IMAGE_NAME:$IMAGE_TAG"

Related

CI/CD script for build & deploy docker image in aws EC2

can I build ,push(to gitlab registry) and deploy the image (to aws EC2) using this CI/CD configuration?
stages:
- build
- deploy
build:
# Use the official docker image.
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
# Default branch leaves tag empty (= latest tag)
# All other branches are tagged with the escaped branch name (commit ref slug)
script:
- |
if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then
tag=""
echo "Running on default branch '$CI_DEFAULT_BRANCH': tag = 'latest'"
else
tag=":$CI_COMMIT_REF_SLUG"
echo "Running on branch '$CI_COMMIT_BRANCH': tag = $tag"
fi
- docker build --pull -t "$CI_REGISTRY_IMAGE${tag}" .
- docker push "$CI_REGISTRY_IMAGE${tag}"
# Run this job in a branch where a Dockerfile exists
deploy:
stage: deploy
before_script:
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
script:
- ssh -o StrictHostKeyChecking=no ubuntu#18.0.0.82 "sudo docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY; sudo docker pull $CI_REGISTRY_IMAGE${tag}; cd /home/crud_app; sudo docker-compose up -d"
after_script:
- sudo docker logout
rules:
- if: $CI_COMMIT_BRANCH
exists:
- Dockerfile
after the script build is getting suceed, deploy gets fail.
(build suceeded)
(deploy got failed)
the configuration must be build and deploy the image
There are a couple of errors, but the overall Pipeline seems good.
You cannot use ssh-add without having the agent running
Why you create the .ssh folder manually if afterwards you're explicitly ignoring the key that is going to be stored under known_hosts?
Using StrictHostKeyChecking=no is dangerous and totally unrecommended.
On the before_script add the following:
before_script:
- eval `ssh-agent`
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- ssh-keyscan -H 18.0.0.82 >> ~/.ssh/known_hosts
Also, don't use sudo on your ubuntu user, better add it to the docker group or connect through SSH to an user that is in the docker group.
In case you don't have already a docker group in your EC2 instance, now it's a good moment to configure it:
Access to your EC2 instance and create the docker group:
$ sudo groupadd docker
Add the ubuntu user to the docker group:
$ sudo usermod -aG docker ubuntu
Now change your script to:
script:
- echo $CI_REGISTRY_PASSWORD > docker_password
- scp docker_password ubuntu#18.0.0.82:~/tmp/docker_password
- ssh ubuntu#18.0.0.82 "cat ~/tmp/docker_password | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY; docker pull $CI_REGISTRY_IMAGE${tag}; cd /home/crud_app; docker-compose up -d; docker logout; rm -f ~/tmp/docker_password"
Also, remember that in the after_script you aren't in the EC2 instance but within the runner image so you don't need to logout, but it would be good to kill the SSH agent tho.
Final Job
deploy:
stage: deploy
before_script:
- eval `ssh-agent`
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- ssh-keyscan -H 18.0.0.82 >> ~/.ssh/known_hosts
script:
- echo $CI_REGISTRY_PASSWORD > docker_password
- scp docker_password ubuntu#18.0.0.82:~/tmp/docker_password
- ssh ubuntu#18.0.0.82 "cat ~/tmp/docker_password | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY; docker pull $CI_REGISTRY_IMAGE${tag}; cd /home/crud_app; docker-compose up -d; docker logout; rm -f ~/tmp/docker_password"
after_script:
- kill $SSH_AGENT_PID
- rm docker_password
rules:
- if: $CI_COMMIT_BRANCH
exists:
- Dockerfile

Trouble deploying multi-container docker application to heroku using travis

So really I have several problems here. This is my travis.yml file and this is the latest run
https://travis-ci.com/github/harryyy27/allies-art-club:
sudo: required
language: generic
services:
- docker
stages:
- dev
- prod
jobs:
include:
- stage: dev
if: NOT(branch=master)
scripts:
- docker build -t harryyy27/allies_art_club/frontend -f ./client/Dockerfile.dev ./client
- docker build -t harryyy27/allies_art_club/backend -f ./src/Dockerfile.dev ./src
- docker run -e CI=true harryyy27/allies_art_club/frontend npm test
- docker run -e CI=true harryyy27/allies_art_club/backend npm test
- stage: prod
if: branch=master
before_deploy:
- docker build -t harryyy27/aac-client ./client
- docker tag harryyy27/aac-client registry.heroku.com/$HEROKU_APP/client
- docker build -t harryyy27/aac-src ./src
- docker tag harryyy27/aac-src registry.heroku.com/$HEROKU_APP/src
- docker build -t harryyy27/aac-nginx ./nginx
- docker tag harryyy27/aac-nginx registry.heroku.com/$HEROKU_APP/nginx
# Log in to docker CLI
- echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_ID" --password-stdin
- curl https://cli-assets.heroku.com/install.sh | sh
- echo "$HEROKU_API" | docker login -u "$HEROKU_USERNAME" --password-stdin registry.heroku.com
deploy:
skip_cleanup: true
provider: script
script:
docker ps -a;
docker push harryyy27/aac-client;
docker push registry.heroku.com/$HEROKU_APP/client;
docker push harryyy27/aac-src;
docker push registry.heroku.com/$HEROKU_APP/src;
docker push harryyy27/aac-nginx;
docker push registry.heroku.com/$HEROKU_APP/nginx;
heroku container:release client src nginx --app $HEROKU_APP;
For some reason I cannot deploy to heroku. The docker push registry.heroku.com/$HEROKU_APP/container_name appears to work along with the echo "$HEROKU_API" | docker login -u "$HEROKU_USERNAME" --password-stdin registry.heroku.com sign in but then when I go to release the heroku containers says "Invalid credentials provided" in the terminal and tells me to login. Is there a way of releasing these containers using the docker CLI on Travis?
If not, would the Heroku CLI help?
So I eventually resolved this by simply changing the $HEROKU_API to $HEROKU_API_KEY. This is an environment variable that when present automatically logs you into the Heroku CLI enabling you to run the scripts required to upload to your docker containers to Heroku. This is the travis.yml I eventually ended up with
sudo: required
language: generic
services:
- docker
stages:
- dev
- prod
jobs:
include:
- stage: dev
if: NOT(branch=master)
scripts:
- docker build -t harryyy27/allies_art_club/frontend -f ./client/Dockerfile.dev ./client
- docker build -t harryyy27/allies_art_club/backend -f ./src/Dockerfile.dev ./src
- docker run -e CI=true harryyy27/allies_art_club/frontend npm test
- docker run -e CI=true harryyy27/allies_art_club/backend npm test
- stage: prod
if: branch=master
before_deploy:
- docker build -t harryyy27/aac-client ./client
- docker tag harryyy27/aac-client registry.heroku.com/$HEROKU_APP/client
- docker build -t harryyy27/aac-src ./src
- docker tag harryyy27/aac-src registry.heroku.com/$HEROKU_APP/src
- docker build -t harryyy27/aac-nginx ./nginx
- docker tag harryyy27/aac-nginx registry.heroku.com/$HEROKU_APP/nginx
# Log in to docker CLI
- echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_ID" --password-stdin
- curl https://cli-assets.heroku.com/install.sh | sh
- echo "$HEROKU_API_KEY" | docker login -u "$HEROKU_USERNAME" --password-stdin registry.heroku.com
- docker push harryyy27/aac-client;
- docker push registry.heroku.com/$HEROKU_APP/client;
- docker push harryyy27/aac-src;
- docker push registry.heroku.com/$HEROKU_APP/src;
- docker push harryyy27/aac-nginx;
- docker push registry.heroku.com/$HEROKU_APP/nginx;
deploy:
skip_cleanup: true
provider: script
script:
heroku container:login;
heroku container:release client src nginx --app $HEROKU_APP;
I do now have errors in Heroku though :P

gitlab-ci: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

I'm installing a personal runner for my projects.
sudo docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \ --non-interactive \ --executor "docker" \ --docker-image docker:stable \ --url "https://gitlab.com" \ --registration-token "my-token" \
--description "docker-runner privileged" \ --tag-list "docker,aws, gara" \ --run-untagged="true" \ --locked="false" \
--access-level="not_protected" \ --docker-privileged
with this runner i can successfully the step test, build and analysis of my maven projet.
For step docker build, I always have:
Cannot connect to the Docker daemon at tcp://docker:2375. Is the
docker daemon running? ERROR: Job failed: exit code 1
Full log:
Running with gitlab-runner 13.1.1 (6fbc7474) on docker-runner
privileged GYDNagVx Preparing the "docker" executor 00:21 Using Docker
executor with image docker:latest ... Starting service docker:dind ...
Pulling docker image docker:dind ... Using docker image
sha256:b3893e48cf281b58a763fb90904fd5d63595f9e5ae5736ee2d892a2dea6a371a
for docker:dind ... Waiting for services to be up and running...
Pulling docker image docker:latest ... Using docker image
sha256:809cc4dba987efb4641d5b6fef38f47abcde1bc5c5e9850616f0ed3552737e8c
for docker:latest ... Preparing environment 00:04 Running on
runner-gydnagvx-project-15477556-concurrent-0 via e59663817b22...
Getting source from Git repository 00:07 Fetching changes with git
depth set to 50... Reinitialized existing Git repository in
/builds/gara-project/back-end-micro-services/msearch/.git/ Checking
out 80fcdd6e as develop... Removing target/ Skipping Git submodules
setup Downloading artifacts 00:18 Downloading artifacts for
maven-build (788252737)... Downloading artifacts from coordinator...
ok id=788252737 responseStatus=200 OK token=jjUorX2D Executing
"step_script" stage of the job script 00:27 $ docker login -u
"${registry_user}" -p "${registry_pass}" "${registry_url}" WARNING!
Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in
/root/.docker/config.json. Configure a credential helper to remove
this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded $ docker build -t
"${registry_url}/${image_name}:${image_tag}" . Cannot connect to the
Docker daemon at tcp://docker:2375. Is the docker daemon running?
ERROR: Job failed: exit code 1
It seems to be a common error but none of solutions found over internet works for me.
My gitla-ci.yml file:
include:
- local: '/templates/.gitlab-ci-common-template.yml'
variables:
SPRING_PROFILES_ACTIVE: test
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
DOCKER_TLS_CERTDIR: ""
default:
image: maven:3.6.3-jdk-8-slim
cache:
key: "$CI_COMMIT_REF_NAME"
paths:
- .m2/repository/
- target/
stages:
- test
- build
- analysis
- docker
- scan
- deploy
- delivery
maven-test:
stage: test
extends:
- .ci_only
- .skip_build
script:
- mvn test
artifacts:
reports:
junit:
- target/surefire-reports/TEST-*.xml
maven-build:
stage: build
extends:
- .ci_only
- .skip_build
script:
- "./mvnw -Pint clean install -DskipTests=true"
artifacts:
paths:
- target/*
expire_in: 1 week
docker-build:
image: docker:latest
services:
- docker:19.03.0-dind
cache: {}
before_script:
- docker login -u "${registry_user}" -p "${registry_pass}" "${registry_url}"
stage: docker
dependencies:
- maven-build
extends:
- .skip_build
- .ci_only
script:
- docker build -t "${registry_url}/${image_name}:${image_tag}" .
- docker run -d --name ${image_tag} "${registry_url}/${image_name}:${image_tag}"
- sleep 60 && docker ps && docker logs ${image_tag}
- if [ $(docker inspect -f '{{.State.Running}}' ${image_tag}) = "true" ]; then echo "Application well started"; else exit 1; fi
- docker rm -f ${image_tag}
- docker push "${registry_url}/${image_name}:${image_tag}"
docker_release_image:
image: docker:latest
stage: delivery
before_script:
- docker login -u "${registry_user}" -p "${registry_pass}" "${registry_url}"
# - docker login -u "${registry_prod_user}" -p "${registry_prod_pass}" "${registry_prod_url}"
services:
- docker:19.03.0-dind
script:
- docker pull "${registry_url}/${image_name}:${image_tag}"
- docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_url}/${image_name}:latest"
# - docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_prod_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker push "${registry_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker push "${registry_url}/${image_name}:latest"
# - docker push "${registry_prod_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker rmi "${registry_url}/${image_name}:${image_tag}"
when: manual
extends:
- .only_tag_on_master
Please how could I correct it ?
Thanks

GitLab CI/CD: building multiarch Docker images

I want an easy way to build multiarch Docker images in a GitLab runner. By easy, I mean that I just would have to add a .gitlab-ci.yml in my project and it would work.
Here is the .gitlab-ci.yml that I wrote. It builds a multiarch image using buildx and then pushes it to the GitLab registry:
image: cl00e9ment/buildx
services:
- name: docker:dind
variables:
PLATFORMS: linux/amd64,linux/arm64
TAG: latest
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
build:
stage: build
script:
- docker buildx build --platform "$PLATFORMS" -t "${CI_REGISTRY_IMAGE}:${TAG}" . --push
The problem is that the linux/arm64 platform isn't available.
Here is how I built the cl00e9ment/buildx image (strongly inspired from snadn/docker-buildx):
Here is the Dockerfile:
FROM docker:latest
ENV DOCKER_CLI_EXPERIMENTAL=enabled
ENV DOCKER_HOST=tcp://docker:2375/
RUN mkdir -p ~/.docker/cli-plugins \
&& wget -qO- https://api.github.com/repos/docker/buildx/releases/latest | grep "browser_download_url.*linux-amd64" | cut -d : -f 2,3 | tr -d '"' | xargs wget -O ~/.docker/cli-plugins/docker-buildx \
&& chmod a+x ~/.docker/cli-plugins/docker-buildx
RUN docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
RUN docker context create buildx \
&& docker buildx create buildx --name mybuilder \
&& docker buildx use mybuilder
RUN docker buildx inspect --bootstrap
...add here is the .gitlab-ci.yml file used to build and push the cl00e9ment/buildx image:
image: docker:latest
services:
- name: docker:dind
before_script:
- docker login -u cl00e9ment -p "$DOCKER_HUB_TOKEN"
build:
stage: build
script:
- docker build --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host -t cl00e9ment/buildx .
- docker run --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host cl00e9ment/buildx docker buildx inspect --bootstrap
- docker push cl00e9ment/buildx
test:
stage: test
script:
- docker run --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host cl00e9ment/buildx docker buildx inspect --bootstrap
So what's happening?
At the end of the build, in the Dockerfile, I run docker buildx inspect --bootstrap to list the available platforms. It gives linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6. So it's all good.
After that, I run it again (just after the build and just before the push) and it still gives linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.
However, on the test stage, when the image is freshly downloaded from Docker Hub on a clean environment, it gives linux/amd64, linux/386.
Why?
There is a lot of outdated and incorrect information on building multiarch images on GitLab CI unfortunately. The seems to change quite frequently as it's still an experimental feature. But as of the time of this post, this is how I got my multiarch build working on GitLab public runners (armv6, armv6, arm64, amd64):
First, one must build and push a Docker image containing the buildx binary. Here is the Dockerfile I am using for that:
FROM docker:latest
ARG BUILDX_VER=0.4.2
RUN mkdir -p /root/.docker/cli-plugins && \
wget -qO ~/.docker/cli-plugins/docker-buildx \
https://github.com/docker/buildx/releases/download/v${BUILDX_VER}/buildx-v${BUILDX_VER}.linux-amd64 && \
chmod +x /root/.docker/cli-plugins/docker-buildx
The current GitLab runner image does not initialize the binfmt handlers correctly despite running the initialization code: https://gitlab.com/gitlab-org/gitlab-runner/-/blob/523854c8/.gitlab/ci/_common.gitlab-ci.yml#L91
So we have to do it in our pipeline. We refer to the comments in MR 1861 of the GitLab Runner code and add in the following magic sauce to our .gitlab-ci.yml:
before_script:
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Then we can run the rest of our pipeline script with docker login, docker buildx build --use, docker buildx build --push ... and so on.
Now the runner is ready to build for multiple architectures.
My final .gitlab-ci.yml can be seen here: https://github.com/oofnikj/nuttssh/blob/multiarch/.gitlab-ci.yml
Ok, I think I know whats going on here: you need to call update-binfmts --enable somewhere to enable the extra formats provided by binfmt_misc for .
I was able to get multiarch images working with buildx on gitlab-ci (after lots of searching) using this repo and its docker images: https://gitlab.com/ericvh/docker-buildx-qemu
However that repo has self dependency on its own docker image repository to build multiarch versions of itself AND it depends on a gitlab-ci template repo for its ci. I'm not super confident in how this web of dependency all began and the owner of that repo is far more skilled than me, but for my uses, I've forked the repo and I'm now trying to modify its CI to be less dependent on external sources.
EDIT: For people from the future this is the Dockerfile:
FROM debian
# Install Docker and qemu
# TODO Use docker stable once it properly supports buildx
RUN apt-get update && apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common && \
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - && \
add-apt-repository "deb https://download.docker.com/linux/debian $(lsb_release -cs) stable" && \
apt-get update && apt-get install -y \
docker-ce-cli \
binfmt-support \
qemu-user-static
# Install buildx plugin
RUN mkdir -p ~/.docker/cli-plugins && \
ARCH=`dpkg --print-architecture` && echo Running on $ARCH && curl -s https://api.github.com/repos/docker/buildx/releases/latest | \
grep "browser_download_url.*linux-$ARCH" | cut -d : -f 2,3 | tr -d \" | \
xargs curl -L -o ~/.docker/cli-plugins/docker-buildx && \
chmod a+x ~/.docker/cli-plugins/docker-buildx
# Write version file
RUN printf "$(docker --version | perl -pe 's/^.*\s(\d+\.\d+\.\d+.*),.*$/$1/')_$(docker buildx version | perl -pe 's/^.*v?(\d+\.\d+\.\d+).*$/$1/')" > /version && \
cat /version
And a stripped down version of .gitlab-ci.yml
build:
image: docker:dind
stage: build
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
# See https://github.com/docker-library/docker/pull/166
DOCKER_TLS_CERTDIR: ""
before_script:
- |
if [[ -z "$CI_COMMIT_TAG" ]]; then
export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG}
export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_SHA}
else
export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE}
export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_TAG}
fi
- echo "$CI_REGISTRY_PASSWORD" | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
script:
- docker build -t "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG" -t "$CI_APPLICATION_REPOSITORY:latest" .
- docker push "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
- docker push "$CI_APPLICATION_REPOSITORY:latest"
EDIT:
Further, I've found that this gitlabci configuration that uses the image built above can use the build cache:
stages:
- build
variables:
CI_BUILD_ARCHS: "linux/amd64,linux/arm/v6,linux/arm/v7"
CI_BUILD_IMAGE: "registry.gitlab.com/gdunstone/docker-buildx-qemu"
build_master:
image: $CI_BUILD_IMAGE
stage: build
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"]
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
# See https://github.com/docker-library/docker/pull/166
DOCKER_TLS_CERTDIR: ""
retry: 2
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
# Use docker-container driver to allow useful features (push/multi-platform)
- update-binfmts --enable # Important: Ensures execution of other binary formats is enabled in the kernel
- docker buildx create --driver docker-container --use
- docker buildx inspect --bootstrap
script:
- >
docker buildx build --platform $CI_BUILD_ARCHS
--cache-from=type=registry,ref=$CI_REGISTRY_IMAGE/cache:latest
--cache-to=type=registry,ref=$CI_REGISTRY_IMAGE/cache:latest
--progress plain
--pull --push
--build-arg CI_PROJECT_PATH=$CI_PROJECT_PATH
-t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
-t "$CI_REGISTRY_IMAGE:latest" .
only:
- master

Stop and rm old docker container and start new with Gitlab CI

Hi I'm using GitLab CI on my Spring Boot application
This is what I want to do :
- execute my test
- build the app
- dockerize it and remove the old container
Here is my .gitlab-ci.yml :
image: maven:latest
services:
- docker:dind
cache:
paths:
- .m2/repository
variables:
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
DOCKER_HOST: tcp://docker:2375
IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
IMAGE_TAG: ${CI_COMMIT_REF_SLUG}
stages:
- test
- build
- release
- deploy
test:
stage: test
script:
- mvn test
project-build:
stage: build
script:
- mvn clean package
release:
image: docker:git
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
stage: release
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
script:
- docker build --tag=$IMAGE_TAG . --pull -t $IMAGE_NAME
- docker push $IMAGE_NAME
only:
- master
deploy-staging:
stage: deploy
image: gitlab/dind:latest
cache: {}
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
before_script:
# add the server as a known host
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- eval "$(ssh-agent -s)"
- ssh-add ~/.ssh/id_rsa
- ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
script:
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}"
# stop container, remove image.
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker stop ${IMAGE_TAG}" || true
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker rm ${IMAGE_TAG}" || true
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker rmi -f ${IMAGE_NAME}" || true
# start new container
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker run --publish=8080:8080 -d ${IMAGE_NAME}"
only:
- master
But the docker stop and docker rm/rmi lines does not work. Here is the stack trace :
108 $ ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
109 # MY_SERVER_IP SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
110 # MY_SERVER_IP SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
111 $ ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}"
112 WARNING! Using --password via the CLI is insecure. Use --password-stdin.
113 WARNING! Your password will be stored unencrypted in /home/deployer/.docker/config.json.
114 Configure a credential helper to remove this warning. See
115 https://docs.docker.com/engine/reference/commandline/login/#credentials-store
116 Login Succeeded
117 $ ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker rm -f ${IMAGE_NAME} 2>/dev/null || exit 0"
118 $ ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker run --publish=8080:8080 -d ${IMAGE_NAME}"
119 0ff2aeeb0bf19b3c528dacaf2f8e6022587c1f3b4d845c1c583731eb76c1b65b
120 docker: Error response from daemon: driver failed programming external connectivity on endpoint pedantic_lamarr (96dd544514d257320160c15adb1b4c0f4a91c7423e20643ea901774d1528c4f1): Bind for 0.0.0.0:8080 failed: port is already allocated.
122
ln: failed to create symbolic link '/sys/fs/cgroup/systemd/name=systemd': Operation not permitted
So to sum up : the 3 first jobs works but if I have a container running in my server, the last one fail because I can't stop and remove the old container
Any idea ?

Resources