Stop and rm old docker container and start new with Gitlab CI - docker

Hi I'm using GitLab CI on my Spring Boot application
This is what I want to do :
- execute my test
- build the app
- dockerize it and remove the old container
Here is my .gitlab-ci.yml :
image: maven:latest
services:
- docker:dind
cache:
paths:
- .m2/repository
variables:
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
DOCKER_HOST: tcp://docker:2375
IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
IMAGE_TAG: ${CI_COMMIT_REF_SLUG}
stages:
- test
- build
- release
- deploy
test:
stage: test
script:
- mvn test
project-build:
stage: build
script:
- mvn clean package
release:
image: docker:git
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
stage: release
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
script:
- docker build --tag=$IMAGE_TAG . --pull -t $IMAGE_NAME
- docker push $IMAGE_NAME
only:
- master
deploy-staging:
stage: deploy
image: gitlab/dind:latest
cache: {}
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
before_script:
# add the server as a known host
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- eval "$(ssh-agent -s)"
- ssh-add ~/.ssh/id_rsa
- ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
script:
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}"
# stop container, remove image.
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker stop ${IMAGE_TAG}" || true
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker rm ${IMAGE_TAG}" || true
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker rmi -f ${IMAGE_NAME}" || true
# start new container
- ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker run --publish=8080:8080 -d ${IMAGE_NAME}"
only:
- master
But the docker stop and docker rm/rmi lines does not work. Here is the stack trace :
108 $ ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
109 # MY_SERVER_IP SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
110 # MY_SERVER_IP SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
111 $ ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}"
112 WARNING! Using --password via the CLI is insecure. Use --password-stdin.
113 WARNING! Your password will be stored unencrypted in /home/deployer/.docker/config.json.
114 Configure a credential helper to remove this warning. See
115 https://docs.docker.com/engine/reference/commandline/login/#credentials-store
116 Login Succeeded
117 $ ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker rm -f ${IMAGE_NAME} 2>/dev/null || exit 0"
118 $ ssh $DEPLOYER_USER#$DEPLOYMENT_SERVER_IP "docker run --publish=8080:8080 -d ${IMAGE_NAME}"
119 0ff2aeeb0bf19b3c528dacaf2f8e6022587c1f3b4d845c1c583731eb76c1b65b
120 docker: Error response from daemon: driver failed programming external connectivity on endpoint pedantic_lamarr (96dd544514d257320160c15adb1b4c0f4a91c7423e20643ea901774d1528c4f1): Bind for 0.0.0.0:8080 failed: port is already allocated.
122
ln: failed to create symbolic link '/sys/fs/cgroup/systemd/name=systemd': Operation not permitted
So to sum up : the 3 first jobs works but if I have a container running in my server, the last one fail because I can't stop and remove the old container
Any idea ?

Related

GitLab CI/CD not taking latest code changes

So I have used GitLab CI/CD to deploy changes to private docker hub repo and using Digital Ocean droplet to run the server using docker but the changes are not being reflected in the docker container running on digital ocean. Here's the config file.
variables:
IMAGE_NAME: codelyzer/test-repo
IMAGE_TAG: test-app-1.0
stages:
- test
- build
- deploy
run_tests:
stage: test
image:
node:16
before_script:
- npm install jest
script:
npm run test
build_image:
stage: build
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_TLS_CERTDIR: "/certs"
before_script:
- docker login -u $REGISTRY_USER -p $REGISTRY_PASS
script:
- docker build -t $IMAGE_NAME:$IMAGE_TAG .
- docker push $IMAGE_NAME:$IMAGE_TAG
deploy:
stage: deploy
before_script:
- chmod 400 $SSH_KEY
script:
- ssh -o StrictHostKeyChecking=no -i $SSH_KEY root#159.89.175.212 "
docker login -u $REGISTRY_USER -p $REGISTRY_PASS &&
docker image prune -f &&
docker ps -aq | xargs docker stop | xargs docker rm &&
docker run -d -p 5001:5001 $IMAGE_NAME:$IMAGE_TAG"
The digital ocean server wasn't fetching the latest image from the repository so I added docker prune as additional step to do.
deploy:
stage: deploy
before_script:
- chmod 400 $SSH_KEY
script:
- ssh -o StrictHostKeyChecking=no -i $SSH_KEY root#159.89.175.212 "
docker login -u $REGISTRY_USER -p $REGISTRY_PASS &&
docker ps -aq | (xargs docker stop || true) | (xargs docker rm || true) &&
docker system prune -a -f &&
docker run -d -p 5001:5001 $IMAGE_NAME:$IMAGE_TAG"

How to pass environment variable to docker run in gitlab ci cd

I am trying to pass the env variable to my node js docker build image ,while running as shown below
stages:
- publish
- deploy
variables:
TAG_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest
TAG_COMMIT: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA
publish:
image: docker:latest
stage: publish
services:
- docker:dind
script:
- touch env.txt
- docker build -t $TAG_COMMIT -t $TAG_LATEST .
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- docker push $TAG_COMMIT
- docker push $TAG_LATEST
deploy:
image: alpine:latest
stage: deploy
tags:
- deployment
script:
- chmod og= $ID_RSA
- apk update && apk add openssh-client
- echo "AWS_ACCESS_KEY_ID"=$AWS_ACCESS_KEY_ID >> "env.txt"
- echo "AWS_S3_BUCKET"=$AWS_S3_BUCKET >> "env.txt"
- echo "AWS_S3_REGION"=$AWS_S3_REGION >> "env.txt"
- echo "AWS_SECRET_ACCESS_KEY"=$AWS_SECRET_ACCESS_KEY >> "env.txt"
- echo "DB_URL"=$DB_URL >> "env.txt"
- echo "JWT_EXPIRES_IN"=$JWT_EXPIRES_IN >> "env.txt"
- echo "OTP_EXPIRE_TIME_SECONDS"=$OTP_EXPIRE_TIME_SECONDS >> "env.txt"
- echo "TWILIO_ACCOUNT_SID"=$TWILIO_ACCOUNT_SID >> "env.txt"
- echo "TWILIO_AUTH_TOKEN"=$TWILIO_AUTH_TOKEN >> "env.txt"
- echo "TWILLIO_SENDER"=$TWILLIO_SENDER >> "env.txt"
- ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER#$SERVER_IP "docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY"
- ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER#$SERVER_IP "docker pull $TAG_COMMIT"
- ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER#$SERVER_IP "docker container rm -f my-app || true"
- ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER#$SERVER_IP "docker run --env-file env.txt -d -p 8080:8080 --name my-app $TAG_COMMIT"
environment:
name: development
url: 90900
only:
- master
I am running this command docker run --env-file env.txt ,but it gives me an error docker: open env.txt: no such file or directory.
How Can I solve the issue ,to pass multiple variables in my docker run command
Which job is failing? In your deploy job, you are creating the env.txt locally and using SSH to do the docker building, but you never scp your local env.txt to $SERVER_USER#$SERVER_ID for the remote process to pick it up.
I had the same issue using Gitlab ci/cd. i.e. Trying to inject env vars that were referenced in the project .env file via the runner (docker executor) into the output docker container.
We don't want to commit any sensitive info into git so one option is to save them on the server in a file and include via the --env-file flag but Gitlab runner creates a new container for every run so not possible to use this as the host server running the yaml script is ephemeral and not the actual server that Gitlab runner was installed onto.
The suggestion by #dmoonfire to scp the file over sounded like a good solution but I couldn't get it to work to copy a file from external to the gitlab runner. I'd need to copy the public key from the executor to the gitlab runner server but the docker executor is ephemeral.
I found the simplest solution to use the Gitlab CI/CD variable settings. It's possible to mask variables and restrict to protected branches or protected tags etc. These get injected into the container so that your .env file can access.

CI/CD script for build & deploy docker image in aws EC2

can I build ,push(to gitlab registry) and deploy the image (to aws EC2) using this CI/CD configuration?
stages:
- build
- deploy
build:
# Use the official docker image.
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
# Default branch leaves tag empty (= latest tag)
# All other branches are tagged with the escaped branch name (commit ref slug)
script:
- |
if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then
tag=""
echo "Running on default branch '$CI_DEFAULT_BRANCH': tag = 'latest'"
else
tag=":$CI_COMMIT_REF_SLUG"
echo "Running on branch '$CI_COMMIT_BRANCH': tag = $tag"
fi
- docker build --pull -t "$CI_REGISTRY_IMAGE${tag}" .
- docker push "$CI_REGISTRY_IMAGE${tag}"
# Run this job in a branch where a Dockerfile exists
deploy:
stage: deploy
before_script:
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
script:
- ssh -o StrictHostKeyChecking=no ubuntu#18.0.0.82 "sudo docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY; sudo docker pull $CI_REGISTRY_IMAGE${tag}; cd /home/crud_app; sudo docker-compose up -d"
after_script:
- sudo docker logout
rules:
- if: $CI_COMMIT_BRANCH
exists:
- Dockerfile
after the script build is getting suceed, deploy gets fail.
(build suceeded)
(deploy got failed)
the configuration must be build and deploy the image
There are a couple of errors, but the overall Pipeline seems good.
You cannot use ssh-add without having the agent running
Why you create the .ssh folder manually if afterwards you're explicitly ignoring the key that is going to be stored under known_hosts?
Using StrictHostKeyChecking=no is dangerous and totally unrecommended.
On the before_script add the following:
before_script:
- eval `ssh-agent`
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- ssh-keyscan -H 18.0.0.82 >> ~/.ssh/known_hosts
Also, don't use sudo on your ubuntu user, better add it to the docker group or connect through SSH to an user that is in the docker group.
In case you don't have already a docker group in your EC2 instance, now it's a good moment to configure it:
Access to your EC2 instance and create the docker group:
$ sudo groupadd docker
Add the ubuntu user to the docker group:
$ sudo usermod -aG docker ubuntu
Now change your script to:
script:
- echo $CI_REGISTRY_PASSWORD > docker_password
- scp docker_password ubuntu#18.0.0.82:~/tmp/docker_password
- ssh ubuntu#18.0.0.82 "cat ~/tmp/docker_password | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY; docker pull $CI_REGISTRY_IMAGE${tag}; cd /home/crud_app; docker-compose up -d; docker logout; rm -f ~/tmp/docker_password"
Also, remember that in the after_script you aren't in the EC2 instance but within the runner image so you don't need to logout, but it would be good to kill the SSH agent tho.
Final Job
deploy:
stage: deploy
before_script:
- eval `ssh-agent`
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- ssh-keyscan -H 18.0.0.82 >> ~/.ssh/known_hosts
script:
- echo $CI_REGISTRY_PASSWORD > docker_password
- scp docker_password ubuntu#18.0.0.82:~/tmp/docker_password
- ssh ubuntu#18.0.0.82 "cat ~/tmp/docker_password | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY; docker pull $CI_REGISTRY_IMAGE${tag}; cd /home/crud_app; docker-compose up -d; docker logout; rm -f ~/tmp/docker_password"
after_script:
- kill $SSH_AGENT_PID
- rm docker_password
rules:
- if: $CI_COMMIT_BRANCH
exists:
- Dockerfile

gitlab-ci: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

I'm installing a personal runner for my projects.
sudo docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \ --non-interactive \ --executor "docker" \ --docker-image docker:stable \ --url "https://gitlab.com" \ --registration-token "my-token" \
--description "docker-runner privileged" \ --tag-list "docker,aws, gara" \ --run-untagged="true" \ --locked="false" \
--access-level="not_protected" \ --docker-privileged
with this runner i can successfully the step test, build and analysis of my maven projet.
For step docker build, I always have:
Cannot connect to the Docker daemon at tcp://docker:2375. Is the
docker daemon running? ERROR: Job failed: exit code 1
Full log:
Running with gitlab-runner 13.1.1 (6fbc7474) on docker-runner
privileged GYDNagVx Preparing the "docker" executor 00:21 Using Docker
executor with image docker:latest ... Starting service docker:dind ...
Pulling docker image docker:dind ... Using docker image
sha256:b3893e48cf281b58a763fb90904fd5d63595f9e5ae5736ee2d892a2dea6a371a
for docker:dind ... Waiting for services to be up and running...
Pulling docker image docker:latest ... Using docker image
sha256:809cc4dba987efb4641d5b6fef38f47abcde1bc5c5e9850616f0ed3552737e8c
for docker:latest ... Preparing environment 00:04 Running on
runner-gydnagvx-project-15477556-concurrent-0 via e59663817b22...
Getting source from Git repository 00:07 Fetching changes with git
depth set to 50... Reinitialized existing Git repository in
/builds/gara-project/back-end-micro-services/msearch/.git/ Checking
out 80fcdd6e as develop... Removing target/ Skipping Git submodules
setup Downloading artifacts 00:18 Downloading artifacts for
maven-build (788252737)... Downloading artifacts from coordinator...
ok id=788252737 responseStatus=200 OK token=jjUorX2D Executing
"step_script" stage of the job script 00:27 $ docker login -u
"${registry_user}" -p "${registry_pass}" "${registry_url}" WARNING!
Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in
/root/.docker/config.json. Configure a credential helper to remove
this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded $ docker build -t
"${registry_url}/${image_name}:${image_tag}" . Cannot connect to the
Docker daemon at tcp://docker:2375. Is the docker daemon running?
ERROR: Job failed: exit code 1
It seems to be a common error but none of solutions found over internet works for me.
My gitla-ci.yml file:
include:
- local: '/templates/.gitlab-ci-common-template.yml'
variables:
SPRING_PROFILES_ACTIVE: test
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
DOCKER_TLS_CERTDIR: ""
default:
image: maven:3.6.3-jdk-8-slim
cache:
key: "$CI_COMMIT_REF_NAME"
paths:
- .m2/repository/
- target/
stages:
- test
- build
- analysis
- docker
- scan
- deploy
- delivery
maven-test:
stage: test
extends:
- .ci_only
- .skip_build
script:
- mvn test
artifacts:
reports:
junit:
- target/surefire-reports/TEST-*.xml
maven-build:
stage: build
extends:
- .ci_only
- .skip_build
script:
- "./mvnw -Pint clean install -DskipTests=true"
artifacts:
paths:
- target/*
expire_in: 1 week
docker-build:
image: docker:latest
services:
- docker:19.03.0-dind
cache: {}
before_script:
- docker login -u "${registry_user}" -p "${registry_pass}" "${registry_url}"
stage: docker
dependencies:
- maven-build
extends:
- .skip_build
- .ci_only
script:
- docker build -t "${registry_url}/${image_name}:${image_tag}" .
- docker run -d --name ${image_tag} "${registry_url}/${image_name}:${image_tag}"
- sleep 60 && docker ps && docker logs ${image_tag}
- if [ $(docker inspect -f '{{.State.Running}}' ${image_tag}) = "true" ]; then echo "Application well started"; else exit 1; fi
- docker rm -f ${image_tag}
- docker push "${registry_url}/${image_name}:${image_tag}"
docker_release_image:
image: docker:latest
stage: delivery
before_script:
- docker login -u "${registry_user}" -p "${registry_pass}" "${registry_url}"
# - docker login -u "${registry_prod_user}" -p "${registry_prod_pass}" "${registry_prod_url}"
services:
- docker:19.03.0-dind
script:
- docker pull "${registry_url}/${image_name}:${image_tag}"
- docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_url}/${image_name}:latest"
# - docker tag "${registry_url}/${image_name}:${image_tag}" "${registry_prod_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker push "${registry_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker push "${registry_url}/${image_name}:latest"
# - docker push "${registry_prod_url}/${image_name}:${CI_COMMIT_TAG/*v/}"
- docker rmi "${registry_url}/${image_name}:${image_tag}"
when: manual
extends:
- .only_tag_on_master
Please how could I correct it ?
Thanks

Gitlab-Runner obviously creates container, but this container is not in list

I have installes gitlab-runner on my staging-server and in my project I have this gitlab-ci.yml:
image: upmcedlp/gitlab-dind-node-java
services:
- docker:dind
stages:
- ver
- init
- tests
- deploy
before_script:
- echo "---------- DOCKER LOGIN"
- docker login -u myusername -p "mypwd" registry.gitlab.com
ver:
stage: ver
script:
- whoami
init:
stage: init
script:
- npm install
testing:
stage: tests
script:
- npm test
deploy_staging:
stage: deploy
script:
- echo "---------- START DEPLOYING STAGING SERVER"
- echo "-> 1) build image"
- docker build -t registry.gitlab.com/admiralcrunch/portal_ci .
- echo "-> 2) push image to registry"
- docker push registry.gitlab.com/admiralcrunch/portal_ci
- echo "-> 3) kill old container"
- docker kill $(docker ps -q) || true
- docker rm $(docker ps -a -q) || true
- echo "-> 4) start new container"
- docker run -dt -p 3000:3000 --name portal registry.gitlab.com/admiralcrunch/portal_ci
- echo "########## END DEPLOYING DOCKER IMAGE"
environment:
name: staging
url: https://stage.latronic.com
only:
- master
deploy_production_1:
stage: deploy
script:
- echo "Deployed to production server 1"
environment:
when: manual
only:
- master
deploy_production_2:
stage: deploy
script:
- echo "Deployed to production server 2"
environment:
when: manual
only:
- master
It is running and I can see how - in the deploy_staging the processes got killed and the docker run -dt -p 3000:3000 --name portal admiralcrunch/portal_ci gots executed without no error. Also I can see the "worker"-container that the gitlab-runner creates on the staging-server.
but when all is finished, there is absolutely no container running(?). Why is that? What am I doing wrong? did I miss sometging?

Resources