I want to deploy a docker stack on my own server. I've written a .gitlab-ci.yml file that currently builds the images in my stack and pushes them to my gitlab registry:
build:
stage: build
image: docker:stable
services:
- docker:dind
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker info
script:
- docker build -t $DOCKER_IMAGE1_TAG -f dir1/Dockerfile ./dir1
- docker push $DOCKER_IMAGE1_TAG
- docker build -t $DOCKER_IMAGE2_TAG -f dir2/Dockerfile ./dir2
- docker push $DOCKER_IMAGE2_TAG
I'm struggling for a way to run the docker deploy command on my own server with the docker-compose.yml file I've written, that successfully pulls the images from my gitlab registry. I figure I could use sshpass to ssh into my server and then copy the docker-compose.yml file across and run docker deploy from there, but I'm not sure what's the best way to allow my server to access the images now located in my gitlab registry:
# Need to ssh into the server, transfer over docker-stack file and run docker swarm deploy
deploy:
stage: deploy
environment:
name: production
image: trion/ng-cli-karma
before_script:
- apt-get update -qq && apt-get install -y -qq sshpass
- eval $(ssh-agent -s)
This is a section of my docker-compse file:
version: "3.2"
services:
octeditor:
image: image # how to set this to the image in my container registry?
ports:
- "3000:3000"
networks:
- front-tier
deploy:
replicas: 1
update_config:
parallelism: 1
failure_action: rollback
placement:
constraints:
- 'node.role == manager'
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
How can I pull the images from my gitlab registry? Is this the preferred way of creating a docker deployment on a remote server, via gitlab ci?
I equally had this difficulty recently, finally I found out that the solution is just to insert the link to the image in the private registry as is the case for me with gitlab.
version: "3.2"
services:
octeditor:
image: registry.gitlab.com/project-or-group/project-name/image-name:tag
ports:
- "3000:3000"
networks:
- front-tier
Related
I want to create CICD that builds docker image and then compose up this image. My runner is in container as a shell executor, but compose must be on host machine (win10).
my actual gitlab.yaml:
build-job:
image: docker:latest
services:
- docker
before_script:
- docker info
stage: build
script:
- cd ...
- cd ....
- docker-compose build
test-job1:
stage: test
script:
- echo "build worked!"
deploy-prod:
stage: deploy
script:
- cd ...
- cd ....
- docker-compose up
environment: production
and my docker-compose for my runner with auto-registration:
version: '3'
name: Worker
services:
register:
container_name: registration
image: gitlab/gitlab-runner
command:
- register
- --non-interactive
- --locked=false
- --name="...."
- --executor=shell
- --docker-volumes=/var/run/docker.sock:/var/run/docker.sock
- --docker-privileged=true
- --docker-volumes=/certs/client
volumes:
- gitlab-runner-config:/etc/gitlab-runner
- /var/run/docker.sock:/var/run/docker.sock
tty: true
stdin_open: true
restart: "no"
environment:
- CI_SERVER_URL=....
- REGISTRATION_TOKEN=....
labels:
- "traefik.enable=false"
worker:
container_name: ....
image: gitlab/gitlab-runner
volumes:
- /usr/bin/docker:/usr/bin/docker
- gitlab-runner-data:/etc/gitlab-runner
- gitlab-runner-data:/home/gitlab-runner
- gitlab-runner-config:/etc/gitlab-runner
restart: always
volumes:
gitlab-runner-config:
external: true
gitlab-runner-data:
external: true
at the end i get my best result - Cannot connect to the Docker daemon at unix:///var/run/docker.sock (at docker info)
i've tried with //usr or giving full $PATH from env in windows 10,
network_mode changed to "host"
a lot of changes with volumes in compose of registration and runner
How could i get it working or at least enable usage of docker commands?
I have an issue with gitlab runner using docker:dind service.
I'm trying to run a docker-compose file with simple volume on a job, here the job :
test_e2e:
image: tmaier/docker-compose
stage: test
services:
- docker:dind
variables:
GIT_STRATEGY: none
GIT_CHECKOUT: "false"
DOCKER_DRIVER: overlay2
before_script:
- ls
script:
- cp .env.dist .env
- docker-compose -f docker-compose.yml -f docker-compose-ci.yml up -d
The job start normally but a container in docker-compose-ci.yml doesn't seem to mount the volume as specified in it, here docker-compose-ci.yml
version: '3.3'
services:
wait_app:
image: dadarek/wait-for-dependencies
networks:
- internal
depends_on:
- traefik
- webapp
command: webapp:3000
cypress:
# the Docker image to use from https://github.com/cypress-io/cypress-docker-images
image: "cypress/included:6.5.0"
networks:
- internal
depends_on:
- traefik
- webapp
- api
- mysql
- redis
environment:
# pass base url to test pointing at the web application
- CYPRESS_baseUrl=http://app.localhost:3000
working_dir: /cypress
volumes:
- ./cypress/:/cypress
Here if I make an "docker exec app_cypress_1 sh -c "ls -al" || 1" of /cypress folder inside the container cypress, I will have nothing even though I do have files in there on the host.
But I tried on a different version of the runner 13.7.0 instead of 13.5.0, and it work as expected.
Where could be the issue ? Is it the gitlab runner are maybe there is another parameter that I can change to make it work ?
Thank you
I am trying to make sure my docker work or not in my Jenkins,
I am running Jenkins in docker and it was running but when I check in Jenkins Pipeline, it said docker: not found
here is my docker-compose.yml
version: '3.7'
services:
jenkins:
image: jenkinsci/blueocean:latest
user: root
privileged: true
restart: always
ports:
- 8080:8080
volumes:
- ./jenkins_home:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
- /usr/bin/docker:/usr/bin/docker
registry:
image: registry
container_name: registry
restart: always
ports:
- 5000:5000
then I run sudo docker-compose up -d
then the Jenkins is running,
can I know why the docker not found ? is my docker-compose wrong ?
You do not need to bind - /usr/bin/docker:/usr/bin/docker, as - /var/run/docker.sock:/var/run/docker.sock is engough to interact with host docker. you should not bind executable with docker container
remove this from the compose file and it should work.
- /usr/bin/docker:/usr/bin/docker
I have below docker-compose.yml
version: "2"
services:
api:
build:
context: .
dockerfile: ./build/dev/Dockerfile
container_name: "project-api"
volumes:
# 1. mount your workdir path
- .:/app
depends_on:
- mongodb
links:
- mongodb
- mysql
nginx:
image: nginx:1.10.3
container_name: "project-nginx"
ports:
- 80:80
restart: always
volumes:
- ./build/dev/nginx.conf:/etc/nginx/conf.d/default.conf
- .:/app
links:
- api
depends_on:
- api
mongodb:
container_name: "project-mongodb"
image: mongo:latest
environment:
- MONGO_DATA_DIR=/data/db
- MONGO_LOG_DIR=/dev/null
ports:
- "27018:27017"
command: mongod --smallfiles --logpath=/dev/null # --quiet
mysql:
container_name: "gamestore-mysql"
image: mysql:5.7.23
ports:
- "3306:3306"
environment:
MYSQL_DATABASE: project_test
MYSQL_USER: user
MYSQL_PASSWORD: user
MYSQL_ROOT_PASSWORD: root
And below .gitlab-ci.yml
test:
stage: test
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay2
before_script:
- apk add --no-cache py-pip
- pip install docker-compose
script:
- docker-compose up -d
- docker-compose exec -T api ls -la
- docker-compose exec -T api composer install
- docker-compose exec -T api php core/init --env=Development --overwrite=y
- docker-compose exec -T api vendor/bin/codecept -c core/common run
- docker-compose exec -T api vendor/bin/codecept -c core/rest run
When i running my gitlab pipeline it's become field because i think gitlab can't work with services runned by docker-compose.
The error says that mysql refuse the connection.
I need this connection because my test written by codeception will test my models and api actions.
I want test my branches every time any one push in them and if pass just in develop deploy into test server and in master deploy on production server.
What is best way to run my test in gitlab ci/cd and then deploy them in my server?
You should use GitLab CI services instead of docker-compose.
You have to pick one image as your main, in which your commands will be run, and other containers just as services.
Sadly CI services cannot have mounted files in gitlab, you have to be able to configure them with env variables, or you need to create you own image with files in it (you can do that CI stage)
I would suggest you to don't use nginx, and use built-in php server for tests. It that's not possible (you have spicifix nginx config), you will need to build yourself nginx image with copied files inside it.
Also for PHP (the api service in docker-compose.yaml i assume), you need to either build the image ahed or copy command from your dockerfile to script.
So the result should be something like:
test:
stage: test
image: custom-php-image #build from ./build/dev/Dockerfile
services:
- name: mysql:5.7.23
alias: gamestore-mysql
- name: mongo:latest
alias: project-mongodb
command: mongod --smallfiles --logpath=/dev/null
variables:
MYSQL_DATABASE: project_test
MYSQL_USER: user
MYSQL_PASSWORD: user
MYSQL_ROOT_PASSWORD: root
MONGO_DATA_DIR: /data/db
MONGO_LOG_DIR: /dev/null
script:
- api ls -la
- composer install
- php core/init --env=Development --overwrite=y
- php -S localhost:8000 # You need to configure your built-in php server probably here
- vendor/bin/codecept -c core/common run
- vendor/bin/codecept -c core/rest run
I don't know your app, so you will probably have to made some tweaks.
More on that:
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-image-and-services-from-gitlab-ciyml
https://docs.gitlab.com/ee/ci/services/
http://php.net/manual/en/features.commandline.webserver.php
The problem
I have made a project with docker compose. It works well on localhost. I want to use this base to test or analyze code with Gitlab Runner. I solved a lot of problems, like install docker compose, run and build selected containers and run commands in container. The first job ran and success (!!!), but the following jobs failed before "before_script":
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
...
Error response from daemon: Conflict.
...
Error response from daemon: Conflict.
I don't understand why. What do I do wrong? I repeat: the first job of the pipeline runs well with "success" message! Each other jobs of the pipeline fail.
Full output:
Running with gitlab-ci-multi-runner 9.4.0 (ef0b1a6)
on XXX Runner (fdc0d656)
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:5096e5a0cba00693905879b09e24a487dc244b56e8e15349fd5b71b432c6ec9ffor docker service...
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Will be retried in 3s ...
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:5096e5a0cba00693905879b09e24a487dc244b56e8e15349fd5b71b432c6ec9f for docker service...
ERROR: Preparation failed: Error response from daemon: Conflict. The container name "/runner-fdc0d656-project-35-concurrent-0-docker" is already in use by container "80918876ffe53e33ce1f069e6e545f03a15469af6596852457f11dbc7a6c5b58". You have to remove (or rename) that container to be able to reuse that name.
Will be retried in 3s ...
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:5096e5a0cba00693905879b09e24a487dc244b56e8e15349fd5b71b432c6ec9f for docker service...
ERROR: Preparation failed: Error response from daemon: Conflict. The container name "/runner-fdc0d656-project-35-concurrent-0-docker" is already in use by container "80918876ffe53e33ce1f069e6e545f03a15469af6596852457f11dbc7a6c5b58". You have to remove (or rename) that container to be able to reuse that name.
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: Conflict. The container name "/runner-fdc0d656-project-35-concurrent-0-docker" is already in use by container "80918876ffe53e33ce1f069e6e545f03a15469af6596852457f11dbc7a6c5b58". You have to remove (or rename) that container to be able to reuse that name.
Files
.gitlab-ci.yml
# Select image from https://hub.docker.com/r/_/php/
image: docker:latest
# Services
services:
- docker:dind
stages:
- build
- test
- deploy
cache:
key: ${CI_BUILD_REF_NAME}
untracked: true
paths:
- vendor
- var
variables:
DOCKER_CMD: docker exec --user user bin
COMPOSE_HTTP_TIMEOUT: 300
before_script:
- apk add --no-cache py-pip bash
- pip install docker-compose
- touch ~/.gitignore
- bin/docker-init.sh
- cp app/config/parameters.gitlab-ci.yml app/config/parameters.yml
- cp app/config/nodejs_parameters.yml.dist app/config/nodejs_paramteres.yml
- chmod -R 777 app/cache app/logs var
# Load only binary and mysql
- docker-compose up -d binary mysql
build:
stage: build
script:
- ${DOCKER_CMD} composer install -n
- ${DOCKER_CMD} php app/console doctrine:database:create --env=test --if-not-exists
- ${DOCKER_CMD} php app/console doctrine:migrations:migrate --env=test
codeSniffer:
stage: test
script:
- ${DOCKER_CMD} bin/php-cs-fixer fix --dry-run --config-file=.php_cs
database:
stage: test
script:
- ${DOCKER_CMD} php app/console doctrine:mapping:info --env=test
- ${DOCKER_CMD} php app/console doctrine:schema:validate --env=test
- ${DOCKER_CMD} php app/console doctrine:fixtures:load --env=test
unittest:
stage: test
script:
- ${DOCKER_CMD} bin/phpunit -c app --debug
deploy_demo:
stage: deploy
script:
- echo "Deploy to staging server"
environment:
name: staging
url: https://staging.example.com
only:
- develop
deploy_prod:
stage: deploy
script:
- echo "Deploy to production server"
environment:
name: production
url: https://example.com
when: manual
only:
- master
docker-compose.yml
version: "2"
services:
web:
image: nginx:latest
ports:
- "${HTTP_PORT}:80"
depends_on:
- mysql
- elasticsearch
- binary
links:
- binary:php
volumes:
- ".:/var/www"
- "./app/config/docker/vhost.conf:/etc/nginx/conf.d/site.conf"
- "${BASE_LOG_DIR}/nginx:/var/log/nginx"
mysql:
image: mysql:5.6
environment:
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: test
ports:
- "${MYSQL_PORT}:3306"
volumes:
- "${BASE_LOG_DIR}/mysql:/var/log/mysql"
- "${BASE_MYSQL_DATA_DIR}:/var/lib/mysql"
- "./app/config/docker/mysql.cnf:/etc/mysql/conf.d/mysql.cnf"
elasticsearch:
image: elasticsearch:1.7.6
ports:
- "${ELASTICSEARCH_PORT}:9200"
volumes:
- "${BASE_ELASTICSEARCH_DATA_DIR}:/usr/share/elasticsearch/data"
binary:
image: fchris82/kunstmaan-test
container_name: bin
volumes:
- ".:/var/www"
- "${BASE_LOG_DIR}/php:/var/log/php"
- "~/.ssh:/home/user/.ssh"
tty: true
environment:
LOCAL_USER_ID: ${LOCAL_USER_ID}
config.toml
[[runners]]
name = "XXX Runner"
url = "https://gitlab.xxx.xx/"
token = "xxxxxxxxxxx"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = true
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
[runners.cache]
OK, I found the problem. I spoilt the configuration. If you use dind service in .gitlab-ci.yml then don't use /var/run/docker.sock volume in config.toml file OR vica versa if you use "socket" method, don't use the dind service.
More informations: https://docs.gitlab.com/ce/ci/docker/using_docker_build.html