Multiple docker-compose files with gitlab shell runner - docker

Question

I am trying to maintain dev, test and master environments on the same host by using Gitlab CI.
.gitlab-ci.yml:
workflow:
rules:
- if: $CI_COMMIT_MESSAGE =~ /-no-ci$/
when: never
- if: $CI_COMMIT_BRANCH == "master"
- if: $CI_COMMIT_BRANCH == "test"
- if: $CI_COMMIT_BRANCH == "dev"
before_script:
- cp /home/gitlab-runner/my-app/.env.$CI_COMMIT_BRANCH ./.env.$CI_COMMIT_BRANCH
- echo '' >> ./.env
- echo 'export CI_COMMIT_BRANCH="${CI_COMMIT_BRANCH}"' >> ./.env.$CI_COMMIT_BRANCH
- "cat .env.$CI_COMMIT_BRANCH | envsubst > .env"
- source .env
- "cat Dockerfile.template | envsubst > Dockerfile"
- rm .env.$CI_COMMIT_BRANCH Dockerfile.template
build-master:
stage: build
script:
- sudo docker-compose build
- sudo docker-compose up -d
only:
- master
tags:
- master
build-test:
stage: build
script:
- sudo docker-compose build
- sudo docker-compose up -d
only:
- test
tags:
- test
build-dev:
stage: build
script:
- sudo docker-compose build
- sudo docker-compose up -d
only:
- dev
tags:
- dev
Since I have to start three containers (webserver, db and pgadmin), I use docker-compose.yml:
version: '3.1'
services:
webserver:
build:
context: ./
restart: always
image: my-app/${CI_COMMIT_BRANCH}
container_name: my-app-webserver-${CI_COMMIT_BRANCH}
volumes:
- ${PATH}:/app/${PATH}
ports:
- ${WEBSERVER_PORT}:${WEBSERVER_PORT}
depends_on:
- db
db:
image: postgres:12.2
restart: always
container_name: my-app-db-${CI_COMMIT_BRANCH}
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- ${HOST_DB_DATA}:/var/lib/postgresql/data
pgadmin:
image: dpage/pgadmin4:4.18
restart: always
container_name: my-app-pgadmin-${CI_COMMIT_BRANCH}
environment:
PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL}
PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD}
PGADMIN_LISTEN_PORT: 80
ports:
- ${PGADMIN_PORT}:80
volumes:
- ${PGADMIN_DATA}:/var/lib/pgadmin/
links:
- "db:pgsql-server"
depends_on:
- db
networks:
default:
name: my-app-network-${CI_COMMIT_BRANCH}
On the host I have /etc/gitlab-runner/config.toml:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "my-app-shell-prod"
url = "https://gitlab.my-app.com/"
token = "abcdefgabcdefg"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
privileged = true
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[[runners]]
name = "my-app-shell-test"
url = "https://gitlab.my-app.com/"
token = "abcdefgabcdefg"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
privileged = true
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[[runners]]
name = "my-app-shell-dev"
url = "https://gitlab.my-app.com/"
token = "abcdefgabcdefg"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
privileged = true
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
My initial thought was to use three different shell runners with tags master, test and dev to maintain independent builds, for each having a corresponding docker-compose.yml.
When I push to dev, pipeline is passed successfully and I have three containers with names ...-dev running. However, if I merge dev to master, another pipeline is passed, but now I have only three ...-master containers running (but in the same time job logs show that -dev have been restarted).
...
134 $ sudo docker-compose up -d
135 Recreating my-app-db-dev ...
136 Recreating my-app-db-dev ... done
137 Recreating my-app-webserver-dev ...
138 Recreating my-app-pgadmin-dev ...
139 Recreating my-app-pgadmin-dev ... done
140 Recreating my-app-webserver-dev ... done
I found an article on overriding docker-compose.yml files. However, when I try to use -f in .gitlab-ci.yml script section, it shows help message in job logs where no -f option is presented.
# docker-compose -v
docker-compose version 1.29.2, build 5becea4c
The question is: how to maintain docker-compose logic without using -f override.yml? Just create three versions of docker-compose.yml in three folders and use [entrypoint] in .gitlab-ci.yml?
Update
Why do I refuse to use docker:dind? I can not find the right way to run a job that starts docker-in-a-docker since again you need three docker-compose.yml files just to start docker:dind containers. Am I missing something?