I am trying to configure GitLab CI to continuously build and deploy my app with Docker and Docker Compose.
When running the CI pipeline, I get the following error messages:
The Compose file './docker-compose.ci.yml' is invalid because:
services.dashboard.ports contains an invalid type, it should be a number, or an object
services.mosquitto.ports contains an invalid type, it should be a number, or an object
services.mosquitto.ports contains an invalid type, it should be a number, or an object
services.mosquitto.ports contains an invalid type, it should be a number, or an object
services.mosquitto.ports value [':', ':', ':'] has non-unique elements
I would like to use environment variables to keep my configuration hidden.
Following is a snippet of my docker-compose.ci.yml:
version: "3.9"
services:
dashboard:
build:
context: ./dashboard
dockerfile: Dockerfile.prod
cache_from:
- "${BACKEND_IMAGE}"
image: "${BACKEND_IMAGE}"
command: gunicorn dashboard.wsgi:application --bind ${DJANGO_HOST}:${DJANGO_PORT}
volumes:
- static_volume:/home/app/web/static
ports:
- "${DJANGO_PORT}:${DJANGO_PORT}"
env_file:
- .env
depends_on:
- postgres
...
mosquitto:
build:
context: ./mosquitto
cache_from:
- "${MOSQUITTO_IMAGE}"
image: "${MOSQUITTO_IMAGE}"
volumes:
- ./mosquitto/config/mosquitto.conf:/mosquitto/config/mosquitto.conf
- ./mosquitto/data:/mosquitto/data
- ./mosquitto/log:/mosquitto/log
- broker_certs:/mosquitto/config/certs
ports:
- "${MQTT_DEFAULT_PORT}:${MQTT_DEFAULT_PORT}"
- "${MQTT_SECURE_PORT}:${MQTT_SECURE_PORT}"
- "${MQTT_WEBSOCKETS_PORT}:${MQTT_WEBSOCKETS_PORT}"
env_file:
- .env
...
In my build stage, I set up the environment variables using a bash script:
.gitlab-ci.yml:
image:
name: docker/compose:1.28.5
entrypoint: [""]
services:
- docker:dind
stages:
- build
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
build:
stage: build
before_script:
...
- chmod +x ./setup_env.sh
- bash ./setup_env.sh
...
...
setup_env.sh:
...
# mosquitto config
echo MQTT_DEFAULT_PORT=$MQTT_DEFAULT_PORT >> .env
echo MQTT_SECURE_PORT=$MQTT_SECURE_PORT >> .env
echo MQTT_WEBSOCKETS_PORT=$MQTT_WEBSOCKETS_PORT >> .env
echo MOSQUITTO_COMMON_NAME=$MOSQUITTO_COMMON_NAME >> .env
...
All my variables are well set up on Gitlab.
Running this with docker-compose locally on my machine doesn't generate any errors.
What am I doing wrong?
The configuration shown above is correct. The problem was that my branch was not protected, and the environments variables on Gitlab are set up for protected branches/tags only. Consequently, they were never picked during the build. In case anyone has a similar issue, make sure your branch/tag is protected.
Related
I get the below error when I run docker-compose up, any pointers why I am getting this error
service "mysqldb-docker" refers to undefined volume mysqldb: invalid compose project
Also, is there a way to pass the $ENV value in CLI to docker-compose up , currently I have a ENV variable that specified dev, uat or prod that I use to specify the db name. Are there better alternatives to do this other than create a .env file explicitly for this
version: '3.8'
services:
mysqldb-docker:
image: '8.0.27'
restart: 'unless-stopped'
ports:
- "3309:3306"
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_PASSWORD=root
- MYSQL_DATABASE=reco-tracker-$ENV
volumes:
- mysqldb:/var/lib/mysql
reco-tracker-docker:
image: 'reco-tracker-docker:v1'
ports:
- "8083:8083"
environment:
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=root
- SPRING_DATASOURCE_URL="jdbc:mysql://mysqldb-docker:3309/reco-tracker-$ENV"
depends_on: [mysqldb-docker]
You must define volumes at the top level like this:
version: '3.8'
services:
mysqldb-docker:
# ...
volumes:
- mysqldb:/var/lib/mysql
volumes:
mysqldb:
You can pass environment variables from your shell straight through to a service’s containers with the ‘environment’ key by not giving them a value
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers
web:
environment:
- ENV
but from my tests you cant write $ENV in the compose file and expect it to read your env
for this you need to call docker-compose that way :
docker-compose run -e ENV web python console.py
see this : https://docs.docker.com/compose/environment-variables/#set-environment-variables-with-docker-compose-run
I have setup Jenkins within a Docker container and I am trying to access that my private Bitbucket repo with that server. I need to copy my SSH key into that container so that Bitbucket recognizes it and I can have my Jenkins server access the repo then.
I have in my docker-compose.yml file the following:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:
However, echo $SSH_PRIVATE_KEY gives /.ssh/id_rsa literally instead of the value stored inside. I have heard the problem with doing this inside the Dockerfile instead would be that it still can be viewed in one of the layers of the image that will be pushed.
My question is how can I set the value of SSH_PRIVATE_KEY to the value of the contents of my file?
I believe this could be a duplicate of How to set environment variable into docker container using docker-compose however that solution does not appear to change anything for me.
You could create an Environment variable in your shell from which you are running your compose :
export SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
and then use it in your compose like :
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY
ports:
- "8080:8080"
- "50000:50000"
It should pick up the value for your environment variable for container from shell environment as specified in the docs :
The value of the variable in the container is taken from the value for the same variable in the shell in which Compose is run.
Possible solution:
environment:
- SSH_PRIVATE_KEY
and call the docker-compose like this:
SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa) docker-compose build
Unfortunately, it's currently not possible to use multiline variables in .env.
Another possibility would be:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
- "/home/user/.ssh/id_rsa:/home/user/.ssh/id_rsa:ro"
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:
I have setup Jenkins within a Docker container and I am trying to access that my private Bitbucket repo with that server. I need to copy my SSH key into that container so that Bitbucket recognizes it and I can have my Jenkins server access the repo then.
I have in my docker-compose.yml file the following:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:
However, echo $SSH_PRIVATE_KEY gives /.ssh/id_rsa literally instead of the value stored inside. I have heard the problem with doing this inside the Dockerfile instead would be that it still can be viewed in one of the layers of the image that will be pushed.
My question is how can I set the value of SSH_PRIVATE_KEY to the value of the contents of my file?
I believe this could be a duplicate of How to set environment variable into docker container using docker-compose however that solution does not appear to change anything for me.
You could create an Environment variable in your shell from which you are running your compose :
export SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
and then use it in your compose like :
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY
ports:
- "8080:8080"
- "50000:50000"
It should pick up the value for your environment variable for container from shell environment as specified in the docs :
The value of the variable in the container is taken from the value for the same variable in the shell in which Compose is run.
Possible solution:
environment:
- SSH_PRIVATE_KEY
and call the docker-compose like this:
SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa) docker-compose build
Unfortunately, it's currently not possible to use multiline variables in .env.
Another possibility would be:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
- "/home/user/.ssh/id_rsa:/home/user/.ssh/id_rsa:ro"
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:
I'm working on a group project involving Docker that has a .env file, which looks like this:
DATABASE_URL=xxx
DJANGO_SETTINGS_MODULE=xxx
SECRET_KEY=xxx
Couldn't this just be declared inside the Dockerfile? If so, what is the advantage of making a .env file?
Not sure if I'm going in the right direction with this, but this Docker Docs page says (emphasis my own):
Your configuration options can contain environment variables. Compose
uses the variable values from the shell environment in which
docker-compose is run. For example, suppose the shell contains
POSTGRES_VERSION=9.3 and you supply this configuration:
db:
`image: "postgres:${POSTGRES_VERSION}"`
When you run docker-compose up with this configuration, Compose looks for the POSTGRES_VERSION environment variable in the shell and substitutes its value in. For this example, Compose resolves the image to postgres:9.3 before running the configuration.
If an environment variable is not set, Compose substitutes with an empty string. In the example above, if POSTGRES_VERSION is not set, the value for the image option is postgres:.
You can set default values for environment variables using a .env file, which Compose automatically looks for. Values set in the shell environment override those set in the .env file.
If we're using a .env file, then wouldn't I see some ${...} syntax in our docker-compose.yml file? I don't see anything like that, though.
Here's our docker-compose.yml file:
version: '3'
services:
server:
build:
context: ./server
dockerfile: Dockerfile
env_file: .env.dev
command: python3 manage.py runserver 0.0.0.0:8000
volumes:
- ./server:/app
ports:
- "8500:8000"
depends_on:
- db
stdin_open: true
tty: true
db:
image: postgres
client:
build:
context: ./client
dockerfile: Dockerfile
command: bash -c "npm install; npm run start"
volumes:
- ./client:/app
- /app/node_modules
ports:
- "3000:3000"
depends_on:
- server
Idea there is probably to have a place to keep secrets separated from docker-compose.yml, which you then can keep in VCS and/or share.
My Docker containers will be set up through a bash script.
In my docker-compose.yml file (which is called through docker-compose up inside the bash file) I have a few arguments:
version: "2"
services:
nginx:
build: ./Dockerfiles/wordpress-nginx
expose:
- "80"
depends_on:
- fpm
links:
- fpm
container_name: wordpress-nginx
args:
- site_title=Example
- admin_email=test#test.co
- site_url=www.testcase001.com
- admin_user=admin
- admin_pass=qwerty
- ssl_domain=www.testcase001.com
- dbname=testcase
- dbuser=testcase
- dbpass=testcase
- dbhost=testcase
fpm:
build: ./Dockerfiles/php-fpm
expose:
- "9000"
container_name: fpm
Now my question is this:
How can I set the arguments from the command line?
For example:
I want to set the site_title argument: docker-compose up --something-to-set-the-site_title-argument
You can use environment variables in your docker-compose.yaml file, e.g.:
args:
- site_title: $SITE_TITLE
In your Bash script, you can then set the title environment variable like this:
#!/bin/bash
SITE_TITLE="My new title"
docker-compose up -d
The $SITE_TITLE environment variable should then be used in your compose configuration.