How to separate environments of a project in docker and docker-compose - ruby-on-rails

I have a couple of basic conceptual question regarding the environments of a webapp.
I am trying to build a dockerized Rails app, having: test, development, staging and production.
My first question is, should the Dockerfile and docker-compose be the same for every environment?
The only thing that would change then would be that when I want to do the testing I pass the RAILS_ENV=test when creating a container, when I want to do the development I pass RAILS_ENV=development and so on.
Would this be the correct idea behind it?
Or can they be different (in my case I am building nginx on production together with app and db but I have just a simple setup with only the app and db for testing and development)
My second question is, when I pass the RAILS_ENV=test for example, should I do it on the Dockerfile (conditionally pass a different environment when building the image):
# Set environment
ARG BUILD_DEVELOPMENT
# if --build-arg BUILD_DEVELOPMENT=1, set RAILS_ENV to 'development' or set to null otherwise.
ENV RAILS_ENV=${BUILD_DEVELOPMENT:+development}
# if RAILS_ENV is null, set it to 'production' (or leave as is otherwise).
ENV RAILS_ENV=${RAILS_ENV:-production}
Or keep the same image and pass the RAILS_ENV when doing the docker-compose ? :
docker-compose -f docker-compose.production.yml run rake db:create db:migrate RAILS_ENV=production
Thank you!

Should the Dockerfile be the same for every environment?
Yes. Build a single image and reuse it in all environments. Do not use Dockerfile ARG to pass in "is it production", or host names, or host-specific user IDs. Especially you should use an identical environment in your pre-production and production environments to avoid deploying an untested image.
Should docker-compose.yml be the same for every environment?
This is the main place you have to control deploy-time options like, for example, where your database is located, or what level of logs should be output. So it makes sense to have a separate Compose file per environment.
Compose supports multiple Compose files. You can use docker-compose -f ... multiple times to specify specific Compose files to use; or if you do not have that option then Compose will read both a docker-compose.yml and a docker-compose.override.yml. So you might have a base docker-compose.yml file that names the images to use
# docker-compose.yml
version: '3.8'
services:
app:
image: registry.example.com/app:${APP_TAG:-latest}
In the question you suggest a docker-compose.prod.yml. That can set $RAILS_ENV and point at your production database:
# docker-compose.prod.yml
services:
app:
environment:
- RAILS_ENV=production
- DB_HOST=db.example.com
- DB_USERNAME=...
# (but don't repeat image:)
You could separately have a docker-compose.dev.yml that launched a local database, and had instructions on how to build the image:
# docker-compose.dev.yaml
version: '3.8'
services:
app:
build: .
environment:
- RAILS_ENV=development
- DB_HOST=db
- DB_USERNAME=db
- DB_PASSWORD=passw0rd
db:
image: postgres:14
environment:
- POSTGRES_USER=db
- POSTGRES_PASSWORD=passw0rd
volumes:
- dbdata:/var/lib/postgresql/data
volumes:
dbdata:
If you use the docker-compose -f option, you need to always mention both files you're using
docker-compose \
-f docker-compose.yml \
-f docker-compose.dev.yml \
run app \
rake db:migrate
You could also symlink docker-compose.override.yml to point at an environment-specific file, and then Compose would be able to find it by default.
ln -sf docker-compose.test.yml docker-compose.override.yml
docker-compose run app rspec

Related

Can docker-compose profiles be used together with docker-compose override to have a common code with configurable environments?

I have the requirement of having a unique docker-compose.yaml / infrastrucutre code which will be versioned across the different deployment stages.
I would like to have some ports exposed in development and not in production. As I learned from other questions this seems to not be possible (using the same .env file that is used to configure other environment variables for the containers).
My idea would be to have my docker-compose.yaml, for example:
version: "3.9"
services:
myservice:
image: myimage
# **
# configuration
# **
ports:
- 80:80
- 19980:19980
Then in production overrideit with a profile docker-compose.production.yaml
version: "3.9"
services:
myservice:
profiles:
- production
ports:
- 80:80
This would allow to have always the same configuration (both .yamlf iles) and switch between them by just calling the docker-compose up command with the production profile (--profile).
My question is, does this work as expected or is the service always overwritten also when the profile flag is not provided?
Compose profiles only affect which services start; they do not have any effect on the options those services use. If you have multiple Compose files then the options in those files are merged according to a set of rules. My expectation is that this would take effect before the profile selection took place.
What you're describing seems like a fairly routine setup for multiple Compose files, without using the profile feature. The most common case I've seen is that a "development" setup strictly adds options to a "production" setup. In your example, both "production" and "development" publish port 80, but only development also publishes the debugger port. There also might be additional environment variables or bind mounts that only make sense in development, but you (usually) are not trying to remove values.
So in this setup, your base docker-compose.yml file would contain the production setup, with the minimum values that are used in all environments.
# docker-compose.yml
version: '3.8'
services:
myservice:
image: myimage
ports:
- '80:80'
Then you'd have a second file that only has the options that are added for the development setup:
# docker-compose.dev.yml / docker-compose.override.yml (see below)
version: '3.8'
services:
myservice:
# (do not need to repeat `image:`; could add `build:`)
ports:
- '19980:19980'
If you name the file docker-compose.override.yml, Compose will use both files automatically, and you need to make sure to push the base file but not the override file to the production environment.
# uses both files, if docker-compose.override.yml is present
docker-compose up -d
If you name it something else, you need to explicitly name all of the files with docker-compose -f options, on every Compose invocation.
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d
docker-compose -f docker-compose.yml -f docker-compose.dev.yml ps
(Or you can set the $COMPOSE_FILE environment variable, but you have to remember to set it in every shell session in every environment.)

Docker compose refuses to apply environment variables

UPDATE
It appears the problem is specifically related to the RUN command in the Dockerfile. If I remove it, the build works fine and the environment variables are clearly being picked up since the password gets applied and I can connect using it. Not sure why the login fails in the RUN command, I've seen many examples using similar code.
I'm working on a very basic docker compose file to setup a dev environment for an app and I started with the database server, which is MS SQL. Here's what the docker-compose.yml file looks like:
version: '3.8'
services:
mssql:
build:
context: .
dockerfile: docker/mssql/Dockerfile
ports:
- '1434:1433'
environment:
ACCEPT_EULA: "Y"
SA_PASSWORD: "YourStrong!Passw0rd"
volumes:
- mssql-data:/var/opt/mssql
As you can see from my dockerfile path, that's in a sub-path and looks like this:
FROM mcr.microsoft.com/mssql/server:2019-latest
COPY ./docker/mssql/TESTDB.bak /var/opt/mssql/backup/TESTDB.bak
RUN ( /opt/mssql/bin/sqlservr --accept-eula & ) | grep -q "Service Broker manager has started" && /opt/mssql-tools/bin/sqlcmd -S localhost,1433 -U SA -P "YourStrong!Passw0rd" -Q 'RESTORE DATABASE TESTDB FROM DISK = "/var/opt/mssql/backup/TESTDB.bak" WITH MOVE "TESTDB_Data" to "/var/opt/mssql/data/TESTDB.mdf", MOVE "TESTDB_Log" to "/var/opt/mssql/data/TESTDB_log.ldf"'
(Yes, I realize that the password in the RUN command is redundant, I had tried to use a variable there earlier and since it wasn't working I hard coded it.)
When I run docker-compose up -d, I always get this error: Login failed for user 'SA'
I wasted way too much time thinking there was actually something wrong with the password until I realized that if I add the environment variables directly in the Dockerfile, it works. So in my Dockerfile, above the RUN command, I can just do this:
ENV ACCEPT_EULA=Y
ENV SA_PASSWORD=YourStrong!Passw0rd
So I concluded that my environment variables simply aren't being read. I tried with quotes, without quotes, using env_file instead, nothing seems to work. I also tried the following format, no luck:
environment
- ACCEPT_EULA=Y
- SA_PASSWORD=YourStrong!Passw0rd
I also tried using MSSQL_SA_PASSWORD instead of SA_PASSWORD, as well as having both in there. I assumed that was unlikely to be the problem though given SA_PASSWORD works fine. Lastly, I tried using a 2017 image in case it was image specific, that didn't work either.
I'm assuming it must be something silly I'm missing. I saw a lot of talk of .env in the root being different, but if I understood correctly people go wrong with that when they try to use environment values in their docker-compose.yml file, which is not what I'm doing here. So I'm about ready to lose my mind on this as it seems like such a simple, basic thing.
I think you're confusing the ENV statement in Dockerfile with the environment variables set when running an image. The key is still in the details of the docs. It notes that they are the same as saying docker run -e, not docker build.
What's causing more confusion, when you use ENV, you are setting defaults for when the image runs later:
https://docs.docker.com/engine/reference/builder/#env
If you haven't yet, I very much recommend getting familiar with building and running your image with docker run and docker build before moving on to compose, it's much less confusing that way.
The issue with your build here stems from a confusion between the build-time and run-time environment variables: with the environment or env_file properties you specify the environment variables to be set for the service container.
But the RUN command in your Dockerfile is executed at the build-time of the image! To pass variables when building a new image you should use build args instead, as you already mentioned in your comment:
services:
mssql:
build:
context: .
dockerfile: docker/mssql/Dockerfile
args:
SA_PASSWORD: "YourStrong!Passw0rd"
# ...
With this you can use the SA_PASSWORD as a build ARG:
FROM mcr.microsoft.com/mssql/server:2019-latest
COPY ./docker/mssql/TESTDB.bak /var/opt/mssql/backup/TESTDB.bak
ARG SA_PASSWORD
RUN ( /opt/mssql/bin/sqlservr --accept-eula & ) | grep -q "Service Broker manager has started" && /opt/mssql-tools/bin/sqlcmd -S localhost,1433 -U SA -P "$SA_PASSWORD" -Q 'RESTORE DATABASE TESTDB FROM DISK = "/var/opt/mssql/backup/TESTDB.bak" WITH MOVE "TESTDB_Data" to "/var/opt/mssql/data/TESTDB.mdf", MOVE "TESTDB_Log" to "/var/opt/mssql/data/TESTDB_log.ldf"'
If you want to move the actual password to a .env file you can use variable substitution in the compose.yml:
services:
mssql:
build:
# ...
args:
SA_PASSWORD: "$SA_PASSWORD"
# ...
In your docker-compose.yml, have you tried:
- ACCEPT_EULA=Y
- SA_PASSWORD=YourStrong!Passw0rd
Both responses above are fine, just a few more things:
SA_PASSWORD is deprecated instead use MSSQL_SA_PASSWORD
It is always nice to define .env files with the variables for instance:
sapassword.env
MSSQL_SA_PASSWORD=YourStrong!Passw0rd
sqlserver.env
ACCEPT_EULA=Y
MSSQL_DATA_DIR=/var/opt/sqlserver/data
MSSQL_LOG_DIR=/var/opt/sqlserver/log
MSSQL_BACKUP_DIR=/var/opt/sqlserver/backup
And in docker-compose.yml instance the env files the following way:
environment:
- sqlserver.env
- sapassword.env

set env variables for all the users is not working in docker

I am setting env variable in dockerfile which is not reflecting for all the users when i start the deployment. Unless i go to the terminal and do source /etc/profile.d/ip.sh then only it gets affected to the user. Without doing this activity can we set this value to all the users. If yes how do we need to achieve it.
Created shell script and added in /etc/profile.d/ip.sh:
IP="1.1.1.1"
export IPADDR=$IP
Dockerfile:
COPY ip.sh /etc/profile.d
RUN chmod 644 /etc/profile.d/ip.sh
RUN . ./etc/profile.d/ip.sh
please read the environment variables section of the docker manual
the reason you should not get what you want is that Docker does not behave the same as full OS therefore you do not get the scripts at startup.
here are some examples from there:
Set environment variables in containers
You can set environment variables in a service’s containers with the ‘environment’ key, just like with docker run -e VARIABLE=VALUE ...:
web:
environment:
- DEBUG=1
The “env_file” configuration option🔗
You can pass multiple environment variables from an external file through to a service’s containers with the ‘env_file’ option, just like with docker run --env-file=FILE ...:
web:
env_file:
- web-variables.env
The “.env” file
You can set default values for any environment variables referenced in the Compose file, or used to configure Compose, in an environment file named .env:
$ cat .env
TAG=v1.5
$ cat docker-compose.yml
version: '3'
services:
web:
image: "webapp:${TAG}"

Why does variables substitution not work when using multiple env_files in docker-compose.yml?

I'm trying to get a docker-compose file working with multiple .env files, and I'm not having any luck. I'm trying to setup three .env files:
global settings that are the same across all container instances
environment-specific settings (stuff just for test or dev)
local settings - overridable things that a developer might need to change in case they have conflicts with, say, a port number
My docker-compose.yml file looks like this:
version: '2'
services:
db:
env_file:
- ./.env
- ./.env.${ENV}
- ./.env.local
image: postgres
ports:
- ${POSTGRES_PORT}:5432
.env looks like this:
POSTGRES_USER=myapp
and the .env.development looks like this:
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_HOST=localhost
POSTGRES_PORT=25432
POSTGRES_DB=myapp_development
.env.local doesn't exist in this case.
After running ENV=development docker-compose up, I receive the following output:
$ ENV=development docker-compose up
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.db.ports is invalid: Invalid port ":5432", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
From that error message, it looks like none of my environment variables are being used. I just upgraded to the newest available docker-compose as well - same errors:
$ docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6
Any ideas here? Would be nice to have a single docker-compose.yml that would work across multiple environments.
In order to apply different/multiple env_files depending on the running environment, such as development/staging/production, I think a better way for docker-compose is to use multiple docker-compose yml files.
For example:
1. Start with a base file that defines the canonical configuration for the services.
docker-compose.yml
web:
image: example/my_web_app:latest
env_file:
- .env
2. Add the override file for development, as its name implies, can contain configuration overrides for existing services or entirely new services.
docker-compose.override.yml
web:
build: .
volumes:
- '.:/code'
ports:
- 8883:80
env_file:
- .env.dev
When you run docker-compose up it reads the overrides automatically.
3. Create another override file for the production environment.
docker-compose.prod.yml
web:
ports:
- 80:80
env_file:
- .env.prod
To deploy with this production Compose file you can run
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up
Note
My Docker version:
$ docker -v
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose -v
docker-compose version 1.22.0, build f46880fe
Reference: https://docs.docker.com/compose/extends/
Keep in mind that there are 2 different environments where you are defining variables. The host machine where you are executing the docker-compose command, and the container itself (running the db service in your case).
Your docker-compose.yml file has access to your host's environment variables. Hence ENV is reachable from the docker-compose command, but not these in your .env files.
On the contrary, the value for ENV is not reachable inside the container, but all variables defined in your .env files will.
I don't know if you really need your db container to access the variables defined on your .env.development. But at least seem that your host machine needs to have the content of that file, so when the docker-compose command is called, the POSTGRES_PORT variable is defined.
To fix your specific problem you would need to define the environment variables on your host machine too, not only for the container. You could do something like this:
#Set for host
ENV=development
#Also sets the variables on the host
source ./.env.$ENV
#POSTGRES_PORT defined in .env.development is used here
docker-compose up
#since env_file also contains .env.development, the variables will be reachable from the container.
Hope that helps.
There is a misconception regarding the .env file and the env_file option in the docker-compose.yml, as it is very ambiguous. Shin points it out very nicely in the github issue docker-compose doesn't use env_file. I will just quote his summary:
Variable substitution in your docker-compose.yml file will be pulled (in decreasing order of priority) from your shell's environment and your .env file.
Variables available in your container are a combination of values found in your env_file files and values described in the environment section of the service.
Those are two entirely separate sets of features.
while reading this page: https://docs.docker.com/compose/environment-variables/
and from my understanding, you should do the following:
for the global variables(that should not change) make an env file like so:
VAR1=VALUE1
VAR2=VALUE2
and for the others(that might change) you should add their name under environment in docker-compose.yml like this:
environment:
- VAR1
- VAR2
this will take the VAR1 and VAR2 values from the shell you are running docker-compose.
I hope this helps.

How can I use environment variables in docker-compose?

I would like to be able to use environment variables inside docker-compose.yml, with values passed in at the time of docker-compose up. This is the example.
I am doing this today with a basic docker run command, which is wrapped around my own script. Is there a way to achieve it with compose, without any such bash wrappers?
proxy:
hostname: $hostname
volumes:
- /mnt/data/logs/$hostname:/logs
- /mnt/data/$hostname:/data
The Docker solution:
Docker-compose 1.5+ has enabled variables substitution: Releases · docker/compose
The latest Docker Compose allows you to access environment variables from your compose file. So you can source your environment variables, then run Compose like so:
set -a
source .my-env
docker-compose up -d
For example, assume we have the following .my-env file:
POSTGRES_VERSION=14
(or pass them via command-line arguments when calling docker-compose, like so: POSTGRES_VERSION=14 docker-compose up -d)
Then you can reference the variables in docker-compose.yml using a ${VARIABLE} syntax, like so:
db:
image: "postgres:${POSTGRES_VERSION}"
And here is more information from the documentation, taken from Compose file specification
When you run docker-compose up with this configuration, Compose looks
for the POSTGRES_VERSION environment variable in the shell and
substitutes its value in. For this example, Compose resolves the image
to postgres:9.3 before running the configuration.
If an environment variable is not set, Compose substitutes with an
empty string. In the example above, if POSTGRES_VERSION is not set,
the value for the image option is postgres:.
Both $VARIABLE and ${VARIABLE} syntax are supported. Extended
shell-style features, such as ${VARIABLE-default} and
${VARIABLE/foo/bar}, are not supported.
If you need to put a literal dollar sign in a configuration value, use
a double dollar sign ($$).
The feature was added in this pull request.
Alternative Docker-based solution: Implicitly sourcing an environment variables file through the docker-compose command
If you want to avoid any Bash wrappers, or having to source a environment variables file explicitly (as demonstrated above), then you can pass a --env-file flag to the docker-compose command with the location of your environment variable file: Use an environment file
Then you can reference it within your docker-compose command without having to source it explicitly:
docker-compose --env-file .my-env up -d
If you don't pass a --env-file flag, the default environment variable file will be .env.
Note the following caveat with this approach:
Values present in the environment at runtime always override those defined inside the .env file. Similarly, values passed via command-line arguments take precedence as well.
So be careful about any environment variables that may override the ones defined in the --env-file!
The Bash solution:
I notice that Docker's automated handling of environment variables can cause confusion. Instead of dealing with environment variables in Docker, let's go back to basics, like Bash! Here is a method using a Bash script and a .env file, with some extra flexibility to demonstrate the utility of environment variables:
POSTGRES_VERSION=14
# Note that the variable below is commented out and will not be used:
# POSTGRES_VERSION=15
# You can even define the compose file in an environment variable like so:
COMPOSE_CONFIG=my-compose-file.yml
# You can define other compose files, and just comment them out
# when not needed:
# COMPOSE_CONFIG=another-compose-file.yml
Then run this Bash script in the same directory, which should deploy everything properly:
#!/bin/bash
docker rm -f `docker ps -aq -f name=myproject_*`
set -a
source .env
cat ${COMPOSE_CONFIG} | envsubst | docker-compose -f - -p "myproject" up -d
Just reference your environment variables in your compose file with the usual Bash syntax (ie ${POSTGRES_VERSION} to insert the POSTGRES_VERSION from the .env file).
While this solution involves Bash, some may prefer it because it has better separation of concerns.
Note the COMPOSE_CONFIG is defined in my .env file and used in my Bash script, but you can easily just replace {$COMPOSE_CONFIG} with the my-compose-file.yml in the Bash script.
Also note that I labeled this deployment by naming all of my containers with the "myproject" prefix. You can use any name you want, but it helps identify your containers so you can easily reference them later. Assuming that your containers are stateless, as they should be, this script will quickly remove and redeploy your containers according to your .env file parameters and your compose YAML file.
Since this answer seems pretty popular, I wrote a blog post that describes my Docker deployment workflow in more depth: Let's Deploy! (Part 1) This might be helpful when you add more complexity to a deployment configuration, like Nginx configurations, Let's Encrypt certificates, and linked containers.
It seems that docker-compose has native support now for default environment variables in a file.
All you need to do is declare your variables in a file named .env and they will be available in docker-compose.yml.
For example, for a .env file with contents:
MY_SECRET_KEY=SOME_SECRET
IMAGE_NAME=docker_image
You could access your variable inside docker-compose.yml or forward them into the container:
my-service:
image: ${IMAGE_NAME}
environment:
MY_SECRET_KEY: ${MY_SECRET_KEY}
Create a template.yml, which is your docker-compose.yml with environment variable.
Suppose your environment variables are in a file 'env.sh'
Put the below piece of code in a sh file and run it.
source env.sh;
rm -rf docker-compose.yml;
envsubst < "template.yml" > "docker-compose.yml";
A new file docker-compose.yml will be generated with the correct values of environment variables.
Sample template.yml file:
oracledb:
image: ${ORACLE_DB_IMAGE}
privileged: true
cpuset: "0"
ports:
- "${ORACLE_DB_PORT}:${ORACLE_DB_PORT}"
command: /bin/sh -c "chmod 777 /tmp/start; /tmp/start"
container_name: ${ORACLE_DB_CONTAINER_NAME}
Sample env.sh file:
#!/bin/bash
export ORACLE_DB_IMAGE=<image-name>
export ORACLE_DB_PORT=<port to be exposed>
export ORACLE_DB_CONTAINER_NAME=ORACLE_DB_SERVER
The best way is to specify environment variables outside the docker-compose.yml file. You can use env_file setting, and define your environment file within the same line. Then doing a docker-compose up again should recreate the containers with the new environment variables.
Here is how my docker-compose.yml looks like:
services:
web:
env_file: variables.env
Note:
docker-compose expects each line in an env file to be in VAR=VAL format. Avoid using export inside the .env file. Also, the .env file should be placed in the folder where the docker-compose command is executed.
The following is applicable for docker-compose 3.x
Set environment variables inside the container
method - 1 Straight method
web:
environment:
- DEBUG=1
POSTGRES_PASSWORD: 'postgres'
POSTGRES_USER: 'postgres'
method - 2 The “.env” file
Create a .env file in the same location as the docker-compose.yml
$ cat .env
TAG=v1.5
POSTGRES_PASSWORD: 'postgres'
and your compose file will be like
$ cat docker-compose.yml
version: '3'
services:
web:
image: "webapp:${TAG}"
postgres_password: "${POSTGRES_PASSWORD}"
source
When using environment variables for volumes you need:
create .env file in the same folder which contains docker-compose.yaml file
declare variable in the .env file:
HOSTNAME=your_hostname
Change $hostname to ${HOSTNAME} at docker-compose.yaml file
proxy:
hostname: ${HOSTNAME}
volumes:
- /mnt/data/logs/${HOSTNAME}:/logs
- /mnt/data/${HOSTNAME}:/data
Of course you can do that dynamically on each build like:
echo "HOSTNAME=your_hostname" > .env && sudo docker-compose up
Don't confuse the .env file and the env_file option!
They serve totally different purposes!
The .env file feeds those environment variables only to your docker compose file, which in turn, can be passed to the containers as well.
But the env_file option only passes those variables to the containers and NOT the docker compose file 😵‍💫
Example
OK, let's say we have this simple compose file:
services:
foo:
image: ubuntu
hostname: suchHostname # <-------------- hard coded 'suchHostname'
volumes:
- /mnt/data/logs/muchLogs:/logs # <--- hard coded 'muchLogs'
- /mnt/data/soBig:/data # <----------- hard coded 'soBig'
We don't want to hard code these anymore! So, we can put them in the current terminal's environment variables and check if docker-compose understands them:
$ export the_hostname="suchHostName"
$ export dir_logs="muchLogs"
$ export dir_data="soBig"
and change the docker-compose.yml file to:
services:
foo:
image: ubuntu
hostname: $the_hostname # <-------------- use $the_hostname
volumes:
- /mnt/data/logs/$dir_logs:/logs # <--- use $dir_logs
- /mnt/data/$dir_data:/data # <-------- usr $dir_data
Now let's check out if it worked with executing $ docker-compose convert and inspecting the output:
name: tmp
services:
foo:
hostname: suchHostName # <------------- $the_hostname
image: ubuntu
networks:
default: null
volumes:
- type: bind
source: /mnt/data/logs/muchLogs # <-- $dir_logs
target: /logs
bind:
create_host_path: true
- type: bind
source: /mnt/data/soBig # <---------- $dir_data
target: /data
bind:
create_host_path: true
networks:
default:
name: tmp_default
OK it works! But let's use the .env file instead. Since docker-compose understands the .env file, let's just create one and set it up:
# .env file (in the same directory as 'docker-compose.yml')
the_hostname="suchHostName"
dir_logs="muchLogs"
dir_data="soBig"
OK, you can test it with a NEW terminal (so that the older environment variables we set with export don't interfere and make sure everything works in a clean terminal) 🖥 Just follow step 4 again and see that it works!
So far so good 😃 However, when you stumble upon the env_file option, it gets confusing 🤔 Let's say that you want to pass a password to the docker compose file (NOT the container).
🙄 In the wrong approach, you might put a password in .secrets file:
# .secrets
somepassword="0P3N$3$#M!"
and then update the docker-compose file as follows:
services:
foo:
image: ubuntu
hostname: $the_hostname
volumes:
- /mnt/data/logs/$dir_logs:/logs
- /mnt/data/$dir_data:/data
# 🔽 BAD:
env_file:
- .env
- .secrets
entrypoint: echo "Hush! This is a secret '$somepassword'"
Now checking it just like step 4 again would result in:
WARN[0000] The "somepassword" variable is not set. Defaulting to a blank string.
name: tmp # ^
services: # |
foo: # |
entrypoint: # |
- echo # |
- Hush! This is a secret '' # <---- 😵‍💫 Oh no!
environment:
dir_data: soBig
dir_logs: muchLogs
somepassword: 0P3N$$3$$#M! # <--- 🤔 Huh?!
the_hostname: suchHostName
hostname: suchHostName
image: ubuntu
networks:
default: null
volumes:
- type: bind
source: /mnt/data/logs/muchLogs
target: /logs
bind:
create_host_path: true
- type: bind
source: /mnt/data/soBig
target: /data
bind:
create_host_path: true
networks:
default:
name: tmp_default
So as you can see, the $somepassord variable is only passed to the container, and NOT the docker compose file.
Wrapping up
You can pass environment variables to docker-compose files in two ways:
By exporting the variable to the terminal before running docker compose.
By putting the variables inside .env file.
The env_file option only passes those extra variables to the containers 📦 and not the compose file 🐳
Since 1.25.4, docker-compose supports the option --env-file that enables you to specify a file containing variables.
Yours should look like this:
hostname=my-host-name
And the command:
docker-compose --env-file /path/to/my-env-file config
To add an environment variable, you may define an env_file (let's call it var.env) as:
ENV_A=A
ENV_B=B
And add it to the docker compose manifest service. Moreover, you can define environment variables directly with environment.
For instance, in docker-compose.yaml:
version: '3.8'
services:
myservice:
build:
context: .
dockerfile: ./docker/Dockerfile.myservice
image: myself/myservice
env_file:
- ./var.env
environment:
- VAR_C=C
- VAR_D=D
volumes:
- $HOME/myfolder:/myfolder
ports:
- "5000:5000"
Please check here for more/updated information: Manuals → Docker →Compose →Environment variables → Overview
Use:
env SOME_VAR="I am some var" OTHER_VAR="I am other var" docker stack deploy -c docker-compose.yml
Use version 3.6:
version: "3.6"
services:
one:
image: "nginx:alpine"
environment:
foo: "bar"
SOME_VAR:
baz: "${OTHER_VAR}"
labels:
some-label: "$SOME_VAR"
two:
image: "nginx:alpine"
environment:
hello: "world"
world: "${SOME_VAR}"
labels:
some-label: "$OTHER_VAR"
I got it from Feature request: Docker stack deploy pass environment variables via cli options #939.
You cannot ... yet. But this is an alternative, think like a docker-composer.yml generator:
https://gist.github.com/Vad1mo/9ab63f28239515d4dafd
Basically a shell script that will replace your variables. Also you can use Grunt task to build your docker compose file at the end of your CI process.
I have a simple bash script I created for this it just means running it on your file before use:
https://github.com/antonosmond/subber
Basically just create your compose file using double curly braces to denote environment variables e.g:
app:
build: "{{APP_PATH}}"
ports:
- "{{APP_PORT_MAP}}"
Anything in double curly braces will be replaced with the environment variable of the same name so if I had the following environment variables set:
APP_PATH=~/my_app/build
APP_PORT_MAP=5000:5000
on running subber docker-compose.yml the resulting file would look like:
app:
build: "~/my_app/build"
ports:
- "5000:5000"
To focus solely on the issue of default and mandatory values for environment variables, and as an update to #modulito's answer:
Using default values and enforcing mandatory values within the docker-compose.yml file is now supported (from the docs):
Both $VARIABLE and ${VARIABLE} syntax are supported. Additionally when using the 2.1 file format, it is possible to provide inline default values using typical shell syntax:
${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment.
${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment.
Similarly, the following syntax allows you to specify mandatory variables:
${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment.
${VARIABLE?err} exits with an error message containing err if VARIABLE is unset in the environment.
Other extended shell-style features, such as ${VARIABLE/foo/bar}, are not supported.
This was written for Docker v20, using the docker compose v2 commands.
I was having a similar roadblock and found that the --env-file parameter ONLY works for docker compose config command. On top of that using the docker compose env_file variable, still forced me to repeat values for the variables, when wanting to reuse them in other places than the Dockerfile such as environment for docker-compose.yml. I just wanted one source of truth, my .env, with the ability to swap them per deployment stage. So here is how I got it to work, basically use docker compose config to generate a base docker-compose.yml file that will pass ARG into Dockerfile's.
.local.env This would be your .env, I have mine split for different deployments.
DEVELOPMENT=1
PLATFORM=arm64
docker-compose.config.yml - This is my core docker compose file.
services:
server:
build:
context: .
dockerfile: docker/apache2/Dockerfile
args:
- PLATFORM=${PLATFORM}
- DEVELOPMENT=${DEVELOPMENT}
environment:
- PLATFORM=${PLATFORM}
- DEVELOPMENT=${DEVELOPMENT}
Now sadly I do need to pass in the variables twice, once for the Dockerfile, the other for environment. However, they are still coming from the single source .local.env so at least I do not need to repeat values.
I then use docker compose config to generate a semi-final docker-compose.yml. This lets me pass in my companion override docker-compose.local.yml for where the final deployment is happening.
docker compose --env-file=.local.env -f docker-compose.config.yml config > docker-compose.yml
This will now let my Dockerfile access the .env variables.
FROM php:5.6-apache
# Make sure to declare after FROM
ARG PLATFORM
ARG DEVELOPMENT
# Access args in strings with $PLATFORM, and can wrap i.e ${PLATFORM}
RUN echo "SetEnv PLATFORM $PLATFORM" > /etc/apache2/conf-enabled/environment.conf
RUN echo "SetEnv DEVELOPMENT $DEVELOPMENT" > /etc/apache2/conf-enabled/environment.conf
This then passes the .env variables from the docker-compose.yml into Dockerfile which then passes it into my Apache HTTP server, which passes it to my final destination, the PHP code.
My next step to then to pass in my docker compose overrides from my deployment stage.
docker-compose.local.yml - This is my docker-compose override.
services:
server:
volumes:
- ./localhost+2.pem:/etc/ssl/certs/localhost+2.pem
- ./localhost+2-key.pem:/etc/ssl/private/localhost+2-key.pem
Lastly, run the docker compose command.
docker compose -f docker-compose.yml -f docker-compose.local.yml up --build
Please note if you change anything in you .env file you will need to re-run the docker compose config and add --build for docker compose up. Since builds are cached it has little impact.
So for my final command I normally run:
docker compose --env-file=.local.env -f docker-compose.config.yml config > docker-compose.yml; docker compose --env-file=.local.env -f docker-compose.yml -f docker-compose.local.yml up --build
As far as I know, this is a work-in-progress. They want to do it, but it's not released yet. See 1377 (the "new" 495 that was mentioned by #Andy).
I ended up implementing the "generate .yml as part of CI" approach as proposed by #Thomas.
Add an environment variable to the .env file
Such as
VERSION=1.0.0
Then save it to deploy.sh
INPUTFILE=docker-compose.yml
RESULT_NAME=docker-compose.product.yml
NAME=test
prepare() {
local inFile=$(pwd)/$INPUTFILE
local outFile=$(pwd)/$RESULT_NAME
cp $inFile $outFile
while read -r line; do
OLD_IFS="$IFS"
IFS="="
pair=($line)
IFS="$OLD_IFS"
sed -i -e "s/\${${pair[0]}}/${pair[1]}/g" $outFile
done <.env
}
deploy() {
docker stack deploy -c $outFile $NAME
}
prepare
deploy
Use .env file to define dynamic values in docker-compse.yml. Be it port or any other value.
Sample docker-compose:
testcore.web:
image: xxxxxxxxxxxxxxx.dkr.ecr.ap-northeast-2.amazonaws.com/testcore:latest
volumes:
- c:/logs:c:/logs
ports:
- ${TEST_CORE_PORT}:80
environment:
- CONSUL_URL=http://${CONSUL_IP}:8500
- HOST=${HOST_ADDRESS}:${TEST_CORE_PORT}
Inside .env file you can define the value of these variables:
CONSUL_IP=172.31.28.151
HOST_ADDRESS=172.31.16.221
TEST_CORE_PORT=10002
I ended up using "sed" in my deploy.sh script to accomplish this, though my requirements were slightly different since docker-compose is being called by Terrafom: Passing Variables to Docker Compose via a Terraform script for an Azure App Service
eval "sed -i 's/MY_VERSION/$VERSION/' ../docker-compose.yaml"
cat ../docker-compose.yaml
terraform init
terraform apply -auto-approve \
-var "app_version=$VERSION" \
-var "client_id=$ARM_CLIENT_ID" \
-var "client_secret=$ARM_CLIENT_SECRET" \
-var "tenant_id=$ARM_TENANT_ID" \
-var "subscription_id=$ARM_SUBSCRIPTION_ID"
eval "sed -i 's/$VERSION/MY_VERSION/' ../docker-compose.yaml"
It's simple like this:
Using command line as mentioned in the documentation:
docker-compose --env-file ./config/.env.dev config
Or using a .env file, I think this is the easiest way:
web:
env_file:
- web-variables.env
Documentation with a sample

Resources