I have the following project directory structure:
myapp/
docker/
docker-compose.yml
.env
src/
<my source code here>
config.properties
Here is my .env file:
ENV=local
SERVICE_DB_HOST=0.0.0.0
SERVICE_DB_PORT=3306
SERVICE_DB_ROOT_PASSWORD=12345
SERVICE_DB_APP_USER=my-service-user
SERVICE_DB_APP_PASSWORD=23456
Here is my docker/docker-compose.yml:
version: "3.7"
services:
myapp-main-db:
env_file:
- ../.env
image: mysql:8
container_name: myapp-main-db
command: --default-authentication-plugin=mysql_native_password
restart: always
ports:
- $SERVICE_DB_PORT:$SERVICE_DB_PORT
environment:
MYSQL_ROOT_PASSWORD: $SERVICE_DB_ROOT_PASSWORD
MYSQL_DATABASE: myapp_service_db_$ENV
MYSQL_USER: $SERVICE_DB_APP_USER
MYSQL_PASSWORD: $SERVICE_DB_APP_PASSWORD
volumes:
- myapp-service-db-data:/var/lib/mysql
volumes:
myapp-service-db-data:
When I run docker-compose -f docker/docker-compose.yml up -d from the project root directory, I see the MySQL container spin up without issue, and verify its running via docker ps.
However I am unable to connect to it using the expected connection string and credentials, which I am sure are correct. This leads me to believe that Docker Compose is not loading the ../.env file as I was expecting it to, and so none of the variables defined in ../.env are being injected as env vars.
Can anybody spot where I'm going awry?
Update
When I run docker-compose config I get:
$ docker-compose config
WARNING: The SERVICE_DB_PORT variable is not set. Defaulting to a blank string.
WARNING: The SERVICE_DB_ROOT_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The ENV variable is not set. Defaulting to a blank string.
WARNING: The SERVICE_DB_APP_USER variable is not set. Defaulting to a blank string.
WARNING: The SERVICE_DB_APP_PASSWORD variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.myapp-main-db.ports contains an invalid type, it should be a number, or an object
According to default behavior of docker, Dockerfile and docker-compose never read any file or value from parent directory. It always read from current directory or from Child directory.
so for your code I suggest please use docker-compose file in source/root directory.
Example:
myapp/
docker-compose.yml
.env
src/
<my source code here>
config.properties
Compose has two separate layers of environment file handling:
When the docker-compose.yml file is read in, it reads a .env file from the same directory as the docker-compose.yml, and uses that to expand $VARIABLE references.
When an individual container is launched, it reads the env_file: and uses that to set environment variables in the container.
Since .env isn't in the same directory as the docker-compose.yml file, Compose won't read it before expanding variables in the file. That leads to the container getting empty strings for the database credentials, and the error because ports: is malformed.
You can either move .env into the same directory as docker-compose.yml
mv .env docker
docker-compose -f docker/docker-compose.yml up
or pass a docker-compose --env-file option
docker-compose -f docker-docker-compose.yml --env-file .env up
You don't need env_file: in the docker-compose.yml file for this particular use and can remove it.
To debug this further, you can docker-compose run a container with an alternate command:. You could for example docker-compose run myapp-main-db env to print out its environment instead of launching a database. The flow I'd expect you to see here is:
Compose reads docker/.env (from the same directory as the docker-compose.yml file), which is absent
Compose replaces variables in the file; I'd expect to see MYSQL_USER to be an empty string
Compose reads the env_file:; the container will have the additional environment variable SERVICE_DB_APP_USER=my-service-user from that file
Quote from the official Docker docs: Using the “--env-file” option.
You can use an .env file from anywhere, using the --env-file option:
By passing the file as an argument, you can store it anywhere and name
it appropriately, for example, .env.ci, .env.dev, .env.prod. Passing
the file path is done using the --env-file option:
docker-compose --env-file ./config/.env.dev up
Obviously, you cannot go to a parent directory of your current directory to get the .env file when you are inside the docker-compose, thus, not like this:
version: "3.7"
services:
myapp-main-db:
env_file:
- ../.env
One way to use an .env file from somewhere else like here at the parent folder is to use the --project-directory parameter of the docker-compose command. According to the official Docker docs:
The .env file path is as follows:
Starting with +v1.28, .env file is placed at the base of the project
directory
For previous versions, it is placed in the current working
directory where the Docker Compose command is executed unless a
--project-directory is defined which overrides the path for the .env file. This inconsistency is addressed in +v1.28 by limiting the
filepath to the project directory."
Which would mean in your case:
docker-compose --project-directory .. up
Or: you can reach any .env file on your system if you pass it as the --env-file parameter of the docker-compose command.
docker-compose --env-file ../.env up
If you use neither --project-directory nor --env-file you need to put the .env file where the docker-compose file is.
I'm trying to get the Env-Variables in Docker-Compose to work. My Files:
env/test.env:
XUSER=you
XHOME=/home/${XUSER}
docker-compose.yml:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
env_file:
- env/test.env
user: "${XUSER}"
docker-compose up --build
docker-compose config
WARNING: The XUSER variable is not set. Defaulting to a blank string.
services:
kernel:
build:
context: xyz
container_name: xyz
environment:
XHOME: /home/you
XUSER: you
image: xyz:latest
user: ''
As you can see user: '' is an empty string, but the env_file works. I found some old Bug reports about this issue, I'm not sure I doing something wrong or not.
Although the other answers are both correct they do not highlight the underlying misunderstanding here enough:
With the env_file option you can specify a file with variables to be injected into the environment in the container.
Using variable substitution in the docker-compose.yml you can access variables in the environment of the docker-compose command, i.e. on the host.
You can set these using the usual mechanisms of your OS/shell, e.g. in bash:
export XUSER=you
docker-compose up
Additionally with docker-compose you can use a .env file in the current directory.
So in your concrete example you should just move env/test.env to .env to add the variables to the environment of docker-compose for variable substitution.
If you also want to add them to the environment in the container you can do it like this:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
# add variables from the docker-compose environment to the container:
environment:
- XUSER=$XUSER
# or even shorter:
- XHOME
# use variable from the docker-compose environment in the config:
user: "${XUSER}"
It says WARNING: The XUSER variable is not set. Defaulting to a blank string. because ${XUSER} doesn't exist at the time this is executed:
user: "${XUSER}"
${XUSER} is not in your environment (you can verify this by running: env | grep XUSER, which should output nothing), and docker-compose didn't find any .env file at the same level or no .env file was passed at the time you ran docker-compose up --build or docker-compose config
Flexible solution:
Rename env/test.env for .env and place it a the root of the folder container your docker-compose file so that docker automatically parses it.
Or use:
docker-compose --env-file path/to/env/test.env up --build
docker-compose --env-file path/to/env/test.env config
Permanent solution:
Export them manually in your environment by running:
export XUSER=you && export XHOME=/home/${XUSER}
Or you use your env/test.env file as a source (note that you'll need to prefix with 'export'):
env/test.env:
export XUSER=you
export XHOME=/home/${XUSER}
And then your run . /path/to/env/test.env or source /path/to/env/test.env
What you need to do is create .env file at the same directory as your docker-compose.yml file, the content of .env is :
XUSER=user1
then run docker-compose config
reference : https://docs.docker.com/compose/environment-variables/
Since +1.28 .env file is placed in project root, not where docker-compose is executed. If you do that the variables will be automatically pulled through to be available to the container.
This works great in dev, especially with a a bind-mount volume to make .env available to compose in project root without also going into image build by including .env in .dockerignore
But in production I was not comfortable including it in my project root especially since I was pulling those project files down from github. The Compose file expects them to be in the production environment to replace for substitution SECRET_VAR=${SECRET_VAR}
So one hack solution was to stick the .env file high in my production directory tree, far away from my project (ideally these would come from an environment store on the hosting service, or another encrypted store), but inject those variables into the container at runtime by using the --env_file flag in Compose up.
The env_file flag works Like this:
docker-compose -f docker-compose.prod.yml --env-file ../.env up -d
Its in the docs
I also started encountering this after upgrading to Docker Desktop 4.12.0. This error started happening for quoted strings inside of .env (when using env_file to load variables in docker-compose.yml). In that case, be sure to use single quotes instead of double quotes, i.e.
MY_VAR='foo$bar'
# ... instead of...
MY_VAR="foo$bar"
Try this, I hope it will work.
You need to escape the variable if you want it to be expanded inside the container, using a double-dollar sign ($${envVariable}).
If however, you want it to be interpreted on your host, the $envVariable needs to be defined in your environment or in the .env file. The env_file option defines environment variables that will be available inside the container only.
I have a ton of docker-compose files. Some defining my microservices, some defining the infrastructural pieces, like db, message bus etc. I need to merge them all into one, big docker-compose.yml file, so that I can easily run it via my CI/CD pipeline. I wrote this shell script to do so, but it seems the docker-compose config command generates a file with absolute paths:
#!/bin/bash
ymls=`ls **/docker-compose.*.yml`
compose_string=""
for file in $ymls
do
compose_string="${compose_string} -f ${file} --project-directory ."
done
echo $compose_string
docker-compose $compose_string config > docker-compose.merged.yml
This gives me the below file contents (shortened for brevity):
services:
curator-services:
build:
args:
ELASTIC_PASSWORD: them-pass
context: E:\repos\them-project\devops\curator
manager-acl-app:
build:
context: E:\repos\them-project
dockerfile: services/ThemProject.Services.ManagerAcl/Dockerfile
Now, the easy solution would be to just run a regexp or something and substitute whatever is in the same line as the word context. The problem with that approach is that I keep my infrastructural definitions in another folder than my microservices Dockerfiles, so the context for the curator-services piece should be context: curator/ and for the manager-acl-app it should be context: ../. Is there a way around those absolute paths with docker-compose config?
You can use --project-directory option with docker-compose config.
Change your script to below
#!/bin/bash
ymls=`ls docker-compose.*.yml`
compose_string=""
DIR_PATH="." #Change it to Project Directory or current directory
for file in $ymls
do
compose_string="${compose_string} -f ${file} --project-directory $DIR_PATH"
done
docker-compose ${compose_string} config > docker-compose.yml
Ref:- https://docs.docker.com/compose/reference/overview/
Please check --project-directory in the above link
I am trying to use the same docker-compose.yml and .env files for both docker-compose and swarm. The variables from the .env file should get parsed, via sed, into a config file by running a run.sh script at boot. This setup works fine when using the docker-compose up command, but they're not getting passed when I use the docker stack deploy command.
How can I pass the variables into the container so that the run.sh script will parse them at boot?
Loading the .env file is a feature of docker-compose that is not part of the docker CLI. You can manually load the contents of this file in your shell before performing the deploy:
set -a; . ./.env; set +a
docker stack deploy -c docker-compose.yml stack_name
Other options include using docker-compose to pre process the compose file:
docker-compose config >docker-compose.processed.yml
Or you could use envsubst to replace the variables to make a compose file with the variables already expanded:
set -a; . ./.env; set +a
envsubst <docker-compose.yml >docker-compose.processed.yml
To pass shell environment variables through to containers use env_file syntax:
web:
env_file:
- web-variables.env
As docs state:
You can pass multiple environment variables from an external file through to a service’s containers with the ‘env_file’ option
However, using .env as external filename may cause unexpected results and is semantically problematic.
Placing .env in the folder where the docker-compose command serves different purpose:
As Docs, Docs2, Docs3 state:
The environment variables you define here are used for variable
substitution in your Compose file
You can set default values for environment variables using a .env
file, which Compose automatically looks for
So if compose file contains:
db:
image: "postgres:${POSTGRES_VERSION}"
You .env would contain:
POSTGRES_VERSION=4.0
This feature indeed works only in compose:
The .env file feature only works when you use the docker-compose up
command and does not work with docker stack deploy
Actually I found the best/easiest way is to just add this argument to the docker-compose.yml file:
env_file:
- .env
I would like to be able to use environment variables inside docker-compose.yml, with values passed in at the time of docker-compose up. This is the example.
I am doing this today with a basic docker run command, which is wrapped around my own script. Is there a way to achieve it with compose, without any such bash wrappers?
proxy:
hostname: $hostname
volumes:
- /mnt/data/logs/$hostname:/logs
- /mnt/data/$hostname:/data
The Docker solution:
Docker-compose 1.5+ has enabled variables substitution: Releases · docker/compose
The latest Docker Compose allows you to access environment variables from your compose file. So you can source your environment variables, then run Compose like so:
set -a
source .my-env
docker-compose up -d
For example, assume we have the following .my-env file:
POSTGRES_VERSION=14
(or pass them via command-line arguments when calling docker-compose, like so: POSTGRES_VERSION=14 docker-compose up -d)
Then you can reference the variables in docker-compose.yml using a ${VARIABLE} syntax, like so:
db:
image: "postgres:${POSTGRES_VERSION}"
And here is more information from the documentation, taken from Compose file specification
When you run docker-compose up with this configuration, Compose looks
for the POSTGRES_VERSION environment variable in the shell and
substitutes its value in. For this example, Compose resolves the image
to postgres:9.3 before running the configuration.
If an environment variable is not set, Compose substitutes with an
empty string. In the example above, if POSTGRES_VERSION is not set,
the value for the image option is postgres:.
Both $VARIABLE and ${VARIABLE} syntax are supported. Extended
shell-style features, such as ${VARIABLE-default} and
${VARIABLE/foo/bar}, are not supported.
If you need to put a literal dollar sign in a configuration value, use
a double dollar sign ($$).
The feature was added in this pull request.
Alternative Docker-based solution: Implicitly sourcing an environment variables file through the docker-compose command
If you want to avoid any Bash wrappers, or having to source a environment variables file explicitly (as demonstrated above), then you can pass a --env-file flag to the docker-compose command with the location of your environment variable file: Use an environment file
Then you can reference it within your docker-compose command without having to source it explicitly:
docker-compose --env-file .my-env up -d
If you don't pass a --env-file flag, the default environment variable file will be .env.
Note the following caveat with this approach:
Values present in the environment at runtime always override those defined inside the .env file. Similarly, values passed via command-line arguments take precedence as well.
So be careful about any environment variables that may override the ones defined in the --env-file!
The Bash solution:
I notice that Docker's automated handling of environment variables can cause confusion. Instead of dealing with environment variables in Docker, let's go back to basics, like Bash! Here is a method using a Bash script and a .env file, with some extra flexibility to demonstrate the utility of environment variables:
POSTGRES_VERSION=14
# Note that the variable below is commented out and will not be used:
# POSTGRES_VERSION=15
# You can even define the compose file in an environment variable like so:
COMPOSE_CONFIG=my-compose-file.yml
# You can define other compose files, and just comment them out
# when not needed:
# COMPOSE_CONFIG=another-compose-file.yml
Then run this Bash script in the same directory, which should deploy everything properly:
#!/bin/bash
docker rm -f `docker ps -aq -f name=myproject_*`
set -a
source .env
cat ${COMPOSE_CONFIG} | envsubst | docker-compose -f - -p "myproject" up -d
Just reference your environment variables in your compose file with the usual Bash syntax (ie ${POSTGRES_VERSION} to insert the POSTGRES_VERSION from the .env file).
While this solution involves Bash, some may prefer it because it has better separation of concerns.
Note the COMPOSE_CONFIG is defined in my .env file and used in my Bash script, but you can easily just replace {$COMPOSE_CONFIG} with the my-compose-file.yml in the Bash script.
Also note that I labeled this deployment by naming all of my containers with the "myproject" prefix. You can use any name you want, but it helps identify your containers so you can easily reference them later. Assuming that your containers are stateless, as they should be, this script will quickly remove and redeploy your containers according to your .env file parameters and your compose YAML file.
Since this answer seems pretty popular, I wrote a blog post that describes my Docker deployment workflow in more depth: Let's Deploy! (Part 1) This might be helpful when you add more complexity to a deployment configuration, like Nginx configurations, Let's Encrypt certificates, and linked containers.
It seems that docker-compose has native support now for default environment variables in a file.
All you need to do is declare your variables in a file named .env and they will be available in docker-compose.yml.
For example, for a .env file with contents:
MY_SECRET_KEY=SOME_SECRET
IMAGE_NAME=docker_image
You could access your variable inside docker-compose.yml or forward them into the container:
my-service:
image: ${IMAGE_NAME}
environment:
MY_SECRET_KEY: ${MY_SECRET_KEY}
Create a template.yml, which is your docker-compose.yml with environment variable.
Suppose your environment variables are in a file 'env.sh'
Put the below piece of code in a sh file and run it.
source env.sh;
rm -rf docker-compose.yml;
envsubst < "template.yml" > "docker-compose.yml";
A new file docker-compose.yml will be generated with the correct values of environment variables.
Sample template.yml file:
oracledb:
image: ${ORACLE_DB_IMAGE}
privileged: true
cpuset: "0"
ports:
- "${ORACLE_DB_PORT}:${ORACLE_DB_PORT}"
command: /bin/sh -c "chmod 777 /tmp/start; /tmp/start"
container_name: ${ORACLE_DB_CONTAINER_NAME}
Sample env.sh file:
#!/bin/bash
export ORACLE_DB_IMAGE=<image-name>
export ORACLE_DB_PORT=<port to be exposed>
export ORACLE_DB_CONTAINER_NAME=ORACLE_DB_SERVER
The best way is to specify environment variables outside the docker-compose.yml file. You can use env_file setting, and define your environment file within the same line. Then doing a docker-compose up again should recreate the containers with the new environment variables.
Here is how my docker-compose.yml looks like:
services:
web:
env_file: variables.env
Note:
docker-compose expects each line in an env file to be in VAR=VAL format. Avoid using export inside the .env file. Also, the .env file should be placed in the folder where the docker-compose command is executed.
The following is applicable for docker-compose 3.x
Set environment variables inside the container
method - 1 Straight method
web:
environment:
- DEBUG=1
POSTGRES_PASSWORD: 'postgres'
POSTGRES_USER: 'postgres'
method - 2 The “.env” file
Create a .env file in the same location as the docker-compose.yml
$ cat .env
TAG=v1.5
POSTGRES_PASSWORD: 'postgres'
and your compose file will be like
$ cat docker-compose.yml
version: '3'
services:
web:
image: "webapp:${TAG}"
postgres_password: "${POSTGRES_PASSWORD}"
source
When using environment variables for volumes you need:
create .env file in the same folder which contains docker-compose.yaml file
declare variable in the .env file:
HOSTNAME=your_hostname
Change $hostname to ${HOSTNAME} at docker-compose.yaml file
proxy:
hostname: ${HOSTNAME}
volumes:
- /mnt/data/logs/${HOSTNAME}:/logs
- /mnt/data/${HOSTNAME}:/data
Of course you can do that dynamically on each build like:
echo "HOSTNAME=your_hostname" > .env && sudo docker-compose up
Don't confuse the .env file and the env_file option!
They serve totally different purposes!
The .env file feeds those environment variables only to your docker compose file, which in turn, can be passed to the containers as well.
But the env_file option only passes those variables to the containers and NOT the docker compose file 😵💫
Example
OK, let's say we have this simple compose file:
services:
foo:
image: ubuntu
hostname: suchHostname # <-------------- hard coded 'suchHostname'
volumes:
- /mnt/data/logs/muchLogs:/logs # <--- hard coded 'muchLogs'
- /mnt/data/soBig:/data # <----------- hard coded 'soBig'
We don't want to hard code these anymore! So, we can put them in the current terminal's environment variables and check if docker-compose understands them:
$ export the_hostname="suchHostName"
$ export dir_logs="muchLogs"
$ export dir_data="soBig"
and change the docker-compose.yml file to:
services:
foo:
image: ubuntu
hostname: $the_hostname # <-------------- use $the_hostname
volumes:
- /mnt/data/logs/$dir_logs:/logs # <--- use $dir_logs
- /mnt/data/$dir_data:/data # <-------- usr $dir_data
Now let's check out if it worked with executing $ docker-compose convert and inspecting the output:
name: tmp
services:
foo:
hostname: suchHostName # <------------- $the_hostname
image: ubuntu
networks:
default: null
volumes:
- type: bind
source: /mnt/data/logs/muchLogs # <-- $dir_logs
target: /logs
bind:
create_host_path: true
- type: bind
source: /mnt/data/soBig # <---------- $dir_data
target: /data
bind:
create_host_path: true
networks:
default:
name: tmp_default
OK it works! But let's use the .env file instead. Since docker-compose understands the .env file, let's just create one and set it up:
# .env file (in the same directory as 'docker-compose.yml')
the_hostname="suchHostName"
dir_logs="muchLogs"
dir_data="soBig"
OK, you can test it with a NEW terminal (so that the older environment variables we set with export don't interfere and make sure everything works in a clean terminal) 🖥 Just follow step 4 again and see that it works!
So far so good 😃 However, when you stumble upon the env_file option, it gets confusing 🤔 Let's say that you want to pass a password to the docker compose file (NOT the container).
🙄 In the wrong approach, you might put a password in .secrets file:
# .secrets
somepassword="0P3N$3$#M!"
and then update the docker-compose file as follows:
services:
foo:
image: ubuntu
hostname: $the_hostname
volumes:
- /mnt/data/logs/$dir_logs:/logs
- /mnt/data/$dir_data:/data
# 🔽 BAD:
env_file:
- .env
- .secrets
entrypoint: echo "Hush! This is a secret '$somepassword'"
Now checking it just like step 4 again would result in:
WARN[0000] The "somepassword" variable is not set. Defaulting to a blank string.
name: tmp # ^
services: # |
foo: # |
entrypoint: # |
- echo # |
- Hush! This is a secret '' # <---- 😵💫 Oh no!
environment:
dir_data: soBig
dir_logs: muchLogs
somepassword: 0P3N$$3$$#M! # <--- 🤔 Huh?!
the_hostname: suchHostName
hostname: suchHostName
image: ubuntu
networks:
default: null
volumes:
- type: bind
source: /mnt/data/logs/muchLogs
target: /logs
bind:
create_host_path: true
- type: bind
source: /mnt/data/soBig
target: /data
bind:
create_host_path: true
networks:
default:
name: tmp_default
So as you can see, the $somepassord variable is only passed to the container, and NOT the docker compose file.
Wrapping up
You can pass environment variables to docker-compose files in two ways:
By exporting the variable to the terminal before running docker compose.
By putting the variables inside .env file.
The env_file option only passes those extra variables to the containers 📦 and not the compose file 🐳
Since 1.25.4, docker-compose supports the option --env-file that enables you to specify a file containing variables.
Yours should look like this:
hostname=my-host-name
And the command:
docker-compose --env-file /path/to/my-env-file config
To add an environment variable, you may define an env_file (let's call it var.env) as:
ENV_A=A
ENV_B=B
And add it to the docker compose manifest service. Moreover, you can define environment variables directly with environment.
For instance, in docker-compose.yaml:
version: '3.8'
services:
myservice:
build:
context: .
dockerfile: ./docker/Dockerfile.myservice
image: myself/myservice
env_file:
- ./var.env
environment:
- VAR_C=C
- VAR_D=D
volumes:
- $HOME/myfolder:/myfolder
ports:
- "5000:5000"
Please check here for more/updated information: Manuals → Docker →Compose →Environment variables → Overview
Use:
env SOME_VAR="I am some var" OTHER_VAR="I am other var" docker stack deploy -c docker-compose.yml
Use version 3.6:
version: "3.6"
services:
one:
image: "nginx:alpine"
environment:
foo: "bar"
SOME_VAR:
baz: "${OTHER_VAR}"
labels:
some-label: "$SOME_VAR"
two:
image: "nginx:alpine"
environment:
hello: "world"
world: "${SOME_VAR}"
labels:
some-label: "$OTHER_VAR"
I got it from Feature request: Docker stack deploy pass environment variables via cli options #939.
You cannot ... yet. But this is an alternative, think like a docker-composer.yml generator:
https://gist.github.com/Vad1mo/9ab63f28239515d4dafd
Basically a shell script that will replace your variables. Also you can use Grunt task to build your docker compose file at the end of your CI process.
I have a simple bash script I created for this it just means running it on your file before use:
https://github.com/antonosmond/subber
Basically just create your compose file using double curly braces to denote environment variables e.g:
app:
build: "{{APP_PATH}}"
ports:
- "{{APP_PORT_MAP}}"
Anything in double curly braces will be replaced with the environment variable of the same name so if I had the following environment variables set:
APP_PATH=~/my_app/build
APP_PORT_MAP=5000:5000
on running subber docker-compose.yml the resulting file would look like:
app:
build: "~/my_app/build"
ports:
- "5000:5000"
To focus solely on the issue of default and mandatory values for environment variables, and as an update to #modulito's answer:
Using default values and enforcing mandatory values within the docker-compose.yml file is now supported (from the docs):
Both $VARIABLE and ${VARIABLE} syntax are supported. Additionally when using the 2.1 file format, it is possible to provide inline default values using typical shell syntax:
${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment.
${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment.
Similarly, the following syntax allows you to specify mandatory variables:
${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment.
${VARIABLE?err} exits with an error message containing err if VARIABLE is unset in the environment.
Other extended shell-style features, such as ${VARIABLE/foo/bar}, are not supported.
This was written for Docker v20, using the docker compose v2 commands.
I was having a similar roadblock and found that the --env-file parameter ONLY works for docker compose config command. On top of that using the docker compose env_file variable, still forced me to repeat values for the variables, when wanting to reuse them in other places than the Dockerfile such as environment for docker-compose.yml. I just wanted one source of truth, my .env, with the ability to swap them per deployment stage. So here is how I got it to work, basically use docker compose config to generate a base docker-compose.yml file that will pass ARG into Dockerfile's.
.local.env This would be your .env, I have mine split for different deployments.
DEVELOPMENT=1
PLATFORM=arm64
docker-compose.config.yml - This is my core docker compose file.
services:
server:
build:
context: .
dockerfile: docker/apache2/Dockerfile
args:
- PLATFORM=${PLATFORM}
- DEVELOPMENT=${DEVELOPMENT}
environment:
- PLATFORM=${PLATFORM}
- DEVELOPMENT=${DEVELOPMENT}
Now sadly I do need to pass in the variables twice, once for the Dockerfile, the other for environment. However, they are still coming from the single source .local.env so at least I do not need to repeat values.
I then use docker compose config to generate a semi-final docker-compose.yml. This lets me pass in my companion override docker-compose.local.yml for where the final deployment is happening.
docker compose --env-file=.local.env -f docker-compose.config.yml config > docker-compose.yml
This will now let my Dockerfile access the .env variables.
FROM php:5.6-apache
# Make sure to declare after FROM
ARG PLATFORM
ARG DEVELOPMENT
# Access args in strings with $PLATFORM, and can wrap i.e ${PLATFORM}
RUN echo "SetEnv PLATFORM $PLATFORM" > /etc/apache2/conf-enabled/environment.conf
RUN echo "SetEnv DEVELOPMENT $DEVELOPMENT" > /etc/apache2/conf-enabled/environment.conf
This then passes the .env variables from the docker-compose.yml into Dockerfile which then passes it into my Apache HTTP server, which passes it to my final destination, the PHP code.
My next step to then to pass in my docker compose overrides from my deployment stage.
docker-compose.local.yml - This is my docker-compose override.
services:
server:
volumes:
- ./localhost+2.pem:/etc/ssl/certs/localhost+2.pem
- ./localhost+2-key.pem:/etc/ssl/private/localhost+2-key.pem
Lastly, run the docker compose command.
docker compose -f docker-compose.yml -f docker-compose.local.yml up --build
Please note if you change anything in you .env file you will need to re-run the docker compose config and add --build for docker compose up. Since builds are cached it has little impact.
So for my final command I normally run:
docker compose --env-file=.local.env -f docker-compose.config.yml config > docker-compose.yml; docker compose --env-file=.local.env -f docker-compose.yml -f docker-compose.local.yml up --build
As far as I know, this is a work-in-progress. They want to do it, but it's not released yet. See 1377 (the "new" 495 that was mentioned by #Andy).
I ended up implementing the "generate .yml as part of CI" approach as proposed by #Thomas.
Add an environment variable to the .env file
Such as
VERSION=1.0.0
Then save it to deploy.sh
INPUTFILE=docker-compose.yml
RESULT_NAME=docker-compose.product.yml
NAME=test
prepare() {
local inFile=$(pwd)/$INPUTFILE
local outFile=$(pwd)/$RESULT_NAME
cp $inFile $outFile
while read -r line; do
OLD_IFS="$IFS"
IFS="="
pair=($line)
IFS="$OLD_IFS"
sed -i -e "s/\${${pair[0]}}/${pair[1]}/g" $outFile
done <.env
}
deploy() {
docker stack deploy -c $outFile $NAME
}
prepare
deploy
Use .env file to define dynamic values in docker-compse.yml. Be it port or any other value.
Sample docker-compose:
testcore.web:
image: xxxxxxxxxxxxxxx.dkr.ecr.ap-northeast-2.amazonaws.com/testcore:latest
volumes:
- c:/logs:c:/logs
ports:
- ${TEST_CORE_PORT}:80
environment:
- CONSUL_URL=http://${CONSUL_IP}:8500
- HOST=${HOST_ADDRESS}:${TEST_CORE_PORT}
Inside .env file you can define the value of these variables:
CONSUL_IP=172.31.28.151
HOST_ADDRESS=172.31.16.221
TEST_CORE_PORT=10002
I ended up using "sed" in my deploy.sh script to accomplish this, though my requirements were slightly different since docker-compose is being called by Terrafom: Passing Variables to Docker Compose via a Terraform script for an Azure App Service
eval "sed -i 's/MY_VERSION/$VERSION/' ../docker-compose.yaml"
cat ../docker-compose.yaml
terraform init
terraform apply -auto-approve \
-var "app_version=$VERSION" \
-var "client_id=$ARM_CLIENT_ID" \
-var "client_secret=$ARM_CLIENT_SECRET" \
-var "tenant_id=$ARM_TENANT_ID" \
-var "subscription_id=$ARM_SUBSCRIPTION_ID"
eval "sed -i 's/$VERSION/MY_VERSION/' ../docker-compose.yaml"
It's simple like this:
Using command line as mentioned in the documentation:
docker-compose --env-file ./config/.env.dev config
Or using a .env file, I think this is the easiest way:
web:
env_file:
- web-variables.env
Documentation with a sample