I have an environment file that contain a variable called Database_User.
In my docker-compose, I have this:
services:
database:
container_name: postgres
hostname: db
image: postgres:12-alpine
environment:
- POSTGRES_USER=${Database_User}
ports:
- "54321:5432"
env_file: project/myproject/.env
But, I am getting an error:
The Database_User variable is not set. Defaulting to a blank string.
To get this right it is important to correctly understand the differences between the environment and env_file properties and the .env file:
environment and env_file let you specify environment variables to be set in the container:
with environment you can specify the variables explicitly in the docker-compose.yml
with env_file you implicitly import all variables from that file
mixing both on the same service is bad practice since it will easily lead to unexpected behavior
the .env file is loaded by docker-compose into the environment of docker-compose itself where it can be used
to alter the behavior of docker-compose with respective CLI environment variables
for variable substitution in the docker-compose.yml
So your current configuration does not do what you probably think it does:
env_file: project/myproject/.env
Will load that .env file into the environment of the container and not of docker-compose. Therefor Database_User won't be set for variable substitution:
environment:
- POSTGRES_USER=${Database_User}
The solution here: remove env_file from your docker-compose.yml and place .env in the same directory you are starting docker-compose from. Alternatively you can specify an .env file by path with the --env-file flag:
docker-compose --env-file project/myproject/.env up
EDIT:
This answer clearly explains the important difference from defining environment variables in an env file vs using variable substitution in a docker-compose file. I have edited my answer for clarity but please make sure you also understand the other answer.
Option 1
You likely have not set the environment variable Database_User for variable substitution to work and need to source wherever you have that defined before running docker-compose.
source ./database_user_defined_here.env
docker-compose up
You can review the docs on environment variable substitution: https://docs.docker.com/compose/environment-variables/#substitute-environment-variables-in-compose-files
Option 2
If you're using an environment file, you shouldn't have to specify the environment section (then just define POSTGRES_USER=... in that file). Then you'd run:
docker-compose --env-file ./your-env-file.env up
Or you can also specify the env file in docker-compose.yml:
database:
...
env_file:
- your-env-file.env
However, you must specify the environment variables explicitly in this file. These docs go over env file syntax.
Related
I try to set variables in my `docker-compose.yml` file in a separate `PORTAL_ENVIRONMENT.env` file
I don't know, what I am doing wrong.
I have this output:
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The HOST variable is not set. Defaulting to a blank string.
WARNING: The PORT2 variable is not set. Defaulting to a blank string.
WARNING: The PORT3 variable is not set. Defaulting to a blank string.
WARNING: The PORT1 variable is not set. Defaulting to a blank string.
docker-compose.yml looks like this:
client_app:
env_file:
- PORTAL_ENVIRONMENT.env
image: 'client:latest'
build:
context: ./
container_name: client
depends_on:
- db
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/${POSTGRES_DB}
- SPRING_DATASOURCE_USERNAME=${POSTGRES_USER}
- SPRING_DATASOURCE_PASSWORD=${POSTGRES_PASSWORD}
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
- HOST_NAME=${HOST}
- CREDIT_POST=${PORT1}
- PRODUCT_POST=${PORT3}
ports:
- ${PORT2}:8080
(there are 3 separate modules,every looks almost the same)
PORTAL_ENVIRONMENT.env looks like this:
HOST=localhost
PORT1=8089
PORT2=8090
PORT3=8091
POSTGRES_DB=create_credit
POSTGRES_USER=user
POSTGRES_PASSWORD=pass123
I run it with:
sudo docker-compose up --force-recreate --build
When specifying your service's environment, you want those environment variables (HOST, POSTGRES_DB, etc) to be accessible to docker-compose at the time that it parses your docker-compose file. To do that, you should put them in a file called just .env. (Alternatively, if they are set in your shell environment at the time you run docker-compose, that is okay too, but you probably want to be keeping them in a .env file.)
Instead, you're trying to use env_file: in the docker-compose file. That specifies that the service that uses the env_file should look in that file and then update its own environment with that information. env_file is like environment:, but it looks at a file. It's just for the container to use, and docker-compose can't use it to set up how to run the container.
If you'd like to also pass variables from a .env file into a container, you can do something like one of these:
environment:
- MY_VARIABLE=${VARIABLE_IN_MY_ENV_FILE}
- MY_VARIABLE_SAFER=${VARIABLE_IN_MY_ENV_FILE:?err}
(the ?err will cause the startup to fail if the environment variable is not set, which is a nice trick.)
If you want the .env file used by docker-compose to not be named .env (maybe you are fond of the name you already have for some reason), docker-compose also has its own --env-file command-line option which you can use to specify the path of the .env file to use.
If you're still curious or confused you could also check out the docker-compose Environment Variables documentation page - that one talks about both environment-setting for docker-compose and for containers, all in the same webpage.
Also make sure you are running docker-compose up in the correct folder where your YML File is, and that you have your .env file in the correct path in relation to your YML file
I'm trying to get the Env-Variables in Docker-Compose to work. My Files:
env/test.env:
XUSER=you
XHOME=/home/${XUSER}
docker-compose.yml:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
env_file:
- env/test.env
user: "${XUSER}"
docker-compose up --build
docker-compose config
WARNING: The XUSER variable is not set. Defaulting to a blank string.
services:
kernel:
build:
context: xyz
container_name: xyz
environment:
XHOME: /home/you
XUSER: you
image: xyz:latest
user: ''
As you can see user: '' is an empty string, but the env_file works. I found some old Bug reports about this issue, I'm not sure I doing something wrong or not.
Although the other answers are both correct they do not highlight the underlying misunderstanding here enough:
With the env_file option you can specify a file with variables to be injected into the environment in the container.
Using variable substitution in the docker-compose.yml you can access variables in the environment of the docker-compose command, i.e. on the host.
You can set these using the usual mechanisms of your OS/shell, e.g. in bash:
export XUSER=you
docker-compose up
Additionally with docker-compose you can use a .env file in the current directory.
So in your concrete example you should just move env/test.env to .env to add the variables to the environment of docker-compose for variable substitution.
If you also want to add them to the environment in the container you can do it like this:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
# add variables from the docker-compose environment to the container:
environment:
- XUSER=$XUSER
# or even shorter:
- XHOME
# use variable from the docker-compose environment in the config:
user: "${XUSER}"
It says WARNING: The XUSER variable is not set. Defaulting to a blank string. because ${XUSER} doesn't exist at the time this is executed:
user: "${XUSER}"
${XUSER} is not in your environment (you can verify this by running: env | grep XUSER, which should output nothing), and docker-compose didn't find any .env file at the same level or no .env file was passed at the time you ran docker-compose up --build or docker-compose config
Flexible solution:
Rename env/test.env for .env and place it a the root of the folder container your docker-compose file so that docker automatically parses it.
Or use:
docker-compose --env-file path/to/env/test.env up --build
docker-compose --env-file path/to/env/test.env config
Permanent solution:
Export them manually in your environment by running:
export XUSER=you && export XHOME=/home/${XUSER}
Or you use your env/test.env file as a source (note that you'll need to prefix with 'export'):
env/test.env:
export XUSER=you
export XHOME=/home/${XUSER}
And then your run . /path/to/env/test.env or source /path/to/env/test.env
What you need to do is create .env file at the same directory as your docker-compose.yml file, the content of .env is :
XUSER=user1
then run docker-compose config
reference : https://docs.docker.com/compose/environment-variables/
Since +1.28 .env file is placed in project root, not where docker-compose is executed. If you do that the variables will be automatically pulled through to be available to the container.
This works great in dev, especially with a a bind-mount volume to make .env available to compose in project root without also going into image build by including .env in .dockerignore
But in production I was not comfortable including it in my project root especially since I was pulling those project files down from github. The Compose file expects them to be in the production environment to replace for substitution SECRET_VAR=${SECRET_VAR}
So one hack solution was to stick the .env file high in my production directory tree, far away from my project (ideally these would come from an environment store on the hosting service, or another encrypted store), but inject those variables into the container at runtime by using the --env_file flag in Compose up.
The env_file flag works Like this:
docker-compose -f docker-compose.prod.yml --env-file ../.env up -d
Its in the docs
I also started encountering this after upgrading to Docker Desktop 4.12.0. This error started happening for quoted strings inside of .env (when using env_file to load variables in docker-compose.yml). In that case, be sure to use single quotes instead of double quotes, i.e.
MY_VAR='foo$bar'
# ... instead of...
MY_VAR="foo$bar"
Try this, I hope it will work.
You need to escape the variable if you want it to be expanded inside the container, using a double-dollar sign ($${envVariable}).
If however, you want it to be interpreted on your host, the $envVariable needs to be defined in your environment or in the .env file. The env_file option defines environment variables that will be available inside the container only.
The docker-compose will automatically read the .env file if it resides in a same with the docker-compose.yml directory. With this in mind, I go ahead and in .env file define the app-container environment variable to store the container URL, like so:
app-container=12345.dkr.ecr.us-west-2.amazonaws.com/container-image:v003
In a docker-compose.yml file I then substitute the container URL with the environment variable app-container.
version: '3'
services:
app1:
build:
context: .
dockerfile: my.dockerfile
image: ${app-container}
It would be great if I could pass the same app-container environment variable to substitute the container's URL defined in Helm's values.yaml file. So, instead of storing the container's URL explicitly
app:
image: 12345.dkr.ecr.us-west-2.amazonaws.com/container-image:v003
the Helm's values.yaml file could use the same app-container environment variable that was already defined in .env file and used by docker-compose. The Helm's values.yaml file could be then simply defined as:
app:
image: ${app-container}
I wonder if sharing the same env file between docker-compose and Helm is possible?
Try this, its little hacky way:
First set the environment variable from .env file.
$ cat .env
appcontainer=12345.dkr.ecr.us-west-2.amazonaws.com/container-image:v003
$ while read LINE; do export "$LINE"; done < .env
NOTE: You need to remove - from app-container name. Otherwise you will get this error while setting environment variable bash: export: app-container=12345.dkr.ecr.us-west-2.amazonaws.com/container-image:v003': not a valid identifier
Now explicitly set that env in values.yaml using --set option.
helm install --set app.image=$appcontainer ./mychart
But ideally it would be good to have separate env for your docker-compose and helm chart.
Hope this helps.
Not directly, you would have to write a script to convert one format into the other.
I am trying to use an environment variable in the docker-compose.yml file. I have a file my-great-env.env.
Here is how they look:
docker-compose.yml
version: '3.4'
services:
blahblah:
images: greatimage
volumes:
- "${MY_PATH}:c:\\FinalFolder"
my-great-env.env
MY_PATH=C:\the\path\to\folder
When I try to docker run this, I get
The MY_PATH variable is not set. Defaulting to a blank string.
How do I use the environment variables defined in the .env file in docker-compose.yml?
There are two aspects of your problem:
if you want to use the environment vars in docker-compose.yml file, you have to put them in file named .env as stated here: https://docs.docker.com/compose/environment-variables/#the-env-file
you can use the env_file configuration option to use any file with variables definitions as described here: https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option BUT! they will be visible only in containers, not in the docker-compose.yml file.
I'm trying to get a docker-compose file working with multiple .env files, and I'm not having any luck. I'm trying to setup three .env files:
global settings that are the same across all container instances
environment-specific settings (stuff just for test or dev)
local settings - overridable things that a developer might need to change in case they have conflicts with, say, a port number
My docker-compose.yml file looks like this:
version: '2'
services:
db:
env_file:
- ./.env
- ./.env.${ENV}
- ./.env.local
image: postgres
ports:
- ${POSTGRES_PORT}:5432
.env looks like this:
POSTGRES_USER=myapp
and the .env.development looks like this:
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_HOST=localhost
POSTGRES_PORT=25432
POSTGRES_DB=myapp_development
.env.local doesn't exist in this case.
After running ENV=development docker-compose up, I receive the following output:
$ ENV=development docker-compose up
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.db.ports is invalid: Invalid port ":5432", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
From that error message, it looks like none of my environment variables are being used. I just upgraded to the newest available docker-compose as well - same errors:
$ docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6
Any ideas here? Would be nice to have a single docker-compose.yml that would work across multiple environments.
In order to apply different/multiple env_files depending on the running environment, such as development/staging/production, I think a better way for docker-compose is to use multiple docker-compose yml files.
For example:
1. Start with a base file that defines the canonical configuration for the services.
docker-compose.yml
web:
image: example/my_web_app:latest
env_file:
- .env
2. Add the override file for development, as its name implies, can contain configuration overrides for existing services or entirely new services.
docker-compose.override.yml
web:
build: .
volumes:
- '.:/code'
ports:
- 8883:80
env_file:
- .env.dev
When you run docker-compose up it reads the overrides automatically.
3. Create another override file for the production environment.
docker-compose.prod.yml
web:
ports:
- 80:80
env_file:
- .env.prod
To deploy with this production Compose file you can run
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up
Note
My Docker version:
$ docker -v
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose -v
docker-compose version 1.22.0, build f46880fe
Reference: https://docs.docker.com/compose/extends/
Keep in mind that there are 2 different environments where you are defining variables. The host machine where you are executing the docker-compose command, and the container itself (running the db service in your case).
Your docker-compose.yml file has access to your host's environment variables. Hence ENV is reachable from the docker-compose command, but not these in your .env files.
On the contrary, the value for ENV is not reachable inside the container, but all variables defined in your .env files will.
I don't know if you really need your db container to access the variables defined on your .env.development. But at least seem that your host machine needs to have the content of that file, so when the docker-compose command is called, the POSTGRES_PORT variable is defined.
To fix your specific problem you would need to define the environment variables on your host machine too, not only for the container. You could do something like this:
#Set for host
ENV=development
#Also sets the variables on the host
source ./.env.$ENV
#POSTGRES_PORT defined in .env.development is used here
docker-compose up
#since env_file also contains .env.development, the variables will be reachable from the container.
Hope that helps.
There is a misconception regarding the .env file and the env_file option in the docker-compose.yml, as it is very ambiguous. Shin points it out very nicely in the github issue docker-compose doesn't use env_file. I will just quote his summary:
Variable substitution in your docker-compose.yml file will be pulled (in decreasing order of priority) from your shell's environment and your .env file.
Variables available in your container are a combination of values found in your env_file files and values described in the environment section of the service.
Those are two entirely separate sets of features.
while reading this page: https://docs.docker.com/compose/environment-variables/
and from my understanding, you should do the following:
for the global variables(that should not change) make an env file like so:
VAR1=VALUE1
VAR2=VALUE2
and for the others(that might change) you should add their name under environment in docker-compose.yml like this:
environment:
- VAR1
- VAR2
this will take the VAR1 and VAR2 values from the shell you are running docker-compose.
I hope this helps.