I want to pass environment variables that is readable by applications spin up by docker-compose up.
What is the proper way of using docker-compose up with varying configuration settings?
I don't want to use .env & environment: config as the environment variables are changing frequently & it is insecure to save tokens in a file.
docker-compose run -e does work a bit, but loses many.
It does not map the ports that defined in docker-compose.yml services.
Also multiple services are defined in docker-compose.yml and I don't want to use depends_on just because docker-compose up doesn't work.
Let's say I define service in docker-compose.yml
service-a:
build:
context: .
dockerfile: DockerfileA
command: node serviceA.js
In my serviceA.js, I simply use the environment variable:
console.log("This is ", process.env.KEY, "running in service A");
When I run docker-compose run -e KEY=DockerComposeRun service-a
I do get the environment variable KEY read by serviceA.js
This is DockerComposeRun running in service A
However I could only get one single service running.
I could have use environment: in docker-compose.yml
environment:
- KEY=DockerComposeUp
But in my use case, each docker compose would have different environment variable values, meaning I would need to edit the file each time before I do docker-compose.
Also, not only single service would use the same environment variable, .env even done a better job, but it is not desired.
There doesn't seem to be a way to do the same for docker-compose up
I have tried KEY=DockerComposeUp docker-compose up,
but what I get is undefined .
Export doesn't work for me as well, it seems they are all about using environment variable for docker-compose.yml instead of for the applications in container
To safely pass sensitive configuration data to your containers you can use Docker secrets. Everything passed through Secrets is encrypted.
You can create and manage secrets using the commands below:
docker secret create
docker secret inspect
docker secret ls
docker secret rm
And use them in your docker-compose file, either referring to existing secrets (external) or use a file:
secrets:
my_first_secret:
file: ./secret_data
my_second_secret:
external: true
You can use environment like this:
service-a:
build:
context: .
dockerfile: DockerfileA
command: node serviceA.js
environment:
KEY=DockerComposeRun
Refer at: https://docs.docker.com/compose/environment-variables/
Related
I'm trying to get the Env-Variables in Docker-Compose to work. My Files:
env/test.env:
XUSER=you
XHOME=/home/${XUSER}
docker-compose.yml:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
env_file:
- env/test.env
user: "${XUSER}"
docker-compose up --build
docker-compose config
WARNING: The XUSER variable is not set. Defaulting to a blank string.
services:
kernel:
build:
context: xyz
container_name: xyz
environment:
XHOME: /home/you
XUSER: you
image: xyz:latest
user: ''
As you can see user: '' is an empty string, but the env_file works. I found some old Bug reports about this issue, I'm not sure I doing something wrong or not.
Although the other answers are both correct they do not highlight the underlying misunderstanding here enough:
With the env_file option you can specify a file with variables to be injected into the environment in the container.
Using variable substitution in the docker-compose.yml you can access variables in the environment of the docker-compose command, i.e. on the host.
You can set these using the usual mechanisms of your OS/shell, e.g. in bash:
export XUSER=you
docker-compose up
Additionally with docker-compose you can use a .env file in the current directory.
So in your concrete example you should just move env/test.env to .env to add the variables to the environment of docker-compose for variable substitution.
If you also want to add them to the environment in the container you can do it like this:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
# add variables from the docker-compose environment to the container:
environment:
- XUSER=$XUSER
# or even shorter:
- XHOME
# use variable from the docker-compose environment in the config:
user: "${XUSER}"
It says WARNING: The XUSER variable is not set. Defaulting to a blank string. because ${XUSER} doesn't exist at the time this is executed:
user: "${XUSER}"
${XUSER} is not in your environment (you can verify this by running: env | grep XUSER, which should output nothing), and docker-compose didn't find any .env file at the same level or no .env file was passed at the time you ran docker-compose up --build or docker-compose config
Flexible solution:
Rename env/test.env for .env and place it a the root of the folder container your docker-compose file so that docker automatically parses it.
Or use:
docker-compose --env-file path/to/env/test.env up --build
docker-compose --env-file path/to/env/test.env config
Permanent solution:
Export them manually in your environment by running:
export XUSER=you && export XHOME=/home/${XUSER}
Or you use your env/test.env file as a source (note that you'll need to prefix with 'export'):
env/test.env:
export XUSER=you
export XHOME=/home/${XUSER}
And then your run . /path/to/env/test.env or source /path/to/env/test.env
What you need to do is create .env file at the same directory as your docker-compose.yml file, the content of .env is :
XUSER=user1
then run docker-compose config
reference : https://docs.docker.com/compose/environment-variables/
Since +1.28 .env file is placed in project root, not where docker-compose is executed. If you do that the variables will be automatically pulled through to be available to the container.
This works great in dev, especially with a a bind-mount volume to make .env available to compose in project root without also going into image build by including .env in .dockerignore
But in production I was not comfortable including it in my project root especially since I was pulling those project files down from github. The Compose file expects them to be in the production environment to replace for substitution SECRET_VAR=${SECRET_VAR}
So one hack solution was to stick the .env file high in my production directory tree, far away from my project (ideally these would come from an environment store on the hosting service, or another encrypted store), but inject those variables into the container at runtime by using the --env_file flag in Compose up.
The env_file flag works Like this:
docker-compose -f docker-compose.prod.yml --env-file ../.env up -d
Its in the docs
I also started encountering this after upgrading to Docker Desktop 4.12.0. This error started happening for quoted strings inside of .env (when using env_file to load variables in docker-compose.yml). In that case, be sure to use single quotes instead of double quotes, i.e.
MY_VAR='foo$bar'
# ... instead of...
MY_VAR="foo$bar"
Try this, I hope it will work.
You need to escape the variable if you want it to be expanded inside the container, using a double-dollar sign ($${envVariable}).
If however, you want it to be interpreted on your host, the $envVariable needs to be defined in your environment or in the .env file. The env_file option defines environment variables that will be available inside the container only.
Is there a way to pass environment variables from one service to the other inside docker-compose.yml ?
services:
testService:
environment:
TEST_KEY: 1234
testServiceTests:
environment:
TEST_KEY: I want to pull in the value 1234 here from service1
No.
However, there's an alternative. You may provide environment variables to all the services within the Docker Compose file by exposing them either from your shell, when you run the Compose or by using a special .env file, See documentation.
Using this approach, you would have a global (for the Compose) environment variable, say GLOBAL_TEST_KEY (it needn't have a different name) and you would be able to share this across multiple services:
services:
testService:
environment:
TEST_KEY: ${GLOBAL_TEST_KEY}
testServiceTests:
environment:
TEST_KEY: ${GLOBAL_TEST_KEY}
And then: docker-compose run -e GLOBAL_TEST_KEY="Some value" ....
Or, create a file called .env alongside docker-compose.yaml and, in .env:
GLOBAL_TEST_KEY="Some value"
And then: docker-compose run ...
NOTE No need to reference .env as it's included by default
I have docker-compose.yml with such content:
version: '3'
services:
some_service:
build:
dockerfile: Dockerfile
ports:
- '8080:${PORT}'
And I have my codeship-steps.yml with:
- type: parallel
steps:
- service: some_service
command: printenv
Also, I have .env file with:
PORT=8080
And when I'm trying to run locally jet steps I getting an error:
strconv.ParseInt: parsing "${PORT}": invalid syntax
I'm trying to pass this env variable in different ways, but I have no success. Is it possible at all or .env variables with Codeship are only for application inside docker and not for configuration?
Environment variables are not available inside the configuration files.
That said, in most cases you also don't need to explicitly specify the external port for an exposed service. Especially in combination with parallel steps this can cause issues with multiple services trying to bind to the same port. Additionally, linked services will always be able to access the some_service service on port 8080.
I'm trying to get a docker-compose file working with multiple .env files, and I'm not having any luck. I'm trying to setup three .env files:
global settings that are the same across all container instances
environment-specific settings (stuff just for test or dev)
local settings - overridable things that a developer might need to change in case they have conflicts with, say, a port number
My docker-compose.yml file looks like this:
version: '2'
services:
db:
env_file:
- ./.env
- ./.env.${ENV}
- ./.env.local
image: postgres
ports:
- ${POSTGRES_PORT}:5432
.env looks like this:
POSTGRES_USER=myapp
and the .env.development looks like this:
POSTGRES_PASSWORD=supersecretpassword
POSTGRES_HOST=localhost
POSTGRES_PORT=25432
POSTGRES_DB=myapp_development
.env.local doesn't exist in this case.
After running ENV=development docker-compose up, I receive the following output:
$ ENV=development docker-compose up
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.db.ports is invalid: Invalid port ":5432", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
From that error message, it looks like none of my environment variables are being used. I just upgraded to the newest available docker-compose as well - same errors:
$ docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6
Any ideas here? Would be nice to have a single docker-compose.yml that would work across multiple environments.
In order to apply different/multiple env_files depending on the running environment, such as development/staging/production, I think a better way for docker-compose is to use multiple docker-compose yml files.
For example:
1. Start with a base file that defines the canonical configuration for the services.
docker-compose.yml
web:
image: example/my_web_app:latest
env_file:
- .env
2. Add the override file for development, as its name implies, can contain configuration overrides for existing services or entirely new services.
docker-compose.override.yml
web:
build: .
volumes:
- '.:/code'
ports:
- 8883:80
env_file:
- .env.dev
When you run docker-compose up it reads the overrides automatically.
3. Create another override file for the production environment.
docker-compose.prod.yml
web:
ports:
- 80:80
env_file:
- .env.prod
To deploy with this production Compose file you can run
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up
Note
My Docker version:
$ docker -v
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose -v
docker-compose version 1.22.0, build f46880fe
Reference: https://docs.docker.com/compose/extends/
Keep in mind that there are 2 different environments where you are defining variables. The host machine where you are executing the docker-compose command, and the container itself (running the db service in your case).
Your docker-compose.yml file has access to your host's environment variables. Hence ENV is reachable from the docker-compose command, but not these in your .env files.
On the contrary, the value for ENV is not reachable inside the container, but all variables defined in your .env files will.
I don't know if you really need your db container to access the variables defined on your .env.development. But at least seem that your host machine needs to have the content of that file, so when the docker-compose command is called, the POSTGRES_PORT variable is defined.
To fix your specific problem you would need to define the environment variables on your host machine too, not only for the container. You could do something like this:
#Set for host
ENV=development
#Also sets the variables on the host
source ./.env.$ENV
#POSTGRES_PORT defined in .env.development is used here
docker-compose up
#since env_file also contains .env.development, the variables will be reachable from the container.
Hope that helps.
There is a misconception regarding the .env file and the env_file option in the docker-compose.yml, as it is very ambiguous. Shin points it out very nicely in the github issue docker-compose doesn't use env_file. I will just quote his summary:
Variable substitution in your docker-compose.yml file will be pulled (in decreasing order of priority) from your shell's environment and your .env file.
Variables available in your container are a combination of values found in your env_file files and values described in the environment section of the service.
Those are two entirely separate sets of features.
while reading this page: https://docs.docker.com/compose/environment-variables/
and from my understanding, you should do the following:
for the global variables(that should not change) make an env file like so:
VAR1=VALUE1
VAR2=VALUE2
and for the others(that might change) you should add their name under environment in docker-compose.yml like this:
environment:
- VAR1
- VAR2
this will take the VAR1 and VAR2 values from the shell you are running docker-compose.
I hope this helps.
I have a docker-compose.yml file and in the terminal I am typing docker-compose up [something] but I would also like to pass an argument to docker-compose.yml. Is this possible? I've read about interpolation variables and tried to specify a variable in the .yml file using ${testval} and then docker-compose up [something] var="test" but I receive the following error:
WARNING: The testval variable is not set. Defaulting to a blank string.
ERROR: No such service: testval=test
Based on dnephin answer, I created this sample repo that you can pass an variable to docker-compose up.
The usage is simple:
MAC / LINUX
TEST= docker-compose up to create and start both app and db container. The api should then be running on your docker daemon on port 3030.
TEST=DO docker-compose up to create and start both app and db container. The api should execute the npm run test inside the package.json file.
WINDOWS (Powershell)
$env:TEST="";docker-compose up to create and start both app and db container. The api should then be running on your docker daemon on port 3030.
$env:TEST="do";docker-compose up to create and start both app and db container. The api should execute the npm run test inside the package.json file.
You need to ensure 2 things:
The docker-compose.yml has the environment variable declared. For example,
services:
app:
image: python3.7
environment:
- "SECRET_KEY=${SECRET_KEY}"
have the variable available in the environment when docker-compose up is called:
SECRET_KEY="not a secret" docker-compose up
Note that this is not equivalent to pass them during build, as it is not advisable to store secrets in docker images.
You need to pass the variables as environment variables:
testvar=test docker-compose up ...
or
export testvar=test
docker-compose up
From the docs:
https://docs.docker.com/compose/reference/up/
https://docs.docker.com/compose/reference/build/
You can't pass arguments to docker-compose up, but you can pass arguments to docker-compose build:
docker-compose build --build-arg KEY1=VALUE1 --build-arg KEY2=VALUE2
I'm not sure what you want to do here, but if what you need is to pass an environmental variable to a specific container docker-compose.yml allows you to do that:
web:
...
environment:
- RAILS_ENV=production
- VIRTUAL_HOST=www.example.com
- VIRTUAL_PORT=3011
This variables will be specific for the container you specified them to, and wil not be shared between containers.
Also "docker-compose up" doesn't take any argument.
When dealing with build argumenets please declare them in compose yml file as follows
services:
app: (name of service
build:
context: docker/app/ (where is your docker build root)
dockerfile: Dockerfile (that is optional)
args:
- COMPOSER_AUTH_TOKEN (name of variable, value will be taken from host environment)
Well before running docker-compose up, export variable as other guys suggested. It will work. I tried. Use docker compose version 3 and above. Have fun
Compose supports declaring default environment variables in an environment file named .env placed in the project directory.
Step 1:
Create a file named .env in the project directory
Step 2:
Declare variables in the form VAR=VAL
NOTE: There is no special handling of quotation mark i.e. TESTVAL='test' means TESTVAL is 'test'(with quotation mark) and not just test. So you'd declare it as TESTVAL=test.
Step 3:
Use the variables in the Compose file as:
environment:
myval=${TESTVAL}
Documentation: Declare default environment variables in file
BONUS: If you are building image on the fly in you docker-compose.yaml, then you can even pass the build args using environment variables. Eg:
version: "3.8"
services:
myapp:
build:
context: ./myDir
dockerfile: ./myDir/myDockerfile
args:
- MYARG=${TESTVAL}
I was trying to find solution for batch file, based on Rafael Delboni answer you can add command inside batch file for calling powershell:
powershell $env:TEST="";docker-compose up ...
but instead of that because it's expensive to call powershell inside batch file you can initialize TEST variable inside batch file and then call your docker-compose command.
Something like this:
set TEST = ...
docker compose up ...