Within the parent directory (./) of docker-compose.yml, I would like to scope some environment variables (e.g. MY_EXTERNAL_PORT) for both docker-compose.yml and for the running container. The simplest possible way I could think to do this (which fails below) is to define MY_EXTERNAL_PORT in the docker-compose.yml as follows:
...
environment:
- MY_EXTERNAL_PORT=1112
ports:
- "${MY_EXTERNAL_PORT}:1111"
...
How can I avoid the following error:
au#pc:~/Containers/mycontainer$ docker-compose up
WARNING: The DOCKER_ISQL_PORT variable is not set. Defaulting to a blank string.
WARNING: The DOCKER_HTTP_PORT variable is not set. Defaulting to a blank string.
Related
I'm running a containerized Milvus Standalone database (Milvus) and I'm trying to find the location of items added to the database. In the docker-compose.yml file, the volume location is defined as follows:
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
Checking my docker server, I do not find an environment variable named DOCKER_VOLUME_DIRECTORY.
What does this definition mean? Also, what does the
:-.
part mean?
It is using Shell Parameter expansion:
${parameter:-word}
If parameter is unset or null, then word is used as a default value.
In this case, as DOCKER_VOLUME_DIRECTORY is not set, the default value of . (the current directory) is used.
$ echo ${DOCKER_VOLUME_DIRECTORY:-.}
.
So the volume will effectively be:
volumes:
- ./volumes/etcd:/etcd
I am new to docker .I recently came across one of the docker-compose file from our org ACR, the ports are defined as variables in the compose file. I DO NOT have the docker file of that image used in docker-compose file.
version: "3"
services:
webapp:
image: p32d1830151.azurecr.io/web/weblogic:0.1
container_name: banker
hostname: banker
ports:
- "${URL_PORT}:8080"
- "${TCP_PORT}:12345"
The advantage of this docker-compose.yml file is that
It can be executed with docker-compose up -d . The default value is taken
It can be executed with docker-compose --env-file d.env up -d , that overrides the default value with the values from env file.
I tried to do achieve the same with my docker images that is different from the same , and it fails with error
docker-compose up -d
WARNING: The URL_PORT variable is not set. Defaulting to a blank string.
WARNING: The TCP_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.webimage.ports contains an invalid type, it should be a number, or an object
services.webimage.ports contains an invalid type, it should be a number, or an object
but it works if I define the port as
ports:
- "URL_PORT:8080"
- "TCP_PORT:12345"
or
ports:
- "URL_PORT:${URL_PORT}"
- "TCP_PORT:${TCP_PORT}"
Has - "${URL_PORT}:8080"
- "${TCP_PORT}:12345" for any...? if so please let me know how to make this work ?
Should something be added to the docker file ?
Do we have some documentation on this ?
How do I attain this flexibility ?
1 How does this work ?
Notice that ${} or single a $ substitutes environment variables inside the docker-compose.yml.
This means when you've set an environment variable like URL_PORT docker-compose will replace $URL_PORT with its value.
Setting the environemnt variable can be done by running export URL_PORT=1234 before you do docker-compose up -d or by placing a .env-file containing URL_PORT=1234 in the current directory.
2 Should something be added to the docker file ?
No you don't have to add anything to the Dockerfile
3 Do we have some documentation on this ?
See: Environment variables in Compose
4 How do I attain this flexibility ?
By setting environment variables.
I try to set variables in my `docker-compose.yml` file in a separate `PORTAL_ENVIRONMENT.env` file
I don't know, what I am doing wrong.
I have this output:
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The HOST variable is not set. Defaulting to a blank string.
WARNING: The PORT2 variable is not set. Defaulting to a blank string.
WARNING: The PORT3 variable is not set. Defaulting to a blank string.
WARNING: The PORT1 variable is not set. Defaulting to a blank string.
docker-compose.yml looks like this:
client_app:
env_file:
- PORTAL_ENVIRONMENT.env
image: 'client:latest'
build:
context: ./
container_name: client
depends_on:
- db
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/${POSTGRES_DB}
- SPRING_DATASOURCE_USERNAME=${POSTGRES_USER}
- SPRING_DATASOURCE_PASSWORD=${POSTGRES_PASSWORD}
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
- HOST_NAME=${HOST}
- CREDIT_POST=${PORT1}
- PRODUCT_POST=${PORT3}
ports:
- ${PORT2}:8080
(there are 3 separate modules,every looks almost the same)
PORTAL_ENVIRONMENT.env looks like this:
HOST=localhost
PORT1=8089
PORT2=8090
PORT3=8091
POSTGRES_DB=create_credit
POSTGRES_USER=user
POSTGRES_PASSWORD=pass123
I run it with:
sudo docker-compose up --force-recreate --build
When specifying your service's environment, you want those environment variables (HOST, POSTGRES_DB, etc) to be accessible to docker-compose at the time that it parses your docker-compose file. To do that, you should put them in a file called just .env. (Alternatively, if they are set in your shell environment at the time you run docker-compose, that is okay too, but you probably want to be keeping them in a .env file.)
Instead, you're trying to use env_file: in the docker-compose file. That specifies that the service that uses the env_file should look in that file and then update its own environment with that information. env_file is like environment:, but it looks at a file. It's just for the container to use, and docker-compose can't use it to set up how to run the container.
If you'd like to also pass variables from a .env file into a container, you can do something like one of these:
environment:
- MY_VARIABLE=${VARIABLE_IN_MY_ENV_FILE}
- MY_VARIABLE_SAFER=${VARIABLE_IN_MY_ENV_FILE:?err}
(the ?err will cause the startup to fail if the environment variable is not set, which is a nice trick.)
If you want the .env file used by docker-compose to not be named .env (maybe you are fond of the name you already have for some reason), docker-compose also has its own --env-file command-line option which you can use to specify the path of the .env file to use.
If you're still curious or confused you could also check out the docker-compose Environment Variables documentation page - that one talks about both environment-setting for docker-compose and for containers, all in the same webpage.
Also make sure you are running docker-compose up in the correct folder where your YML File is, and that you have your .env file in the correct path in relation to your YML file
I have an environment file that contain a variable called Database_User.
In my docker-compose, I have this:
services:
database:
container_name: postgres
hostname: db
image: postgres:12-alpine
environment:
- POSTGRES_USER=${Database_User}
ports:
- "54321:5432"
env_file: project/myproject/.env
But, I am getting an error:
The Database_User variable is not set. Defaulting to a blank string.
To get this right it is important to correctly understand the differences between the environment and env_file properties and the .env file:
environment and env_file let you specify environment variables to be set in the container:
with environment you can specify the variables explicitly in the docker-compose.yml
with env_file you implicitly import all variables from that file
mixing both on the same service is bad practice since it will easily lead to unexpected behavior
the .env file is loaded by docker-compose into the environment of docker-compose itself where it can be used
to alter the behavior of docker-compose with respective CLI environment variables
for variable substitution in the docker-compose.yml
So your current configuration does not do what you probably think it does:
env_file: project/myproject/.env
Will load that .env file into the environment of the container and not of docker-compose. Therefor Database_User won't be set for variable substitution:
environment:
- POSTGRES_USER=${Database_User}
The solution here: remove env_file from your docker-compose.yml and place .env in the same directory you are starting docker-compose from. Alternatively you can specify an .env file by path with the --env-file flag:
docker-compose --env-file project/myproject/.env up
EDIT:
This answer clearly explains the important difference from defining environment variables in an env file vs using variable substitution in a docker-compose file. I have edited my answer for clarity but please make sure you also understand the other answer.
Option 1
You likely have not set the environment variable Database_User for variable substitution to work and need to source wherever you have that defined before running docker-compose.
source ./database_user_defined_here.env
docker-compose up
You can review the docs on environment variable substitution: https://docs.docker.com/compose/environment-variables/#substitute-environment-variables-in-compose-files
Option 2
If you're using an environment file, you shouldn't have to specify the environment section (then just define POSTGRES_USER=... in that file). Then you'd run:
docker-compose --env-file ./your-env-file.env up
Or you can also specify the env file in docker-compose.yml:
database:
...
env_file:
- your-env-file.env
However, you must specify the environment variables explicitly in this file. These docs go over env file syntax.
Within my docker-compose.yml file I have a service with lines like the following
environment:
- ENV1=hello
- ENV2=world
command: -f ./tmp/config.toml
volumes:
- ./config/config_x.toml:/tmp/config.toml
I want to make it so that if ENV1 is defined (i.e. not an empty string), then mount the volume
- ./config/config_x.toml:/tmp/config.toml
otherwise, mount the volume
- ./config/config_y.toml:/tmp/config.toml
What would be the best way of doing this?
You can add an env variable which holds the name of the volume you want to mount. For ex:
environment:
- ENV1=myvolume
- ENV2=world
command: -f ./tmp/config.toml
volumes:
- ./${myvolume}/config_x.toml:/tmp/config.toml
Docker-compose doesn't have explicit conditional statement(if) concepts.
To achieve what you want, you have to define a single property key which the value is different according to your target environment.
Besides, the interpolation of variable in a docker-compose template is done broadly in two ways :
set the variables in the caller shell
use an .env file (while it doesn't work with docker stack deploy).
Using .env is not enough for your case because you need to have a file by environment with the expected value for the same property key.
A possible approach would be probably to set variables in the shell.
Either manually : SRC_VOLUME=config_x.toml docker-compose up
and SRC_VOLUME=config_y.toml docker-compose up for the second.
Or by defining two files that contain the key=value that you have to write into your shell environment.
For example :
SRC_VOLUME=config_x.toml
and that in the second :
SRC_VOLUME=config_y.toml
In any case you just need to use a placeholder in the docker-compose template :
volumes:
- ./config/${SRC_VOLUME}:/tmp/config.toml