I am trying to parameterize a docker-compose file using .env. Doc
docker-compose.yml
version: '2.3'
networks:
default: { external: true, name: $NETWORK_NAME }
services:
rabbitmq_local:
image: 'rabbitmq:3.6-management-alpine'
ports:
# The standard AMQP protocol port
- ${RABBIT_PORT}:5672
# HTTP management UI
- '15672:15672'
.env file
NETWORK_NAME=uv_atp_network
RABBIT_PORT=5672
RABBIT_HTTP_MANAGEMENT_PORT=15672
Parameterizing NETWORK_NAME works, but parameterizing RABBIT_PORT doesn't, with
The Compose file 'docker-compose.yml' is invalid because:
services.rabbitmq_local.ports contains an invalid type, it should be a number, or an object
This makes me suspect RABBIT_PORT is interpreted as string rather than a number.
How can I parameterize it correctly?
EDIT
I found that forcing the variable to be mandatory
- ${RABBIT_PORT:?unspecified_rabbit_port}:5672
gives the error, meaning it is unset or empty.
What am I doing wrong?
It seems that when running with pytest and pytest-docker-compose the .env file has to be in the root folder of pytest, along with the pytest.ini file.
Running from docker-compose from command-line doesn't have that limitation in docker 1.24.
After relocating the file, the variables could be resolved.
Related
I'm trying to generate multiple instances of the same git repository with a generic docker-compose.yml file and multiple .env files.
For this somewhere in the code I generate a temporary folder which contains:
.env:
APP_PORT="3000"
APP_NAME="app-name"
REPO_NAME="repo-name"
docker-compose.yml
version: '3.6'
services:
web-app:
image: golang:alpine
environment:
- APP_PORT
- APP_NAME
- REDIS_HOST=db-app
ports:
- ${APP_PORT}:${APP_PORT}
volumes:
- /opt/docker/repositories/${REPO_NAME}:/app
command: sh -c "cd /app && go run ./"
db-app:
image: redis:alpine
then running docker-compose config in this directory gives me the following output :
services:
db-app:
image: redis:alpine
web-app:
command: sh -c "cd /app && go run ./"
environment:
APP_NAME: app-name
APP_PORT: '3000'
REDIS_HOST: db-app
image: golang:alpine
ports:
- published: 3000
target: 3000
volumes:
- /opt/docker/repositories/repo-name:/app:rw
version: '3.6'
This did not only interpolate env variables, it also changed some fields such as ports with published and target, and a :rw at the end of my volume.
This is all done in Go, and when I try to unmarshal the output into a Go struct with yaml fields, it is not recognized as a valid docker-compose file because of the ports field (which is supposed to be an array of strings).
How can I make it so docker-compose config only replaces the ${APP_PORT} with its value and not add these extra unwanted fields ?
Reading the source code, I found this in the config types:
def legacy_repr(self):
return normalize_port_dict(self.repr())
Which is the representation you need. So I searched for legacy_repr in the source code and found this:
if 'ports' in service_dict:
service_dict['ports'] = [
p.legacy_repr() if p.external_ip or version < VERSION else p
for p in service_dict['ports']
]
So apparently, to trigger the use of the legacy representation, you either need to have an external IP address or need to do something with the version. I tried to downgrade the docker-compose.yaml file version but it didn't change anything (maybe it's the docker-compose CLI's version instead).
Reading the spec of the docker-compose config file, in the ports section, you can specify the IP address in the short syntax:
[HOST:]CONTAINER[/PROTOCOL] where:
HOST is [IP:](port | range)
CONTAINER is port | range
PROTOCOL to restrict port to specified protocol. tcp and udp values are defined by the specification, Compose implementations MAY offer support for platform-specific protocol names.
So a solution is to replace ${APP_PORT}:${APP_PORT} by:
0.0.0.0:${APP_PORT}:${APP_PORT}
By setting the external IP address to 0.0.0.0 you are not restricting anything and you force the use of the legacy representation.
Host: Centos 7
Docker version: 19.03.12, build 48a66213fe
Docker compose version: version 1.26.2, build eefe0d31
I am using the advice given here on specifying environment variables in the shell and then passing those into the compose file. So my docker-compose file looks like this:
version: '3.8'
services:
springboot:
image: <my image>
ports:
- "8445:8445"
depends_on:
- "database"
environment:
- SPRING_PROFILES_ACTIVE=dev
database:
image: "mongo"
Here's the error that I've run into:
ERROR: for 39b165c237b9_deploy_springboot_1 Cannot create container for service springboot: invalid volume specification: 'a9d7debde5ffcba2a023c12d5f1e822567e9b4047a1bda76efeefbfc80c1f622:app/data:rw': invalid mount config for type "volume": invalid mount path: 'app/data' mount path must be absolute
The corresponding Docker file does have a reference to VOLUME app/data, however that doesn't seem controversial.
If I take out the environment block, it seems to work fine (though the application starts without having the right config, but at least it doesn't error).
So what is the right way to pass on shell environment variables to the container?
EDIT
Thanks to #David Maze, it was indeed the VOLUME declaration in the original Docker file - nothing to do with docker-compose. Deleted it and it now works. Thanks!
there are 2 ways to specify directories in docker.
absolute path i.e. from the root e.g. /var/lib
relative path i.e. relative to the compose file e.g. ./app/data
Try replacing SPRING_PROFILES_ACTIVE=devby SPRING_PROFILES_ACTIVE: dev Instead, so replace = with : . And you should also add the volume to the docker-compose file:
version: "3.8"
services:
.
.
volumes:
- Path/to/your/volume
environment:
ENV_VAR1: Value1
ENV_VAR2: Value2
.
.
.
And according the error message, the path must be absolute.
docker stack deploy isnt respecting the extra_hosts parameter in my compose file. when i do a simple docker-compose up the entry is created in the /etc/hosts however when i do docker deploy –compose-file docker-compose.yml myapp it ignores extra_hosts, any insights?
Below is the docker-compose.xml:
version: '3'
services:
web:
image: user-service
deploy:
labels:
- the label
build:
context: ./
environment:
DATABASE_URL: jdbc:postgresql://dbhost:5432/postgres
ports:
- 9002:9002
extra_hosts:
- "dbhost: ${DB_HOST}"
networks:
- wellness_swarm
env_file:
- .env
networks:
wellness_swarm:
external:
name: wellness_swarm
the docker-compose config also displays the compose file properly.
This may not be a direct answer to the question as it doesn't use env variables but what I found was that the extra_hosts block in the compose file was ignored in swarm mode if entered in the format above.
i.e. this works for me and puts entries in /etc/hosts in the container:
extra_hosts:
retisdev: 10.48.161.44
retistesting: 10.48.161.44
whereas when entered in the other format it gets ignored when deploying as a service
extra_hosts:
- "retisdev=10.48.161.44"
- "retistesting=10.48.161.44"
I think it's an ordering issue. The ${} variable you've got in the compose file runs during the YAML processing before the service definition is created. Then stack deploy processes the .env file for running in the container as envvars, but the YAML variable is needed first...
To fix that, you should use the docker-compose config command first, to process the YAML, and then use the output of that to send to the stack deploy.
docker-compose config will show you the output you're likely wanting.
Then do a pipe to get a one-liner.
docker-compose config | docker stack deploy -c - myapp
Note: Ideally you wouldn't use the extra_hosts, but rather put the envvar directly in the connection string. Your way seems like unnecessary complexity and isn't the usual way I see a connection string created.
e.g.
version: '3'
services:
web:
image: user-service
deploy:
labels:
- the label
build:
context: ./
environment:
DATABASE_URL: jdbc:postgresql://${DB_HOST}:5432/postgres
ports:
- 9002:9002
networks:
- wellness_swarm
env_file:
- .env
networks:
wellness_swarm:
external:
name: wellness_swarm
As i see https://github.com/moby/moby/issues/29133 seems like it is by design where in the compose command takes into consideration the environment variables mentioned in .env file however the deploy command ignores that :( why is that so, pretty lame reasons!
I simply want to use a environment variable loaded from file in my docker-compose file. But after running the container, I only got
WARNING: The TESTVAR variable is not set. Defaulting to a blank string.
Only found this topic, but I'm using a later version of docker like there (docker-compose: 1.14.0, docker: 17.05.0-ce). And I changed the encoding to ISO 8859-1, since I found a github issue where strange behavior with encodings was detected. Both doesn't work.
My docker-compose file
version: '2'
services:
mysql:
container_name: test_${TESTVAR}
build: mysql
mem_limit: 1G
env_file:
- credentials.env
credentials.env contains only TESTVAR=test123. To start, I run docker-compose up mysql and I also tried to specify the environment variables directly in the compose file like this:
environment:
- TESTVAR=1234
Not working, too.
If you want to use variables in the docker-compose.yml you can do it with .env file, docker docs
$ cat .env
TAG=v1.5
TESTVAR=123
$ cat docker-compose.yml
version: '3'
services:
web:
image: "webapp:${TAG}"
environment: ["TESTVAR=${TESTVAR}"]
Is it possible to specify the env file that docker compose uses for variable substitution? Currently it defaults to ".env" but I have different flavours of compose files which should have different envs.
You can use inheritance for this. If you have one "base" service where you set up the environment, all of your other services can inherit from that.
Example:
version: "2"
services:
base:
env_file:
- my_env.txt
web:
extends:
service: base
image: foo
database:
extends:
service: base
image: foo-db
The above example has everything in the same file, but you can also split this up into multiple files, where the base service would reside in a base.yaml file. You just need to add file: base.yaml to the extends section. Please see the documentation here.
I use this approach for setting the proxy variables for all containers. I have a proxy.yaml file that defines a proxy-app service that picks up the proxy environment variables from the shell. All of my real services extend the proxy-app service and thus inherit the environment settings from that service.
The --env-file command-line argument and the env_file docker-compose.yml variable specify the env file to use for the container, not for the container build. To set a different file (e.g. alt.env) for the build itself, use this:
env $(cat alt.env) docker-compose up --build
According to the documentation, it's now possible to load an environment file (contrary to a per-service file), docker-compose will then export the env variables defined in this env file prior to starting any service, they can then be used in the docker-compose.yml config file itself:
version: "3.7"
services:
node:
environment:
APP_ENV: "${APP_ENV}"
NODE_ENV: "${NODE_ENV}"
ports:
- "${HOST_EXPOSED_NODEJS_DEBUG_PORT}:9229"
volumes:
- type: bind
source: ./project
target: /var/www/project
read_only: false
Since docker-compose 1.25 it's also possible to specify a custom .env file with the --env-file flag (unfortunately it's currently not possible to specify multiple .env files with the --env-file flag)