I have a docker-compose.yml file that defines the volumes section like this:
volumes:
seqfs:
driver: azure_file
driver_opts:
share_name: seqtest
storage_account_name: stacctest
storage_account_key: ${STORAGE_ACCOUNT_KEY}
I am trying to pass in STORAGE_ACCOUNT_KEY during the build command:
docker-compose -f docker-compose.yml build --build-arg STORAGE_ACCOUNT_KEY="##########"
But an error is returned:
The STORAGE_ACCOUNT_KEY variable is not set. Defaulting to a blank string.
Please note I do not want to save STORAGE_ACCOUNT_KEY into a file such as .env for security reasons -- I want to pass it from the command line.
How can I pass an argument to the volumes section in my docker-compose.yml?
Regarding the error you get:
docker-compose -f docker-compose.yml build --build-arg STORAGE_ACCOUNT_KEY="##########"
But an error is returned:
The STORAGE_ACCOUNT_KEY variable is not set. Defaulting to a blank string.
the fact is that --build-arg only deals with ARG directives within the Dockerfile.
(BTW it is simpler to run docker-compose up --build
rather than running docker-compose build then docker-compose up)
the only solutions I can think about to achieve what you want are:
either put STORAGE_ACCOUNT_KEY=foobar in an .env file
and run docker-compose up --build
or put STORAGE_ACCOUNT_KEY=foobar in an other.env file
and run docker-compose --env-file=other.env up --build
(Note: the docker/compose syntax contains two different kinds of "env-file", for details on this subtlety, see this answer of mine: Pass variables from .env file to dockerfile through docker-compose.)
or run STORAGE_ACCOUNT_KEY=foobar docker-compose up --build
Concluding remarks:
You said you do not want to save STORAGE_ACCOUNT_KEY into a file such as .env for security reasons, so at first sight you would only accept solution 3. above.
However, note that it is unlikely that 3. is more secure if you assume you don't trust the environment of your docker host. Indeed, .env files may be protected thanks to file permissions (and .gitignore FWIW), while the foobar string above automatically leaks in the name of the running processes (try e.g. ps aux | grep STORAGE_ACCOUNT_KEY).
Related
In docker run one can do
docker run --env-file <(env | grep ^APP_) ...
Is there a similar way for docker-compose?
I would like to avoid physical env file.
The equivalent of --env-file option of the docker cli in docker-compose is the env_file configuration option in the docker-compose file. But I think this requires a physical .env file.
If you want use the environment variables of your host machine, you can define them in docker-compose (with an optional fallback value):
version: "3.9"
services:
app:
image: myapp
environment:
- APP_MYVAR=${APP_MYVAR-fallbackvalue}
It's not so convenient as doing a grep of your ^APP_ vars, but one way to avoid the physical file.
You can do it by supplying multiple compose files, as documented here. In your case the first one is a physical docker-compose.yml and the second one a Compose containing only the environment variables for the needed service. Obviously, env variables must be properly formatted, so a sed that prepends the string - is necessary because they are added as YAML list.
docker-compose -f docker-compose.yml -f <(printf "services:
your_service:
environment:\n$(env | grep ^APP_ | sed -e "s/^/ - /")"
) up
This is how Docker behaves:
When you supply multiple files, Compose combines them into a single configuration. Compose builds the configuration in the order you supply the files. Subsequent files override and add to their predecessors
I start to work with a new project using Docker and I can see there is no docker-compose.yml file there, but a few files like docker-compose.myname1.yml, docker-compose.myname2.yml instead. Trying to run docker-compose up I get an error:
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
So I wonder if that's possible for a Docker project to work without the docker-compose.yml file and if it is, what conditions should be met. Maybe that's the matter of some specific version or environment?
https://docs.docker.com/compose/reference/
"Options:
-f, --file FILE Specify an alternate compose file
(default: docker-compose.yml)"
so in your case that would be:
docker-compose -f dockcer-compose.myname1.yml up
docker-compose -f dockcer-compose.myname2.yml up
There are two different issues presented in your question:
The reason for getting an error
When running docker-compose up, you must be in a directory that contains docker-compose.yml (or '.yaml), as indicated by the error you are receiving.
The reason you are getting an error is either because there is no such file in the current directory, or (less likely) the environment variable COMPOSE_FILE is set to something else.
Different names for the docker-compose file
In order to use an alternative file, you can take one of these approaches:
Run docker-compose with the --file argument:
$ docker-compose --help
-f, --file FILE Specify an alternate compose file
(default: docker-compose.yml)
Set the COMPOSE_FILE environment variable prior to running any docker-compose command.
I'm trying to get the Env-Variables in Docker-Compose to work. My Files:
env/test.env:
XUSER=you
XHOME=/home/${XUSER}
docker-compose.yml:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
env_file:
- env/test.env
user: "${XUSER}"
docker-compose up --build
docker-compose config
WARNING: The XUSER variable is not set. Defaulting to a blank string.
services:
kernel:
build:
context: xyz
container_name: xyz
environment:
XHOME: /home/you
XUSER: you
image: xyz:latest
user: ''
As you can see user: '' is an empty string, but the env_file works. I found some old Bug reports about this issue, I'm not sure I doing something wrong or not.
Although the other answers are both correct they do not highlight the underlying misunderstanding here enough:
With the env_file option you can specify a file with variables to be injected into the environment in the container.
Using variable substitution in the docker-compose.yml you can access variables in the environment of the docker-compose command, i.e. on the host.
You can set these using the usual mechanisms of your OS/shell, e.g. in bash:
export XUSER=you
docker-compose up
Additionally with docker-compose you can use a .env file in the current directory.
So in your concrete example you should just move env/test.env to .env to add the variables to the environment of docker-compose for variable substitution.
If you also want to add them to the environment in the container you can do it like this:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
# add variables from the docker-compose environment to the container:
environment:
- XUSER=$XUSER
# or even shorter:
- XHOME
# use variable from the docker-compose environment in the config:
user: "${XUSER}"
It says WARNING: The XUSER variable is not set. Defaulting to a blank string. because ${XUSER} doesn't exist at the time this is executed:
user: "${XUSER}"
${XUSER} is not in your environment (you can verify this by running: env | grep XUSER, which should output nothing), and docker-compose didn't find any .env file at the same level or no .env file was passed at the time you ran docker-compose up --build or docker-compose config
Flexible solution:
Rename env/test.env for .env and place it a the root of the folder container your docker-compose file so that docker automatically parses it.
Or use:
docker-compose --env-file path/to/env/test.env up --build
docker-compose --env-file path/to/env/test.env config
Permanent solution:
Export them manually in your environment by running:
export XUSER=you && export XHOME=/home/${XUSER}
Or you use your env/test.env file as a source (note that you'll need to prefix with 'export'):
env/test.env:
export XUSER=you
export XHOME=/home/${XUSER}
And then your run . /path/to/env/test.env or source /path/to/env/test.env
What you need to do is create .env file at the same directory as your docker-compose.yml file, the content of .env is :
XUSER=user1
then run docker-compose config
reference : https://docs.docker.com/compose/environment-variables/
Since +1.28 .env file is placed in project root, not where docker-compose is executed. If you do that the variables will be automatically pulled through to be available to the container.
This works great in dev, especially with a a bind-mount volume to make .env available to compose in project root without also going into image build by including .env in .dockerignore
But in production I was not comfortable including it in my project root especially since I was pulling those project files down from github. The Compose file expects them to be in the production environment to replace for substitution SECRET_VAR=${SECRET_VAR}
So one hack solution was to stick the .env file high in my production directory tree, far away from my project (ideally these would come from an environment store on the hosting service, or another encrypted store), but inject those variables into the container at runtime by using the --env_file flag in Compose up.
The env_file flag works Like this:
docker-compose -f docker-compose.prod.yml --env-file ../.env up -d
Its in the docs
I also started encountering this after upgrading to Docker Desktop 4.12.0. This error started happening for quoted strings inside of .env (when using env_file to load variables in docker-compose.yml). In that case, be sure to use single quotes instead of double quotes, i.e.
MY_VAR='foo$bar'
# ... instead of...
MY_VAR="foo$bar"
Try this, I hope it will work.
You need to escape the variable if you want it to be expanded inside the container, using a double-dollar sign ($${envVariable}).
If however, you want it to be interpreted on your host, the $envVariable needs to be defined in your environment or in the .env file. The env_file option defines environment variables that will be available inside the container only.
I am trying to use the same docker-compose.yml and .env files for both docker-compose and swarm. The variables from the .env file should get parsed, via sed, into a config file by running a run.sh script at boot. This setup works fine when using the docker-compose up command, but they're not getting passed when I use the docker stack deploy command.
How can I pass the variables into the container so that the run.sh script will parse them at boot?
Loading the .env file is a feature of docker-compose that is not part of the docker CLI. You can manually load the contents of this file in your shell before performing the deploy:
set -a; . ./.env; set +a
docker stack deploy -c docker-compose.yml stack_name
Other options include using docker-compose to pre process the compose file:
docker-compose config >docker-compose.processed.yml
Or you could use envsubst to replace the variables to make a compose file with the variables already expanded:
set -a; . ./.env; set +a
envsubst <docker-compose.yml >docker-compose.processed.yml
To pass shell environment variables through to containers use env_file syntax:
web:
env_file:
- web-variables.env
As docs state:
You can pass multiple environment variables from an external file through to a service’s containers with the ‘env_file’ option
However, using .env as external filename may cause unexpected results and is semantically problematic.
Placing .env in the folder where the docker-compose command serves different purpose:
As Docs, Docs2, Docs3 state:
The environment variables you define here are used for variable
substitution in your Compose file
You can set default values for environment variables using a .env
file, which Compose automatically looks for
So if compose file contains:
db:
image: "postgres:${POSTGRES_VERSION}"
You .env would contain:
POSTGRES_VERSION=4.0
This feature indeed works only in compose:
The .env file feature only works when you use the docker-compose up
command and does not work with docker stack deploy
Actually I found the best/easiest way is to just add this argument to the docker-compose.yml file:
env_file:
- .env
I use docker and docker compose to package scientific tools into easily/universally executable modules. One example is a docker that packages a rather complicated python library into a container that runs a jupyter notebook server; the idea is that other scientists who are not terribly tech-savvy can clone a github repository, run docker-compose up then do their analyses without having to install the library, configure various plugins and other dependencies, etc.
I have this all working fine except that I'm having issues getting the volume mounts to work in a coherent fashion. The reason for this is that the library inside the docker container handles multiple kinds of datasets, which users will store in several separate directories that are conventionally tracked through shell environment variables. (Please don't tell me this is a bad way to do this--it's the way things are done in the field, not the way I've chosen to do things.) So, for example, if the user stores FreeSurfer data, they will have an environment variable named SUBJECTS_DIR that points to the directory containing the data; if they store HCP data, they will have an environment variable HCP_SUBJECTS_DIR. However, they may have both, either, or neither of these set (as well as a few others).
I would like to be able to put something like this in my docker-compose.yml file in order to handle these cases:
version: '3'
services:
my_fancy_library:
build: .
ports:
- "8080:8888"
environment:
- HCP_SUBJECTS_DIR="/hcp_subjects"
- SUBJECTS_DIR="/freesurfer_subjects"
volumes:
- "$SUBJECTS_DIR:/freesurfer_subjects"
- "$HCP_SUBJECTS_DIR:/hcp_subjects"
In testing this, if the user has both environment variables set, everything works swimmingly. However, if they don't have one of these set, I get an error about not mounting directories that are fewer than 2 characters long (which I interpret to be a complaint about mounting a volume specified by ":/hcp_subjects").
This question asks basically the same thing, and the answer points to here, which, if I'm understanding it right, basically explains how to have multiple docker-compose files that are resolved in some fashion. This isn't really a viable solution for my case for a few reasons:
This tool is designed for use by people who don't necessarily know anything about docker, docker-compose, or related utilities, so expecting them to write/edit their own docker-compose.yml file is a problem
There are more than just two of these directories (I have shown two as an example) and I can't realistically make a docker-compose file for every possible combination of these paths being declared or not declared
Honestly, this solution seems really clunky given that the information needed is right there in the variables that docker-compose is already reading.
The only decent solution I've been able to come up with is to ask the users to run a script ./run.sh instead of docker-compose up; the script examines the environment variables, writes out its own docker-compose.yml file with the appropriate volumes, and runs docker-compose up itself. This also seems somewhat clunky, but it works.
Does anyone know of a way to conditionally mount a set of volumes based on the state of the environment variables when docker-compose up is run?
You can set defaults for environment variable in a .env-file shipped alongside with a docker-compose.yml [1].
By setting your environment variables to /dev/null by default and then handling this case in the containerized application, you should be able to achieve what you need.
Example
$ tree -a
.
├── docker-compose.yml
├── Dockerfile
├── .env
└── run.sh
docker-compose.yml
version: "3"
services:
test:
build: .
environment:
- VOL_DST=${VOL_DST}
volumes:
- "${VOL_SRC}:${VOL_DST}"
Dockerfile
FROM alpine
COPY run.sh /run.sh
ENTRYPOINT ["/run.sh"]
.env
VOL_SRC=/dev/null
VOL_DST=/volume
run.sh
#!/usr/bin/env sh
set -euo pipefail
if [ ! -d ${VOL_DST} ]; then
echo "${VOL_DST} not mounted"
else
echo "${VOL_DST} mounted"
fi
Testing
Environment variable VOL_SRC not defined:
$ docker-compose up
Starting test_test_1 ... done
Attaching to test_test_1
test_1 | /volume not mounted
test_test_1 exited with code 0
Environment variable VOL_SRC defined:
$ VOL_SRC="./" docker-compose up
Recreating test_test_1 ... done
Attaching to test_test_1
test_1 | /volume mounted
[1] https://docs.docker.com/compose/environment-variables/#the-env-file
Even though #Ente's answer solves the problem, here is an alternative solution when you have more complex differences between environments.
Docker compose supports multiple docker-compose files for configuration overriding in different environments.
This is useful if you have different named volumes you need to potentially mount on the same path depending on the environment.
You can modify existing services or even add new ones, for instance:
# docker-compose.yml
version: '3.3'
services:
service-a:
image: "image-name"
volumes:
- type: volume
source: vprod
target: /data
ports:
- "80:8080"
volumes:
vprod:
vdev:
And then you have the override file to change the volume mapping:
# docker-compose.override.yml
services:
service-a:
volumes:
- type: volume
source: vdev
target: /data
When running docker-compose up -d both configurations will be merged with the override file taking precedence.
Docker compose picks up docker-compose.yml and docker-compose.override.yml by default, if you have more files, or files with different names, you need to specify them in order:
docker-compose -f docker-compose.yml -f docker-compose.custon.yml -f docker-compose.dev.yml up -d