I really did not know how to word the title.
In my system I have two instances:
a Prod server
a Dev server
Dev used mostly for testing. In each case I have two versions of AMQP both having different hostnames.
To avoid duplication or unnecessary time rewriting the same code in multiple projects I wanted to use the env file that docker compose has, though everywhere I read, no one discusses this case. That case being that depending on where a stack is deployed is which env file it would use and that env file existing on the swarm itself rather than the individual projects.
Hopefully, I didn't miss anything when explaining this. Summary being two swarms each having their own env file that the containers deployed to it can use. Also if I need to reword anything, I will do so.
you can have multiple .env files and assign them to services in docker-compose.yml like
web:
env_file:
- web-variables.env
nginx:
env_file:
- nginx-variables.env
and if you want to change them for development environemnt you could override the docker-compose.yml with docker-compose.development.ymlfile and then start it with
docker-compose -f docker-compose.yml -f docker-compose.development.yml up -d
Related
In my docker-compose.yml, I defined two services, app and db.
version: "3.7"
services:
app:
image: my_app
container_name: my-app
ports:
- ${MY_PORT}:${MY_PORT}
env_file:
- ./app.env
...
depends_on:
- db
environment:
- DATABASE_URL=${DB_URL}
db:
image: my_db
container_name: my-db
env_file:
- ./db.env
ports:
- ${DB_PORT}:${DB_PORT}
As you can see above, I have defined two env files, app.env and db.env in the env_file option of app and db services.
app.env:
MY_PORT=8081
db.env:
DB_PORT=4040
DB_URL=postgres://myapp:app#db:4040/myapp
I want to check if my docker-compose can successfully read the environment variables. So, I run the command docker-compose config. However the output is
$ docker-compose config
WARNING: The MY_PORT variable is not set. Defaulting to a blank string.
WARNING: The DB_URL variable is not set. Defaulting to a blank string.
WARNING: The DB_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.app.ports is invalid: Invalid port ":", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
services.db.ports is invalid: Invalid port ":", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
Why my docker compose can't read environment variables from those env files I declared in the env_file option in my docker-compose.yml?
Besides, I have another question, that's I understand that normally the env file shouldn't be version controlled since it could contain credentials. How normally should the env file be used for different environment e.g. development, staging and production environments? Imaging different environment has different values for those variables. Could someone please provide some examples?
The reason this is failing, is that the environment variables that you are defining the the external named app.env and db.env files, and specifying in the env_file option, are only being set inside the container that is started - and are not used for variable expansion inside the docker-compose.yml file when parsed by docker-compose.
This is easily confused with the option of supplying a file named .env in the same location as the docker-compose.yml file. Since docker-compose will look for a file specifically named .env next to the docker-compose.yml file (or next to the file that you are specifying with the -f switch) - and use the environment variables in that file for variable expansion in the docker-compose.yml file, before parsing it.
In other words:
The env_file option
Will set environment variables inside your container, is is just a convenience feature that allows you to externalise the environment variables from the docker-compose.yml file
Environment variables in these files will NOT be used for variable expansion in the docker-compose.yml file before parsed by docker-compose.
The .env file
Will be used for environment variable expansion inside the docker-compose.yml file before parsing.
Will NOT set environment variables inside the started container.
Suggested solution to the first question
If you migrate your values into a single .env file and place it in the same directory as your docker-compose.yml file, this should work.
Second question
As I understand your second question, you are asking how the .env file, or the env_file option should be used to configure your services for your different environments.
I do not think that there is a simple and single answer to this. It can be solved in a number of ways. But it also depends on what you are deploying to? Is it kubernetes? Docker swarm? Or just a single node docker host?
Kubernetes and Docker swarm have different means of helping you out with this.
Kubernetes secrets
Docker swarm secrets
Those are highly secure solutions, where operators of the secrets can be limited, and the secrets will not be seen by developers or operators that do not have access.
But for the single node docker host, not operating in swarm mode (secrets only work in swarm mode), there really isn't a lot of fancy options. You will have to manage this pretty manually in your build and deploy pipes as far as I am aware.
You are right that the sensitive configuration of your services, should not go in the same repository as the service definition. Things like root password for a database, or credentials to your service discovery service for your production environment do not need to live next to the sources.
Traditionally, another repository would contain this - giving you the oppotunity to limit the group of people that have this access. The build/deployment server/service will check out the new revision of your service, build it perhaps, and then check out the configuration repository and start the services with the configurations from there. And, make sure to remove the configuration files afterwards.
That would be the solution I would recommend for a single node docker host deployment regime - two repositories, and some scripting that ensures that the correct .env file is put in place during deployment, and removed again.
I hope this is helpful?
I use docker and docker compose to package scientific tools into easily/universally executable modules. One example is a docker that packages a rather complicated python library into a container that runs a jupyter notebook server; the idea is that other scientists who are not terribly tech-savvy can clone a github repository, run docker-compose up then do their analyses without having to install the library, configure various plugins and other dependencies, etc.
I have this all working fine except that I'm having issues getting the volume mounts to work in a coherent fashion. The reason for this is that the library inside the docker container handles multiple kinds of datasets, which users will store in several separate directories that are conventionally tracked through shell environment variables. (Please don't tell me this is a bad way to do this--it's the way things are done in the field, not the way I've chosen to do things.) So, for example, if the user stores FreeSurfer data, they will have an environment variable named SUBJECTS_DIR that points to the directory containing the data; if they store HCP data, they will have an environment variable HCP_SUBJECTS_DIR. However, they may have both, either, or neither of these set (as well as a few others).
I would like to be able to put something like this in my docker-compose.yml file in order to handle these cases:
version: '3'
services:
my_fancy_library:
build: .
ports:
- "8080:8888"
environment:
- HCP_SUBJECTS_DIR="/hcp_subjects"
- SUBJECTS_DIR="/freesurfer_subjects"
volumes:
- "$SUBJECTS_DIR:/freesurfer_subjects"
- "$HCP_SUBJECTS_DIR:/hcp_subjects"
In testing this, if the user has both environment variables set, everything works swimmingly. However, if they don't have one of these set, I get an error about not mounting directories that are fewer than 2 characters long (which I interpret to be a complaint about mounting a volume specified by ":/hcp_subjects").
This question asks basically the same thing, and the answer points to here, which, if I'm understanding it right, basically explains how to have multiple docker-compose files that are resolved in some fashion. This isn't really a viable solution for my case for a few reasons:
This tool is designed for use by people who don't necessarily know anything about docker, docker-compose, or related utilities, so expecting them to write/edit their own docker-compose.yml file is a problem
There are more than just two of these directories (I have shown two as an example) and I can't realistically make a docker-compose file for every possible combination of these paths being declared or not declared
Honestly, this solution seems really clunky given that the information needed is right there in the variables that docker-compose is already reading.
The only decent solution I've been able to come up with is to ask the users to run a script ./run.sh instead of docker-compose up; the script examines the environment variables, writes out its own docker-compose.yml file with the appropriate volumes, and runs docker-compose up itself. This also seems somewhat clunky, but it works.
Does anyone know of a way to conditionally mount a set of volumes based on the state of the environment variables when docker-compose up is run?
You can set defaults for environment variable in a .env-file shipped alongside with a docker-compose.yml [1].
By setting your environment variables to /dev/null by default and then handling this case in the containerized application, you should be able to achieve what you need.
Example
$ tree -a
.
├── docker-compose.yml
├── Dockerfile
├── .env
└── run.sh
docker-compose.yml
version: "3"
services:
test:
build: .
environment:
- VOL_DST=${VOL_DST}
volumes:
- "${VOL_SRC}:${VOL_DST}"
Dockerfile
FROM alpine
COPY run.sh /run.sh
ENTRYPOINT ["/run.sh"]
.env
VOL_SRC=/dev/null
VOL_DST=/volume
run.sh
#!/usr/bin/env sh
set -euo pipefail
if [ ! -d ${VOL_DST} ]; then
echo "${VOL_DST} not mounted"
else
echo "${VOL_DST} mounted"
fi
Testing
Environment variable VOL_SRC not defined:
$ docker-compose up
Starting test_test_1 ... done
Attaching to test_test_1
test_1 | /volume not mounted
test_test_1 exited with code 0
Environment variable VOL_SRC defined:
$ VOL_SRC="./" docker-compose up
Recreating test_test_1 ... done
Attaching to test_test_1
test_1 | /volume mounted
[1] https://docs.docker.com/compose/environment-variables/#the-env-file
Even though #Ente's answer solves the problem, here is an alternative solution when you have more complex differences between environments.
Docker compose supports multiple docker-compose files for configuration overriding in different environments.
This is useful if you have different named volumes you need to potentially mount on the same path depending on the environment.
You can modify existing services or even add new ones, for instance:
# docker-compose.yml
version: '3.3'
services:
service-a:
image: "image-name"
volumes:
- type: volume
source: vprod
target: /data
ports:
- "80:8080"
volumes:
vprod:
vdev:
And then you have the override file to change the volume mapping:
# docker-compose.override.yml
services:
service-a:
volumes:
- type: volume
source: vdev
target: /data
When running docker-compose up -d both configurations will be merged with the override file taking precedence.
Docker compose picks up docker-compose.yml and docker-compose.override.yml by default, if you have more files, or files with different names, you need to specify them in order:
docker-compose -f docker-compose.yml -f docker-compose.custon.yml -f docker-compose.dev.yml up -d
I am searching for a scaleable solution to the problem of having numerous possible environments for a containerised app. Let's say am creating a web app and I have the usual deployment environments, develop, testing, production but I also have multiple instances of the app for different clients, client1, client2, client3 etc.
This quickly becomes a mess if I have to create separate docker-compose files:
docker-compose-client1-develop.yml
docker-compose-client1-testing.yml
docker-compose-client1-production.yml
docker-compose-client2-develop.yml
...
Breaking the client specific configuration into a .env file and dockers variable substitution gets me most of the way there, I can now have one docker-compose.yml file and just do:
services:
webapp:
image: 'myimage:latest'
env_file:
- ./clients/${CLIENT}.env # client specific .env file
environment:
- DEPLOY # develop, testing, production
so now I just need the CLIENT and DEPLOY environment variables set when I run docker-compose up which is fine, but I'm wondering about a convenient way to pass those environment variables in to docker-compose. There's the potential (at least during development) for a decent amount of context-switching. Is there a tidy way to pass in different CLIENT and DEPLOY env vars to docker-compose up every time I run it?
What you are trying to achieve is to set environment variables per-command.
Are you running on Linux? Take a look at env command. Just prepend your docker-compose command line like this:
env CLIENT=client1 DEPLOY=production docker-compose ...
On Windows, you may have to do something more complicated (like this), but there could be simpler ways.
Have you tried docker-compose file extending?
For instance you can have base docker-compose.yml file which is the production one and multiple extending files where you only change what needs to be overloaded:
docker-compose.dev.yml
version: '2'
services:
webapp:
env_file: path_to_the_file_env
Then you simply use both:
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up
To spin up the production it's as easy as:
docker-compose up
I personally use this technique a lot in many of my projects.
I'm trying to wrap my head around Docker, but I'm having a hard time figuring it out. I tried to implement it in my small project (MERN stack), and I was thinking how do you distinct between development, (maybe staging), and production environments.
I saw one example where they used 2 Docker files, and 2 docker-compose files, (each pair for one env, so Dockerfile + docker-compose.yml for prod, Dockerfile-dev + docker-compose-dev.yml for dev).
But this just seems like a bit of an overkill for me. I would prefer to have it only in two files.
Also one of the problem is that e.g. for development I want to install nodemon globally, but not for poduction.
In perfect solution I imagine running something like that
docker-compose -e ENV=dev build
docker-compose -e ENV=dev up
Keep in mind, that I still don't fully get docker, so if you caught some of mine misconceptions about docker, you can point them out.
You could take some clues from "Using Compose in production"
You’ll almost certainly want to make changes to your app configuration that are more appropriate to a live environment. These changes may include:
Removing any volume bindings for application code, so that code stays inside the container and can’t be changed from outside
Binding to different ports on the host
Setting environment variables differently (e.g., to decrease the verbosity of logging, or to enable email sending)
Specifying a restart policy (e.g., restart: always) to avoid downtime
Adding extra services (e.g., a log aggregator)
The advice is then not quite similar to the example you mention:
For this reason, you’ll probably want to define an additional Compose file, say production.yml, which specifies production-appropriate configuration. This configuration file only needs to include the changes you’d like to make from the original Compose file.
docker-compose -f docker-compose.yml -f production.yml up -d
This overriding mechanism is better than trying to mix dev and prod logic in one compose file, with environment variable to try and select one.
Note: If you name your second dockerfile docker-compose.override.yml, a simple docker-compose up would read the overrides automatically.
But in your case, a name based on the environment is clearer.
Docker Compose will read docker-compose.yml and docker-compose.override.yml by default. Understanding-Multiple-Compose-Files
You can set a default docker-compose.yml and different overwrite compose file. For example, docker-compose.prod.yml docker-compose.test.yml. Keep them in the same place.
Then create a symbolic link named docker-compose.override.yml for each env.
Track docker-compose.{env}.yml files and add docker-compose.override.yml to .gitignore.
In prod env: ln -s ./docker-compose.prod.yml ./docker-compose.override.yml
In test env: ln -s ./docker-compose.test.yml ./docker-compose.override.yml
The project structure will then look like this:
project\
- docker-compose.yml # tracked
- docker-compose.prod.yml # tracked
- docker-compose.test.yml # tracked
- docker-compose.override.yml # ignored & linked to override composefile for current env
- src/
- ...
Then you have done. In each environment, you can use the compose-file with the same command docker-compose up
If you are not sure, use docker-compose config to check if it's been override properly.
There seems to be sparse conflicting information around on this subject. Im new to Docker and need some help. I have several docker containers to run an application, some require different config files for local development as they do for production. I don't seem to be able to find a neat way to automate this with Docker.
My containers that include custom config are Nginx, Freeradius and my code/data container is Laravel therefore requires a .env.php file (L4.2 at the moment).
I have tried Dockers environment variables in docker compose:
docker-compose.yml:
freeradius:
env_file: ./env/freeradius.env
./env/freeradius.env
DB_HOST=123.56.12.123
DB_DATABASE=my_database
DB_USER=me
DB_PASS=itsasecret
Except I can't pick those variables up in /etc/freeradius/mods-enabled/sql where they need to be.
How can I get Docker to run as a 'local' container with local config, or as a 'production' container with production config without having to actually build different containers, and without having to attach to each container to manually config them. I need it automated as this is to eventually be used on quite a large production environment which will have a large cluster of servers with many instances.
Happy to learn Ansible if this is how people achieve this.
If you can't use environment variables to configure the application (which is my understanding of the problem), then the other option is to use volumes to provide the config files.
You can use either "data volume containers" (which are containers with the sole purpose of sharing files and directories) with volumes_from, or you can use a named volume.
Data Volume container
If the go with the "data volume container" route, you would create a container with all the environment configuration files. Every service that needs a file uses volumes_from: - config. In dev you'd have something like:
configs:
build: dev-configs/
freeradius:
volumes_from:
- configs
The dev-configs directory will need a Dockerfile to build the image, which will have a bunch of VOLUME directives for all the config paths.
For production (and other environments) you can create an override file which replaces the configs service with a different container:
docker-compose.prod.yml:
configs:
build: prod-configs/
You'll probably have other settings you want to change between dev and prod, which can go into this file as well. Then you run compose with the override file:
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d
You can learn more about this here: http://docs.docker.com/compose/extends/#multiple-compose-files
Named Volume
If you go with the "named volume" route, it's a bit easier to configure. On dev you create a volume with docker volume create thename and put some files into it. In your config you use it directly:
freeradius:
volumes:
- thename:/etc/freeradius/mods-enabled/sql
In production you'll either need to create that named volume on every host, or use a volume driver plugin that supports multihost (I believe flocker is one example of this).
Runtime configs using Dockerize
Finally, another option that doesn't involve volumes is to use https://github.com/jwilder/dockerize which lets you generate the configs at runtime from environment variables.