If I'm using Docker with nginx for hosting a web app, how can I use either
Variables in my docker-compose.yml file
Environment variables such as HOSTNAME=example.com.
So that when I build the container, it will insert the value into my nginx.conf file that I copy over when I build the container.
You can use environment variables is your compose file. According to official docs
Your configuration options can contain environment variables. Compose uses the variable values from the shell environment in which docker-compose is run. For example, suppose the shell contains POSTGRES_VERSION=9.3 and you supply this configuration:
db: image: "postgres:${POSTGRES_VERSION}"
When you run docker-compose up with this configuration, Compose looks for the POSTGRES_VERSION environment variable in the shell and substitutes its value in.
See the docs for more information. You will find various other approaches to supply environment variables in the link like passing them through env_file etc.
Related
In my docker-compose.yml, I defined two services, app and db.
version: "3.7"
services:
app:
image: my_app
container_name: my-app
ports:
- ${MY_PORT}:${MY_PORT}
env_file:
- ./app.env
...
depends_on:
- db
environment:
- DATABASE_URL=${DB_URL}
db:
image: my_db
container_name: my-db
env_file:
- ./db.env
ports:
- ${DB_PORT}:${DB_PORT}
As you can see above, I have defined two env files, app.env and db.env in the env_file option of app and db services.
app.env:
MY_PORT=8081
db.env:
DB_PORT=4040
DB_URL=postgres://myapp:app#db:4040/myapp
I want to check if my docker-compose can successfully read the environment variables. So, I run the command docker-compose config. However the output is
$ docker-compose config
WARNING: The MY_PORT variable is not set. Defaulting to a blank string.
WARNING: The DB_URL variable is not set. Defaulting to a blank string.
WARNING: The DB_PORT variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.app.ports is invalid: Invalid port ":", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
services.db.ports is invalid: Invalid port ":", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]
Why my docker compose can't read environment variables from those env files I declared in the env_file option in my docker-compose.yml?
Besides, I have another question, that's I understand that normally the env file shouldn't be version controlled since it could contain credentials. How normally should the env file be used for different environment e.g. development, staging and production environments? Imaging different environment has different values for those variables. Could someone please provide some examples?
The reason this is failing, is that the environment variables that you are defining the the external named app.env and db.env files, and specifying in the env_file option, are only being set inside the container that is started - and are not used for variable expansion inside the docker-compose.yml file when parsed by docker-compose.
This is easily confused with the option of supplying a file named .env in the same location as the docker-compose.yml file. Since docker-compose will look for a file specifically named .env next to the docker-compose.yml file (or next to the file that you are specifying with the -f switch) - and use the environment variables in that file for variable expansion in the docker-compose.yml file, before parsing it.
In other words:
The env_file option
Will set environment variables inside your container, is is just a convenience feature that allows you to externalise the environment variables from the docker-compose.yml file
Environment variables in these files will NOT be used for variable expansion in the docker-compose.yml file before parsed by docker-compose.
The .env file
Will be used for environment variable expansion inside the docker-compose.yml file before parsing.
Will NOT set environment variables inside the started container.
Suggested solution to the first question
If you migrate your values into a single .env file and place it in the same directory as your docker-compose.yml file, this should work.
Second question
As I understand your second question, you are asking how the .env file, or the env_file option should be used to configure your services for your different environments.
I do not think that there is a simple and single answer to this. It can be solved in a number of ways. But it also depends on what you are deploying to? Is it kubernetes? Docker swarm? Or just a single node docker host?
Kubernetes and Docker swarm have different means of helping you out with this.
Kubernetes secrets
Docker swarm secrets
Those are highly secure solutions, where operators of the secrets can be limited, and the secrets will not be seen by developers or operators that do not have access.
But for the single node docker host, not operating in swarm mode (secrets only work in swarm mode), there really isn't a lot of fancy options. You will have to manage this pretty manually in your build and deploy pipes as far as I am aware.
You are right that the sensitive configuration of your services, should not go in the same repository as the service definition. Things like root password for a database, or credentials to your service discovery service for your production environment do not need to live next to the sources.
Traditionally, another repository would contain this - giving you the oppotunity to limit the group of people that have this access. The build/deployment server/service will check out the new revision of your service, build it perhaps, and then check out the configuration repository and start the services with the configurations from there. And, make sure to remove the configuration files afterwards.
That would be the solution I would recommend for a single node docker host deployment regime - two repositories, and some scripting that ensures that the correct .env file is put in place during deployment, and removed again.
I hope this is helpful?
I am trying to use the same docker-compose.yml and .env files for both docker-compose and swarm. The variables from the .env file should get parsed, via sed, into a config file by running a run.sh script at boot. This setup works fine when using the docker-compose up command, but they're not getting passed when I use the docker stack deploy command.
How can I pass the variables into the container so that the run.sh script will parse them at boot?
Loading the .env file is a feature of docker-compose that is not part of the docker CLI. You can manually load the contents of this file in your shell before performing the deploy:
set -a; . ./.env; set +a
docker stack deploy -c docker-compose.yml stack_name
Other options include using docker-compose to pre process the compose file:
docker-compose config >docker-compose.processed.yml
Or you could use envsubst to replace the variables to make a compose file with the variables already expanded:
set -a; . ./.env; set +a
envsubst <docker-compose.yml >docker-compose.processed.yml
To pass shell environment variables through to containers use env_file syntax:
web:
env_file:
- web-variables.env
As docs state:
You can pass multiple environment variables from an external file through to a service’s containers with the ‘env_file’ option
However, using .env as external filename may cause unexpected results and is semantically problematic.
Placing .env in the folder where the docker-compose command serves different purpose:
As Docs, Docs2, Docs3 state:
The environment variables you define here are used for variable
substitution in your Compose file
You can set default values for environment variables using a .env
file, which Compose automatically looks for
So if compose file contains:
db:
image: "postgres:${POSTGRES_VERSION}"
You .env would contain:
POSTGRES_VERSION=4.0
This feature indeed works only in compose:
The .env file feature only works when you use the docker-compose up
command and does not work with docker stack deploy
Actually I found the best/easiest way is to just add this argument to the docker-compose.yml file:
env_file:
- .env
After reading the config point of the 12 factor app I decided to override my config file containing default value with environment variable.
I have 3 Dockerfiles, one for an API, one for a front-end and one for a worker. I have one docker-compose.yml to run those 3 services plus a database.
Now I'm wondering if I should define the environment variables in Dockerfiles or docker-compose.yml ? What's the difference between using one rather than another ?
See this:
You can set environment variables in a service’s containers with the 'environment' key, just like with docker run -e VARIABLE=VALUE ...
Also, you can use ENV in dockerfile to define a environment variable.
The difference is:
Environment variable define in Dockerfile will not only used in docker build, it will also persist into container. This means if you did not set -e when docker run, it will still have environment variable same as defined in Dockerfile.
While environment variable define in docker-compose.yaml just used for docker run.
Maybe next example could make you understand more clear:
Dockerfile:
FROM alpine
ENV http_proxy http://123
docker-compose.yaml:
app:
environment:
- http_proxy=http://123
If you define environment variable in Dockerfile, all containers used this image will also has the http_proxy as http://123. But the real situation maybe when you build the image, you need this proxy. But, the container maybe run by other people maybe not need this proxy or just have another http_proxy, so they had to remove the http_proxy in entrypoint or just change to another value in docker-compose.yaml.
If you define environment variable in docker-compose.yaml, then user could just choose his own http_proxy when do docker-compose up, http_proxy will not be set if user did not configure it docker-compose.yaml.
I have a docker-compose file that allows me to pass the environment variables as a file (.env file). As I have multiple ENV variables, Is there any option in Dockerfile like env_file in docker-compose for passing multiple environment variables during docker build?
This is the docker-compose.yml
services:
web:
image: "node"
links:
- "db"
env_file: "env.app"
AFAIK, there is no such way to inject environment variables using a file during the build step using Dockerfile. However, in most cases, people do end up using an entrypoint script & injecting variables during the docker run or docker-compose up.
In case it's a necessity you might need to write a shell wrapper which will change the values in the Dockerfile dynamically by taking a key-value pair text file as an input or make it something as below but the ENV file name need to be included in Dockerfile.
COPY my-env-vars /
RUN export $(cat my-env-vars | xargs)
It's an open issue - https://github.com/moby/moby/issues/28617
PS - You need to be extra careful while using this approach because the secrets are baked into the image itself.
I have writte a docker-compose.yml file to create a multi-container app with nodejs and mongodb (so 2 containers), and I want to make some options configurable, as the choice of server address and port.
To do that, I have written what follows in docker-compose.yml to set them as env variables:
..
web:
environment:
- PORT=3000
- ADDRESS=xxx.xxx.xxx.xxx
..
Within the source code of my application, I use process.env.PORT and process.env.ADDRESS to refer to these variables.
But how can I do if I want to change those values, and for example set PORT=3001?
I have to use docker-compose build and docker-compose up again and build all the application, included mongodb container?
I have to use docker-compose build and docker-compose up again and build all the application, included mongodb container?
Not build, just up. They are runtime options, not build options. Changing them in the docker-compose.yml file and then doing a docker-compose up again should recreate the containers with the new environment variables.
Alternatively, you can specify environment variables outside the docker-compose.yml file to make for easier changes:
One of these methods is to use a .env file in the same folder that you execute docker-compose form (see https://docs.docker.com/compose/env-file/ for information on it).
Another option is to interpolate environment variables from your shell. You could specify them such as - PORT=${APP_PORT} and then export APP_PORT=3000 on your shell before running docker-compose. See https://docs.docker.com/compose/environment-variables/ for more information.