I'm working on a group project involving Docker that has a .env file, which looks like this:
DATABASE_URL=xxx
DJANGO_SETTINGS_MODULE=xxx
SECRET_KEY=xxx
Couldn't this just be declared inside the Dockerfile? If so, what is the advantage of making a .env file?
Not sure if I'm going in the right direction with this, but this Docker Docs page says (emphasis my own):
Your configuration options can contain environment variables. Compose
uses the variable values from the shell environment in which
docker-compose is run. For example, suppose the shell contains
POSTGRES_VERSION=9.3 and you supply this configuration:
db:
`image: "postgres:${POSTGRES_VERSION}"`
When you run docker-compose up with this configuration, Compose looks for the POSTGRES_VERSION environment variable in the shell and substitutes its value in. For this example, Compose resolves the image to postgres:9.3 before running the configuration.
If an environment variable is not set, Compose substitutes with an empty string. In the example above, if POSTGRES_VERSION is not set, the value for the image option is postgres:.
You can set default values for environment variables using a .env file, which Compose automatically looks for. Values set in the shell environment override those set in the .env file.
If we're using a .env file, then wouldn't I see some ${...} syntax in our docker-compose.yml file? I don't see anything like that, though.
Here's our docker-compose.yml file:
version: '3'
services:
server:
build:
context: ./server
dockerfile: Dockerfile
env_file: .env.dev
command: python3 manage.py runserver 0.0.0.0:8000
volumes:
- ./server:/app
ports:
- "8500:8000"
depends_on:
- db
stdin_open: true
tty: true
db:
image: postgres
client:
build:
context: ./client
dockerfile: Dockerfile
command: bash -c "npm install; npm run start"
volumes:
- ./client:/app
- /app/node_modules
ports:
- "3000:3000"
depends_on:
- server
Idea there is probably to have a place to keep secrets separated from docker-compose.yml, which you then can keep in VCS and/or share.
Related
I have a docker compose setup where I want to use environment variables from env file in my dockerfile. I want to use these variables during the build time since I use this version number in concatenating the string in order to form a download URL.
Here I wrote part of the files I'm using just to keep the focus on the point of my question.
.env
MY_APP_VER=v1.2.3
docker-compose.yml
version: "2"
services:
my-app:
build: .
container_name: my_app
environment:
- my_app_version=$MY_APP_VER
Dockerfile
FROM scratch
ENV my_app_ver=$my_app_version
RUN echo $my_app_ver
I have checked various sources but without any success. I'm not sure if this is even possible or am I using the wrong syntax (should I use quotes or no e.g. "$my_app_ver" or curly brackets ${my_app_ver}).
For version 3.8 you can do it in the following way
version: '3.8'
services:
my-app:
build: .
ports:
- ${CONTAINER_PORT}:${PORT} # for example
env_file: .env
container_name: my-app-${NODE_ENV} # for example
environment:
MYSQL_DATABASE: ${DB_NAME} # for example
my_app_version: ${MY_APP_VER} # for your case
Find more information in documentation
Also, you can find more information about the usage of env variables in Dockerfile and docker-compose here
There is an option called env-file in docker-compose, that you can leverage: https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option
version: "3"
services:
my-app:
build: .
container_name: my_app
env_file:
- .env.dev
Be aware, that the .env file is loaded by default, if it is present in the current context. So you only have to use env_file, if it is named differently or is in a different folder.
I use docker and also use docker-compose for tie each container.
In my python flask code, refer environment variable like this.
import os
from app import db, create_app
app = create_app(os.getenv('FLASK_CONFIGURATION') or 'development')
if __name__ == '__main__':
print(os.getenv('FLASK_CONFIGURATION'))
app.run(host='0.0.0.0', debug=True)
And docker-compose.yml here.
version: '3.7'
services:
nginx:
build:
context: .
dockerfile: docker/nginx/dockerfile
container_name: nginx
hostname: nginx-prod
ports:
- '80:80'
networks:
- backend
links:
- web_project
depends_on:
- web_project
environment:
- FLASK_CONFIGURATION=production
mongodb:
build:
context: .
dockerfile: docker/mongodb/dockerfile
container_name: mongodb
hostname: mongodb-prod
ports:
- '27017:27017'
networks:
- backend
web_project:
build:
context: .
dockerfile: docker/web-prod/dockerfile
container_name: web_project
hostname: web_project_prod
ports:
- '5000:5000'
networks:
- backend
tty: true
depends_on:
- mongodb
links:
- mongodb
environment:
- FLASK_CONFIGURATION=production
networks:
backend:
driver: 'bridge'
I set FLASK_CONFIGURATION=production via environment command.
But when I execute, maybe FLASK_CONFIGURATION=production doesn't work.
I also tried to ENV FLASK_CONFIGURATION production to each dockerfile. (doesn't work too)
Strange thing is, When I enter to my container via bash(docker exec -it bash) and check the environment variable with export, it was set perfectly.
Is there any wrong code in my docker settings?
Thanks.
[SOLVED]
It is caused by supervisor.
When using supervisor, it's shell is isolated with original.
So we have to define our environment variables into supervisor.conf
Your flask code is looks ok, and as you said ... in bash this ENV variable exists,
My advice to you is to find way to put this variable to .env file in your project.
I will explain why i'm saying it regarding similar issue that i had with cron:
The cron run in his "own world" because the system run and execute it, and because of it he don't share those ENV variables that the bash of the main container process holding.
So i assume (please give feed back if not) that flask run too in similar way in his "own world" and don't have access to those ENV that Docker set.
So, there for, i created bash script that read all ENV variable and write them to the .env file of the project, this script run after the container created.
In this way, no matter from where and how you run the code/script ... those ENV variables will always be exists.
I have the following in my docker-compose.yml file
geth-testnet:
build:
context: .
dockerfile: Dockerfile
args:
GETH_REPO: 'https://github.com/ethereum/go-ethereum'
GETH_VERSION: 'v1.8.12'
RPC_LISTEN_PORT: 8546
command: "--rpcport ${RPC_LISTEN_PORT}"
entrypoint:
- "geth"
tty: true
image: geth-node-testnet:v1.8.12
container_name: geth-node-testnet
ports:
- '8546:8546'
volumes:
- /root/.ethereum
When I run, docker-compose up --build, expect it to run the following command:
geth -rpcport 8546
However, I get the following error
flag needs an argument: -rpcport
So, the value for RPC_LISTEN_PORT is not correctly substituted.
I have ARG RPC_LISTEN_PORT in my dockerfile
Double checked your question, seems command: "--rpcport ${RPC_LISTEN_PORT}" cannot utilize the value you put in docker-compose.yml.
So afford two solutions:
export RPC_LISTEN_PORT=8546 in bash before you do compose command.
New a .env file in the same folder, put RPC_LISTEN_PORT=8546 to it.
I'm using docker compose to run my application. And for do that I need to set the hosts inside container (it's depends on the environment i'm running).
My approach was:
Create an environment file and set the variable:
#application.env
SERVER_IP=10.10.9.134
My docker compose file looks like:
version: '2'
services:
api:
container_name: myApplication
env_file:
- application.env
build: ./myApplication/
entrypoint: ./docker/api-startup.sh
ports:
- "8080:8080"
depends_on:
- redis
extra_hosts: &extra_hosts
myip: $SERVER_IP
But my problem is that the variable SERVER_IP is never replaced.
When I run docker-compose config I see:
services:
api:
build:
context: /...../myApplication
container_name: myApplication
depends_on:
- redis
entrypoint: ./docker/api-startup.sh
environment:
SERVER_IP: 10.10.9.134
extra_hosts:
myip: ''
ports:
- 8080:8080
I've tried to replace the variable reference using $SERVER_IP or ${SERVER_IP} but it didn't work.
I created a file .env, added single line HOST=test.example.com, then did this in docker-compose:
extra_hosts:
- myip:${HOST}
docker-compose config then shows
extra_hosts:
myip: test.example.com
To do this I followed the documentation from Docker-compose environment variables the section about .env file
UPDATE
According to the Docker documentation,
Note: If your service specifies a build option, variables defined in
environment files will not be automatically visible during the build.
Use the args sub-option of build to define build-time environment
variables.
It basically means if you place your variables in .env file, you can use them for substitution in docker-compose.yml, but if you use env_file option for the particular container, you can only see the variables inside the Docker container, not during the build. It is also logical, env_file replaces docker run --env-file=FILE ... and nothing else.
So, you can only place your values into .env. Alternatively, as William described, you can use host's environment variables.
EDIT
Try the following:
version: '2'
services:
api:
container_name: myApplication
env_file:
- application.env
build: ./myApplication/
entrypoint: ./docker/api-startup.sh
ports:
- "8080:8080"
depends_on:
- redis
extra_hosts:
- "myip:${SERVER_IP}"
Ensure curly bracers and that the environment variable exists on the host os.
When I run docker-compose build && docker-compose up redis, with environment specified in docker-compose.yaml and RUN env in the Dockerfile, the environment variables I set don't get printed.
Why does this not work?
I'm using docker-compose version 1.4.2.
Here are the relevant files:
docker-compose.yaml with environment as a list of KEY=value pairs:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO='bar'
docker-compose.yaml with environment as a dictionary:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO: 'bar'
Dockerfile:
FROM redis:2.6
MAINTAINER me#email.com
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
That's normal
docker-compose only sets the environment variables specified in the environment directive in the docker-compose.yaml file during the run phase of the container, and not during the build phase.
So if you do docker-compose run --entrypoint "/bin/bash" redis -c env you will be able to see your env variables.
If you want to set variables inside your Dockerfile (to be able to see them during the build phase) you can add inside your dockerfile before your RUN env:
ENV FOO bar
Well
I have tested and found following solutions for docker compose with env file or without env file. I will show you two different approach
Lets say you have following docker compose yml file
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
env_file: docker.env
Now you need to setup the postgres variable in a file called docker.env. Remember you need to keep the docker_compose.yml file and docker.env file in same folder.
Next, In the docker.env file you need to have the database variable and value like this:
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=myapp_db
Now hit docker-compose up . It should work.
Lets say now you dont like to specify the env file name in the docker-compose.yml file. So you have to write docker-compose.yml file like this:
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
environments:
- POSTGRES_USER=${PGU}
-POSTGRES_PASSWORD=${PGP}
-POSTGRES_DB=${PGD}
Now your docker.env file should look like this:
PGU=postgres
PGP=postgres
PGD=myapp_db
now hit docker-compose --env-file docker.env up
you are good to go.
This is because you were using environment when (I guess) you wanted to use args inside the build block:
redis:
build:
context: ../storage/redis
args:
- FOO: 'bar'
ports:
- "6379:6379"
Your Dockerfile would define FUN in the (image) environment:
FROM redis:2.6
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
# Read FUN from (build) arguments
# (may define a default: ARG FUN='wow')
ARG FUN
# Define env variable FUN with value from ARG
ENV FUN=$FUN
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
The environment block is used to define variables for the running container (when docker-compose up, NOT when docker-compose build).