docker does really confuse me....
I am trying to use the "environments" variable as well as the .env file to pass variables into the container that will be created. Sadly without success.
What my setup is:
docker-compose.yml
# dockerfile version 3.8 -> 18.07.2020
version: "3.8"
# services basicly means scalable amount of containers
services:
testalpine:
image: testenvalpine
build:
context: .
dockerfile: test-dockerfile
args:
- DB_NAME=nextcloud
- DB_USER=nextcloud
# todo only use https and self sign certificate
ports:
- "80:80"
environment:
- env1=hello
- env2=world
networks:
- nextcloudnetwork
# todo include redis and mariadb
networks:
nextcloudnetwork:
# std driver seems to be overlay?
driver: overlay
Dockerfile:
test-dockerfile
FROM alpine:latest
LABEL maintainer="xddq <donthavemyownemailyet:(((>"
ARG DB_NAME=default
ARG DB_USER=default
ENV env1=dockerfile env2=$DB_NAME
ENTRYPOINT [ "sh", "-c", \
"echo DB_NAME: $DB_NAME DB_USER: $DB_USER env1: $env1 env2: $env2" ]
My .env file
DB_NAME=nextcloud
DB_USER=nextcloud
The output I did EXPECT:
DB_NAME:nextcloud DB_USER: nextcloud env1: hello env2:nextcloud
The output I got:
DB_NAME: DB_USER: env1: dockerfile env2: nextcloud
Does this mean ".env" and ENV variable in docker-compose are completely useless for the env variables inside the container that will be created? I mean I could only get any result using the ARG variable..? :/
greetings
The .env is not automatically passed to the container, you need to declare it in your docker-compose.yml using env_file: Explaination here. The environment inside the dockerfile should be overriden by the ones in your docker-compose file not sure why this is not the case.
Related
I get the below error when I run docker-compose up, any pointers why I am getting this error
service "mysqldb-docker" refers to undefined volume mysqldb: invalid compose project
Also, is there a way to pass the $ENV value in CLI to docker-compose up , currently I have a ENV variable that specified dev, uat or prod that I use to specify the db name. Are there better alternatives to do this other than create a .env file explicitly for this
version: '3.8'
services:
mysqldb-docker:
image: '8.0.27'
restart: 'unless-stopped'
ports:
- "3309:3306"
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_PASSWORD=root
- MYSQL_DATABASE=reco-tracker-$ENV
volumes:
- mysqldb:/var/lib/mysql
reco-tracker-docker:
image: 'reco-tracker-docker:v1'
ports:
- "8083:8083"
environment:
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=root
- SPRING_DATASOURCE_URL="jdbc:mysql://mysqldb-docker:3309/reco-tracker-$ENV"
depends_on: [mysqldb-docker]
You must define volumes at the top level like this:
version: '3.8'
services:
mysqldb-docker:
# ...
volumes:
- mysqldb:/var/lib/mysql
volumes:
mysqldb:
You can pass environment variables from your shell straight through to a service’s containers with the ‘environment’ key by not giving them a value
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers
web:
environment:
- ENV
but from my tests you cant write $ENV in the compose file and expect it to read your env
for this you need to call docker-compose that way :
docker-compose run -e ENV web python console.py
see this : https://docs.docker.com/compose/environment-variables/#set-environment-variables-with-docker-compose-run
I'm trying to pass some host's environment variables content as args to the docker-compose file, through a .env file. But the variable is interpreted as a string.
Follows the content of my files:
.env:
USER=$USER
UID=$UID
GID=$GID
docker-compose.yml:
version: "2"
services:
opencv_python:
build:
args:
- username=${USER}
- uid=${UID}
- gid=${GID}
context: .
dockerfile: opencv_base.Dockerfile
container_name: ocv-data-augmentation
image: ocv-data-augmentation
environment:
DISPLAY: $DISPLAY
QT_X11_NO_MITSHM: 1
volumes:
- "../project:/home/&{USER}/data_augmentation/" # Host : Container
- "/tmp/.X11-unix:/tmp/.X11-unix"
tty: true
And this is the output of the command docker-compose config:
services:
opencv_python:
build:
args:
gid: $$GID
uid: $$UID
username: fsalvagnini
context: /home/fsalvagnini/Documents/containers/data_augmentation/dockerfiles
dockerfile: opencv_base.Dockerfile
container_name: ocv-data-augmentation
environment:
DISPLAY: :1
QT_X11_NO_MITSHM: 1
image: ocv-data-augmentation
tty: true
volumes:
- /home/fsalvagnini/Documents/containers/data_augmentation/project:/home/&{USER}/data_augmentation:rw
- /tmp/.X11-unix:/tmp/.X11-unix:rw
version: '2.0'
If you need to use the .env file and assuming that all the env variables are defined, you just need to follow one step:
.env file
source .env
The above statement will source all the variables defined in .env and hence the env variables will be accessible to docker-compose.
Just added thing, you should also look at ${VARIABLE:-default} just in case you need to pass a default value.
More documentation here
According to the docker-compose manual:
When you set the same environment variable in multiple files, here’s
the priority used by Compose to choose which value to use:
Compose file
Shell environment variables
Environment file
Dockerfile
Variable is not defined
So if shell environment variables are not set, then env file will be used.
For your case, if you need to use shell env vars, you don't need to create .env file. To solve your issue, you need to export the variables before invoking docker-compose.
export GID
export UID
export DISPLAY
docker-compose config
output:
services:
opencv_python:
build:
args:
gid: '20'
uid: '501'
username: enix
context: /Users/enix/source/devops/stackoverflow
dockerfile: opencv_base.Dockerfile
container_name: ocv-data-augmentation
environment:
DISPLAY: :1
QT_X11_NO_MITSHM: 1
image: ocv-data-augmentation
tty: true
volumes:
- /Users/enix/source/devops/project:/home/&{USER}/data_augmentation:rw
- /tmp/.X11-unix:/tmp/.X11-unix:rw
version: '2.0'
I use docker and also use docker-compose for tie each container.
In my python flask code, refer environment variable like this.
import os
from app import db, create_app
app = create_app(os.getenv('FLASK_CONFIGURATION') or 'development')
if __name__ == '__main__':
print(os.getenv('FLASK_CONFIGURATION'))
app.run(host='0.0.0.0', debug=True)
And docker-compose.yml here.
version: '3.7'
services:
nginx:
build:
context: .
dockerfile: docker/nginx/dockerfile
container_name: nginx
hostname: nginx-prod
ports:
- '80:80'
networks:
- backend
links:
- web_project
depends_on:
- web_project
environment:
- FLASK_CONFIGURATION=production
mongodb:
build:
context: .
dockerfile: docker/mongodb/dockerfile
container_name: mongodb
hostname: mongodb-prod
ports:
- '27017:27017'
networks:
- backend
web_project:
build:
context: .
dockerfile: docker/web-prod/dockerfile
container_name: web_project
hostname: web_project_prod
ports:
- '5000:5000'
networks:
- backend
tty: true
depends_on:
- mongodb
links:
- mongodb
environment:
- FLASK_CONFIGURATION=production
networks:
backend:
driver: 'bridge'
I set FLASK_CONFIGURATION=production via environment command.
But when I execute, maybe FLASK_CONFIGURATION=production doesn't work.
I also tried to ENV FLASK_CONFIGURATION production to each dockerfile. (doesn't work too)
Strange thing is, When I enter to my container via bash(docker exec -it bash) and check the environment variable with export, it was set perfectly.
Is there any wrong code in my docker settings?
Thanks.
[SOLVED]
It is caused by supervisor.
When using supervisor, it's shell is isolated with original.
So we have to define our environment variables into supervisor.conf
Your flask code is looks ok, and as you said ... in bash this ENV variable exists,
My advice to you is to find way to put this variable to .env file in your project.
I will explain why i'm saying it regarding similar issue that i had with cron:
The cron run in his "own world" because the system run and execute it, and because of it he don't share those ENV variables that the bash of the main container process holding.
So i assume (please give feed back if not) that flask run too in similar way in his "own world" and don't have access to those ENV that Docker set.
So, there for, i created bash script that read all ENV variable and write them to the .env file of the project, this script run after the container created.
In this way, no matter from where and how you run the code/script ... those ENV variables will always be exists.
I'm working on a group project involving Docker that has a .env file, which looks like this:
DATABASE_URL=xxx
DJANGO_SETTINGS_MODULE=xxx
SECRET_KEY=xxx
Couldn't this just be declared inside the Dockerfile? If so, what is the advantage of making a .env file?
Not sure if I'm going in the right direction with this, but this Docker Docs page says (emphasis my own):
Your configuration options can contain environment variables. Compose
uses the variable values from the shell environment in which
docker-compose is run. For example, suppose the shell contains
POSTGRES_VERSION=9.3 and you supply this configuration:
db:
`image: "postgres:${POSTGRES_VERSION}"`
When you run docker-compose up with this configuration, Compose looks for the POSTGRES_VERSION environment variable in the shell and substitutes its value in. For this example, Compose resolves the image to postgres:9.3 before running the configuration.
If an environment variable is not set, Compose substitutes with an empty string. In the example above, if POSTGRES_VERSION is not set, the value for the image option is postgres:.
You can set default values for environment variables using a .env file, which Compose automatically looks for. Values set in the shell environment override those set in the .env file.
If we're using a .env file, then wouldn't I see some ${...} syntax in our docker-compose.yml file? I don't see anything like that, though.
Here's our docker-compose.yml file:
version: '3'
services:
server:
build:
context: ./server
dockerfile: Dockerfile
env_file: .env.dev
command: python3 manage.py runserver 0.0.0.0:8000
volumes:
- ./server:/app
ports:
- "8500:8000"
depends_on:
- db
stdin_open: true
tty: true
db:
image: postgres
client:
build:
context: ./client
dockerfile: Dockerfile
command: bash -c "npm install; npm run start"
volumes:
- ./client:/app
- /app/node_modules
ports:
- "3000:3000"
depends_on:
- server
Idea there is probably to have a place to keep secrets separated from docker-compose.yml, which you then can keep in VCS and/or share.
When I run docker-compose build && docker-compose up redis, with environment specified in docker-compose.yaml and RUN env in the Dockerfile, the environment variables I set don't get printed.
Why does this not work?
I'm using docker-compose version 1.4.2.
Here are the relevant files:
docker-compose.yaml with environment as a list of KEY=value pairs:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO='bar'
docker-compose.yaml with environment as a dictionary:
redis:
build: ../storage/redis
ports:
- "6379:6379"
environment:
- FOO: 'bar'
Dockerfile:
FROM redis:2.6
MAINTAINER me#email.com
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
That's normal
docker-compose only sets the environment variables specified in the environment directive in the docker-compose.yaml file during the run phase of the container, and not during the build phase.
So if you do docker-compose run --entrypoint "/bin/bash" redis -c env you will be able to see your env variables.
If you want to set variables inside your Dockerfile (to be able to see them during the build phase) you can add inside your dockerfile before your RUN env:
ENV FOO bar
Well
I have tested and found following solutions for docker compose with env file or without env file. I will show you two different approach
Lets say you have following docker compose yml file
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
env_file: docker.env
Now you need to setup the postgres variable in a file called docker.env. Remember you need to keep the docker_compose.yml file and docker.env file in same folder.
Next, In the docker.env file you need to have the database variable and value like this:
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=myapp_db
Now hit docker-compose up . It should work.
Lets say now you dont like to specify the env file name in the docker-compose.yml file. So you have to write docker-compose.yml file like this:
version: '3.8'
services:
db:
image: postgres:13
volumes:
- "./volumes/postgres:/var/lib/postgresql/data"
ports:
- "5432:5432"
environments:
- POSTGRES_USER=${PGU}
-POSTGRES_PASSWORD=${PGP}
-POSTGRES_DB=${PGD}
Now your docker.env file should look like this:
PGU=postgres
PGP=postgres
PGD=myapp_db
now hit docker-compose --env-file docker.env up
you are good to go.
This is because you were using environment when (I guess) you wanted to use args inside the build block:
redis:
build:
context: ../storage/redis
args:
- FOO: 'bar'
ports:
- "6379:6379"
Your Dockerfile would define FUN in the (image) environment:
FROM redis:2.6
RUN mkdir -p /var/redis && chown -R redis:redis /var/redis
# Read FUN from (build) arguments
# (may define a default: ARG FUN='wow')
ARG FUN
# Define env variable FUN with value from ARG
ENV FUN=$FUN
RUN echo '-------------- env ---------------'
RUN env
COPY redis.conf /usr/local/etc/redis/redis.conf
EXPOSE 6379
ENTRYPOINT ["redis-server", "/usr/local/etc/redis/redis.conf"]
The environment block is used to define variables for the running container (when docker-compose up, NOT when docker-compose build).