How to copy file contents to a service during docker-compose up? - docker

I have 2 files:
.env
docker-compose.yml
docker-compose.yml looks like this:
version: '3'
services:
database:
image: mysql:5.7
myapp:
image: me/some-image
depends_on:
- database
env_file: .env
myapp is a web service app that needs a .env file or optionally it can access the environment variables if no .env file is present.
As of now, the myapp is accessing the environment variables because I don't want the .env file to be included in the image build for security reasons. What I did is to pass a env_file: .env to the myapp service in the docker-compose.yml file so it will rely to the environment variables of the service instead of a .env file.
Now, I really want to add a .env file to the myapp service when running docker-compose up. Take note that the myapp web service will throw an error if it didnt find a .env file and the option is to look for a .env file instead of getting from the environment variables of the container.
Is there a way to create a .env file when running docker-compose up and copy the contents of the .env file on the host? Thank you in advance.

You can use bind mount to mount file into the container
change the target location to the one your app requires
version: '3'
services:
database:
image: mysql:5.7
myapp:
image: me/some-image
depends_on:
- database
env_file: .env
volumes:
- type: bind
source: ./.env
target: /envfile/.env
readonly: true

You shouldn't be storing the environment variables in the container.
According to TwelveFactor-
The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
You are correctly passing the env file. The env file needs to be in the format of <key>=<value>.
For example -
DB_USERNAME=user
DB_PASSWORD=password
DB_SCHEMA=db
DB_PORT=3306
DB_HOST=db
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
In your application, you don't need to care about if the .env file is present, you simply read the environment variable.
Example in python -
import os
db_username = os.environ['DB_USERNAME']
You should not be coupling your environment variable to a file. It should read from the runtime environment.

Related

Can't set a different name for .env file inside docker-compose

I'm refactoring and writing a new dockercompose for our project and I'm facing some really strange problems.
My dockercompose has different services inside, each one has the own .env file. All the .env files are inside the same directory of the docker-compose file of course.
To keep it simple for my test, now I'm working with just ONE service inside the docker-compose.
If I name the .env file just ".env", everything works fine BUT if I try to name it "userservice.env", the docker compose CAN find the file but it can't read some variables such as the git hub access key and username.
That's my docker-compose:
version: '3.8'
services:
db:
image: mariadb:10.5.2
restart: always
environment:
MYSQL_ROOT_PASSWORD: password
container_name: exampledb
ports:
- "4040:3306"
volumes:
- ./initmysqldb:/docker-entrypoint-initdb.d
adminer:
image: adminer
restart: always
ports:
- "${DOCKER_ADMINER_HOST_PORT:-38088}:8080"
app-user-area:
image: user_area
build:
context: ./user-area
network: host
args:
GITLAB_ACCESS_KEY: "${GITLAB_ACCESS_KEY}"
GITLAB_LOGIN: "${GITLAB_LOGIN}"
NODE_VER: "${NODE_VER}"
ports:
- "${DOCKER_APP_HOST_PORT:-8088}:4000"
- "${DOCKER_DELVE_HOST_PORT:-48088}:40000"
env_file:
- userservice.env
restart: always
links:
- db
- adminer
The .env file:
....other info
# GITLAB
# ------------------------------------------------------------------------------
GITLAB_ACCESS_KEY=git_access_key
GITLAB_LOGIN=git_login
...other info
The error that the console gives to me if I name the file "userservice.env" (and then I call it as well inside the dockercompose of course) is:
The "GITLAB_ACCESS_KEY" variable is not set. Defaulting to a blank string.
And that's happened only if the ".env" file is called as "userservice.env". The strange thing is that apparently the dockercompose CAN find the .env file (if I name it "example.env" without change the name inside the docker-compose, the process can't run with "userservice.env not found" error) but it can't just read the environment variable inside.
In the end, other strange things is that if I add other services and one of them has a ".env" file and the others have "custom1.env", "custom2.env"... files, everythings works fine! It looks like that docker needs at least ONE file called ".env".
Any help would be appreciated.
Completely my fault, I was mixing .env and env_file.
To solve my problem I created a .env inside the same directory of the docker-compose with all the variables called by the compose.
Inside the env_file section I pass the .env file used by the services.
Basically, we must keep in mind that .env is different from env_file

Injecting environment variable to docker containers using docker-compose

Is there a way to inject environment variables to all of my docker-compose services, without explicitly declaring them in each service's configuration?
1)using the env_file directive
you can use the env_file directive within your docker-compose.yml file
You can pass multiple environment variables from an external file
through to a service’s containers with the ‘env_file’ option, just
like with docker run --env-file=FILE ...:
you will have to declare the env file for each service that will use it and that is it.
example :
the docker-compose.yml file :
version: "3"
services:
database:
image: "ubuntu"
tty: true
env_file:
- same-variables.env
web:
image: "ubuntu"
tty: true
env_file:
- same-variables.env
the same-variables.env file
IS_DOCKER_COMPOSE=yes
then if you do opening a terminal :
docker exec -it <docker_container> "echo $IS_DOCKER_COMPOSE"
result will be :
yes
2)using the .env file in project root
According to the doc: https://docs.docker.com/compose/environment-variables/#the-env-file
You can set default values for any environment variables referenced in
the Compose file, or used to configure Compose, in an environment file
named .env. The .env file path is as follows
Create an .env file as the root of your project as follow.
example :
your .env file :
TAG=v1.5
your docker-compose.yml file :
version: '3'
services:
web:
image: "webapp:$TAG"
organization of your project should be :
root_folder
|-.env
|-docker-compose.yaml
with the .env file with all your variable
the env file will work for all of them at the same time

How to share global env vars in Docker?

I have two environments development and production, I use two files with respective vars of each environment: .env.development and .env.production. I'm use, too, docker-compose to load this variables.
.env.development
COMPOSE_PROJECT_NAME=luna
RAILS_ENV=development
DATABASE_URL=postgresql://user:pass#lunapostgres:5432/luna?encoding=utf8&pool=5&timeout=5000
REDIS_CACHE_URL=redis://:pass#redis:6379/0/cache
ACTIVE_JOB_QUEUE_PREFIX=luna:jobs
ACTIVE_JOB_URL=redis://:pass#redis:6379/0
AUTH_BASE_URL=auth.com
SOLAR_BASE_URL=http://test.url
# SOLAR_BASE_URL=http://api/api/v1/
BUNDLE_PATH=/box
BIND_ON=0.0.0.0:3000
SENTRY_DSN=http://xxxxxxx
PAGER=more
ACCESS_TOKEN=xxx
VERIFY_TOKEN=xxx
DIALOGFLOW_CLIENT_ACCESS_TOKEN=xxx
DIALOGFLOW_DEV_ACCESS_TOKEN=xxx
RAILS_MAX_THREADS=1
WEB_CONCURRENCY=1
REQUEST_TIMEOUT=5
DOMAIN=localhost:3000
BASE_URL=localhost:300
SECRET_TOKEN=xxx
LOG_LEVEL=debug
SOLAR_MENTOS_DEBUG=true
.env.production
COMPOSE_PROJECT_NAME=luna
RAILS_ENV=production
RACK_ENV=production
DATABASE_URL=postgresql://user:pass#lunapostgres:5432/luna?encoding=utf8&pool=5&timeout=5000
REDIS_CACHE_URL=redis://:pass#redis:6379/0/cache
ACTIVE_JOB_QUEUE_PREFIX=luna:jobs
ACTIVE_JOB_URL=redis://:pass#redis:6379/0
AUTH_BASE_URL=auth.com
SOLAR_BASE_URL=http://test.url
# SOLAR_BASE_URL=http://api/api/v1/
BUNDLE_PATH=/box
BIND_ON=0.0.0.0:3000
SENTRY_DSN=http://xxxxxxx
ACCESS_TOKEN=yyy
APP_SECRET=yyy
VERIFY_TOKEN=yyy
DIALOGFLOW_CLIENT_ACCESS_TOKEN=yyy
DIALOGFLOW_DEV_ACCESS_TOKEN=yyy
RAILS_SERVE_STATIC_FILES=true
RAILS_LOG_TO_STDOUT=true
WEB_CONCURRENCY=5
REQUEST_TIMEOUT=5
RAILS_MAX_THREADS=5
DOMAIN=production.com
BASE_URL=https://production.com
SECRET_TOKEN=yyy
LOG_LEVEL=info
# ----------------------------------------
DEVISE_SECRET_KEY='yyy'
GOOGLE_ANALYTICS_UA='yyy'
docker-compose.override.yml
version: '2'
services:
app:
env_file:
- '.env.development'
docker-compose.production.yml
version: '2'
services:
app:
env_file:
- '.env.production'
I would like share equivalents environment variables between my containers, and keep different variables in your respective environment.
Use extendable environment files.
version: '2'
services:
app:
env_file:
- 'base.env'
- 'production.env'
From the Docs
When you set the same environment variable in multiple files, here’s
the priority used by Compose to choose which value to use:
Compose file,
Environment file
Dockerfile
Variable is not defined
The docs are not clear on the use of multiple files so I ran a test. The last environment files overrides previously set variables. If you want to override default values, do so in the last file or in the compose file.
base.env
TEST_VARIABLE=base
production.env
TEST_VARIABLE=production
docker-compose.yml
version: '2.1'
services:
test:
image: alpine
env_file:
- 'base.env'
- 'production.env'
Running docker-compose run --rm test env gives us TEST_VARIABLE=production. Thus, the second file overrides the first.

How to use environment variables inside docker .env file?

I have a cache folder and the path is stored in an evironment variable: $LOCAL_CACHE (export LOCAL_CACHE=/home/me/path/to/any/cache/folder)
Now I want to append some additional folders inside my .env file:
My .env file:
LOCAL_COMPOSER_DIR=${LOCAL_CACHE}/composer
LOCAL_NPM_DIR=${LOCAL_CACHE}/npm
LOCAL_BOWER_DIR=${LOCAL_CACHE}/bower
My docker_compose.yml looks like that:
version: '2'
services:
composer:
env_file: .env
image: composer/composer
volumes:
- ./src:${APP_ROOT}
- ${LOCAL_COMPOSER_DIR}:/composer
working_dir: ${APP_ROOT}
When I start the service with docker-compose run composer instal. ... it stops with following error: ERROR: Named volume "$LOCAL_CACHE"/composer":/composer:rw" is used in service "composer" but no declaration was found in the volumes section.
I'm not shure but it seems, the .env file doesn't support the use of variables.
Yes. Into .env file you can not use variables. Suddenly but true. Actual for docker compose version 1.8.
But you can use variables into docker-compose.yml file like
volumes:
- ${LOCAL_COMPOSER_DIR}/${CACHE_DIR}:/composer/${CACHE_DIR}
This answer suggest us another way.

Specify the env file docker compose uses

Is it possible to specify the env file that docker compose uses for variable substitution? Currently it defaults to ".env" but I have different flavours of compose files which should have different envs.
You can use inheritance for this. If you have one "base" service where you set up the environment, all of your other services can inherit from that.
Example:
version: "2"
services:
base:
env_file:
- my_env.txt
web:
extends:
service: base
image: foo
database:
extends:
service: base
image: foo-db
The above example has everything in the same file, but you can also split this up into multiple files, where the base service would reside in a base.yaml file. You just need to add file: base.yaml to the extends section. Please see the documentation here.
I use this approach for setting the proxy variables for all containers. I have a proxy.yaml file that defines a proxy-app service that picks up the proxy environment variables from the shell. All of my real services extend the proxy-app service and thus inherit the environment settings from that service.
The --env-file command-line argument and the env_file docker-compose.yml variable specify the env file to use for the container, not for the container build. To set a different file (e.g. alt.env) for the build itself, use this:
env $(cat alt.env) docker-compose up --build
According to the documentation, it's now possible to load an environment file (contrary to a per-service file), docker-compose will then export the env variables defined in this env file prior to starting any service, they can then be used in the docker-compose.yml config file itself:
version: "3.7"
services:
node:
environment:
APP_ENV: "${APP_ENV}"
NODE_ENV: "${NODE_ENV}"
ports:
- "${HOST_EXPOSED_NODEJS_DEBUG_PORT}:9229"
volumes:
- type: bind
source: ./project
target: /var/www/project
read_only: false
Since docker-compose 1.25 it's also possible to specify a custom .env file with the --env-file flag (unfortunately it's currently not possible to specify multiple .env files with the --env-file flag)

Resources