I'm aware of the variable substitutions available, where I could use a .env at the root of the project and that would be done, but in this case I'm adapting an existing project, where existing .env file locations are expected and I would like to prevent having to have var entries on multiple files!
See documentation for more info, and all the code is available as WIP on the docker-support branch of the repo, but I'll succinctly describe the project and issue below:
Project structure
|- root
| |- .env # mongo and mongo-express vars (not on git!)
| |- docker-compose.yaml # build and ups a staging env
| |- docker-compose.prod.yaml # future wip
| |- api # the saas-api service
| |- Dockerfile # if 'docked' directly should build production
| |- .env # api relative vars (not on git!)
| |- app # the saas-app service
| |- Dockerfile # if 'docked' directly should build production
| |- .env # api relative vars (not on git!)
Or see the whole thing here, it works great by the way for the moment, but there's one problem with saas-app when building an image for staging/production that I could identify so far.
Issue
At build time Next.js builds a static version of the pages using webpack to do it's thing about process.env substitution, so it requires the actual eventual running vars to be included at docker build stage so next.js doesnt need to rebuild again at runtime and also so that I can safely spawn multiple instances when traffic requires!
I'm aware that if at runtime the same vars are not sent it will have to rebuild again defying the point of this exercise, but that's precisely what I'm trying to prevent here, to that if the wrong values are sent it's on us an not the project!
And I also need to consider Next.js BUILD ID managemement, but that's for another time/question.
Attempts
I've been testing with including the ARG and ENV declarations for each of the variables expected by the app on it's Dockerfile, e.g.:
ARG GA_TRACKING_ID=
ENV GA_TRACKING_ID ${GA_TRACKING_ID}
This works as expected, however it forces me to manually declare them on the docker-compose.yml file, which is not ideal:
saas-app:
build:
context: app
args:
GA_TRACKING_ID: UA-xXxXXXX-X
I cannot use variable substitution here because my root .env does not include this var, it's on ./app/.env, and I also tested leaving the value empty but it is not picking it up from the env_file or enviroment definitions, which I believe is as expected.
I've pastbinned a full output of docker-compose config with the existing version on the repository:
Ideally, I'd like:
saas-app:
build:
args:
LOG_LEVEL: notice
NODE_ENV: development
PORT: '3000'
context: /home/pedro/src/opensource/saas-boilerplate/app
command: yarn start
container_name: saas-app
depends_on:
- saas-api
environment:
...
To become:
saas-app:
build:
args:
LOG_LEVEL: notice
NODE_ENV: development
PORT: '3000'
BUCKET_FOR_POSTS: xxxxxx
BUCKET_FOR_TEAM_AVATARS: xxxxxx
GA_TRACKING_ID: ''
LAMBDA_API_ENDPOINT: xxxxxxapi
NODE_ENV: development
STRIPEPUBLISHABLEKEY: pk_test_xxxxxxxxxxxxxxx
URL_API: http://api.saas.localhost:8000
URL_APP: http://app.saas.localhost:3000
context: /home/pedro/src/opensource/saas-boilerplate/app
command: yarn start
container_name: saas-app
depends_on:
- saas-api
environment:
...
Questions
How would I be able to achieve this, if possible, but:
Without merging the existing .env files into a single root, or having to duplicate vars on multiple files.
Without manually declaring the values on the compose file, or having to infer them on the command e.g. docker-compose build --build-arg GA_TRACKING_ID=UA-xXxXXXX-X?
Without having to COPY each .env file during the build stage, because it doesn't feel right and/or secure?
Maybe a args_file on the compose build options feature request for the compose team seems to me to be a valid, would you also say so?
Or perhaps have a root option on the compose file where you could set more than one .env file for variable substituion?
Or perhaps another solution i'm not seeing? Any ideas?
I wouldn't mind sending each .env file as a config or secret, it's a cleaner solution than splitting the compose files, is anyone running such an example for production?
Rather than trying to pass around and merge values in multiple .env's would you consider making one master .env and having the API and APP services inherit the same root .env?
I've managed to achieve a compromise that does not affect any of the existing development workflows, nor does it allow for app to build without env variables (a requirement that will be more crucial for production builds).
I've basically decided to reuse the internal ability of docker to read the .env file and use those in variable substitution on the compose file, here's an example:
# compose
COMPOSE_TAG_NAME=stage
# common to api and app (build and run)
LOG_LEVEL=notice
NODE_ENV=development
URL_APP=http://app.saas.localhost:3000
URL_API=http://api.saas.localhost:8000
API_PORT=8000
APP_PORT=3000
# api (run)
MONGO_URL=mongodb://saas:secret#saas-mongo:27017/saas
SESSION_NAME=saas.localhost.sid
SESSION_SECRET=3NvS3Cr3t!
COOKIE_DOMAIN=.saas.localhost
GOOGLE_CLIENTID=
GOOGLE_CLIENTSECRET=
AMAZON_ACCESSKEYID=
AMAZON_SECRETACCESSKEY=
EMAIL_SUPPORT_FROM_ADDRESS=
MAILCHIMP_API_KEY=
MAILCHIMP_REGION=
MAILCHIMP_SAAS_ALL_LIST_ID=
STRIPE_TEST_SECRETKEY=
STRIPE_LIVE_SECRETKEY=
STRIPE_TEST_PUBLISHABLEKEY=
STRIPE_LIVE_PUBLISHABLEKEY=
STRIPE_TEST_PLANID=
STRIPE_LIVE_PLANID=
STRIPE_LIVE_ENDPOINTSECRET=
# app (build and run)
STRIPEPUBLISHABLEKEY=
BUCKET_FOR_POSTS=
BUCKET_FOR_TEAM_AVATARS=
LAMBDA_API_ENDPOINT=
GA_TRACKING_ID=
See the updated docker-compose.yml I've also made use of Extension fields to make sure only the correct and valid vars are sent across on build and run.
It breaks rule 1. from the question, but I feel it's a good enough compromise, because it no longer relies on the other .env files, that would potentically be development keys most of the time anyway!
Unfortunately we will need to mantain the compose file if the vars change in the future, and the same .env file has to be used for a production build, but since that will probably be done externally on some CI/CD, that does not worry much.
I'm posting this but not fully closing the question, if anyone else could chip in with a better idea, I'd be greatly appreciated.
Related
So I have been tasked with taking an existing dockerized version of a service, and creating docker images from this repository.
Creating the images is not the problem however, since the build command starts it up no problem. The issue is that this dockerfile copies an .env file during build, that holds variables that must be customizable after the build process is done (expected db and other endpoint info).
Is there some way to set that file to automatically be changed to reflect the environmental variables used in the docker run command? (I do want to note, that the docker image does copy the .env file into the working directory, it is not docker-compose reading that .env file)
I am sure that there has to be an easy way to do this, but all the tutorials I am pulling up just show you how to declare these variables, not how to get the files in docker to use them! Most of the code being run is javascript, and uses npm and yarn if that makes any difference...
docker does not provide any way to update files from environment variables on container start. But I don't think this is what you need anyway:
As I understand a .env file with default values is copied into the image at build time and you want to be able to change some of the values at runtime via container environment variables?
Usually such an .env file is read by the application and complemented by any variables set in the environment, i.e. you can override values from the file with environment variables. For javascript projects dotenv is a popular module to do this.
So to override say an API_ENDPOINT variable specified in .env you simply need to pass an environment variable with the same name and desired value to the container:
docker run -e API_ENDPOINT=/other/endpoint ...
If for some reason your applications do not work according to this convention and you actually need to change the values in the .env file you will need to write a custom script that updates/generates .env from the values of passed environment variables and use this script as ENTRYPOINT
I'm working on a new service on my stack.
So when I work on the new-service branch, I have in docker-compose.yml:
version: "3.6"
services:
app:
image: whatever
new_service:
image: whatever
I do docker-compose up and start working.
But then I sometimes need to make hot fixes on the master branch, where new_service is not there.
When I try to do docker-compose up, I get the following error:
ERROR: The Compose file is invalid because:
Service new_service has neither an image nor a build context specified. At least one must be provided.
I guess, I could edit my docker-compose.yml file to add fake new_service entry only to start my server, but that's not an acceptable solution to me.
Shouldn't docker-compose re-read the service list on every start? Or can't I at least force the refresh like docker-compose up --reset?
Ok my bad.
docker-compose does work as expected, but I had a
service:
new_service:
command: yarn dev
entry in my docker-compose.override.yml which is .gitignored, hence the error.
I wanted to delete the question, but then I considered others may do the mistake, and landing here would help them too.
We'll see. :-)
Can Docker's COPY or RUN cp be used in a Dockerfile to overwrite a default config file with a docker-specific version of the file?
In a Rails project, our config folder has multiple versions of database.yml for different environments:
# projectname/config/
database.yml # an unused default placeholder
database_for_docker_2.yml
database_for_vagrant.yml
For different dev environments (vagrant+virtualbox vs docker) during initialization of the machine/container we copy the appropriate version of the .yml to database.yml
In the Dockerfile, after this section:
WORKDIR /my_app
RUN bundle install
COPY . /my_app
we tried:
RUN cp ./config/database_docker_2.yml /my_app/config/database.yml
but the file does not seem to be copied, the default version of database.yml is used when we spin up the container.
we then tried:
COPY ./config/database_docker_2.yml /my_app/config/database.yml
the file still does not seem to be copied, the default version of the file gets used when we spin up the container.
What DOES work is adding another entry to the volume section of docker-compose.yml specifically for that one file:
volumes:
- .:/my_app
- ./config/database_docker_2.yml:/my_app/config/database.yml
but we prefer to manage the placement of env-specific versions of files in the Dockerfile (as opposed to littering the docker-compose.yml with such env-specific files)
The command COPY ./config/database_docker_2.yml /my_app/config/database.yml probably works, there is no reason it shouldn't assuming the source exists.
What I suspect happens, is that when you are testing it, you already have a volume with .:/my_app, which then shows you the local folder, and not the in-container folder.
Run it without the volume, and I believe you will in fact see that it copied it into the container, as you intended.
On a side note:
If you are not yet locked in your way of handling this multiple database config, I would consider re-evaluating your situation, and trying to find a solution that does not require you to change database.yml for each environment. One way, would be to have the database.yml use an environment variable (usually DATABASE_URL) and then you have one docker-compose for all, and one database.yml for all, and you only configure environment with environment variables.
I want to know where can I see the variables I have defined. For example, I have three files:
.env
BLA=1
docker-compose.xml
IS_ONE=${BLA}
Dockerfile
RUN echo "$BLA" >> file.txt
I want to know the relation, for example:
Do I need the variable defined in all files? or there a relation like
docker-compose.xml can only see .evn variables?
Dockerfile can only see docker-compose.xml variables?
And, where do I need to declare a variable to use in a bash script automatically triggered in Dockerfile (if it's possible), or use later in the console of the container.
Lots of examples here: https://docs.docker.com/compose/environment-variables/
If you need to customize docker-compose itself then .env file is used.
For example, I want to include build id in my container name, my .env file would have
BUILD_ID=20
and docker-compose.yml would have
containername: "foo-${BUILD_ID}"
If I want to pass environment vars into container when it runs, I use
environment:
- FOO_VAR=bar
If I want to pass in multiple environment vars into container,
docker-compose.yml would have
env_file: my_container_env
is the easy way if the values of the FOO_VAR are going to be dynamically generated.
Whether to use set env values in docker-compose directly or use file depends on:
Are the values dynamic? If yes, use file and populate correct values
in the file with some script (like jenkins could set my BUILD_ID in
my .env file)
Are they 'secret'? If yes, use file and don't checkin the .env or my_container_env file into the repo - you have to manage those files
separately
I have several containers that I run together with docker-compose.
One of them, is mysql, which requires some variables to be set. I have those in a .env file:
MYSQL_USER='my_user'
MYSQL_PASSWORD='my_password'
MYSQL_ROOT_PASSWORD='supersecretpassword'
MYSQL_DATABASE='my_database'
And I am able to start the mysql container successfully.
The problem comes when I want to use another service for db migrations, which require the following variables set in the .env file:
SERVICE_DBUSER='my_user'
SERVICE_DBPASSWORD='my_password'
SERVICE_DBNAME='my_database'
And what I would like to write (this doesn't work), to avoid repetition, is something like:
SERVICE_DBUSER="$MYSQL_USER"
SERVICE_DBPASSWORD="$MYSQL_PASSWORD"
SERVICE_DBNAME="$MYSQL_DATABASE"
But docker doesn't recognize that and doesn't perform the substitution. In the docker docs, it also states that expects each line in an env file to be in VAR=VAL format.
My question is, is it possible to avoid the repetition?
Many thanks.
Compose will substitute environment variables into the YAML compose file when you reference them with $VARIABLE or ${VARIABLE}.
You can still use the .env file to set a default environment. But when you want to reference a variable, put it in the environment: section of the compose yaml:
environment:
SERVICE_DBUSER: "${MYSQL_USER}"
SERVICE_DBPASSWORD: "${MYSQL_PASSWORD}"
SERVICE_DBNAME: "${MYSQL_DATABASE}"
Then if you set, or source an alternate environment when running docker-compose you will get the new values substituted in.
$ MYSQL_USER="other" MYSQL_PASSWORD="opass" docker-compose start