Normally when using environment variables in Docker you would specify them with ENV command in the Dockerfile or give the .env file with the --env-file option in a docker run command.
When creating a Vertex AI notebook executor this latter option is not available while the first option requires maintenance of the Dockerfile when an env variable changes.
We can specify the env variables as parameters in a .yaml when creating the executor, but that requires again an extra step editing in the process when a env variable changes.
How can we pass the content of the .env file to the container at runtime with no extra effort in Vertex AI?
Related
I have a Dockerfile that sets environment variables that are common to all environments, whether dev, test, or production ones, but I have to set another environment variable that is only applicable to my development environment, so I can't set it in the Dockerfile because such file is managed by the version control, so the change would be deployed to all environments.
How can add an environment variable to a docker container only in my local development environment?
In case that the env variable can be specified when the image is being used, then just supplying the variable then makes more sense. For instance, if you are locally testing the image, by using the docker cli, you can set the variable with:
docker run -e KEY=VALUE $image
If you are using other tools to test the image, there are always other methods to set env keys.
If it's required for you to have set the variable at build time, you can specify built args inside the Dockerfile.
An example for that would be:
FROM someimage:v1
ARG DEV_ONLY_VAR
ENV KEY=$DEV_ONLY_VAR
Using this, you can specify the build arg DEV_ONLY_VAR in the build command by writing:
docker build --build-arg DEV_ONLY_VAR=VALUE .
Note, even without the ENV KEY=$DEV_ONLY_VAR line the build arg will be available like a env variable during build time, on other run steps.
More on build args here
I am using Docker composé ver 1.5 or 6 and nginx image. I want to parameterize the nginx.config. To do that I want to create a var from $(basename some path). But the problem is Docker does not accept dynamic vars like that in env part of the Dockerfile. Another problem is that I also cannot map those variables on build run as Docker compose does not accept dynamic, scripted vars. How to overcome that issue?
From nginx
ENV myvar=$(basename /)
This is impossible to build the image.
The other way was
ARG myvar
ENV myvar2=myvar
But my version of Docker compose allows only to set
Environment: myvar=$(basename mypathinthevolume/)
That also does not seem to work
After reading the config point of the 12 factor app I decided to override my config file containing default value with environment variable.
I have 3 Dockerfiles, one for an API, one for a front-end and one for a worker. I have one docker-compose.yml to run those 3 services plus a database.
Now I'm wondering if I should define the environment variables in Dockerfiles or docker-compose.yml ? What's the difference between using one rather than another ?
See this:
You can set environment variables in a service’s containers with the 'environment' key, just like with docker run -e VARIABLE=VALUE ...
Also, you can use ENV in dockerfile to define a environment variable.
The difference is:
Environment variable define in Dockerfile will not only used in docker build, it will also persist into container. This means if you did not set -e when docker run, it will still have environment variable same as defined in Dockerfile.
While environment variable define in docker-compose.yaml just used for docker run.
Maybe next example could make you understand more clear:
Dockerfile:
FROM alpine
ENV http_proxy http://123
docker-compose.yaml:
app:
environment:
- http_proxy=http://123
If you define environment variable in Dockerfile, all containers used this image will also has the http_proxy as http://123. But the real situation maybe when you build the image, you need this proxy. But, the container maybe run by other people maybe not need this proxy or just have another http_proxy, so they had to remove the http_proxy in entrypoint or just change to another value in docker-compose.yaml.
If you define environment variable in docker-compose.yaml, then user could just choose his own http_proxy when do docker-compose up, http_proxy will not be set if user did not configure it docker-compose.yaml.
I have a docker-compose file that allows me to pass the environment variables as a file (.env file). As I have multiple ENV variables, Is there any option in Dockerfile like env_file in docker-compose for passing multiple environment variables during docker build?
This is the docker-compose.yml
services:
web:
image: "node"
links:
- "db"
env_file: "env.app"
AFAIK, there is no such way to inject environment variables using a file during the build step using Dockerfile. However, in most cases, people do end up using an entrypoint script & injecting variables during the docker run or docker-compose up.
In case it's a necessity you might need to write a shell wrapper which will change the values in the Dockerfile dynamically by taking a key-value pair text file as an input or make it something as below but the ENV file name need to be included in Dockerfile.
COPY my-env-vars /
RUN export $(cat my-env-vars | xargs)
It's an open issue - https://github.com/moby/moby/issues/28617
PS - You need to be extra careful while using this approach because the secrets are baked into the image itself.
I have writte a docker-compose.yml file to create a multi-container app with nodejs and mongodb (so 2 containers), and I want to make some options configurable, as the choice of server address and port.
To do that, I have written what follows in docker-compose.yml to set them as env variables:
..
web:
environment:
- PORT=3000
- ADDRESS=xxx.xxx.xxx.xxx
..
Within the source code of my application, I use process.env.PORT and process.env.ADDRESS to refer to these variables.
But how can I do if I want to change those values, and for example set PORT=3001?
I have to use docker-compose build and docker-compose up again and build all the application, included mongodb container?
I have to use docker-compose build and docker-compose up again and build all the application, included mongodb container?
Not build, just up. They are runtime options, not build options. Changing them in the docker-compose.yml file and then doing a docker-compose up again should recreate the containers with the new environment variables.
Alternatively, you can specify environment variables outside the docker-compose.yml file to make for easier changes:
One of these methods is to use a .env file in the same folder that you execute docker-compose form (see https://docs.docker.com/compose/env-file/ for information on it).
Another option is to interpolate environment variables from your shell. You could specify them such as - PORT=${APP_PORT} and then export APP_PORT=3000 on your shell before running docker-compose. See https://docs.docker.com/compose/environment-variables/ for more information.