Objective
I have an env variable script file that looks like:
#!/bin/sh
export FOO="public"
export BAR="private"
I would like to source the env variables to be available when a docker image is being built. I am aware that I can use ARG and ENV with build args, but I have too many Env Variables, and I am afraid that will be a lengthy list.
It's worth mentioning that I only need the env variables to install a specific step in my docker file (will highlight in the Dockerfile below), and do not necessarily want them to be available in the built image after that.
What I have tried so far
I have tried having a script (envs.sh) that export env vars like:
#!/bin/sh
export DOG="woof"
export CAT="meow"
My Docker file looks like:
FROM fishtownanalytics/dbt:0.18.1
# Define working directory
# Load ENV Vars
COPY envs.sh envs.sh
CMD ["sh", "envs.sh"]
# Install packages required
CMD ["sh", "-c", "envs.sh"]
RUN dbt deps # I need to env variables to be available for this step
# Exposing DBT Port
EXPOSE 8081
But that did not seem to work. How can I export env variables as a script to the docker file?
In the general case, you can't set environment variables in a RUN command: each RUN command runs a new shell in a new container, and any environment variables you set there will get lost at the end of that RUN step.
However, you say you only need the variables at one specific step in your Dockerfile. In that special case, you can run the setup script and the actual command in the same RUN step:
FROM fishtownanalytics/dbt:0.18.1
COPY envs.sh envs.sh
RUN . ./envs.sh \
&& dbt deps
# Anything that envs.sh `export`ed is lost _after_ the RUN step
(CMD is irrelevant here: it only provides the default command that gets run when you launch a container from the built image, and doesn't have any effect on RUN steps. It also looks like the image declares an ENTRYPOINT so that you can only run dbt subcommands as CMD, not normal shell commands. I also use the standard . to read in a script file instead of source, since not every container has a shell that provides that non-standard extension.)
Your CMD call runs a new shell (sh) that defines those variables and then dies, leaving the current process unchanged. If you want those environment variables to apply to the current process, you could source it:
CMD ["source", "envs.sh"]
Related
I have this Dockerfile
FROM node:14.17.1
ARG GITHUB_TOKEN
ARG REACT_APP_BASE_URL
ARG DATABASE_URL
ARG BASE_URL
RUN mkdir /usr/src/app
WORKDIR /usr/src/app
ENV PATH /usr/src/app/node_modules/.bin:$PATH
ENV GITHUB_TOKEN=${GITHUB_TOKEN}
ENV REACT_APP_BASE_URL=${REACT_APP_BASE_URL}
ENV DATABASE_URL=${DATABASE_URL}
ENV BASE_URL=${BASE_URL}
ENV PORT 80
COPY . /usr/src/app
RUN npm install
RUN npm run build
EXPOSE 80
CMD ["npm", "start"]
But I don't like having to set each environment variable. Is is possible to make all of them available without needing to set one by one?
We need to pay attention to next two items before continue:
As mentioned by #Lukman in comments, TOKEN is not a good item to be stored in image unless you totally for internal use, you decide.
Even we did not specify environment one by one in Dockerfile, we still need to define them in some other place, as program itself can't know what environment you really need.
If you no problem with above, let's go on. Basically, I think define the environment (Here, use ENV1, ENV2 as example) in a script, then source them in container, and let app have ways to access these variables is what you needed.
env.sh:
export ENV1=1
export ENV2=2
app.js:
#!/usr/bin/env node
var env1 = process.env.ENV1;
var env2 = process.env.ENV2;
console.log(env1);
console.log(env2);
entrypoint.sh:
#!/bin/bash
source /usr/src/app/env.sh
exec node /usr/src/app/app.js
Dockerfile:
FROM node:14.17.1
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN chmod -R 755 /usr/src/app
CMD ["/usr/src/app/entrypoint.sh"]
Execution:
$ docker build -t abc:1 .
$ docker run --rm abc:1
1
2
Explain:
We change CMD or ENTRYPOINT in Dockerfile to use customized entrypoint.sh, in this entrypoint.sh, we will first source env.sh which make ENV1 and ENV2 visible to subprocess of entrypoint.sh.
Then, we use exec to replace current process as node app.js, so PID1 becomes node app.js now, meanwhile app.js still could get the environment defined in env.sh.
With above, we no need to define variables in Dockerfile one by one, but still our app could get the environment.
Here's a different (easy) way.
Start by making your file. Here I'm choosing to use everything on my this is messy and not recommended. It's a useful bit of code though so I thought I'd add it.
env | sed 's/^/export /' > env.sh
edit it so you only have what you need
vi env.sh
Use the below to import files into the container. Change pwd to whichever folder you want to share. Using this carelessly may result in you sharing to many files*
sudo docker run -it -v `pwd`:`pwd` ubuntu
Assign appropriate file permissions. I'm using 777 which means anyone can read, write, execute - for demonstration purposes. But you only need execute privileges.
Run this command and make sure you add the full stop.
. /LOCATION/env.sh
If you're confused to where your file is just type pwd in the host console.
You can just add those commands where appropriate to your Dockerfile to automate the process. If I recall there is a VOLUME flag for Dockerfile.
I want to know if what I'm doing is considered best practice, or is there a better way:
What I want
I want to have a Docker image, which will have some environment variables predefined when run as a container. I want to do that by running some shell script that will export those variables.
What I'm doing
My dockerfile looks like this:
Dockerfile code..
..
..
RUN useradd -m develop
RUN echo ". /env.sh" >> /home/develop/.bashrc
USER develop
Is that a good way?
Using the Dockerfile ENV directive would be much more usable than putting environment variables into a file.
ENV SOME_VARIABLE=some-value
# Do not use .bashrc at all
# Do not `RUN .` or `RUN source`
Most ways to use Docker don't involve running shell dotfiles like .bashrc. Adding settings there won't usually have any effect. In a Dockerfile, any environment variable settings in a RUN instruction will get lost at the end of that line, including files you read in using the shell . command (or the equivalent but non-standard source).
For example, given the Dockerfile you show, a docker run invocation like this never invokes a shell at all and never reads the .bashrc file:
docker run the-image env \
| grep A_VARIABLE_FROM_THE_BASHRC
There are some workarounds to this (my answer to How to source a script with environment variables in a docker build process? describes a startup-time entrypoint wrapper) but the two best ways are to (a) restructure your application to need fewer environment variables and have sensible defaults if they're not set, and (b) use ENV instead of a file of environment-variable settings.
** Check edits
Yes will probably work, if in the right order.
# Add user
RUN useradd -m develop
#switch to user
USER develop
#run script as user.
#RUN echo "./env.sh" >> /home/develop/.bashrc
RUN /bin/bash -c "source ./env.sh"
Although the duplicated RUN useradd is not necessary at all
I have a dockerfile that has a entrypoint.sh file which exports some Postgres variable.
Then I want to start the parent docker container which is referenced in "FROM pactfoundation/pact-broker" image. Looking at github for it's Dockerfile github pact broker it has CMD ["config.ru"] at the end. So I did similar to that in my Dockerfile:
FROM pactfoundation/pact-broker
COPY entrypoint.sh .
CMD ["config.ru"]
When I execute my docker run command:
docker run --rm -e POSTGRES_PORT=5433 -e POSTGRES_DBNAME=pactsd -e POSTGRES_URL=localhost -e POSTGRES_PASSWORD=1234 -e POSTGRES_USERNAME=postgres --name pact sonamsamdupkhangsar/pact:test -d
I see my entrypoint.sh echo statement and the container is dead.
setting pact broker database variables
How do I start the parent container after setting my envrionment variables in my entrypoint.sh file?
I also tried with the following:
FROM pactfoundation/pact-broker
ENV PACT_BROKER_DATABASE_NAME=${POSTGRES_DBNAME}
ENV PACT_BROKER_DATABASE_USERNAME=${POSTGRES_USERNAME}
ENV PACT_BROKER_DATABASE_PASSWORD=${POSTGRES_PASSWORD}
ENV PACT_BROKER_DATABASE_HOST=${POSTGRES_URL}
ENV PACT_BROKER_DATABASE_NAME=${POSTGRES_DBNAME}
ENV PACT_BROKER_DATABASE_PORT=$POSTGRES_PORT
RUN echo "PACT_BROKER_DATABASE_PORT: $PACT_BROKER_DATABASE_PORT"
Yet, when I run my built docker image I still don't see the variables being set. I tried both approaches for "${}" and "$" for env var setting.
You've to set your environment variables using the ENV in your docker file.
As each step executed at different containers which altogether builds the image if you set via shell scripts it won't work. Consider using the ENV command to set it
Ref: DOCKERFILE ENV
What is happening is that those environment variables that you are passing in at run-time with '-e' parameter are not yet defined at build-time as the ENV instructions are executed at build-time only.
E.g. at build-time this line you have:
ENV PACT_BROKER_DATABASE_NAME=${POSTGRES_DBNAME}
becomes this line:
ENV PACT_BROKER_DATABASE_NAME=
as '${POSTGRES_DBNAME}' evaluates to empty at build-time. Then, when run-time happens, you are defining all your POSTGRES_ environment variables as parameters so they will indeed exist in the container, BUT no further instructions will be executed to set the PACT_BROKER_ environment variables to any other values.
Proposed solution: I would recommend the simplest approach if you can make it work to just use the environment variables 'directly' however you define them as parameters. I.e. either change the names of your '-e' parameters to PACT_BROKER_'s or use the POSTGRES_ environment variables in your container. Either way you would remove the ENV lines from the Dockerfile.
If you really-really need to set the environment variables to other names at run-time, then you should be able to do this by writing to the appropriate 'startup' file in the Dockerfile (making sure to literally write the '$'s to the file so they could be dereferenced at run-time).
The ENV instruction sets the environment variable to the value . This value will be in the environment for all subsequent instructions in the build stage
Below instruction:
ENV PATH=$PATH:$HOME/go/bin
does not append PATH variable
$HOME/go/bin is /root/go/bin
How to append $HOME/go/bin to $PATH? in below docker file
FROM golang:1.14.10
MAINTAINER xyz
ENV GOPATH=
ENV PATH=$PATH:$HOME/go/bin
RUN echo $PATH
Apparently Docker doesn't let you use environment variables defined outside of your Dockerfile within an ENV or ARG declaration.
As a workaround, you can pass the names/directories to your Dockerfile explicitly using ARG:
FROM golang:1.14.10
# set default to `root`
ARG USERNAME=root
ENV PATH=$PATH:/$USERNAME/go/bin
RUN echo $PATH
You can then pass the USERNAME via docker build --build-arg USERNAME=myuser
Depending on your usecase you can also do this using a RUN or ENTRYPOINT.
I think the confusion is this: when you say $HOME in ENV, home isn't defined yet. But when you say RUN echo $HOME, home is defined by the shell in the base image.
PATH is working and causing confusion, because it's defined by the base image you're using with FROM.
ENV is used to define default variables for the image that will be built, and that will be accessible in RUN statements. Think of it this way: the Dockerfile can provide variables to the container, but the container cannot provide variables to the Dockerfile.
Really, I would just hardcode in /root if root is the user you want to run from. The variables provided by the build are meant to be defaults if you want to do something fancy and dynamic, you probably are better off injecting a script into your image, and running that.
As per mentioned here in this link
I tried using this format and it worked with me:
ENV gradle=/opt/gradle/gradle-6.6.1/bin
ENV PATH=${gradle}:${PATH}
Say I have a Dockerfile:
.
.
RUN echo 'source /root/script.sh' >> /etc/bash.bashrc
(The script adds some env variables)
If I:
1) Do this:
docker run -it -v /home/user/script.sh:/root/script.sh image
It takes me to shell where if I call "env" I see the variable set by the script
But if I:
2) Do this:
docker run -it -v /home/user/script.sh:/root/script.sh image env
It prints out env and exits and my variable is missing
What am I missing? I need the variable to exists even if I specify a command/script like "env" at the end of the docker run command
When you run a command like
docker run ... image command
Docker directly runs the command you give; it doesn’t launch any kind of shell, and there’s no opportunity for a .bashrc or similar file to be read.
I’d suggest two things here:
If your program does need environment variables set in some form, set them directly using Dockerfile ENV directives. Don’t try to edit .bashrc or /etc/profile or any other shell dotfile; they won’t reliably get run.
As much as you can install things in places so that you don’t need to change environment variables. For instance, Python supports a “virtual environment” concept that allows an isolated library environment, which requires changing $PATH and similar things; but Docker provides the same isolation on its own, so just install things into the “global” package space.
If you really can’t manage either of these things, then you can write an entrypoint script that sets environment variables and then launches the container’s command. This might look like
#!/bin/sh
. /root/script.sh
exec "$#"
And then you could include this in your Dockerfile like
...
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/app/myapp"]
(If you need to use docker exec to get a debugging shell in the container, that won’t be a child process of the entrypoint and won’t get its environment variables.)