I have a Dockerfile that requires two environment variables:
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
I'm passing them from the host through my compose file using:
build:
# ...
args:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
My problem is that in some flows there is an IAM role set (and no env vars) and I don't want to use the environment variables. But even when they don't exist on the host they seem to be set to empty strings during the build process.
I've tried this:
run if [ -z "$AWS_ACCESS_KEY_ID" ]; then unset AWS_ACCESS_KEY_ID; fi
run if [ -z "$AWS_SECRET_ACCESS_KEY" ]; then unset AWS_SECRET_ACCESS_KEY; fi
run env # see if set
But it doesn't work (the variables are still set even if not set in host env).
I'd welcome another solution on mixing env vars and IAM roles when building dockers.
Different run in Dockerfile not impact each other, to make your aims, suggest to combine them to one run, something likes follows, FYI:
run if [ -z "$AWS_ACCESS_KEY_ID" ]; then unset AWS_ACCESS_KEY_ID; fi && \
if [ -z "$AWS_SECRET_ACCESS_KEY" ]; then unset AWS_SECRET_ACCESS_KEY; fi && \
env
Then, you will find no AWS_ACCESS_KEY_ID was set to empty value in env.
And, when try, suggest use docker-compose build --no-cache to test.
Finally, why you see empty value?
I made a experiment, seems if no ENV set in Dockerfile, meanwhile, no env for this variable set in HOST, the ARG in Dockerfile will automatically be changed to one ENV variable when docker build, as the ARG did not set a value, so it's empty.
Related
I have a Dockerfile in which I set some environment variables (for use when I run the container). Some of the environment variables depend on previous environment variables. I want to be DRY and avoid having to hard-code the value of environment variables multiple times, when I could substitute in a variable.
In this simple example, the PYTHONPATH environment variable uses values from the PROJ_DIR environment variable.
FROM python:3.8.4
ENV PROJ_DIR=/myproj/ \
PYTHONPATH=${PROJ_DIR}:${PYTHONPATH}
However, when I actually run the container, the PROJ_DIR correctly gets set, but the dependent variable, PYTHONPATH, does not get set.
docker build -f Dockerfile . -t test-docker
docker run --rm -it test-docker:latest bash
root#60fc899899a1:/# export | grep -i proj
declare -x PROJ_DIR="/myproj/"
root#60fc899899a1:/# export | grep -i pythonpath
declare -x PYTHONPATH=":"
How do I use previously set environment variables in a Dockerfile?
For this to work you would need to spearate out the variables and use multiline assignment.
ENV PROJ_DIR /myproj/
ENV PYTHONPATH ${PROJ_DIR}:${PYTHONPATH}
Throughout the entire instruction, environment variable substitution will use the same value for each variable. In your case PROJ_DIR is yet to be assigned a value, so it returns empty in PYTHONPATH varaible.
To be more clear, in:
ENV x=hello
ENV x=world z=$x
z will have value hello and not world.
Due to multiline there will be not be any additonal layers getting created as ENV layers do get squashed.
Hope that helps.
You can reuse environment variables that were previously set outside but not inside an ENV, since all statements in the same ENV pick up the value defined before the ENV is processed. So you should split it into separate ENVs.
ENV PROJ_DIR=/myproj/
ENV PYTHONPATH=${PROJ_DIR}:${PYTHONPATH}
I need to set the output of a RUN command to be an environment variable that is available inside my container after it's built:
...
RUN ...
GO111MODULE=on go get github.com/emersion/hydroxide/cmd/hydroxide && \
echo "the_password" | hydroxide auth the_username#protonmail.com > bridge_password.txt && \
BRIDGE_PASSWORD=`sed -n 's/Password: Bridge password: //p' bridge_password.txt` && \
hydroxide smtp
...
Above, I need $BRIDGE_PASSWORD to be available in my NodeJS project (process.env.BRIDGE_PASSWORD). How can I achieve this?
You need to:
Declare your env var with the ENV command in your Dockerfile.
Substitute your current RUN with a command in your image entrypoint so that the variable gets set correctly everytime your start a new container and is available to any launched command.
I need to set some environment variable for all users and processes inside docker container. It should be set at container start, not in Dockerfile, because it depends on running environment.
So the simple Dockerfile
FROM ubuntu
RUN echo 'export TEST=test' >> '/root/.bashrc'
works well for interactive sessions
docker run -ti test bash
then
env
and there is TEST=test
but when docker run -ti test env there is no TEST
I was trying
RUN echo 'export TEST=test' >> '/etc/environment'
RUN echo 'TEST="test"' >> '/etc/environment'
RUN echo 'export TEST=test' >> /etc/profile.d/1.sh
ENTRYPOINT export TEST=test
Nothing helps.
Why I need this. I have http_proxy variable inside container automatically set by docker, I need to set another variables, based on it, i.e. JAVA_OPT, do it system wide, for all users and processes, and in running environment, not at build time.
I would create a script which would be an entrypoint:
#!/bin/bash
# if env variable is not set, set it
if [ -z $VAR ];
then
# env variable is not set
export VAR=$(a command that gives the var value);
fi
# pass the arguments received by the entrypoint.sh
# to /bin/bash with command (-c) option
/bin/bash -c $#
And in Dockerfile I would set the entrypoint:
ENTRYPOINT entrypoint.sh
Now every time I run docker run -it <image> <any command> it uses my script as entrypoint so will always run it before the command then pass the arguments to the right place which is /bin/bash.
Improvements
The above script is enough to work if you are always using the entrypoint with arguments, otherwise your $# variable will be empty and will give you an error /bin/bash: -c: option requires an argument. A easy fix is an if statement:
if [ ! -z $# ];
then
/bin/bash -c $#;
fi
Setting the parameter in ENTRYPOINT would solve this issue.
In docker file pass parameter in ENTRYPOINT
For some certain reasons, I have to set "http_proxy" and "https_proxy" ENV in my dockerfile. I would like to now unset them because there are also some building process can't be done through the proxy.
# dockerfile
# ... some process
ENV http_proxy=http://...
ENV https_proxy=http://...
# ... some process that needs the proxy to finish
UNSET ENV http_proxy # how to I unset the proxy ENV here?
UNSET ENV https_proxy
# ... some process that can't use the proxy
It depends on what effect you are trying to achieve.
Note that, as a matter of pragmatics (i.e. how developers actually speak), "unsetting a variable" can mean two things: removing it from the environment, or setting the variable to an empty value. Technically, these are two different operations. In practice though I have not run into a case where the software I'm trying to control differentiates between the variable being absent from the environment, and the variable being present in the environment but set to an empty value. I generally can use either method to get the same result.
If you don't care whether the variable is in the layers produced by Docker, but leaving it with a non-empty value causes problems in later build steps.
For this case, you can use ENV VAR_NAME= at the point in your Dockerfile from which you want to unset the variable. Syntactic note: Docker allows two syntaxes for ENV: this ENV VAR=1 is the same as ENV VAR 1. You can separate the variable name from the value with a space or an equal sign. When you want to "unset" a variable by setting it to an empty value you must use the equal sign syntax or you get an error at build time.
So for instance, you could do this:
ENV NOT_SENSITIVE some_value
RUN something
ENV NOT_SENSITIVE=
RUN something_else
When something runs, NOT_SENSITIVE is set to some_value. When something_else runs, NOT_SENSITIVE is set to the empty string.
It is important to note that doing unset NOT_SENSITIVE as a shell command will not affect anything else than what executes in this shell. Here's an example:
ENV NOT_SENSITIVE some_value
RUN unset NOT_SENSITIVE && printenv NOT_SENSITIVE || echo "does not exist"
RUN printenv NOT_SENSITIVE
The first RUN will print does not exist because NOT_SENSITIVE is unset when printenv executes and because it is unset printenv returns a non-zero exit code which causes the echo to execute. The second RUN is not affected by the unset in the first RUN. It will print some_value to the screen.
But what if I need to remove the variable from the environment, not just set it to an empty value?
In this case using ENV VAR_NAME= won't work. I don't know of any way to tell Docker "from this point on, you must remove this variable from the environment, not just set it to an empty value".
If you still want to use ENV to set your variable, then you'll have to start each RUN in which you want the variable to be unset with unset VAR_NAME, which will unset it for that specific RUN only.
If you want to prevent the variable from being present in the layers produced by Docker.
Suppose that variable contains a secret and the layer could fall into the hands of people who should not have the secret. In this case you CANNOT use ENV to set the variable. A variable set with ENV is baked into the layers to which it applies and cannot be removed from those layers. In particular, (assuming the variable is named SENSITIVE) running
RUN unset SENSITIVE
does not do anything to remove it from the layer. The unset command above only removes SENSITIVE from the shell process that RUN starts. It affects only that shell. It won't affect shells spawned by CMD, ENTRYPOINT, or any command provided through running docker run at the command line.
In order to prevent the layers from containing the secret, I would use docker build --secret= and RUN --mount=type=secret.... For instance, assuming that I've stored my secret in a file named sensitive, I could have a RUN like this:
RUN --mount=type=secret,id=sensitive,target=/root/sensitive \
export SENSITIVE=$(cat /root/sensitive) \
&& [[... do stuff that requires SENSITIVE ]] \
Note that the command given to RUN does not need to end with unset SENSITIVE. Due to the way processes and their environments are managed, setting SENSITIVE in the shell spawned by RUN does not have any effect beyond what that shell itself spawns. Environment changes in this shell won't affect future shells nor will it affect what Docker bakes into the layers it creates.
Then the build can be run with:
$ DOCKER_BUILDKIT=1 docker build --secret id=secret,src=path/to/sensitive [...]
The environment for the docker build command needs DOCKER_BUILDKIT=1 to use BuildKit because this method of passing secrets is only available if Docker uses BuildKit to build the images.
If one needs env vars during the image build but they should not persist, just clear them. In the following example, the running container shows empty env vars.
Dockerfile
# set proxy
ARG http_proxy
ARG https_proxy
ARG no_proxy
ENV http_proxy=$http_proxy
ENV https_proxy=$http_proxy
ENV no_proxy=$no_proxy
# ... do stuff that needs the proxy during the build, like apt-get, curl, et al.
# unset proxy
ENV http_proxy=
ENV https_proxy=
ENV no_proxy=
build.sh
docker build -t the-image \
--build-arg http_proxy="$http_proxy" \
--build-arg https_proxy="$http_proxy" \
--build-arg no_proxy="$no_proxy" \
--no-cache \
.
run.sh
docker run --rm -i \
the-image \
sh << COMMANDS
env
COMMANDS
Output
no_proxy=
https_proxy=
http_proxy=
...
According to docker docs you need to use shell command instead:
FROM alpine
RUN export ADMIN_USER="mark" \
&& echo $ADMIN_USER > ./mark \
&& unset ADMIN_USER
CMD sh
See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#env for more details.
Short-answer:
Try to avoid unnecessary environment variables, so you don't need to unset them.
In case you have to unset for a command you can do the following:
RUN unset http_proxy https_proxy no_proxy \
&& execute_your_command_here
In case you have to unset for the built image you can do the following:
FROM ubuntu_with_http_proxy
ENV http_proxy= \
https_proxy= \
no_proxy=
Once environment variables are set using the ENV instruction we can't really unset them as it is detailed:
Each ENV line creates a new intermediate layer, just like RUN commands. This means that even if you unset the environment variable in a future layer, it still persists in this layer and its value can be dumped.
See: Best practices for writing Dockerfiles
Details:
I prefer to define http_proxy as an argument during build like the following:
FROM ubuntu:20.04
ARG http_proxy=http://host.docker.internal:3128
ARG https_proxy=http://host.docker.internal:3128
ARG no_proxy=.your.domain,localhost,127.0.0.1,.docker.internal
On corporate proxy we need authentication anyways, so we need to configure local proxy server listening on 127.0.0.1:3128 witch is accessible over host.docker.internal:3128 from containers. This way it also works on docker desktop if we connect to corporate network over VPN (with local/home network blocked).
Setting no_proxy is also important to avoid flooding the proxy server.
See the following article for more details on no_proxy related topics:
Can we standardize NO_PROXY?
Sometimes it is also good to read the related documentation:
ENV
ARG
In case we need to configure those environment variables we can use the following command:
during build (link):
docker build ... --build-arg http_proxy='http://alternative.proxy:3128/' ...
during runs (link):
docker run ... -env http_proxy='http://alternative.proxy:3128/' ...
Also note that we don't even need to define proxy related arguments since those are already predefine according to the following section:
Dockerfile reference - Predefined ARGs
You can add below lines in the Dockerfile
ENV http_proxy ""
ENV https_proxy ""
I found the secret approach didn't work because I needed the env variable to persist in the container when I ran it in interactive mode but then needed to completely remove the variable for a later stage build for production.
What worked was in building for the development phase I appended the environment variable to the /root/.basrc file as
RUN echo export AWS_PROFILE=role-name >> /root/.bashrc
``
In the production stage of the build I then removed the last line of /root/.bashrc:
RUN sed -i '$ d' /root/.bashrc
I have a situation where I need to set an ENV based on runtime condition like thus:
RUN if [ "$RUNTIME" = "prod" ] then VARIABLE="Some Production URL"; else VARIABLE="Some QA URL"; fi;
ENV={VARIABLE}
Been looking at different solutions but none of them seem to be panning out (for example the basic one where VARIABLE is lost when RUN exits). What would be an elegant way to achieve this?
It is an unfortunate constraint that you only have this "dev/qa/prod" environment variable. However, it is possible to achieve what you want.
First, you might consider baking your environment specific configuration into the image for all environments. (Normally I would discourage to do this!)
For example you can COPY three files into your image:
dev-env.sh: contains your dev config in the form:
ELASTICSEARCH_URL=http://elastic-dev:123
qa-env.sh (similar)
prod-env.sh (similar)
Then you evaluate at run-time (not at build-time) in which environment you are: You add an ENTRYPOINT script to your image which will source the correct file, depending on the ENVIRONMENT_NAME variable.
Dockerfile (part):
ENTRYPOINT ["docker-entrypoint.sh"]
docker-entrypoint.sh (copied into WORKDIR of the image):
#!/bin/bash
set -e
if [ "$ENVIRONMENT_NAME" = "prod" ]; then
source prod-env.sh
fi
# else if qa ... , else if dev ..., else fail
exec "$#"
This script will run when you launch the docker container, so this approach is no option for you if you need the variables to be available in Dockerfile-instructions (at build-time).
Another (build-time) workaround is described here and consists of using temporary files to store environment variables across multiple image layers.
The literal conditional execution can be achieved with multistage build and ONBUILD.
ARG mode=prod
FROM alpine as img_prod
ONBUILD ENV ELASTICSEARH_URL=whatever/for/prod
FROM alpine as img_qa
ONBUILD ENV ELASTICSEARH_URL=whatever/for/qa
FROM img_${mode}
...
Then you build with docker build --build-arg mode=qa .
Wouldn't passing env var with docker run be the solution you need? Something like this:
docker run -e YOUR_VARIABLE="Some Production URL" ...