I have a Dockerfile in which files in a directory are downloaded:
RUN wget https://www.classe.cornell.edu/~cesrulib/downloads/tarballs/ -r -l1 --no-parent -A tgz \
--cut=99 -nH -nv --show-progress --progress=bar:force:noscroll
I know that there is exactly one file here of the form "bmad_dist_YYYY_MMDD.tgz" where "YYYY_MMDD" is a date. For example, the file might be named "bmad_dist_2020_0707.tgz". I want to set a bash variable to the file name without the ".tgz" extension. If this was outside of docker I could use:
FULLNAME=$(ls -1 bmad_dist_*.tgz)
BMADDIST="${FULLNAME%.*}"
So I tried in the dockerfile:
ENV FULLNAME $(ls -1 bmad_dist_*.tgz)
ENV BMADDIST "${FULLNAME%.*}"
But this does not work. Is it possible to do what I want?
Shell expansion does not happen in Dockerfile ENV. Then workaround that you can try is to pass the name during Docker build.
Grab the filename during build name and discard the file or you can try --spider for wget to just get the filename.
ARG FULLNAME
ENV FULLNAME=${FULLNAME}
Then pass the full name dynamically during build time.
For example
docker build --build-args FULLNAME=$(wget -nv https://upload.wikimedia.org/wikipedia/commons/5/54/Golden_Gate_Bridge_0002.jpg 2>&1 |cut -d\" -f2) -t my_image .
The ENV ... ... syntax is mainly for plaintext content, docker build arguments, or other environment variables. It does not support a subshell like your example.
It is also not possible to use RUN export ... and have that variable defined in downstream image layers.
The best route may be to write the name to a file in the filesystem and read from that file instead of an environment variable. Or, if an environment variable is crucial, you could set an environment variable from the contents of that file in an ENTRYPOINT script.
Related
This question already has answers here:
Allowed characters in Linux environment variable names
(8 answers)
Closed 2 years ago.
Docker version: 20.10.2, build 2291f61
In a Dockerfile, is it possible to use an ENV directive for which the key contains embedded periods? For example:
ENV story.paragraph.port 2029
And if it is possible to declare ENV variables with keys that have periods, then is it later possible to reference them, using familiar shell-interpolation, in the same Dockerfile?
EXPOSE $story.paragraph.port
The latter EXPOSE directive breaks for me.
My use-case: I have a python script that loads its configuration from an INI file. Eg, I might have configuration properties like these:
story.paragraph.word=helloworld
story.paragraph.length=256
The python logic recognizes configuration settings both from the INI file (by default) or, alternatively, in overrides specified in the environment. The idea is that a container instance could specify its own environment variables for story.paragraph.word or story.paragraph.length, and that those values would override the default configuration.
Periods in env is not a valid identifier from unix prespective.
Setting env with periods inside a docker container is possible at runtime, but you can't directly access that env; you will need some workaround.
$ docker run --name test -itd -e story.paragraph.word=helloworld alpine sh
da4214ac2377cf1b3ce3af515f30e96dfadabf0140f541c06ee4a176a2bef746
$ docker exec -it test sh
/ # env | grep story
story.paragraph.word=helloworld <=== env is set
/ # echo $story.paragraph.word
.paragraph.word <=== can't access the env
/ # echo ${story.paragraph.word}
sh: syntax error: bad substitution
/ # env | grep 'story.paragraph.word' | cut -f 2 -d '=' <=== need such workaround
helloworld
/ # $
I'm creating a Dockerfile that needs to execute a command, let's call it foo
In order to execute foo, I need to create a .cfc in current directory with token information to call this foo service.
So basically I should do something like
ENV FOO_TOKEN token
ENV FOO_HOST host
ENV FOO_SHARED_DIRECTORY directory
ENV LIBS_TARGET target
and then put the first three variables in a .cfg file and then launch a command using the last variable as target.
Given that if run more than one CMD in a Dockerfile, only the last one will be considered, how should I do that?
My ideal execution is docker run -e "FOO_TOKEN=aaaaaaa" -e "FOO_HOST=myhost" -e "FOO_SHARED_DIRECTORY=Shared" -e "LIBS_TARGET=target/scala-2.11/*.jar" -it --rm --name my-ci-deploy foo/foo:latest
If you wanted to keep everything in the Dockerfile (something I think is rather desirable), you can do something nasty like:
ENV SCRIPT=IyEvdXNyL2Jpbi9lbnYgYmFzaApwZG9fc3Fsc3J2PTAKc3Vkbz0KdmVuZG9yPSQoIGxzYl9yZWxlYXNlIC1p
RUN echo -n "$SCRIPT" | base64 -d | /usr/bin/env bash
Where the contents of SCRIPT= are derived by piping your shell script thusly:
cat my_script.sh | base64 --wrap=0
You may have to adjust the /usr/bin/env bash if you have a really minimal (Alpine) setup.
I have the following line in my Dockerfile which is supposed to capture the display number of the host:
RUN DISPLAY_NUMBER="$(echo $DISPLAY | cut -d. -f1 | cut -d: -f2)" && echo $DISPLAY_NUMBER
When I tried to build the Dockerfile, the DISPLAY_NUMBER is empty. But however when I run the same command directly in the terminal I get the see the result. Is there anything that I'm doing wrong here?
Commands specified with RUN are executed when the image is built. There is no display during build hence the output is empty.
You can exchange RUN with ENTRYPOINT then the command is executed when the docker starts.
But how to forward the hosts display to the container is another matter entirely.
Host environment variables cannot be passed during build, only at run-time.
Only build args can be specified by:
first "declaring the arg"
ARG DISPLAY_NUMBER
and then running
docker build . --no-cache -t disp --build-arg DISPLAY_NUMBER=$DISPLAY_NUMBER
You can work around this issue using the envsubst trick
RUN echo $DISPLAY_NUMBER
And on the command line:
envsubst < Dockerfile | docker build . -f -
Which will rewrite the Dockerfile in memory and pass it to Docker with the environment variable changed.
Edit: Note that this solution is pretty useless though, because you probably
want to do this during run-time anyways, because this value should depend on not on where the image is built, but rather where it is run.
I would personally move that logic into your ENTRYPOINT or CMD script.
I am creating an image that will run a java application inside. Java application process creation command takes parameters from a configuration file inside of the image. I want to use environment variables to set those config file contents. I don't know how to modify those values. When I just simply copy the file it just copies the env variable name.
FROM base-image
ARG SERVICE=test
ENV SERVICE $SERVICE
COPY runtime.properties /tmp/
RUN chmod 700 /tmp/runtime.properties
# here i am creating java process using those properties
runtime.properties
# few lines
SERVICE_NAME='${SERVICE}'
# few lines
Java will read static property files literally and doesn't do any interpolation of these files before running. There are a few options available to you.
One is to add to the Dockerfile a step to search and replace the value in the file.
FROM java:alpine
ARG SERVICE=test
ENV SERVICE $SERVICE
COPY runtime.properties /tmp/
RUN sed -i -e 's/${SERVICE}/asd/g' /tmp/runtime.properties
RUN chmod 700 /tmp/runtime.properties
Another option is to change the properties file to a java class and read the environment variable directly. This gives the advantage of having the default value in the code for standalone running.
public enum LocalConfig {
INSTANCE;
private String service = System.getenv("SERVICE") ==null ? "test" : System.getenv("SERVICE");
}
Yet another option if you have lots of environment variables is to use envsubst, this will replace all of the environment variables in the file. But this depends on what your base image is. https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html
FROM java
ARG SERVICE=test
ENV SERVICE $SERVICE
COPY runtime.properties /tmp/
RUN envsubst < /tmp/runtime.properties > /tmp/runtime.properties
RUN chmod 700 /tmp/runtime.properties
The last option I can think of is interpreting the environment variables after you inport the file. There is a good thread on that here: Regarding application.properties file and environment variable
You can write the environment variable to runtime.properties by using bash:
RUN echo "SERVICE_NAME=$SERVICE" > /tmp/runtime.properties
or if static content already exists in that file, substitute > with >> to append.
I need to fill a variable in dockerfile with the result of a command
Like in bash var=$(date)
EDIT 1
date is a example.
in my case i use FROM phusion/baseimage:0.9.17 so i want at each building use the last version so i use this
curl -v --silent api.github.com/repos/phusion/baseimage-docker/tags 2>&1 | grep -oh 'rel-.*",' | head -1 | sed 's/",//' | sed 's/rel-//' ==> 0.9.17.
but i don't know how i parse it in var with dockerfile for this result
ENV verbaseimage=curl...
FROM phusion/baseimage:$verbaseimage
RESULT
In my use case
FROM phusion/baseimage:latest
But the question remains unresolved for other case
I had same issue and found way to set environment variable as result of function by using RUN command in dockerfile.
For example i need to set SECRET_KEY_BASE for Rails app just once without changing as would when i run:
docker run -e SECRET_KEY_BASE="$(openssl rand -hex 64)"
Instead it i write to Dockerfile string like:
RUN bash -l -c 'echo export SECRET_KEY_BASE="$(openssl rand -hex 64)" >> /etc/bash.bashrc'
and my env variable available from root, even after bash login.
or may be
RUN /bin/bash -l -c 'echo export SECRET_KEY_BASE="$(openssl rand -hex 64)" > /etc/profile.d/docker_init.sh'
then it variable available in CMD and ENTRYPOINT commands
Docker cache it as layer and change only if you change some strings before it.
You also can try different ways to set environment variable.
The old workaround is mentioned here (issue 2637: Feature request: expand Dockerfile ENV $VARIABLES in WORKDIR):
One work around that I've used, is to have a file in my context called "build-env". What I do is source it and run my desired command in the same RUN step. So for example:
build-env:
VERSION=stable
Dockerfile:
FROM radial/axle-base:latest
ADD build-env /build-env
RUN source build-env && mkdir /$VERSION
RUN ls /
But for date, that might not be as precise as you want.
Other workarounds are in issue 2022 "Dockerfile with variable interpolation".
In docker 1.9 (end of October 2015), you will have "support for build-time environment variables to the 'build' API (PR 9176)" and "Support for passing build-time variables in build context (PR 15182)".
docker build --build-arg=[]: Set build-time variables
You can use ENV instructions in a Dockerfile to define variable values. These values persist in the built image. However, often persistence is not what you want. Users want to specify variables differently depending on which host they build an image on.
A good example is http_proxy or source versions for pulling intermediate files. The ARG instruction lets Dockerfile authors define values that users can set at build-time using the ---build-arg flag:
$ docker build --build-arg HTTP_PROXY=http://10.20.30.2:1234 .
This flag allows you to pass the build-time variables that are accessed like regular environment variables in the RUN instruction of the Dockerfile.
Also, these values don't persist in the intermediate or final images like ENV values do.
so I want at each building use the last version so I use this
curl -v --silent api.github.com/repos/phusion/baseimage-docker/tags 2>&1 | grep -oh 'rel-.*",' | head -1 | sed 's/",//' | sed 's/rel-//' ==> 0.9.17.
If you want to use the last version of that image, all you need to do is use the tag 'latest' with the FROM directive:
FROM phusion/baseimage:latest
See also "The misunderstood Docker tag: latest": it doesn't always reference the actual latest build, but in this instance, it should work.
If you really want to use the curl|parse option, use it to generate a Dockerfile with the right value (as in a template processed to generate the right file).
Don't try to use it directly in the Dockerfile.
I wanted to set an ENV or LABEL variable from a computation in the Dockerfile, e.g. to make some computed installation options visible in docker inspect.
There does not seem to be any way to do that, and this issue suggests that it's a security design choice.
A Dockerfile can set an ENV variable to $X, ${X:-default}, or ${X:+substitute} where that $X must be another ENV or ARG variable.
A single RUN command can set and use shell variables, but that goes away at the end of the RUN command when that container layer shuts down.
A RUN command can write computed data into files, but the Dockerfile still can't get that data into an ENV or LABEL even if the file is ~/.bashrc. (File contents can, of course, be used by code running in the Container.)
The build can at least RUN echo $X to record choices to the build log -- unless that step comes from the build cache, in which case the RUN step doesn't run.
Please do correct me if there's a way out.
Partially connected to question. If one wants to use the result of some command later on it is possible within single RUN statement as follows:
RUN CUR_DIR=`pwd` && \
echo $CUR_DIR