How to make a build arg mandatory during Docker build? - docker

Is there any way to make a build argument mandatory during docker build? The expected behaviour would be for the build to fail if the argument is missing.
For example, for the following Dockerfile:
FROM ubuntu
ARG MY_VARIABLE
ENV MY_VARIABLE $MY_VARIABLE
RUN ...
I would like the build to fail at ARG MY_VARIABLE when built with docker build -t my-tag . and pass when built with docker build -t my-tag --build-arg MY_VARIABLE=my_value ..
Is there any way to achieve that behaviour? Setting a default value doesn't really do the trick in my case.
(I'm running Docker 1.11.1 on darwin/amd64.)
EDIT:
One way of doing that I can think of is to run a command that fails when MY_VARIABLE is empty, e.g.:
FROM ubuntu
ARG MY_VARIABLE
RUN test -n "$MY_VARIABLE"
ENV MY_VARIABLE $MY_VARIABLE
RUN ...
but it doesn't seem to be a very idiomatic solution to the problem at hand.

I tested with RUN test -n <ARGvariablename> what #konradstrack mentioned in the original (edit) post... that seems do the job of mandating the variable to be passed as the build time argument for the docker build command:
FROM ubuntu
ARG MY_VARIABLE
RUN test -n "$MY_VARIABLE"
ENV MY_VARIABLE $MY_VARIABLE

You can also use shell parameter expansion to achieve this.
Let's say your mandatory build argument is called MANDATORY_BUILD_ARGUMENT, and you want it to be set and non-empty, your Dockerfile could look like this:
FROM debian:stretch-slim
MAINTAINER Evel Knievel <evel#kniev.el>
ARG MANDATORY_BUILD_ARGUMENT
RUN \
# Check for mandatory build arguments
: "${MANDATORY_BUILD_ARGUMENT:?Build argument needs to be set and non-empty.}" \
# Install libraries
&& apt-get update \
&& apt-get install -y \
cowsay \
fortune \
# Cleanup
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/var/tmp/* \
/tmp/* \
CMD ["/bin/bash", "-c", "/usr/games/fortune | /usr/games/cowsay"]
Of course, you would also want to use the build-argument for something, unlike I did, but still, I recommend building this Dockerfile and taking it for a test-run :)
EDIT
As mentioned in #Jeffrey Wen's answer, to make sure that this errors out on a centos:7 image (and possibly others, I admittedly haven't tested this on other images than stretch-slim):
Ensure that you're executing the RUN command with the bash shell.
RUN ["/bin/bash", "-c", ": ${MYUID:?Build argument needs to be set and not null.}"]

Another simple way:
RUN test -n "$MY_VARIABLE" || (echo "MY_VARIABLE not set" && false)

Long time ago I had a need to introduce a required (mandatory) ARG, and for better UX include the check at the beginning:
FROM ubuntu:bionic
ARG MY_ARG
RUN [ -z "$MY_ARG" ] && echo "MY_ARG is required" && exit 1 || true
...
RUN ./use-my-arg.sh
But this busts the build cache for every single layer after the initial MY_ARG, because MY_ARG=VALUE is prepended to every RUN command afterwards.
Whenever I changed MY_ARG it would end up rebuilding the whole image, instead of rerunning the last RUN command only.
To bring caching back, I have changed my build to a multi-staged one:
The first stage uses MY_ARG and checks it's presence.
The second stage proceeds as usual and declares ARG MY_ARG right at the end.
FROM alpine:3.11.5
ARG MY_ARG
RUN [ -z "$MY_ARG" ] && echo "MY_ARG is required" && exit 1 || true
FROM ubuntu:bionic
...
ARG MY_ARG
RUN ./use-my-arg.sh
Since ARG MY_ARG in the second stage is declared right before it's used, all the previous steps in that stage are unaffected, thus cache properly.

You could do something like this...
FROM ubuntu:14.04
ONBUILD ARG MY_VARIABLE
ONBUILD RUN if [ -z "$MY_VARIABLE" ]; then echo "NOT SET - ERROR"; exit 1; else : ; fi
Then docker build -t my_variable_base .
Then build your images based on this...
FROM my_variable_base
...
It's not super clean, but at least it abstracts the 'bleh' stuff away to the base image.

I cannot comment yet because I do not have 50 reputation, but I would like to add onto #Jan Nash's solution because I had a little difficulty getting it to work with my image.
If you copy/paste #Jan Nash's solution, it will work and spit out the error message that the build argument is not specified.
What I want to add
When I tried getting it to work on a CentOS 7 image (centos:7), Docker ran the RUN command without erroring out.
Solution
Ensure that you're executing the RUN command with the bash shell.
RUN ["/bin/bash", "-c", ": ${MYUID:?Build argument needs to be set and not null.}"]
I hope that helps for future incoming people. Otherwise, I believe #Jan Nash's solution is just brilliant.

In case anybody is looking for a the solution but with docker compose build, I used mandatory variables.
version: "3.9"
services:
my-service:
build:
context: .
args:
- ENVVAR=${ENVVAR:?See build instructions}
After running docker compose build:
Before exporting ENVVAR: Invalid template: "required variable ENVVAR is missing a value: See build instructions"
After exporting ENVVAR: build proceeds
Support for Required Environment variables
Compose Environment Variables

None of these answers worked for me. I wanted ${MY_VARIABLE:?} but did not want to print anything, so I did like this:
ARG MY_VARIABLE
RUN test -n ${MY_VARIABLE:?}
Nothing is printed on success. On error you see this, which is a good enough error:
ERROR RUN test -n ${MY_VARIABLE:?}
/bin/sh: MY_VARIABLE: parameter not set or null
executor failed running [/bin/sh -c test -n ${MY_VARIABLE:?}]: >exit code: 2

Related

How to set breakpoint in Dockerfile itself?

Searching up the above shows many results about how to set breakpoints for apps running in docker containers, yet I'm interested in setting a breakpoint in the Dockerfile itself, such that the docker build is paused at the breakpoint. For an example Dockerfile:
FROM ubuntu:20.04
RUN echo "hello"
RUN echo "bye"
I'm looking for a way to set a breakpoint on the RUN echo "bye" such that when I debug this Dockerfile, the image will build non-interactively up to the RUN echo "bye" point, exclusive. After then, I would be able to interactively run commands with the container. In the actual Dockerfile I have, there are RUNs before the breakpoint that change the file system of the image being built, and I want to analyze the filesystem of the image at the breakpoint by being able to interactively run commands like cd / ls / find at the time of the breakpoint.
You can't set a breakpoint per se, but you can get an interactive shell at an arbitrary point in your build sequence (between steps).
Let's build your image:
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM ubuntu:20.04
---> 1e4467b07108
Step 2/3 : RUN echo "hello"
---> Running in 917b34190e35
hello
Removing intermediate container 917b34190e35
---> 12ebbdc1e72d
Step 3/3 : RUN echo "bye"
---> Running in c2a4a71ae444
bye
Removing intermediate container c2a4a71ae444
---> 3c52993b0185
Successfully built 3c52993b0185
Each of the lines that says ---> 0123456789ab with a hex ID has a valid image ID. So from here you can
docker run --rm -it 12ebbdc1e72d sh
which will give you an interactive shell on the partial image resulting from the first RUN command.
There's no requirement that the build as a whole succeed. If a RUN step fails, you can use this technique to get an interactive shell on the image immediately before that step and re-run the command by hand. If you have a very long RUN command, you may need to break it into two to be able to get a debugging shell at a specific point within the command sequence.
I don't think this is possible directly - that feature has been discussed and rejected.
What I generally do to debug a Dockerfile is to comment all of the steps after the "breakpoint", then run docker build followed by docker run -it image bash or docker run -it image sh (depending on whether you have bash installed inside the container).
Then, I have an interactive shell, and I can run commands to debug why later stages are failing.
I agree that being able to set a breakpoint and poke around would be a handy feature, though.
You can run commands in intermediate containers using Remote shell debugging tricks.
Make sure your container images include basic utilities like netcat (nc) and fuser. These utilities enable "calling home" from any intermediate container image. At home you'll answer calls with netcat (or socat). This netcat will send your commands to containers, and print their outcomes. This debugging approach will work even on Dockerfiles that are built on unknown worker nodes somewhere in cloud.
Example:
FROM debian:testing-slim
# Set environment variables for calling home from breakpoints (BP)
ENV BP_HOME=<IP-ADDRESS-OF-YOUR-HOST>
ENV BP_PORT=33720
ENV BP_CALLHOME='BP_FIFO=/tmp/$BP.$BP_HOME.$BP_PORT; (rm -f $BP_FIFO; mkfifo $BP_FIFO) && (echo "\"c\" continues"; echo -n "($BP) "; tail -f $BP_FIFO) | nc $BP_HOME $BP_PORT | while read cmd; do if test "$cmd" = "c" ; then echo -n "" >$BP_FIFO; sleep 0.1; fuser -k $BP_FIFO >/dev/null 2>&1; break; else eval $cmd >$BP_FIFO 2>&1; echo -n "($BP) " >$BP_FIFO; fi; done'
# Install needed utils (netcat, fuser)
RUN apt update && apt install -y netcat psmisc
# Now you are ready to run "eval $BP_CALLHOME" wherever you want to call home.
RUN BP=before-hello eval $BP_CALLHOME
RUN echo "hello"
RUN BP=after-hello eval $BP_CALLHOME
RUN echo "bye"
Start waiting for and answering calls from a Dockerfile before launching a Docker build. On home host run nc -k -l -p 33720 (alternatively socat STDIN TCP-LISTEN:33720,reuseaddr,fork).
This is how above example looks like at home:
$ nc -k -l -p 33720
"c" continues
(before-hello) echo *
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
(before-hello) id
uid=0(root) gid=0(root) groups=0(root)
(before-hello) c
"c" continues
(after-hello)
...
The recent (May 2022) project ktock/buildg offers breakpoints.
See "Interactive debugger for Dockerfile" from Kohei Tokunaga
buildg is a tool to interactively debug Dockerfile based on BuildKit.
Source-level inspection
Breakpoints and step execution
Interactive shell on a step with your own debugigng tools
Based on BuildKit (needs unmerged patches)
Supports rootless
The command break, b LINE_NUMBER sets a breakpoint.
Example:
$ buildg.sh debug --image=ubuntu:22.04 /tmp/ctx
WARN[2022-05-09T01:40:21Z] using host network as the default
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 195B done
#2 DONE 0.1s
#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 3.0s
#4 [build1 1/2] FROM docker.io/library/busybox#sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8
#4 resolve docker.io/library/busybox#sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8 0.0s done
#4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 772.81kB / 772.81kB 0.2s done
Filename: "Dockerfile"
2| RUN echo hello > /hello
3|
4| FROM busybox AS build2
=> 5| RUN echo hi > /hi
6|
7| FROM scratch
8| COPY --from=build1 /hello /
>>> break 2
>>> breakpoints
[0]: line 2
>>> continue
#4 extracting sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0.0s done
#4 DONE 0.3s
...
From PR 24:
Add --cache-reuse option which allows sharing the build cache among invocation of buildg debug to make the 2nd-time debugging faster.
This is useful to speed up running buildg multiple times for debugging an errored step.
Note that breakpoints on cached steps are ignored as of now.
Because of this limitation, this feature is optional as of now. We should fix this limitation and make it the default behaviour in the future.
Man, Docker makes things hard. Here's a workaround I cooked up:
Insert FROM scratch where you want the break point.
Run docker build . --stage=<n-1> where <n> is the number of FROM commands before your "breakpoint". Eg, if it's a single stage build, use --stage=0.
Alternatively, if you have already named the stage where you want the break point with FROM <image> AS <stage> then you can use --stage=<stage> instead.
Docker has cached all your successful layers anyway (even if you can't see them), and because the FROM "breakpoint" comes before the (potentially unsuccessful) point of interest, the build should all come from cache and be very fast.
So for example, if my Dockerfile looks like this:
FROM debian:bullseye AS build
RUN apt-get update && apt-get install -y \
build-essential cmake ninja-build \
libfontconfig1-dev libdbus-1-dev libfreetype6-dev libicu-dev libinput-dev libxkbcommon-dev libsqlite3-dev libssl-dev libpng-dev libjpeg-dev libglib2.0-dev
<SNIP lots of other setup commands>
ADD my_source.tar.xz /
WORKDIR /my_source
RUN ./configure -option1 -option2
RUN cmake --build . --parallel
RUN cmake --install .
FROM alpine
COPY --from=build /my_build /my_build
...
Then I can add a "breakpoint" like this:
FROM debian:bullseye AS build
RUN apt-get update && apt-get install -y \
build-essential cmake ninja-build \
libfontconfig1-dev libdbus-1-dev libfreetype6-dev libicu-dev libinput-dev libxkbcommon-dev libsqlite3-dev libssl-dev libpng-dev libjpeg-dev libglib2.0-dev
<SNIP lots of other setup commands>
ADD my_source.tar.xz /
WORKDIR /my_source
#### BREAKPOINT ###
FROM scratch
#### BREAKPOINT ###
RUN ./configure -option1 -option2
RUN cmake --build . --parallel
RUN cmake --install .
FROM alpine
COPY --from=build /my_build /my_build
...
and trigger it with docker build . --stage=build

Replacing character in dockerfile variable - error: missing ':' in substitution

In my docker file I have a variable defined for a version number with dots which I want to replace with underscores for a further usage.
ARG ABC_VERSION=1.2.3
ARG SOME_OTHER_VARIABLE=/dir_name/abc_${ABC_VERSION//./_}
Unfortunately, I get the following error.
failed to process "/dir_name/abc_${ABC_VERSION//./_}": missing ':' in substitution
I need the version number several times within the dockerfile multiple times with the '.' and one time with the '_' and I do not like to define two variables.
Does someone know how to solve this problem?
Edit:
One portion of the actual code where I would like to make use of the feature to replace characters is looking like this.
ARG EXPAT_VERSION=2.1.0
# ...
RUN wget https://github.com/libexpat/libexpat/releases/download/R_${EXPAT_VERSION//./_}/expat-${EXPAT_VERSION}.tar.gz \
&& tar xzf expat-${EXPAT_VERSION}.tar.gz \
&& cp -R expat-${EXPAT_VERSION}/lib ./xmp_sdk/third-party/expat \
&& rm -r expat-${EXPAT_VERSION} && rm expat-${EXPAT_VERSION}.tar.gz
I saw that something similiar is used in the tensorflow-gpu dockerfiles:
ARG CUDA=10.1
#...
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cuda-command-line-tools-${CUDA/./-} #...
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dockerfiles/dockerfiles/devel-gpu.Dockerfile
The ${parameter/pattern/string} syntax is actually a Bash feature (cf. shell parameter expansion), not a POSIX feature.
According to the official documentation, the Dockerfile directives only supports:
$var
${var}
${var:-word} → if var is not set then word is the result;
${var:+word} → if var is set then word is the result, otherwise the empty string
Workaround 1
So the problem does not have a "direct" solution, but if the variable you would like to substitute will be used, in the end, in some shell command (in a RUN, ENTRYPOINT or CMD directive), you could just as well keep the initial value as is (with no substitution), then substitute it later on?
I mean for example, the following Dockerfile:
FROM debian
ARG ABC_VERSION=1.2.3
ENV SOME_OTHER_VARIABLE=/app/${ABC_VERSION}
WORKDIR /app
RUN /bin/bash -c 'touch "${SOME_OTHER_VARIABLE//./_}"'
# RUN touch "${SOME_OTHER_VARIABLE//./_}"
# would raise /bin/sh: 1: Bad substitution
CMD ["/bin/bash", "-c", "ls -hal \"${SOME_OTHER_VARIABLE//./_}\""]
As an aside:
I replaced ARG SOME_OTHER_VARIABLE with ENV SOME_OTHER_VARIABLE just to be able to use it from CMD.
It can be recalled that ENTRYPOINT and CMD directives should rather be written in exec form − CMD ["…", "…"] − rather in shell form (see e.g. that question: CMD doesn't run after ENTRYPOINT in Dockerfile).
Workaround 2
Or as an alternative workaround, you may want to split your version number in major, minor, patch, to write something like this?
ARG MAJOR=1
ARG MINOR=2
ARG PATCH=3
ARG ABC_VERSION=$MAJOR.$MINOR.$PATCH
ARG SOME_OTHER_VARIABLE=/dir_name/abc_${MAJOR}_${MINOR}_${PATCH}
…
A more concise syntax for workaround 1
Following the OP's edit, I guess one concern is the relative verbosity of this line that I mentioned in the "workaround 1":
…
RUN /bin/bash -c 'touch "${SOME_OTHER_VARIABLE//./_}"'
To alleviate this, Docker allows one to replace the implied shell (by default sh) with Bash, which does support the shell parameter expansion you are interested in. The key point is the following directive that has to be written before the RUN command (and which was precisely part of the Dockerfile the OP mentioned):
SHELL ["/bin/bash", "-c"]
Thus, the Dockerfile becomes:
…
ARG ABC_VERSION=1.2.3
SHELL ["/bin/bash", "-c"]
RUN touch "/dir_name/abc_${ABC_VERSION//./_}" \
&& ls -hal "/dir_name/abc_${ABC_VERSION//./_}"
or taking advantage of some temporary environment variable:
…
ARG ABC_VERSION=1.2.3
SHELL ["/bin/bash", "-c"]
RUN export SOME_OTHER_VARIABLE="/dir_name/abc_${ABC_VERSION//./_}" \
&& touch "$SOME_OTHER_VARIABLE" \
&& ls -hal "$SOME_OTHER_VARIABLE"

docker build --build-arg SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)" returning empty

I want to be able to read the contents of the file ~/.ssh/id_rsa and pass the same to my build stage of the image. When I use the command docker build --build-arg SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)" and then I try to echo that inside the container during a build, I get empty.
RUN echo "$SSH_PRIVATE_KEY" > /priv_key \
&& cat /priv_key
the result is
Step 6/14 : RUN echo "$SSH_PRIVATE_KEY" > /priv_key && cat /priv_key
---> Running in c8d6e3c88cd8
Removing intermediate container c8d6e3c88cd8
In the dockerfile I have ARG SSH_PRIVATE_KEY.
But when I use a dummy text like docker build --build-arg SSH_PRIVATE_KEY="dummy text" I can see it in the logs.
This causes my private key to be in invalid format since it is empty.
RUN echo "${SSH_PRIVATE_KEY}" >> /root/.ssh/id_rsa
What am I doing wrong or what is it that am not doing? Thank you
I went ahead and used ONVAULT toool to handle the ssh keys. https://github.com/dockito/vault.
Also, I had misconfigured my .ssh/config file. The new file looks like this
Host *
IgnoreUnknown AddKeysToAgent,UseKeychain
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_rsa
I hope it helps someone in future.
I could solve this by placing the ARG after defining the base image:
FROM ubuntu:18.04 as builder
ARG SSH_PRV_KEY
instead of
ARG SSH_PRV_KEY
FROM ubuntu:18.04 as builder

Default Docker Argument to UUID

Is it possible to default an ARG to a new unique ID if one has not been provided as a Build Argument?
Failing that is it possible to fail on build if a build argument is not provided?
Failing that is it possible to fail on build if a build argument is
not provided?
Yes, but then you should set any default value to ARGS in Dockerfile, or if set default value then you can ignore the default value in condition.
FROM alpine
ARG NODE_ENV
RUN if [ -z ${NODE_ENV} ];then \
echo "No value provided for build ARGS, please provide NODE_ENV, like --build-arg NODE_ENV=dev" \
exit 1; \
else \
echo "NODE_ENV is passed, value is ${NODE_ENV}"; \
fi
So if you tried like
docker build --build-arg NODE_ENV=test -t test_arg .
It will build the image, but if you tired like
docker build -t test_arg .
In this, the build will fail and will print a message to pass NODE_ENV

Is there a way to use If condition in Dockefile?

I am new to docker, I am looking for a way to execute a command in docker container depends on the environment.
In Dockerfile, I have 2 commands, command_a and command_b. If the env = 'prod' run command_a, else command_b. How can I achieve this?
I tried like below:
RUN if [ $env = "prod" ] ; then echo command_a; else echo cpmmand_b; fi;
How can I achieve the desired behaviour?
PS:
I know that echo should not be there.
Docker 17.05 and later supports a kind of conditionals using multi-stage build and build args. Have a look at https://medium.com/#tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae
From the blog post:
ARG BUILD_VERSION=1
FROM alpine AS base
RUN ...
FROM base AS branch-version-1
RUN touch version1
FROM base AS branch-version-2
RUN touch version2
FROM branch-version-${BUILD_VERSION} AS after-condition
FROM after-condition
RUN ...
And then use docker build --build-arg BUILD_VERSION=value ...

Resources