Replacing character in dockerfile variable - error: missing ':' in substitution - docker

In my docker file I have a variable defined for a version number with dots which I want to replace with underscores for a further usage.
ARG ABC_VERSION=1.2.3
ARG SOME_OTHER_VARIABLE=/dir_name/abc_${ABC_VERSION//./_}
Unfortunately, I get the following error.
failed to process "/dir_name/abc_${ABC_VERSION//./_}": missing ':' in substitution
I need the version number several times within the dockerfile multiple times with the '.' and one time with the '_' and I do not like to define two variables.
Does someone know how to solve this problem?
Edit:
One portion of the actual code where I would like to make use of the feature to replace characters is looking like this.
ARG EXPAT_VERSION=2.1.0
# ...
RUN wget https://github.com/libexpat/libexpat/releases/download/R_${EXPAT_VERSION//./_}/expat-${EXPAT_VERSION}.tar.gz \
&& tar xzf expat-${EXPAT_VERSION}.tar.gz \
&& cp -R expat-${EXPAT_VERSION}/lib ./xmp_sdk/third-party/expat \
&& rm -r expat-${EXPAT_VERSION} && rm expat-${EXPAT_VERSION}.tar.gz
I saw that something similiar is used in the tensorflow-gpu dockerfiles:
ARG CUDA=10.1
#...
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cuda-command-line-tools-${CUDA/./-} #...
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dockerfiles/dockerfiles/devel-gpu.Dockerfile

The ${parameter/pattern/string} syntax is actually a Bash feature (cf. shell parameter expansion), not a POSIX feature.
According to the official documentation, the Dockerfile directives only supports:
$var
${var}
${var:-word} → if var is not set then word is the result;
${var:+word} → if var is set then word is the result, otherwise the empty string
Workaround 1
So the problem does not have a "direct" solution, but if the variable you would like to substitute will be used, in the end, in some shell command (in a RUN, ENTRYPOINT or CMD directive), you could just as well keep the initial value as is (with no substitution), then substitute it later on?
I mean for example, the following Dockerfile:
FROM debian
ARG ABC_VERSION=1.2.3
ENV SOME_OTHER_VARIABLE=/app/${ABC_VERSION}
WORKDIR /app
RUN /bin/bash -c 'touch "${SOME_OTHER_VARIABLE//./_}"'
# RUN touch "${SOME_OTHER_VARIABLE//./_}"
# would raise /bin/sh: 1: Bad substitution
CMD ["/bin/bash", "-c", "ls -hal \"${SOME_OTHER_VARIABLE//./_}\""]
As an aside:
I replaced ARG SOME_OTHER_VARIABLE with ENV SOME_OTHER_VARIABLE just to be able to use it from CMD.
It can be recalled that ENTRYPOINT and CMD directives should rather be written in exec form − CMD ["…", "…"] − rather in shell form (see e.g. that question: CMD doesn't run after ENTRYPOINT in Dockerfile).
Workaround 2
Or as an alternative workaround, you may want to split your version number in major, minor, patch, to write something like this?
ARG MAJOR=1
ARG MINOR=2
ARG PATCH=3
ARG ABC_VERSION=$MAJOR.$MINOR.$PATCH
ARG SOME_OTHER_VARIABLE=/dir_name/abc_${MAJOR}_${MINOR}_${PATCH}
…
A more concise syntax for workaround 1
Following the OP's edit, I guess one concern is the relative verbosity of this line that I mentioned in the "workaround 1":
…
RUN /bin/bash -c 'touch "${SOME_OTHER_VARIABLE//./_}"'
To alleviate this, Docker allows one to replace the implied shell (by default sh) with Bash, which does support the shell parameter expansion you are interested in. The key point is the following directive that has to be written before the RUN command (and which was precisely part of the Dockerfile the OP mentioned):
SHELL ["/bin/bash", "-c"]
Thus, the Dockerfile becomes:
…
ARG ABC_VERSION=1.2.3
SHELL ["/bin/bash", "-c"]
RUN touch "/dir_name/abc_${ABC_VERSION//./_}" \
&& ls -hal "/dir_name/abc_${ABC_VERSION//./_}"
or taking advantage of some temporary environment variable:
…
ARG ABC_VERSION=1.2.3
SHELL ["/bin/bash", "-c"]
RUN export SOME_OTHER_VARIABLE="/dir_name/abc_${ABC_VERSION//./_}" \
&& touch "$SOME_OTHER_VARIABLE" \
&& ls -hal "$SOME_OTHER_VARIABLE"

Related

Difference between RUN bash -c and open form of RUN

Some Dockerfiles have
RUN bash -c "apt-get update -qq && ... \"
while others write without quotes like
RUN apt-get update -qq && ... \
What is the difference between these variants? Is one of them more preferable over another one?
You should just write RUN apt-get update ... without manually inserting a sh -c wrapper.
The RUN, CMD, and ENTRYPOINT directives all share the same syntax. It's best documented for ENTRYPOINT but all three commands work the same way. There are two ways to write commands for them: you can either provide a specific set of command words as a JSON array (exec form), or you can write a string and Docker will automatically wrap it sh -c (shell form). For example:
# Create a directory with a space in its name
RUN mkdir "a directory"
# JSON-array form: each array element is a shell word
RUN ["ls", "-ld", "a directory"]
# String form: Docker provides a shell, so these two are equivalent
RUN ls -ld 'a directory'
RUN ["/bin/sh", "-c", "ls -ld 'a directory'"]
This makes your first form redundant: if you RUN bash -c '...', it's a string, and Docker automatically wraps it in sh -c. So you get in effect
# RUN bash -c '...'
RUN ["/bin/sh", "-c", "bash -c '...'"]
GNU bash has a number of extensions that are not POSIX-standard syntax and it's possible to run into trouble with these, particularly on an Alpine-based image where /bin/sh is a minimal shell from the BusyBox toolset. I could see this as an attempt to force a shell command to run using bash rather than the default shell. For most things that appear in Dockerfiles, they won't usually be so complex that they can't be easily rewritten in standard syntax.
# Needs bash for the non-standard `source` syntax
RUN bash -c 'source ./venv/bin/activate && pip list'
# But you can use the standard `.` instead
RUN . ./venv/bin/activate && pip list
If you must have bash interpreting RUN lines then I'd suggest using the SHELL directive to change the command that's used to interpret bare strings.
Style-wise, I also occasionally see JSON-array syntax that begins with an explicit CMD ["/bin/sh", "-c", "..."]. There's no reason to write this out; it's shorter and no less clear to use the string form.

Issue Running an Initialization Script with Nginx Docker Container

I am creating a Nginx docker image that I'll be using as a reverse proxy component in ECS/Fargate in AWS. I'm using the official Nginx image as the base image (1.17.5).
When the container starts I'm trying to run a bash script from an ENTRYPOINT to go out to the AWS Parameter Store and retrieve certificate info. This work fine, however when I try to add a parameter to pass to the bash script (e.g. ENTRYPOINT ["installcerts.sh", "AppName"] it executes the script but the container terminates without error.
I want the container to continue on to start up Nginx after the parameterized batch script.
Here is my Docker File:
FROM nginx:1.17.5
# Install AWS CLI/BOTO3, JQ
RUN apt-get update && apt-get install -y && apt-get install awscli -y && apt-get install jq -y
# Copy Nginx config to etc/nginx
COPY proxy_ssl.conf /etc/nginx/conf.d/
VOLUME ["/etc/nginx/conf/d"]
# Copy entrypoint bash script to install certs from the AWS Parameter Store
COPY installcerts.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/installcerts.sh
#Pull certs from Parameter Store
ENTRYPOINT ["/usr/local/bin/installcerts.sh", "AppName"]
CMD ["nginx", "-g", "daemon off;"]
And here is my "installcerts.sh" script showing utilizing the parameter passed in from ENTRYPOINT.
#!/usr/bin/env bash
-e
echo Installing certs...
aws ssm get-parameters --name /Certificate/$1/CRT | jq '.Parameters[0].Value' -r > /etc/nginx/conf.d/app.crt
aws ssm get-parameters --name /Certificate/$1/KEY | jq '.Parameters[0].Value' -r > /etc/nginx/conf.d/app.key
echo Exiting script.
exec "$#"
The "exec "$#" in the bash script is needed, but honestly, I don't completely understand how or why this works even after hours of trying to track it down.
The short story is:
If I use this, the container does what i want it to do but I can't send a parameter to the bash script.
ENTRYPOINT ["/usr/local/bin/installcerts.sh"]
But If I use this, the script will run WITH the parameter successfully, but the container exits and Nginx doesn't start up.
ENTRYPOINT ["/usr/local/bin/installcerts.sh", "AppName"]
What am I doing wrong?
When you set an ENTRYPOINT on your image, then docker passes that script the value of CMD (or whatever you pass on the command line after the image name). For example, if you have:
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/myprogram"]
Then docker runs, effectively:
/entrypoint.sh /usr/bin/myprogram
That is, Docker itself never runs /usr/bin/myprogram: it is entirely up to the ENTRYPOINT script to do that. That is what the exec "$#" is for. This is a shell variable the evaluates to:
Expands to the positional parameters, starting from one.
(bash(1) man page, in the "Special Parameters" section)
In our example, this would evaluate to:
exec /usr/bin/myprogram
...which replaces the current script with /usr/bin/myprogram. But if we were to set ENTRYPOINT as you have in your question:
ENTRYPOINT ["/entrypoint.sh", "appName"]
Then exec "$#" will in fact evaluate to:
exec appName /usr/bin/myprogram
And since appName isn't a valid command, the container will simply fail.
There are a few ways of dealing with this:
Do you really need to pass parameters to your ENTRYPOINT script? What about using environment variables instead?
If you always pass parameters to your script, you can use the shift shell command to drop those from the positional parameters before using $#. For example, for a script that expects two parameters:
param1=$1
param2=$2
shift 2
...do stuff here...
exec "$#"
...but this only works if you always pass two parameters.
You can implement command line option processing in your script using e.g. the getopts command:
while getopts a:b: ch; do
case $ch in
(a) param1=$OPTARG
;;
(b) param2=$OPTARG
;;
esac
done
shift $(( $OPTIND - 1 ))
...do stuff here...
exec "$#"
Having said that: I would opt for option 1 (use environment variables) as being the simplest solution:
docker run -e PARAM1="some value" ...
And then in your ENTRYPOINT script you can just use the $PARAM1 variable where you need it.

ARG or ENV, which one to use in this case?

This could be maybe a trivial question but reading docs for ARG and ENV doesn't put things clear to me.
I am building a PHP-FPM container and I want to give the ability for enable/disable some extensions on user needs.
Would be great if this could be done in the Dockerfile by adding conditionals and passing flags on the build command perhaps but AFAIK is not supported.
In my case and my personal approach is to run a small script when container starts, something like the following:
#!/bin/sh
set -e
RESTART="false"
# This script will be placed in /config/init/ and run when container starts.
if [ "$INSTALL_XDEBUG" == "true" ]; then
printf "\nInstalling Xdebug ...\n"
yum install -y php71-php-pecl-xdebug
RESTART="true"
fi
...
if [ "$RESTART" == "true" ]; then
printf "\nRestarting php-fpm ...\n"
supervisorctl restart php-fpm
fi
exec "$#"
This is how my Dockerfile looks like:
FROM reynierpm/centos7-supervisor
ENV TERM=xterm \
PATH="/root/.composer/vendor/bin:${PATH}" \
INSTALL_COMPOSER="false" \
COMPOSER_ALLOW_SUPERUSER=1 \
COMPOSER_ALLOW_XDEBUG=1 \
COMPOSER_DISABLE_XDEBUG_WARN=1 \
COMPOSER_HOME="/root/.composer" \
COMPOSER_CACHE_DIR="/root/.composer/cache" \
SYMFONY_INSTALLER="false" \
SYMFONY_PROJECT="false" \
INSTALL_XDEBUG="false" \
INSTALL_MONGO="false" \
INSTALL_REDIS="false" \
INSTALL_HTTP_REQUEST="false" \
INSTALL_UPLOAD_PROGRESS="false" \
INSTALL_XATTR="false"
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
https://rpms.remirepo.net/enterprise/remi-release-7.rpm
RUN yum install -y \
yum-utils \
git \
zip \
unzip \
nano \
wget \
php71-php-fpm \
php71-php-cli \
php71-php-common \
php71-php-gd \
php71-php-intl \
php71-php-json \
php71-php-mbstring \
php71-php-mcrypt \
php71-php-mysqlnd \
php71-php-pdo \
php71-php-pear \
php71-php-xml \
php71-pecl-apcu \
php71-php-pecl-apfd \
php71-php-pecl-memcache \
php71-php-pecl-memcached \
php71-php-pecl-zip && \
yum clean all && rm -rf /tmp/yum*
RUN ln -sfF /opt/remi/php71/enable /etc/profile.d/php71-paths.sh && \
ln -sfF /opt/remi/php71/root/usr/bin/{pear,pecl,phar,php,php-cgi,phpize} /usr/local/bin/. && \
mv -f /etc/opt/remi/php71/php.ini /etc/php.ini && \
ln -s /etc/php.ini /etc/opt/remi/php71/php.ini && \
rm -rf /etc/php.d && \
mv /etc/opt/remi/php71/php.d /etc/. && \
ln -s /etc/php.d /etc/opt/remi/php71/php.d
COPY container-files /
RUN chmod +x /config/bootstrap.sh
WORKDIR /data/www
EXPOSE 9001
Currently this is working but ... If I want to add let's say 20 (a random number) of extensions or any other feature that can be enable|disable then I will end with 20 non necessary ENV (because Dockerfile doesn't support .env files) definition whose only purpose would be set this flag for let the script knows what to do then ...
Is this the right way to do it?
Should I use ENV for this purpose?
I am open to ideas if you have a different approach for achieve this please let me know about it
From Dockerfile reference:
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.
The ENV instruction sets the environment variable <key> to the value <value>.
The environment variables set using ENV will persist when a container is run from the resulting image.
So if you need build-time customization, ARG is your best choice.
If you need run-time customization (to run the same image with different settings), ENV is well-suited.
If I want to add let's say 20 (a random number) of extensions or any other feature that can be enable|disable
Given the number of combinations involved, using ENV to set those features at runtime is best here.
But you can combine both by:
building an image with a specific ARG
using that ARG as an ENV
That is, with a Dockerfile including:
ARG var
ENV var=${var}
You can then either build an image with a specific var value at build-time (docker build --build-arg var=xxx), or run a container with a specific runtime value (docker run -e var=yyy)
So if want to set the value of an environment variable to something different for every build then we can pass these values during build time and we don't need to change our docker file every time.
While ENV, once set cannot be overwritten through command line values. So, if we want to have our environment variable to have different values for different builds then we could use ARG and set default values in our docker file. And when we want to overwrite these values then we can do so using --build-args at every build without changing our docker file.
For more details, you can refer this.
Why to use ARG or ENV ?
Let's say we have a jar file and we want to make a docker image of it. So, we can ship it to any docker engine.
We can write a Dockerfile.
Dockerfile
FROM eclipse-temurin:17-jdk-alpine
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
Now, if we want to build the docker image using Maven, we can pass the JAR_FILE using the --build-arg as target/*.jar
docker build --build-arg JAR_FILE=target/*.jar -t myorg/myapp
However, if we are using Gradle; the above command doesn't work and we've to pass a different path: build/libs/
docker build --build-arg JAR_FILE=build/libs/*.jar -t myorg/myapp .
Once you have chosen a build system, we don’t need the ARG. We can hard code the JAR location.
For Maven, that would be as follows:
Dockerfile
FROM eclipse-temurin:17-jdk-alpine
VOLUME /tmp
COPY target/*.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
here, we can build an image with the following command:
docker build -t image:tag .
When to use `ENV`?
If we want to set some values at running containers and reflect that to the image like the Port Number that your application can run/listen on. We can set that using the ENV.
Both ARG and ENV seem very similar. Both can be accessed from within our Dockerfile commands in the same manner.
Example:
ARG VAR_A 5
ENV VAR_B 6
RUN echo $VAR_A
RUN echo $VAR_B
Personal Option!
There is a tradeoff between choosing ARG over ENV. If you choose ARG you can't change it later during the run. However, if you chose ENV you can modify the value at the container.
I personally prefer ARG over ENV wherever I can, like,
In the above Example:
I have used ARG as the build system maven or Gradle impacts during build rather than runtime. It thus encapsulates a lot of details and provided a minimum set of arguments for the runtime.
For more details, you can refer to this.

How to export an environment variable to a docker image?

I can define "static" environment variables in a Dockerfile with ENV, but is it possible to pass some value at build time to this variable? I'm attempting something like this, which doesn't work:
FROM phusion/baseimage
RUN mkdir -p /foo/2016/bin && \
FOOPATH=`ls -d /foo/20*/bin` && \
export FOOPATH
ENV PATH $PATH:$FOOPATH
Of course, in the real use case I'd be running/unpacking something that creates a directory whose name will change with different versions, dates, etc., and I'd like to avoid modifying the Dockerfile every time the directory name changes.
Edit: Since it appears it's not possible, the best workaround so far is using a symlink:
FROM phusion/baseimage
RUN mkdir -p /foo/2016/bin && \
FOOPATH=`ls -d /foo/20*/bin` && \
ln -s $FOOPATH /mypath
ENV PATH $PATH:/mypath
To pass a value in at build time, use an ARG.
FROM phusion/baseimage
RUN mkdir -p /foo/2016/bin && \
FOOPATH=`ls -d /foo/20*/bin` && \
export FOOPATH
ARG FOOPATH
ENV PATH $PATH:${FOOPATH}
Then you can run docker build --build-arg FOOPATH=/dir -t myimage .
Edit: from you comment, my answer above won't solve your issue. There's nothing in the Dockerfile you can update from the output of the run command, the output isn't parsed, only the resulting filesystem is saved. For this, I think you're best off in your run command writing the path to the image and read in from your /etc/profile or a custom entrypoint script. That depends on how you want to launch your container and the base image.

How to make a build arg mandatory during Docker build?

Is there any way to make a build argument mandatory during docker build? The expected behaviour would be for the build to fail if the argument is missing.
For example, for the following Dockerfile:
FROM ubuntu
ARG MY_VARIABLE
ENV MY_VARIABLE $MY_VARIABLE
RUN ...
I would like the build to fail at ARG MY_VARIABLE when built with docker build -t my-tag . and pass when built with docker build -t my-tag --build-arg MY_VARIABLE=my_value ..
Is there any way to achieve that behaviour? Setting a default value doesn't really do the trick in my case.
(I'm running Docker 1.11.1 on darwin/amd64.)
EDIT:
One way of doing that I can think of is to run a command that fails when MY_VARIABLE is empty, e.g.:
FROM ubuntu
ARG MY_VARIABLE
RUN test -n "$MY_VARIABLE"
ENV MY_VARIABLE $MY_VARIABLE
RUN ...
but it doesn't seem to be a very idiomatic solution to the problem at hand.
I tested with RUN test -n <ARGvariablename> what #konradstrack mentioned in the original (edit) post... that seems do the job of mandating the variable to be passed as the build time argument for the docker build command:
FROM ubuntu
ARG MY_VARIABLE
RUN test -n "$MY_VARIABLE"
ENV MY_VARIABLE $MY_VARIABLE
You can also use shell parameter expansion to achieve this.
Let's say your mandatory build argument is called MANDATORY_BUILD_ARGUMENT, and you want it to be set and non-empty, your Dockerfile could look like this:
FROM debian:stretch-slim
MAINTAINER Evel Knievel <evel#kniev.el>
ARG MANDATORY_BUILD_ARGUMENT
RUN \
# Check for mandatory build arguments
: "${MANDATORY_BUILD_ARGUMENT:?Build argument needs to be set and non-empty.}" \
# Install libraries
&& apt-get update \
&& apt-get install -y \
cowsay \
fortune \
# Cleanup
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/var/tmp/* \
/tmp/* \
CMD ["/bin/bash", "-c", "/usr/games/fortune | /usr/games/cowsay"]
Of course, you would also want to use the build-argument for something, unlike I did, but still, I recommend building this Dockerfile and taking it for a test-run :)
EDIT
As mentioned in #Jeffrey Wen's answer, to make sure that this errors out on a centos:7 image (and possibly others, I admittedly haven't tested this on other images than stretch-slim):
Ensure that you're executing the RUN command with the bash shell.
RUN ["/bin/bash", "-c", ": ${MYUID:?Build argument needs to be set and not null.}"]
Another simple way:
RUN test -n "$MY_VARIABLE" || (echo "MY_VARIABLE not set" && false)
Long time ago I had a need to introduce a required (mandatory) ARG, and for better UX include the check at the beginning:
FROM ubuntu:bionic
ARG MY_ARG
RUN [ -z "$MY_ARG" ] && echo "MY_ARG is required" && exit 1 || true
...
RUN ./use-my-arg.sh
But this busts the build cache for every single layer after the initial MY_ARG, because MY_ARG=VALUE is prepended to every RUN command afterwards.
Whenever I changed MY_ARG it would end up rebuilding the whole image, instead of rerunning the last RUN command only.
To bring caching back, I have changed my build to a multi-staged one:
The first stage uses MY_ARG and checks it's presence.
The second stage proceeds as usual and declares ARG MY_ARG right at the end.
FROM alpine:3.11.5
ARG MY_ARG
RUN [ -z "$MY_ARG" ] && echo "MY_ARG is required" && exit 1 || true
FROM ubuntu:bionic
...
ARG MY_ARG
RUN ./use-my-arg.sh
Since ARG MY_ARG in the second stage is declared right before it's used, all the previous steps in that stage are unaffected, thus cache properly.
You could do something like this...
FROM ubuntu:14.04
ONBUILD ARG MY_VARIABLE
ONBUILD RUN if [ -z "$MY_VARIABLE" ]; then echo "NOT SET - ERROR"; exit 1; else : ; fi
Then docker build -t my_variable_base .
Then build your images based on this...
FROM my_variable_base
...
It's not super clean, but at least it abstracts the 'bleh' stuff away to the base image.
I cannot comment yet because I do not have 50 reputation, but I would like to add onto #Jan Nash's solution because I had a little difficulty getting it to work with my image.
If you copy/paste #Jan Nash's solution, it will work and spit out the error message that the build argument is not specified.
What I want to add
When I tried getting it to work on a CentOS 7 image (centos:7), Docker ran the RUN command without erroring out.
Solution
Ensure that you're executing the RUN command with the bash shell.
RUN ["/bin/bash", "-c", ": ${MYUID:?Build argument needs to be set and not null.}"]
I hope that helps for future incoming people. Otherwise, I believe #Jan Nash's solution is just brilliant.
In case anybody is looking for a the solution but with docker compose build, I used mandatory variables.
version: "3.9"
services:
my-service:
build:
context: .
args:
- ENVVAR=${ENVVAR:?See build instructions}
After running docker compose build:
Before exporting ENVVAR: Invalid template: "required variable ENVVAR is missing a value: See build instructions"
After exporting ENVVAR: build proceeds
Support for Required Environment variables
Compose Environment Variables
None of these answers worked for me. I wanted ${MY_VARIABLE:?} but did not want to print anything, so I did like this:
ARG MY_VARIABLE
RUN test -n ${MY_VARIABLE:?}
Nothing is printed on success. On error you see this, which is a good enough error:
ERROR RUN test -n ${MY_VARIABLE:?}
/bin/sh: MY_VARIABLE: parameter not set or null
executor failed running [/bin/sh -c test -n ${MY_VARIABLE:?}]: >exit code: 2

Resources