How to run sed command and save the result to one new Variable in docker.
The sed will replace the last occurrence of '.' and replace with '_'
Example :
JOB_NAME_WITH_VERSION = test_git_0.1 and wanted result is ZIP_FILE_NAME = test_git_0_1
--Dockerfile
RUN ZIP_FILE_NAME=$(echo ${JOB_NAME_WITH_VERSION} | sed 's/\(.*\)\./\1_/') && export ZIP_FILE_NAME
RUN echo "Zip file Name found : $ZIP_FILE_NAME"
I tried this in my docker file but the result is empty
Zip file Name found :
The issue here is that every RUN command results in a new layer, so whatever shell variable was declared in previous layers is subsequently lost.
Compare this:
FROM ubuntu
RUN JOB="FOOBAR"
RUN echo "${JOB}"
$ docker build .
...
Step 3/3 : RUN echo "${JOB}"
---> Running in c4b7d1632c7e
...
to this:
FROM ubuntu
RUN JOB="FOOBAR" && echo "${JOB}"
$ docker build .
...
Step 2/2 : RUN JOB="FOOBAR" && echo "${JOB}"
---> Running in c11049d1687f
FOOBAR
...
so as a workaround, if using a single RUN command is not an option for whatever reason, write the variable to disk and read it when needed, e.g.:
FROM ubuntu
RUN JOB="FOOBAR" && echo "${JOB}" > /tmp/job_var
RUN cat /tmp/job_var
$ docker build .
...
Step 3/3 : RUN cat /tmp/job_var
---> Running in a346c30c2cd5
FOOBAR
...
Each RUN statement in a Dockerfile is run in a separate shell. So once a statement is done, all environment variables are lost. Even if they are exported.
To do what you want to do, you can combine your RUN statements like this
RUN ZIP_FILE_NAME=$(echo ${JOB_NAME_WITH_VERSION} | sed 's/\(.*\)\./\1_/') && \
export ZIP_FILE_NAME && \
echo "Zip file Name found : $ZIP_FILE_NAME"
As your variable is lost once the RUN statement is finished, your environment variable won't be available in your container when it runs. To have an environment variable available there, you need to use the ENV statement.
This is a MWE of my Dockerfile: (I'm hand-composing, not using docker-compose)
# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2019-amd64
SHELL ["cmd", "/S", "/C"]
RUN set zz=foo && echo %zz%
When I build the container, I expect this to print foo, but instead I get %zz% which seems to indicate that echo doesn't see the variable zz set.
Step 20/24 : RUN set zz=foo && echo %zz%
---> Running in b1f462e81c5c
%zz%
Removing intermediate container b1f462e81c5c
---> 9c0abf6eb928
If I run this on the command line, it works as expected:
C:\> cmd /S /C set zz=foo && echo %zz%
foo
How can I make the RUN instruction work like the command prompt?
My actual use case, in case this is an XY problem: I'm reading the contents of a user-provided file into a variable and then using that in a command which sets some global config file.
RUN `
set /p pat=<your_pat.txt && `
git config --global url."https://%pat%#github.com/".insteadOf "https://github.com/"
I would like to pass argument (from the docker command) to the shell script inside the Dockerfile.
This is the docker command line.
docker build --file=DockerfileTest --build-arg docker_image=PX-release-migration --tag=test-image:latest --rm=true .
This is a script that is called inside the Dockerfile.
#!/bin/sh -e
image_name=$1
echo "docker image is $image_name"
if [[ ($image_name == '') || ($image_name == *"-dev-"*) ]]; then
echo "This is development"
cp src/main/resources/application-dev.properties src/main/resources/application.properties
elif [[ $image_name == *"-preprod-"* ]]; then
echo "This is preprod"
cp src/main/resources/application-stg.properties src/main/resources/application.properties
elif [[ $image_name == *"-release-"* ]]; then
echo "This is production"
cp src/main/resources/application-prod.properties src/main/resources/application.properties
fi
When I execute separately the script, it works, but it doe
This is docker file.
ARG spring_env=local
ARG docker_image=-local-
FROM maven:3.6.1-jdk-8
COPY . /apps/demo
WORKDIR /apps/demo
RUN chmod +x /apps/demo/initialize_env.sh
RUN ./initialize_env.sh $docker_image
RUN echo "spring_env is ${spring_env}"
So basically, i would like to use a different spring application properties file during the build depending on the docker_image name. If a docker image name contains 'release', i would like to package application-prod.properties during the build.
This is the error message that I am getting.
Step 1/8 : ARG spring_env=local
Step 2/8 : ARG docker_image=-local-
Step 3/8 : FROM maven:3.6.1-jdk-8
---> 4c81be38db66
Step 4/8 : COPY . /apps/demo
---> 41439197c465
Step 5/8 : WORKDIR /apps/demo
---> Running in 56bd408c2eb1
Removing intermediate container 56bd408c2eb1
---> 4c4025bf5f64
Step 6/8 : RUN chmod +x /apps/demo/initialize_env.sh
---> Running in 18dc3a5c1a54
Removing intermediate container 18dc3a5c1a54
---> 60d2037a0209
Step 7/8 : RUN ./initialize_env.sh $docker_image
---> Running in 2e049b2cf630
docker image is
./initialize_env.sh: 5: ./initialize_env.sh: Syntax error: word unexpected (expecting ")")
The command '/bin/sh -c ./initialize_env.sh $docker_image' returned a non-zero code: 2
When I execute separately the script, it works, but it doesn't inside the docker container.
Tip: Use ShellCheck to check scripts for syntax errors.
#!/bin/sh -e
if [[ ($image_name == '') || ($image_name == *"-dev-"*) ]]; then
[[ is bash syntax but your script is declared to use plain sh. It works on your machine presumably because sh is really symlinked to bash, but inside the container that's not the case. maven:3.6.1-jdk-8 is based on debian:stretch which uses dash instead of bash.
Change the shebang line. You can also delete the parentheses; they're superfluous.
#!/bin/bash -e
if [[ $image_name == '' || $image_name == *"-dev-"* ]]; then
You could also use a case block to simplify the repetitive checks.
case "$image_name" in
''|*-dev-*)
echo "This is development"
cp src/main/resources/application-dev.properties src/main/resources/application.properties
;;
*-preprod-*)
echo "This is preprod"
cp src/main/resources/application-stg.properties src/main/resources/application.properties
;;
*-release-*)
echo "This is production"
cp src/main/resources/application-prod.properties src/main/resources/application.properties
;;
esac
I have a base image which exports a number of variables in it's ENTRYPOINT shell script docker-entry.sh.
Now I'm consuming the base image and running CMD with a mongo-entry.sh. I was hoping to access the variables exported from the ENTRYPOINT in the mongo-entry.sh.
I'm able to see the the log statements with the variables from docker-entry.sh being printed out, but, I'm unable to access it in mongo-entry.sh
I've already tried calling the docker-entry.sh and mongo-entry.sh as ENTRYPOINT in my DockerFile
Code:
Base Image ENTRYPOINT docker-entry.sh
export var1 = "var1"
Current image
FROM 'baseimage'
COPY ["mongo-entry.sh"]
CMD ["mongo-entry.sh"]
mongo-entry.sh
echo "var1=$VAR1"
Ouptut
var1=
I suspect there are 1-2 issues:
ENTRYPOINT and CMD aren't inherited from the FROM image
You may be using the exec form of ENTRYPOINT|CMD
There are 2 forms of ENTRYPOINT and CMD, the shell and exec forms. The exec form "does not invoke a command shell" and so you won't get environment variable processing:
https://docs.docker.com/engine/reference/builder/#cmd
Here's an example which shows ENV values passing through FROM'd images:
echo '
FROM bash
ENV DOG=Freddie
ENTRYPOINT echo "dog=${DOG}"
' | docker build --tag=1st --file=- .
echo '
FROM 1st
ENV CAT=Emmett
ENTRYPOINT echo "dog=${DOG}" "cat=${CAT}"
' | docker build --tag=2nd --file=- .
docker run 1st
docker run 2nd
Results in:
dog=Freddie
dog=Freddie cat=Emmett
NB ${DOG} (and its value) is inherited from 1st
Is there any way to make a build argument mandatory during docker build? The expected behaviour would be for the build to fail if the argument is missing.
For example, for the following Dockerfile:
FROM ubuntu
ARG MY_VARIABLE
ENV MY_VARIABLE $MY_VARIABLE
RUN ...
I would like the build to fail at ARG MY_VARIABLE when built with docker build -t my-tag . and pass when built with docker build -t my-tag --build-arg MY_VARIABLE=my_value ..
Is there any way to achieve that behaviour? Setting a default value doesn't really do the trick in my case.
(I'm running Docker 1.11.1 on darwin/amd64.)
EDIT:
One way of doing that I can think of is to run a command that fails when MY_VARIABLE is empty, e.g.:
FROM ubuntu
ARG MY_VARIABLE
RUN test -n "$MY_VARIABLE"
ENV MY_VARIABLE $MY_VARIABLE
RUN ...
but it doesn't seem to be a very idiomatic solution to the problem at hand.
I tested with RUN test -n <ARGvariablename> what #konradstrack mentioned in the original (edit) post... that seems do the job of mandating the variable to be passed as the build time argument for the docker build command:
FROM ubuntu
ARG MY_VARIABLE
RUN test -n "$MY_VARIABLE"
ENV MY_VARIABLE $MY_VARIABLE
You can also use shell parameter expansion to achieve this.
Let's say your mandatory build argument is called MANDATORY_BUILD_ARGUMENT, and you want it to be set and non-empty, your Dockerfile could look like this:
FROM debian:stretch-slim
MAINTAINER Evel Knievel <evel#kniev.el>
ARG MANDATORY_BUILD_ARGUMENT
RUN \
# Check for mandatory build arguments
: "${MANDATORY_BUILD_ARGUMENT:?Build argument needs to be set and non-empty.}" \
# Install libraries
&& apt-get update \
&& apt-get install -y \
cowsay \
fortune \
# Cleanup
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/var/tmp/* \
/tmp/* \
CMD ["/bin/bash", "-c", "/usr/games/fortune | /usr/games/cowsay"]
Of course, you would also want to use the build-argument for something, unlike I did, but still, I recommend building this Dockerfile and taking it for a test-run :)
EDIT
As mentioned in #Jeffrey Wen's answer, to make sure that this errors out on a centos:7 image (and possibly others, I admittedly haven't tested this on other images than stretch-slim):
Ensure that you're executing the RUN command with the bash shell.
RUN ["/bin/bash", "-c", ": ${MYUID:?Build argument needs to be set and not null.}"]
Another simple way:
RUN test -n "$MY_VARIABLE" || (echo "MY_VARIABLE not set" && false)
Long time ago I had a need to introduce a required (mandatory) ARG, and for better UX include the check at the beginning:
FROM ubuntu:bionic
ARG MY_ARG
RUN [ -z "$MY_ARG" ] && echo "MY_ARG is required" && exit 1 || true
...
RUN ./use-my-arg.sh
But this busts the build cache for every single layer after the initial MY_ARG, because MY_ARG=VALUE is prepended to every RUN command afterwards.
Whenever I changed MY_ARG it would end up rebuilding the whole image, instead of rerunning the last RUN command only.
To bring caching back, I have changed my build to a multi-staged one:
The first stage uses MY_ARG and checks it's presence.
The second stage proceeds as usual and declares ARG MY_ARG right at the end.
FROM alpine:3.11.5
ARG MY_ARG
RUN [ -z "$MY_ARG" ] && echo "MY_ARG is required" && exit 1 || true
FROM ubuntu:bionic
...
ARG MY_ARG
RUN ./use-my-arg.sh
Since ARG MY_ARG in the second stage is declared right before it's used, all the previous steps in that stage are unaffected, thus cache properly.
You could do something like this...
FROM ubuntu:14.04
ONBUILD ARG MY_VARIABLE
ONBUILD RUN if [ -z "$MY_VARIABLE" ]; then echo "NOT SET - ERROR"; exit 1; else : ; fi
Then docker build -t my_variable_base .
Then build your images based on this...
FROM my_variable_base
...
It's not super clean, but at least it abstracts the 'bleh' stuff away to the base image.
I cannot comment yet because I do not have 50 reputation, but I would like to add onto #Jan Nash's solution because I had a little difficulty getting it to work with my image.
If you copy/paste #Jan Nash's solution, it will work and spit out the error message that the build argument is not specified.
What I want to add
When I tried getting it to work on a CentOS 7 image (centos:7), Docker ran the RUN command without erroring out.
Solution
Ensure that you're executing the RUN command with the bash shell.
RUN ["/bin/bash", "-c", ": ${MYUID:?Build argument needs to be set and not null.}"]
I hope that helps for future incoming people. Otherwise, I believe #Jan Nash's solution is just brilliant.
In case anybody is looking for a the solution but with docker compose build, I used mandatory variables.
version: "3.9"
services:
my-service:
build:
context: .
args:
- ENVVAR=${ENVVAR:?See build instructions}
After running docker compose build:
Before exporting ENVVAR: Invalid template: "required variable ENVVAR is missing a value: See build instructions"
After exporting ENVVAR: build proceeds
Support for Required Environment variables
Compose Environment Variables
None of these answers worked for me. I wanted ${MY_VARIABLE:?} but did not want to print anything, so I did like this:
ARG MY_VARIABLE
RUN test -n ${MY_VARIABLE:?}
Nothing is printed on success. On error you see this, which is a good enough error:
ERROR RUN test -n ${MY_VARIABLE:?}
/bin/sh: MY_VARIABLE: parameter not set or null
executor failed running [/bin/sh -c test -n ${MY_VARIABLE:?}]: >exit code: 2