Why can't I print an ARG with echo in Dockerfile? [duplicate] - docker

I'm using Docker 18.05.0~ce~3-0~ubuntu and I'd like to pass a build argument to the FROM as well as other lines in my Dockerfile. You would expect the below to work:
ARG FROM_IMAGE=ubuntu:bionic
FROM $FROM_IMAGE
COPY sources_list/$FROM_IMAGE /etc/apt/sources.list
It works for the second line (FROM), but it behaves like it is unset in the COPY line:
Step 1/3 : ARG FROM_IMAGE=ubuntu:bionic
Step 2/3 : FROM $FROM_IMAGE
---> 8626492fecd3
[...]
Step 3/3 : COPY sources_list/${SOURCES_LIST_FILE} /etc/apt/sources.list
failed to copy files: failed to copy directory: mkdir
/var/lib/docker/overlay2/0536b4e280ddca2fec18db9d79fa625a8be86efdbaaea5b3dbbefcdaaab3f669/merged/etc/apt/sources.list:
not a directory
If add another, separate build arg, it works for the same COPY line:
ARG FROM_IMAGE=ubuntu:bionic
FROM $FROM_IMAGE
ARG SOURCES_LIST_FILE
COPY sources_list/${SOURCES_LIST_FILE} /etc/apt/sources.list
Step 4/4 : COPY sources_list/${SOURCES_LIST_FILE} /etc/apt/sources.list
---> 7f974fffe929
Why can't I use the FROM_IMAGE build arg twice, on and after a FROM line? I fail to find any documented restriction of this sort.

Why can't I use the FROM_IMAGE build arg twice, on and after a FROM line?
There is a real difference depending on where you put ARG related to FROM line:
any ARG before the first FROM can be used in any FROM line
any ARG within a build stage (after a FROM) can be used in that build stage
This is related to build stages mechanics and some reference of actual behavior can be found here: https://github.com/docker/cli/pull/333, and a discussion on why documentation and build mechanics is a bit confusing on ARG usage is here: https://github.com/moby/moby/issues/34129

Related

Dockerfile build ARG in COPY --from=

I am trying to set up a build process for a project and am running into an issue with using arg in the COPY command.
Part of the process is the build of a library into an image, that is used by multiple other images. The problem occurs in the following lines:
ARG BUILD_CONFIG=dev
COPY --from=company/mu_library:${BUILD_CONFIG} /some/path /other/path
According to the error message, the ${BUILD_CONFIG} is never translated into dev. When adding an echo line beforehand, the echo prints dev.
ARG BUILD_CONFIG=dev
RUN echo ${BUILD_CONFIG}
COPY --from=company/mu_library:${BUILD_CONFIG} /some/path /other/path
Does anyone have an idea how to go around it without creating duplicate stages in the dockerfile that all point to separate tags?
EDIT:
Minimal Dockerfile
FROM node:12.15:0 as prod
ARG BUILD_CONFIG=dev
RUN echo ${BUILD_CONFIG}
COPY --from=test/test-library:${BUILD_CONFIG} /work/dist /work/library/dist
CMD[ "bash" ]
Error:
invalid from flag value test/test-library:${BUILD_CONFIG}: invalid reference format
At last check, you can't use a build arg there, but you can use it in a top level from line using the multi-stage syntax. Then, you also need to define the build arg at the top level:
ARG BUILD_CONFIG=dev
FROM test/test-library:${BUILD_CONFIG} as test-library
FROM node:12.15:0 as prod
ARG BUILD_CONFIG
RUN echo ${BUILD_CONFIG}
COPY --from=test-library /work/dist /work/library/dist
CMD[ "bash" ]

docker pass build arg to the shell script which needs to be run during the docker building

The problem is i cannot get the docker build arg value in the shell script while running the docker build.
My docker build command:
DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain \
-t test \
--build-arg WHL_PATH=/fake/path \
.
Dockerfile
ARG WHL_PATH
FROM python:3.8.8
COPY test.sh .
RUN ./test.sh $WHL_PATH
and in the test.sh the "$1" is empty...., if in the Dockerfile i put some constant value then i will be able to see that value in the $1, but with docker build arg or set the build arg as ENV VAR are always empty...
Where am i doing wrong, how should i achieve this?
Docker version 20.10.5, build 55c4c88
Build args are scoped. Before the first FROM step they only apply to the FROM steps to adjust the image you use. Within each stage, an ARG step applies to the remaining steps within that stage. So the fix is to reorder your steps:
FROM python:3.8.8
COPY test.sh .
ARG WHL_PATH
RUN ./test.sh $WHL_PATH
Oops, i never realised the position of the ARG instruction matters, basically:
any ARG before the first FROM can be used in any FROM line
any ARG within a build stage (after a FROM) can be used in that build stage
After i moved the ARG WHL_PATH after the line FROM xxx it works perfectly, hope this can save some of your time in the future.
And i was inspired by this answer actually: https://stackoverflow.com/a/50292255/7658313

When is a call to "RUN ln" able to be cached by Docker?

I have the following result from docker build .. Why doesn't Docker ever seem to cache step 4? Can I make it use cache with some tweak?
(The rest of the file as far as I know is irrelevant, but this failure to cache also results in steps 5 and following to build without cache.)
+ docker build .
Sending build context to Docker daemon 3.678MB
Step 1/36 : FROM mcr.microsoft.com/dotnet/core/sdk:3.1 as builder
---> c4155a9104a8
Step 2/36 : ARG branch_build_commit
---> Using cache
---> 2a8f43280a27
Step 3/36 : WORKDIR /tmp
---> Using cache
---> f3bf42ec98ab
Step 4/36 : RUN ln -snf /usr/share/zoneinfo/America/Chicago /etc/localtime && echo America/Chicago > /etc/timezone
---> Running in 32bd1e2cff9e
Removing intermediate container 32bd1e2cff9e
---> 8d49354c0de6
The Dockerfile documentation notes, under Impact [of ARG] on build caching:
...all RUN instructions following an ARG instruction use the ARG variable implicitly (as an environment variable), thus can cause a cache miss.
If this is in the context of a CI system and each build is normally passed a docker build --build-arg branch_build_commit=$(git rev-parse HEAD) . or something similar, then every build will have a different ARG value and RUN instructions will always cause a cache miss.
On the other hand, it's merely convention that ARG comes early in a Dockerfile so that it can be found. The Dockerfile documentation on Scope [of ARG] has some specific notes that ARG values can only be used after the ARG statement, and has an example of ARG not being first. So you can make caching work again by moving the ARG statement later in the file:
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 as builder
# These statements don't use the ARG value so they can come first.
WORKDIR /tmp
RUN ln -snf /usr/share/zoneinfo/America/Chicago /etc/localtime && echo America/Chicago > /etc/timezone
# Then you can declare the ARG value later
ARG branch_build_commit
# And any uses must come after the ARG statement.
RUN echo ${branch_build_commit:-unknown} > /etc/build-version

Is there a better way than repeating --build-arg on the command line when building a Dockerfile?

My single docker build command line is getting quite long. I wonder if there's a better way than settings all the ARG's in the Dockerfile and then --build-arg through each of them on the docker build command.
Unfortunately, there are no built-in possibility to specify file with build args to use docker build yet.
But anyway we can use simple bash wrapper for that:
1. We create file with build args. Something like args_list:
month=March
day=26
year=2018
2. An example of the Dockerfile is:
FROM alpine:3.7
ARG month
ARG day
ARG year
RUN echo "Today is the $day/$month/$year."
3. And build our above docker image with the following one line script:
command="docker build ."; for i in $(cat args_list); do command+=" --build-arg $i"; done; $command
The output shows that everything is successfully injected:
Sending build context to Docker daemon 3.072kB
Step 1/5 : FROM alpine:3.7
---> 3fd9065eaf02
Step 2/5 : ARG month
---> Running in 5c6e695c3fab
Removing intermediate container 5c6e695c3fab
---> 6193d0d5665f
Step 3/5 : ARG day
---> Running in da0bc3e65890
Removing intermediate container da0bc3e65890
---> 41a90a4dd893
Step 4/5 : ARG year
---> Running in 83f6ae68390d
Removing intermediate container 83f6ae68390d
---> 1fd4beef3c53
Step 5/5 : RUN echo "Today is the $day/$month/$year."
---> Running in b5eaa6a5a80c
Today is the 26/March/2018.
Removing intermediate container b5eaa6a5a80c
---> de7581671706
Successfully built de7581671706
This way we can simply add new build args into the "build args" file and our CI system will build proper images without changes on its side.

ARG substitution in RUN command not working for Dockerfile

In my Dockerfile I have the following:
ARG a-version
RUN wget -q -O /tmp/alle.tar.gz http://someserver/server/$a-version/a-server-$a-version.tar.gz && \
mkdir /opt/apps/$a-version
However when building this with:
--build-arg http_proxy=http://myproxy","--build-arg a-version=a","--build-arg b-version=b"
Step 10/15 : RUN wget... is shown with $a-version in the path instead of the substituted value and the build fails.
I have followed the instructions shown here but must be missing something else.
My questions is, what could be causing this issue and how can i solve
it?
Another thing to be careful about is that after every FROM statements all the ARGs get collected and are no longer available. Be careful with multi-stage builds.
You can reuse ARG with omitted default value inside FROM to get through this problem:
ARG VERSION=latest
FROM busybox:$VERSION
ARG VERSION
RUN echo $VERSION > image_version
Example taken from docs:
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
Don't use - in variable names.
Docker build will always show you the line as is written down in the Dockerfile, despite the variable value.
So use this variable name a_version:
ARG a_version
See this example:
Dockerfile:
FROM alpine
ARG a_version
RUN echo $a_version
Build:
$ docker build . --build-arg a_version=1234
Sending build context to Docker daemon 2.048 kB
Step 1/3 : FROM alpine
---> a41a7446062d
Step 2/3 : ARG a_version
---> Running in c55e98cab494
---> 53dedab7de75
Removing intermediate container c55e98cab494
Step 3/3 : RUN echo $a_version <<< note this <<
---> Running in 56b8aeddf77b
1234 <<<< and this <<
---> 89badadc1ccf
Removing intermediate container 56b8aeddf77b
Successfully built 89badadc1ccf
I had the same problem using Windows containers for Windows.
Instead of doing this (Which works in linux containers)
FROM alpine
ARG TARGETPLATFORM
RUN echo "I'm building for $TARGETPLATFORM"
You need to do this
FROM mcr.microsoft.com/windows/servercore
ARG TARGETPLATFORM
RUN echo "I'm building for %TARGETPLATFORM%"
Just change the variable resolution according to the OS.
I spent much time to have the argument substitution working, but the solution was really simple. The substitution within RUN needs the argument reference to be enclosed in double quotes.
ARG CONFIGURATION=Debug
RUN dotnet publish "Project.csproj" -c "$CONFIGURATION" -o /app/publish
The only way I was able to substitute an ARG in a Windows Container was to prefix with $env:, as mentioned here.
An example of my Dockerfile is below. Notice that the ARG PAT is defined after the FROM so that it's in scope for its use in the RUN nuget sources add command (as Hongtao suggested). The only successful way I found to supply the personal access token was using $env:PAT
FROM mcr.microsoft.com/dotnet/framework/sdk:4.7.2 AS build
WORKDIR /app
ARG PAT
# copy csproj and restore as distinct layers
COPY *.sln .
COPY WebApi/*.csproj ./WebApi/
COPY WebApi/*.config ./WebApi/
RUN nuget sources add -name AppDev -source https://mysource.pkgs.visualstudio.com/_packaging/AppDev/nuget/v2 -username usern -password $env:PAT
RUN nuget restore
# copy everything else and build app
COPY WebApi/. ./WebApi/
WORKDIR /app/WebApi
RUN msbuild /p:Configuration=Release
FROM mcr.microsoft.com/dotnet/framework/aspnet:4.7.2 AS runtime
WORKDIR /inetpub/wwwroot
COPY --from=build /app/WebApi/. ./
The actual Docker command looks like this:
docker build --build-arg PAT=mypatgoeshere -t webapi .
I had the same problem accessing build-args in my RUN command. Turns out that the line containing the ARG definition should not be the first line. The working Dockerfile snippet looks like this:
FROM centos:7
MAINTAINER xxxxx
ARG SERVER_IPS
Earlier, I had placed the ARG definition as the first line of Dockerfile . My docker version is v19.
There are many answers, which make sense.
But the main thing is missed.
The way, how to use build arguments depends on the base image.
For Linux image, it will work with $ARG
For Windows, depending on image, it can be either $env:ARG(e.g. for mcr.microsoft.com/dotnet/framework/sdk:4.8) or %ARG% (e.g. for mcr.microsoft.com/windows/nanoserver:1809)
For me it was argument's order:
docker build . -f somepath/to/Dockerfile --build-arg FOO=BAR
did not work, but:
docker build --build-arg FOO=BAR . -f somepath/to/Dockerfile
did.

Resources