Dockerfile build ARG in COPY --from= - docker

I am trying to set up a build process for a project and am running into an issue with using arg in the COPY command.
Part of the process is the build of a library into an image, that is used by multiple other images. The problem occurs in the following lines:
ARG BUILD_CONFIG=dev
COPY --from=company/mu_library:${BUILD_CONFIG} /some/path /other/path
According to the error message, the ${BUILD_CONFIG} is never translated into dev. When adding an echo line beforehand, the echo prints dev.
ARG BUILD_CONFIG=dev
RUN echo ${BUILD_CONFIG}
COPY --from=company/mu_library:${BUILD_CONFIG} /some/path /other/path
Does anyone have an idea how to go around it without creating duplicate stages in the dockerfile that all point to separate tags?
EDIT:
Minimal Dockerfile
FROM node:12.15:0 as prod
ARG BUILD_CONFIG=dev
RUN echo ${BUILD_CONFIG}
COPY --from=test/test-library:${BUILD_CONFIG} /work/dist /work/library/dist
CMD[ "bash" ]
Error:
invalid from flag value test/test-library:${BUILD_CONFIG}: invalid reference format

At last check, you can't use a build arg there, but you can use it in a top level from line using the multi-stage syntax. Then, you also need to define the build arg at the top level:
ARG BUILD_CONFIG=dev
FROM test/test-library:${BUILD_CONFIG} as test-library
FROM node:12.15:0 as prod
ARG BUILD_CONFIG
RUN echo ${BUILD_CONFIG}
COPY --from=test-library /work/dist /work/library/dist
CMD[ "bash" ]

Related

Docker ARG tricky behavior

Docker build tricky behavior:
Dockerfile
ARG php_version="7.2"
ARG flavor="stretch"
FROM php:${php_version}-fpm-${flavor}
#ARG php_version="7.2"
ENV php_v $php_version
CMD echo $php_v
If you build it and run:
docker build -t args:1.0 .
docker run -it --name testargs args:1.0
Output is empty string instead of the expected "7.2"
But if the third ARG line is uncommented we get "7.2"
Why does it behave this way?
Each FROM in a Dockerfile represents a new build stage and an ARG declared before the FROM is not available for use in the newer build stages starting with another FROM.
To quote the relevant section of the doc:
An ARG declared before a FROM is outside of a build stage, so it can’t
be used in any instruction after a FROM.

docker pass build arg to the shell script which needs to be run during the docker building

The problem is i cannot get the docker build arg value in the shell script while running the docker build.
My docker build command:
DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain \
-t test \
--build-arg WHL_PATH=/fake/path \
.
Dockerfile
ARG WHL_PATH
FROM python:3.8.8
COPY test.sh .
RUN ./test.sh $WHL_PATH
and in the test.sh the "$1" is empty...., if in the Dockerfile i put some constant value then i will be able to see that value in the $1, but with docker build arg or set the build arg as ENV VAR are always empty...
Where am i doing wrong, how should i achieve this?
Docker version 20.10.5, build 55c4c88
Build args are scoped. Before the first FROM step they only apply to the FROM steps to adjust the image you use. Within each stage, an ARG step applies to the remaining steps within that stage. So the fix is to reorder your steps:
FROM python:3.8.8
COPY test.sh .
ARG WHL_PATH
RUN ./test.sh $WHL_PATH
Oops, i never realised the position of the ARG instruction matters, basically:
any ARG before the first FROM can be used in any FROM line
any ARG within a build stage (after a FROM) can be used in that build stage
After i moved the ARG WHL_PATH after the line FROM xxx it works perfectly, hope this can save some of your time in the future.
And i was inspired by this answer actually: https://stackoverflow.com/a/50292255/7658313

Why can't I print an ARG with echo in Dockerfile? [duplicate]

I'm using Docker 18.05.0~ce~3-0~ubuntu and I'd like to pass a build argument to the FROM as well as other lines in my Dockerfile. You would expect the below to work:
ARG FROM_IMAGE=ubuntu:bionic
FROM $FROM_IMAGE
COPY sources_list/$FROM_IMAGE /etc/apt/sources.list
It works for the second line (FROM), but it behaves like it is unset in the COPY line:
Step 1/3 : ARG FROM_IMAGE=ubuntu:bionic
Step 2/3 : FROM $FROM_IMAGE
---> 8626492fecd3
[...]
Step 3/3 : COPY sources_list/${SOURCES_LIST_FILE} /etc/apt/sources.list
failed to copy files: failed to copy directory: mkdir
/var/lib/docker/overlay2/0536b4e280ddca2fec18db9d79fa625a8be86efdbaaea5b3dbbefcdaaab3f669/merged/etc/apt/sources.list:
not a directory
If add another, separate build arg, it works for the same COPY line:
ARG FROM_IMAGE=ubuntu:bionic
FROM $FROM_IMAGE
ARG SOURCES_LIST_FILE
COPY sources_list/${SOURCES_LIST_FILE} /etc/apt/sources.list
Step 4/4 : COPY sources_list/${SOURCES_LIST_FILE} /etc/apt/sources.list
---> 7f974fffe929
Why can't I use the FROM_IMAGE build arg twice, on and after a FROM line? I fail to find any documented restriction of this sort.
Why can't I use the FROM_IMAGE build arg twice, on and after a FROM line?
There is a real difference depending on where you put ARG related to FROM line:
any ARG before the first FROM can be used in any FROM line
any ARG within a build stage (after a FROM) can be used in that build stage
This is related to build stages mechanics and some reference of actual behavior can be found here: https://github.com/docker/cli/pull/333, and a discussion on why documentation and build mechanics is a bit confusing on ARG usage is here: https://github.com/moby/moby/issues/34129

docker: Variables in two stage build

I am using a two stage build for a docker image;
I want to make a value in my second stage configurable, so I was thinking of using ARG.
However I am not sure the --build-arg command line option applies to other than the first stage, so I came up with this
### First stage
FROM some_base_image
ARG MYUSERNAME=foo
### Second stage
FROM another_base_image
ARG MYUSERNAME=$MYUSERNAME
but that didn't work;
any suggestions how to pass via the build command line some --build-args that should be usable from the second stage?
ARG instructions have a scope, it's described here in the docs: https://docs.docker.com/engine/reference/builder/#scope
An ARG instruction goes out of scope at the end of the build stage where it was defined. To use an arg in multiple stages, each stage must include the ARG instruction.
So you can use an ARG in multiple stages, you just need to use the ARG instruction again as shown in the example:
FROM busybox
ARG SETTINGS
RUN ./run/setup $SETTINGS
FROM busybox
ARG SETTINGS
RUN ./run/other $SETTINGS

ARG substitution in RUN command not working for Dockerfile

In my Dockerfile I have the following:
ARG a-version
RUN wget -q -O /tmp/alle.tar.gz http://someserver/server/$a-version/a-server-$a-version.tar.gz && \
mkdir /opt/apps/$a-version
However when building this with:
--build-arg http_proxy=http://myproxy","--build-arg a-version=a","--build-arg b-version=b"
Step 10/15 : RUN wget... is shown with $a-version in the path instead of the substituted value and the build fails.
I have followed the instructions shown here but must be missing something else.
My questions is, what could be causing this issue and how can i solve
it?
Another thing to be careful about is that after every FROM statements all the ARGs get collected and are no longer available. Be careful with multi-stage builds.
You can reuse ARG with omitted default value inside FROM to get through this problem:
ARG VERSION=latest
FROM busybox:$VERSION
ARG VERSION
RUN echo $VERSION > image_version
Example taken from docs:
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
Don't use - in variable names.
Docker build will always show you the line as is written down in the Dockerfile, despite the variable value.
So use this variable name a_version:
ARG a_version
See this example:
Dockerfile:
FROM alpine
ARG a_version
RUN echo $a_version
Build:
$ docker build . --build-arg a_version=1234
Sending build context to Docker daemon 2.048 kB
Step 1/3 : FROM alpine
---> a41a7446062d
Step 2/3 : ARG a_version
---> Running in c55e98cab494
---> 53dedab7de75
Removing intermediate container c55e98cab494
Step 3/3 : RUN echo $a_version <<< note this <<
---> Running in 56b8aeddf77b
1234 <<<< and this <<
---> 89badadc1ccf
Removing intermediate container 56b8aeddf77b
Successfully built 89badadc1ccf
I had the same problem using Windows containers for Windows.
Instead of doing this (Which works in linux containers)
FROM alpine
ARG TARGETPLATFORM
RUN echo "I'm building for $TARGETPLATFORM"
You need to do this
FROM mcr.microsoft.com/windows/servercore
ARG TARGETPLATFORM
RUN echo "I'm building for %TARGETPLATFORM%"
Just change the variable resolution according to the OS.
I spent much time to have the argument substitution working, but the solution was really simple. The substitution within RUN needs the argument reference to be enclosed in double quotes.
ARG CONFIGURATION=Debug
RUN dotnet publish "Project.csproj" -c "$CONFIGURATION" -o /app/publish
The only way I was able to substitute an ARG in a Windows Container was to prefix with $env:, as mentioned here.
An example of my Dockerfile is below. Notice that the ARG PAT is defined after the FROM so that it's in scope for its use in the RUN nuget sources add command (as Hongtao suggested). The only successful way I found to supply the personal access token was using $env:PAT
FROM mcr.microsoft.com/dotnet/framework/sdk:4.7.2 AS build
WORKDIR /app
ARG PAT
# copy csproj and restore as distinct layers
COPY *.sln .
COPY WebApi/*.csproj ./WebApi/
COPY WebApi/*.config ./WebApi/
RUN nuget sources add -name AppDev -source https://mysource.pkgs.visualstudio.com/_packaging/AppDev/nuget/v2 -username usern -password $env:PAT
RUN nuget restore
# copy everything else and build app
COPY WebApi/. ./WebApi/
WORKDIR /app/WebApi
RUN msbuild /p:Configuration=Release
FROM mcr.microsoft.com/dotnet/framework/aspnet:4.7.2 AS runtime
WORKDIR /inetpub/wwwroot
COPY --from=build /app/WebApi/. ./
The actual Docker command looks like this:
docker build --build-arg PAT=mypatgoeshere -t webapi .
I had the same problem accessing build-args in my RUN command. Turns out that the line containing the ARG definition should not be the first line. The working Dockerfile snippet looks like this:
FROM centos:7
MAINTAINER xxxxx
ARG SERVER_IPS
Earlier, I had placed the ARG definition as the first line of Dockerfile . My docker version is v19.
There are many answers, which make sense.
But the main thing is missed.
The way, how to use build arguments depends on the base image.
For Linux image, it will work with $ARG
For Windows, depending on image, it can be either $env:ARG(e.g. for mcr.microsoft.com/dotnet/framework/sdk:4.8) or %ARG% (e.g. for mcr.microsoft.com/windows/nanoserver:1809)
For me it was argument's order:
docker build . -f somepath/to/Dockerfile --build-arg FOO=BAR
did not work, but:
docker build --build-arg FOO=BAR . -f somepath/to/Dockerfile
did.

Resources