Why does docker rebuild all layers every time I change build args - docker

Question

I have a docker file which has a lot of layers. At the top of the file I have some args like
FROM ubuntu:18.04
ARGS USER=test-user
ARGS UID=1000
#ARGS PW=test-user
# Then several Layers which does not use any ARGS. Example
LABEL version="1.0"
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
RUN mkdir ~/mapped-volume
RUN apt-get update && apt-get install -y wget bzip2 ca-certificates build-essential curl git-core htop pkg-config unzip unrar tree freetds-dev vim \
sudo nodejs npm net-tools flex perl automake bison libtool byacc
# And so on
# And finally towards the end
# Setup User
RUN useradd -m -d /home/${USER} --uid ${UID} -G sudo -s /bin/bash ${USER}
# && echo "${USER}:${PW}" | chpasswd
# Couple of more commands to change dir, entry point etc. Example
When I build this docker file with any arg value different from the last build and/or after small changes in the last two layers, the build builds everything again. It does not use cached layer. The command I use to build is something like this
docker build --build-arg USER=new-user --build-arg UID=$UID -t my-image:1.0 .
And every time I change the values the build goes all through again. With a truncated top like below
UID -t my-image:1.0 .
Sending build context to Docker daemon 44.54kB
Step 1/23 : FROM ubuntu:18.04
---> ccc6e87d482b
Step 2/23 : ARG USER=ml-user
---> Using cache
---> 6c0c5d5c5056
Step 3/23 : ARG UID=1000
---> Using cache
---> b25867c282c7
Step 4/23 : LABEL version="1.0"
---> Running in 1ffff70d56c1
Removing intermediate container 1ffff70d56c1
---> 0f1277def3ca
Step 5/23 : ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
---> Running in 49d08c41b233
Removing intermediate container 49d08c41b233
---> f5b345573c1f
Step 6/23 : RUN mkdir ~/mapped-volume
---> Running in e4f8a5956450
Removing intermediate container e4f8a5956450
---> 1b22731d9051
Step 7/23 : RUN apt-get update && apt-get install -y wget bzip2 ca-certificates build-essential curl git-core htop pkg-config unzip unrar tree freetds-dev vim sudo nodejs npm net-tools flex perl automake bison libtool byacc
---> Running in ffc297de6234
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
So from step 7 it keeps doing all steps without using the cache of that layer which should have a bunch of packages
Why? How can I stop this? Previously when I did not have args, this layer and other layers used to be picked up from cache.

Answer 1

Move your args to just before you need them. Docker does not replace args in the RUN commands before running them. Instead, the args are passed as environment variables and expanded by the shell within the temporary container. Because of that, a change to an arg is a change to the environment, and a miss of the build cache for that step. Once one step misses the cache, all following steps must be rebuilt.
FROM ubuntu:18.04
# Then several Layers which does not use any ARGS. Example
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
RUN mkdir ~/mapped-volume
RUN apt-get update && apt-get install -y wget bzip2 ca-certificates build-essential curl git-core htop pkg-config unzip unrar tree freetds-dev vim \
sudo nodejs npm net-tools flex perl automake bison libtool byacc
# And so on
# And finally towards the end
# Setup User
ARGS USER=test-user
ARGS UID=1000
RUN useradd -m -d /home/${USER} --uid ${UID} -G sudo -s /bin/bash ${USER}
# && echo "${USER}:${PW}" | chpasswd
# Couple of more commands to change dir, entry point etc. Example
LABEL version="1.0"
Also, labels, environment variables that aren't needed at build time, exposed ports, and any other meta data is often best left to the end of the Dockerfile since they have minimal impact on build time and there's no need to miss the cache when they change.