I know docker has a --no-cache=true option to force a clean build of a docker image. For me however, all I'd really like to do is force the last step to run in my dockerfile, which is a CMD command that runs a shell script.
For whatever reason, when I modify that script and save it, a typical docker build will reuse the cached version of that step. Is there a way to force docker not to do so, just on that one portion?
Note that this would invalidate the cache for all Dockerfile directives after that line. This is requested in Issue 1996 (not yet implemented, and now (2021) closed), and issue 42799 (mentioned by ub-marco in the comments).
The current workaround is:
FROM foo
ARG CACHE_DATE=2016-01-01
<your command without cache>
docker build --build-arg CACHE_DATE=$(date) ....
That would invalidate cache after the ARG CACHE_DATE line for every build.
acdcjunior reports in the comments having to use:
docker build --build-arg CACHE_DATE=$(date +%Y-%m-%d_%H:%M:%S)
Another workaround from azul:
Here's what I am using to rebuild in CI if changes in git happened:
export LAST_SERVER_COMMIT=`git ls-remote $REPO "refs/heads/$BRANCH" | grep -o "^\S\+"`
docker build --build-arg LAST_SERVER_COMMIT="$LAST_SERVER_COMMIT"
And then in the Dockerfile:
ARG LAST_SERVER_COMMIT
RUN git clone ...
This will only rebuild the following layers if the git repo actually changed.
Related
Does Docker link images at build time or execution time? I am presenting a very simple case here to demonstrate my dilemma but I am presently rebuilding a chain of Docker files every time I change the base docker image and Im not sure if I even need to.
I have two docker files:
my base-docker-file contains
# this is not based on another image
RUN build commands here
which I build with
docker build --no-cache -t base-image -f base-docker-file.
and my from-docker-file here:
# this is based on the base-image
FROM base-image
RUN build commands here
which I build with
docker build --no-cache -t from-image -f from-docker-file.
Based on this simple setup, do I need to rebuild my from-image whenever I make a change to my base-image or does image linking occur at run time?
you will have to build each time if you make a change to the base-image. You could add these individual commands in a script to be executed together rather than running them separate everytime
1. Multi-stage build
Multi-stage build allows you to isolate images in one Dockerfile. 1
Example:
# syntax=docker/dockerfile:1
FROM alpine:latest AS builder
RUN apk --no-cache add build-base
FROM builder
COPY source1.cpp source.cpp
RUN g++ -o /binary source.cpp
2. Turn off no-cache when building the base-image.
Simple. If there are no changes in the base image, it will not rebuild due to cache.
Our application take a very long time to compile. Any change to the Dockerfile triggers a full-recomplile (which takes a very long time) The Dockerfile is a muli-stage build. I'm trying to work on the second stage. Is there any way to tell docker build to begin at the second stage?
FROM debian:latest AS builder
# 10-20 mins worth of stuff here
FROM alpine:latest AS runner
WORKDIR /
COPY --from=builder /work/myapp.zip .
RUN unzip myapp.zip -d /myapp
# and more stuff that I'm working on here
Is there some way to do docker build --begin-with runner?
Docker build run from the last changed stage..
See here might help you
rebuild docker image from specific step
Actually the Docker build cache should handle such situations automatically.
However this implementation has it's limits. It may not give you exactly what you want, but may be close.
Check out https://www.baeldung.com/linux/docker-build-cache
You can use --target [STAGE-NAME] from the build command according to the documentation, e.g
docker build -f Dockerfile --target runner -t imagename:1.0 .
This will compile just the runner stage
We have a Dockerfile, where at a certain point would like no caching to happen.
Currently we are using
ENV CACHE_BUST=$($RANDOM)
Upon further inspection, funny enough that gets cached:
Step 1/1 : ENV CACHE_BUST=$($RANDOM)
---> Using cache
Is there any way from inside the Dockerfile to bust the cache without passing in a unique build-arg (Like docker build . --build-arg CACHE_BUST=$(date +%s)) in the build step?
Update: Reviewing this one, it looks like you injected the cache busting option incorrectly in two ways:
ENV is not an ARG
The $(x) syntax is not a variable expansion, you need curly brackets (${}), not parenthesis ($()).
To break the cache on the next run line, the syntax is:
ARG CACHE_BUST
RUN echo "command with external dependencies"
And then build with:
docker build --build-arg CACHE_BUST=$(date +%s) .
Why does that work? Because during the build, the values for ARG are injected into RUN commands as environment variables. Changing an environment variable results in a cache miss on the new build.
To bust the cache, one of the inputs needs to change. If the command being run is the same, the cache will be reused even if the command has external dependencies that have changed, since docker cannot see those external dependencies.
Options to work around this include:
Passing a build arg that changes (e.g. setting it to a date stamp).
Changing a file that gets included into the image with COPY or ADD.
Running your build with the --no-cache option.
Since you do not want to do option 1, there is a way to do option 3 on a specific line, but only if you can split up your Dockerfile into 2 parts. The first Dockerfile has all the lines as you have today up to the point you want to break the cache. Then the second Dockerfile has a FROM line to depend on the first Dockerfile, and you build that with the --no-cache option. E.g.
Dockerfile1:
FROM base
RUN normal steps
Dockerfile2
FROM intermediate
RUN curl external.jar>file.jar
RUN other lines that cannot be cached
CMD your cmd
Then build with:
docker build -f Dockerfile1 -t intermediate .
docker build -f Dockerfile2 -t final --no-cache .
The only other option I can think of is to make a new frontend with BuildKit that allows you to inject an explicit cache break, or unique variable that results in a cache break.
You can add ADD layer with the downloading of some dynamic page from stable source in the beginning of Dockerfile. Image will be always re-built without using a cache.
Just an example of Dockerfile:
FROM alpine:3.9
ADD https://google.com cache_bust
RUN apk add --no-cache wget
p.s. I believe you are aware of docker build --no-cache option.
Is there a way to use external volumes during the docker image build ?
I have a situation where I would like to use a configuration inside a external volume during the docker image build time. Is that possible?
(edited to reflect current Docker CLI behavior)
If by 'docker image build' you mean running a single 'docker build ...' command: no, there is no way to do that (at least, not in the most recent documentation that I have read). However, nothing prevents you from performing the step that needs the external volume using direct docker commands and then commit the container and tag it just as 'docker build' would. Assuming this is the last step in your build, put all other commands (that don't need the volume) into a Dockerfile and then do this:
tmp_img=`docker build .`
tmp_container=`docker run -d -v $my_ext_volume:$my_mount_path --entrypoint=(your volume-dependent build command here) $tmp_img`
docker wait "$tmp_container"
docker commit $tmp_container my_repo/image_tag:latest
docker rm "$tmp_container"
This does the same as having a RUN command in the Dockerfile, but with the added volume mount. The commit command in the example also tags the image.
It is a bit more complex if you need to have other Dockerfile commands after the volume-dependent one, but in most cases you can combine run commands and re-arrange your install in a way that leaves the manual run-with-volume command last, to keep things simple.
You can copy the file into the docker image (ADD) and rm as one of the last steps
podman is an alternative to Docker that has an api that is the same BUT also supports mounting volumes at buildtime.
I use this to load data into testdatabases without having to copy the data into the image first.
You can use ADD combined with ARG (build time parameters) to access files or directories during the build without having to hardcode their location.
ARG MAVEN_SETTINGS=settings.xml
ADD $MAVEN_SETTINGS ./
And now you can change the file location during the build with:
docker build --build-arg MAVEN_SETTINGS=someotherfile.xml
We are not restricted to Docker to build OCI images.
With buildah it's possible to mount volumes from the host that won't be persisted in the final image. Useful for configuration and secrets.
buildah bud --volume /home/test:/myvol:ro -t imageName .
I have a bunch of Dockerfiles that are build from a common automated place using the same build command:
docker build -t $name:$tag --build-arg BRANCH=$branch .
Some of the Dockerfiles contain this:
ARG BRANCH=master
And that argument is used for some steps of the image build.
But for some Dockerfiles which doesn't need that argument I get this error at the end:
One or more build-args [BRANCH] were not consumed, failing build.
How can I overcome this problem without including the argument to all the Dockerfiles?
Have you considered grepping your Dockerfile for BRANCH and using it result to decide if you should supply your ARG or not?
You could replace your automation build trigger with something like:
if grep BRANCH Dockerfile; then docker build -t $name:$tag --build-arg BRANCH=$branch .; else docker build -t $name:$tag . ; fi
I don't see any documented way to avoid this error without changing your input or your Dockerfile. robertobado already covers changing your input. As a second option, you can include an effectively unused build arg at the end of your Dockerfile which would have a very minor impact on your build.
ARG BRANCH=undefined
RUN echo "Built from branch ${BRANCH}"
Since this doesn't modify the filesystem, I believe the image checksum will be identical.