Using cache for cmake build process in Docker build - docker

Question

I need one application in docker image which requires some specific version of libraries that have to be built from source.
So I am building it during the Docker build process.
Problem is, that it takes so long time (about 30mins).
I am wondering if it's possible to save it to the cache layer and skip it if the build process is done next time.
Here is the critical part of code from Dockerfile:
ADD https://sqlite.org/2022/sqlite-autoconf-3380200.tar.gz sqlite-autoconf-3380200.tar.gz
RUN tar -xvzf sqlite-autoconf-3380200.tar.gz
WORKDIR sqlite-autoconf-3380200
RUN ./configure
RUN make
RUN make install
WORKDIR /tmp
ADD https://download.osgeo.org/proj/proj-9.0.0.tar.gz proj-9.0.0.tar.gz
RUN tar -xvzf proj-9.0.0.tar.gz
WORKDIR proj-9.0.0
RUN mkdir build
WORKDIR build
RUN cmake ..
RUN cmake --build .
RUN cmake --build . --target install
RUN projsync --system-directory --list-files

Answer 1

The important detail about Docker layer caching is that, if any of the previous steps have changed, then all of the following steps will be rebuilt. So for your setup, if you change anything in one of the earlier dependencies, it will cause all of the later steps to be rebuilt again.
This is a case where Docker multi-stage builds can help. The idea is that you'd build each library in its own image, and therefore each library build can be independently cached. You can then copy all of the build results into a final image.
The specific approach I'll describe here assumes (a) all components install into /usr/local, (b) /usr/local is initially empty, and (c) there aren't conflicts between the different library installations. You should be able to adapt it to other filesystem layouts.
Everything below is in the same Dockerfile.
I'd make a very first stage selecting a base Linux-distribution image. If you know you'll always need to install something – TLS CA certificates, mandatory package updates – you can put it here. Having this helps ensure that everything is being built against a consistent base.
FROM ubuntu:20.04 AS base
# empty in this example
Since you have multiple things you need to build, a next stage will install any build-time dependencies. The C toolchain and its dependencies are large, so having this separate saves time and space since the toolchain can be shared across the later stages.
FROM base AS build-deps
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
build-essential \
cmake
# libfoo-dev
Now for each individual library, you have a separate build stage that downloads the source, builds it, and installs it into /usr/local.
FROM build-deps AS sqlite
WORKDIR /sqlite
ADD https://sqlite.org/2022/sqlite-autoconf-3380200.tar.gz sqlite-autoconf-3380200.tar.gz
...
RUN make install
FROM build-deps AS proj
WORKDIR /proj
ADD https://download.osgeo.org/proj/proj-9.0.0.tar.gz proj-9.0.0.tar.gz
...
RUN cmake --build . --target install
To actually build your application, you'll need the C toolchain, plus you'll also need these various libraries.
FROM build-deps AS app
COPY --from=sqlite /usr/local/ /usr/local/
COPY --from=proj /usr/local/ /usr/local/
WORKDIR /app
COPY ./ ./
RUN ./configure && make && make install
Once you've done all of this, in the app image, the /usr/local tree will have all of the installed libraries (COPYed from the previous image) plus your application. So for the final stage, start from the original OS image (without the C toolchain) and COPY the /usr/local tree in (without the original sources).
FROM base
COPY --from=app /usr/local/ /usr/local/
EXPOSE 12345
CMD ["myapp"] # in `/usr/local/bin`
Let's say you update to a newer patch version of proj. In the sqlite path, the base and build-deps layers haven't changed and the ADD and RUN commands are the same, so this stage runs entirely from cache. proj is rebuilt. That will cause the COPY --from=proj step to invalidate the cache in the app stage, and you'll rebuild your application against the newer library.