.NET package restore in Docker cached separately from build - docker

Question

How does one build a Docker image of a .NET 5/C# app so that the restored NuGet packages are cached properly? By proper caching I mean that when sources (but not project files) are changed, the layer containing restored packages is still taken from cache during docker build.
It is a best practice in Docker to perform package restore before adding the full sources and building the app itself as it makes it possible to cache the restore separately, which significantly speeds up the builds. I know that not only the packages directory, but also the bin and obj directories of individual projects have to be preserved from dotnet restore to dotnet publish --no-restore so that everything works together. I also know that once the cache is busted, all following layers are built anew.
My issue is that I cannot come up with a way to COPY just the *.csproj. If I copy more than just the *.csproj, source changes bust the cache. I could copy them into one place outside the docker build and simply COPY them inside the build, but I want to be able to build the image even outside the pipeline, manually, with a reasonably simple command. (Is it an unreasonable requirement?)
For the web app that consists of multiple projects in a pretty standard folder structure src/*/*.csproj, I came up with this attempt that tries to compensate for too many files being copied into the image (which still busts the cache):
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build-env
WORKDIR /src
COPY NuGet.Config NuGet.Config
COPY src/ src/
RUN find . -name NuGet.Config -prune -o \! -type d \! -name \*.csproj -exec rm -f '{}' + \
&& find -depth -type d -empty -exec rmdir '{}' \;
RUN dotnet restore src/Company.Product.Component.App/Company.Product.Component.App.csproj
COPY src/ src/
RUN dotnet publish src/Company.Product.Component.App/Company.Product.Component.App.csproj -c Release --no-restore -o /out
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS run-env
WORKDIR /app
COPY --from=build-env /out .
ENTRYPOINT ["dotnet", "Company.Product.Component.App.dll"]
I also tried splitting the build-env stage into two just after the restore, copying the /root/.nuget/packages and /src to the build stage, but that did not help either.
The first RUN line and the one immediately before should be replaced with something that copies just the *.csproj, but I don't know what that is. The obvious laborious solution is to have a separate COPY line for each *.csproj, but that does not feel right as projects tend to be added and removed so it makes the Dockerfile hard to maintain. I have tried COPY src/*/*.csproj src/ and then fixing the flattened paths, which is a trick that I googled, but it did not work for me as my Docker processes the wildcards just in file names and interprets directory names literally, emitting an error for nonexistent src/* directory. I am using Docker Desktop 3.5.2 (66501), which uses the BuildKit backend to build the images, but I am open to changing the tooling if it helps.
This leaves me clueless about how to satisfy the relatively simple set of my requirements. My options seem exhausted. Have I missed something? Do I have to accept a tradeoff and drop some of my requirements?

Answer 1

The lack of support for wildcards in directory names is likely a missing feature in BuildKit. The issue has already been reported at moby/buildkit GitHub as #1900.
Till the issue is fixed, disable BuildKit if you don't need any of its features. Either
set the environment variable DOCKER_BUILDKIT to zero (0), or
edit the Docker daemon config so that the "buildkit" feature is set to false and restart the daemon.
In Docker Desktop, the config is easily accessible in Settings > Docker Engine. This method of turning off the feature is recommended by the Docker Desktop 3.2.0 release notes where BuildKit was first enabled by default.
Once BuildKit is disabled, replace
COPY src/ src/
RUN find . -name NuGet.Config -prune -o \! -type d \! -name \*.csproj -exec rm -f '{}' + \
&& find -depth -type d -empty -exec rmdir '{}' \;
with
COPY src/*/*.csproj src/
RUN for from in src/*.csproj; do to=$(echo "$from" | sed 's/\/\([^/]*\)\.csproj$/\/\1&/') \
&& mkdir -p "$(dirname "$to")" && mv "$from" "$to"; done
The COPY will succeed without busting the cache and the RUN will fix the paths. It relies on the fact the projects are in the "src" directory, each in a separate directory of the same name as the project file.
This is basically the solution at the bottom of VonC's answer to a related question. The answer also mentions Moby issue #15858, which has an interesting discussion on the topic.
There is a dotnet tool for the paths fixup, too, but I have not tested it.
An alternate solution that does not require disabling BuildKit is to split the original stage in two right after cleaning up the copied files, i.e. just before restore (not after!).
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS projects-env
WORKDIR /src
COPY NuGet.Config NuGet.Config
COPY src/ src/
RUN find . -name NuGet.Config -prune -o \! -type d \! -name \*.csproj -exec rm -f '{}' + \
&& find . -depth -type d -empty -exec rmdir '{}' \;
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build-env
WORKDIR /src
COPY --from=projects-env /src /src
RUN dotnet restore src/Company.Product.Component.App/Company.Product.Component.App.csproj
COPY src/ src/
RUN dotnet publish src/Company.Product.Component.App/Company.Product.Component.App.csproj -c Release --no-restore -o /out
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS run-env
WORKDIR /app
COPY --from=build-env /out .
ENTRYPOINT ["dotnet", "Company.Product.Component.App.dll"]
The COPY src/ src/ layer in the sources-env is invalidated by source changes, but cache invalidation works separately for each stage. As the files copied over to the build-env are the same across builds, the COPY --from=projects-env cache is not invalidated, so the RUN dotnet restore layer is taken from the cache, too.
I suspect there are other solutions using the BuildKit mounts (RUN --mount=...), but I haven't tested any.

Answer 2

Here's the alternative way to solve the problem.
First, copy the .sln and .csproj files (depends on the solution folder structure)
COPY *.sln ./
COPY **/*.csproj ./
COPY **/**/**/*.csproj ./
After that run the following script:
RUN dotnet sln list | grep ".csproj" \
| while read -r line; do \
mkdir -p $(dirname $line); \
mv $(basename $line) $(dirname $line); \
done;
The script simply moves .csproj files to the same locations they are in the host filesystem.