I'm building my docker images in GitLab CI and trying to cache the build so that if the app dependencies don't change, I don't need to re-build the entire image, but for some reason the Docker layer caching isn't working.
Dockerfile
FROM public.ecr.aws/lambda/python:3.9
RUN pip install poetry==1.3.2
WORKDIR /app
COPY ./pyproject.toml ./poetry.lock ./
RUN poetry config virtualenvs.create false && poetry install \
--no-interaction --no-ansi --without dev
ENV PYTHONPATH=/app
COPY . .
CMD ["app.handler"]
GitLab CI Log Output
$ docker build --cache-from "${CI_REGISTRY_IMAGE}:latest" -t $IMAGE_NAME .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 321B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 145B done
#2 DONE 0.0s
#3 [internal] load metadata for public.ecr.aws/lambda/python:3.9
#3 DONE 0.5s
#4 importing cache manifest from registry.gitlab.com/foo/bar:latest
#4 DONE 0.0s
#5 [1/6] FROM public.ecr.aws/lambda/python:3.9#sha256:7e3bcdc955c2c3cab1101bbbfd55849c0f56a6b5a21a149d50df91deacad6aac
#5 resolve public.ecr.aws/lambda/python:3.9#sha256:7e3bcdc955c2c3cab1101bbbfd55849c0f56a6b5a21a149d50df91deacad6aac 0.0s done
#5 sha256:ebf75d2390460a5a59d7c52cc4d6e7f6840c610b399c069b0ccf531792d77c7d 3.00kB / 3.00kB done
#5 sha256:7e3bcdc955c2c3cab1101bbbfd55849c0f56a6b5a21a149d50df91deacad6aac 772B / 772B done
#5 sha256:e3d5decbab4ddca757c7de36637b50826165efddecfe8b7ef76db761f51b790d 1.58kB / 1.58kB done
#5 DONE 0.3s
#6 [internal] load build context
#6 transferring context: 1.16MB 0.2s done
#6 DONE 0.3s
...
The pipeline is successfully picking up on the latest build as you can see from the importing cache manifest from registry.gitlab.com/foo/bar:latest line. However, it doesn't seem to be caching any of the build steps.
Can anyone see a reason why the cache might be missing in this case?
Related
I have two ContainerApps in the same solution, both build and run locally in Docker desktop, and 1 of them publishes to an Azure ContainerApp properly. The other one appears identical structurally and w.r.t. dockerfile/projectFile but fails to publish with the "failed to compute cache key" project copy error
Windows 11
VisualStudio 17.3.4
Docker Desktop 4.12.0 / WSL2
\SolutionDir
\FR911.Worker.Project1
Dockerfile
FR911.Worker.Project1.csproj
\FR911.Worker.Project2
Dockerfile
FR911.Worker.Project2.csproj
Dockerfile for Project1 (publishes correctly)
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["FR911.Worker.Project1/FR911.Worker.Project1.csproj", "FR911.Worker.Project1/"]
RUN dotnet restore "FR911.Worker.Project1/FR911.Worker.Project1.csproj"
COPY . .
WORKDIR "/src/FR911.Worker.Project1"
RUN dotnet build "FR911.Worker.Project1.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "FR911.Worker.Project1.csproj" -c Release -o /app/publish /p:UseAppHost=false
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "FR911.Worker.Project1.dll"]
Dockerfile for Project2 (does not publish)
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["FR911.Worker.Project2/FR911.Worker.Project2.csproj", "FR911.Worker.Project2/"]
RUN dotnet restore "FR911.Worker.Project2/FR911.Worker.Project2.csproj"
COPY . .
WORKDIR "/src/FR911.Worker.Project2"
RUN dotnet build "FR911.Worker.Project2.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "FR911.Worker.Project2.csproj" -c Release -o /app/publish /p:UseAppHost=false
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "FR911.Worker.Project2.dll"]
Log for Project1 Publish (publishes correctly)
FR911.Worker.Project1 -> G:\FR911Cloud\FR911.Worker.Project1\bin\Release\net6.0\FR911.Worker.Project1.dll
FR911.Worker.Project1 -> G:\FR911Cloud\FR911.Worker.Project1\obj\Docker\publish\
Docker version 20.10.17, build 100c701
docker build -f "G:\FR911Cloud\FR911.Worker.Project1\Dockerfile" --force-rm -t fr911workerProject1 --label "com.microsoft.created-by=visual-studio" --label "com.microsoft.visual-studio.project-name=FR911.Worker.Project1" "G:\FR911Cloud"
#1 [internal] load build definition from Dockerfile
#1 sha256:f35cad84d8f344f22ff5c01ec0f2b5593a1e8e8106c5a291cbb81d75702e8f4b
#1 transferring dockerfile: 980B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:773696df5d1b3a2e3082d19ade268ad9cee78057bcc7d5bf617dbf42632a68c3
#2 transferring context: 35B 0.0s done
#2 DONE 0.0s
#4 [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:6.0
#4 sha256:ac4494cbca04ddb415c76edcbcc7688784c2a6ea65dd656286c013738aa3b75f
#4 DONE 0.0s
#3 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0
#3 sha256:9eb4f6c3944cfcbfe18b9f1a753c769fc35341309a8d4a21f8937f47e94c712b
#3 DONE 0.3s
#8 [build 1/7] FROM mcr.microsoft.com/dotnet/sdk:6.0#sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37
#8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a
#8 DONE 0.0s
#5 [base 1/2] FROM mcr.microsoft.com/dotnet/aspnet:6.0
#5 sha256:50f1ddc10932c4a74c7af5704e931a9489c710faea4f2381fe2380827a900e00
#5 DONE 0.0s
#10 [internal] load build context
#10 sha256:a7dbe6d5c5594ebe893dc072a634555501969a46c9ae9d648db74ea23d0ee810
#10 transferring context: 673.15kB 1.4s done
#10 DONE 1.4s
#9 [build 2/7] WORKDIR /src
#9 sha256:fd721b9ab8612450c39dfab43831d16b893f71b294b4d92a8c1d6fdaf0c47a22
#9 CACHED
#11 [build 3/7] COPY [FR911.Worker.Project1/FR911.Worker.Project1.csproj, FR911.Worker.Project1/]
#11 sha256:91f73142833c22e9b36d6dfb095a3421596ba684c3076da61c19a7d7aae5f1bc
#11 CACHED
... continues until success
Log for Project2 Publish (fails)
FR911.Worker.Project2 -> G:\FR911Cloud\FR911.Worker.Project2\bin\Release\net6.0\FR911.Worker.Project2.dll
FR911.Worker.Project2 -> G:\FR911Cloud\FR911.Worker.Project2\obj\Docker\publish\
Docker version 20.10.17, build 100c701
docker build -f "G:\FR911Cloud\FR911.Worker.Project2\Dockerfile" --force-rm -t fr911workerProject2 --label "com.microsoft.created-by=visual-studio" --label "com.microsoft.visual-studio.project-name=FR911.Worker.Project2" "G:\FR911Cloud"
#1 [internal] load build definition from Dockerfile
#1 sha256:69dc10bf751883ff6fa38a281a2a6e3570552aeded8c5f82fc34e54ad9c820c0
#1 transferring dockerfile: 32B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:5089ff4c45602d16118aed124a38ee56bbe29b79f636b726f36a16f813470f64
#2 transferring context: 35B 0.0s done
#2 DONE 0.0s
#4 [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:6.0
#4 sha256:ac4494cbca04ddb415c76edcbcc7688784c2a6ea65dd656286c013738aa3b75f
#4 DONE 0.0s
#3 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0
#3 sha256:9eb4f6c3944cfcbfe18b9f1a753c769fc35341309a8d4a21f8937f47e94c712b
#3 DONE 0.3s
#8 [build 1/7] FROM mcr.microsoft.com/dotnet/sdk:6.0#sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37
#8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a
#8 DONE 0.0s
#9 [build 2/7] WORKDIR /src
#9 sha256:fd721b9ab8612450c39dfab43831d16b893f71b294b4d92a8c1d6fdaf0c47a22
#9 CACHED
#5 [base 1/2] FROM mcr.microsoft.com/dotnet/aspnet:6.0
#5 sha256:50f1ddc10932c4a74c7af5704e931a9489c710faea4f2381fe2380827a900e00
#5 DONE 0.0s
#10 [internal] load build context
#10 sha256:193c5f727995c82adbaa58aed42c8a05308b1cbab22f31bea74fad6ffe5c2e83
#10 transferring context: 673.11kB 1.4s done
#10 DONE 1.4s
#11 [build 3/7] COPY [FR911.Worker.Project2/FR911.Worker.Project2.csproj, FR911.Worker.Project2/]
#11 sha256:687c29b5b257b0efc68aa34321eca5a6576a579c63371f320e619daa0c9c705c
#11 ERROR: "/FR911.Worker.Project2/FR911.Worker.Project2.csproj" not found: not found
------
> [build 3/7] COPY [FR911.Worker.Project2/FR911.Worker.Project2.csproj, FR911.Worker.Project2/]:
------
failed to compute cache key: "/FR911.Worker.Project2/FR911.Worker.Project2.csproj" not found: not found
The publish profiles are also identical except for the container & environment names, and the project files both target net6.0 and Linux. The Docker command that VisualStudio runs to build each also seems identical(except for the path & name):
docker build -f "G:\FR911Cloud\FR911.Worker.Project2\Dockerfile" --force-rm -t fr911workerProject2 --label "com.microsoft.created-by=visual-studio" --label "com.microsoft.visual-studio.project-name=FR911.Worker.Project2" "G:\FR911Cloud"
I'm having trouble finding out why one works and the other fails - presumably because it cannot find the project at step 3 of the build from the dockerfile COPY ["FR911.Worker.Project2/FR911.Worker.Project2.csproj", "FR911.Worker.Project2/"]
It was a casing issue in the Dockerfile on the project file name.
COPY ["FR911.Worker.Project2/FR911.Worker.Project2.csproj", "FR911.Worker.Project2/"]
is not the same as
COPY ["FR911.Worker.Project2/FR911.Worker.project2.csproj", "FR911.Worker.Project2/"]
I am trying to build a docker image for my sample-go app.
I am running it from the sample-app folder itself and using the goland editor's terminal. But the build is failing and giving me certain errors.
My docker file looks like this:
FROM alpine:latest
RUN mkdir -p /src/build
WORKDIR /src/build
RUN apk add --no-cache tzdata ca-certificates
COPY ./configs /configs
COPY main /main
EXPOSE 8000
CMD ["/main"]
command for building:
docker build --no-cache --progress=plain - < Dockerfile
Error And Logs:
#1 [internal] load build definition from Dockerfile
#1 sha256:8bb9ee83603259cf748d90ce42602f12527fa720d7417da22799b2ad4e503497
#1 transferring dockerfile: 222B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:f93d938488588cd0e0a94d9d343fe69dcfd28d0cb1da95ad7aab00aac50235c3
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/alpine:latest
#3 sha256:13549c58a76bcb5dac9d52bc368a8fb6b5cf7659f94e3fa6294917b85546978d
#3 DONE 0.0s
#10 [1/6] FROM docker.io/library/alpine:latest
#10 sha256:d20daa00e252bfb345a1b4f53b6bb332aafe702d8de5e583a76fcd09ba7ea1c1
#10 CACHED
#7 [internal] load build context
#7 sha256:0f7a8a6082a837c139acc2855e1b745bba9f28cc96709d45cd0b7be42442c0e8
#7 transferring context: 2B done
#7 DONE 0.0s
#4 [2/6] RUN mkdir -p /src/build
#4 sha256:b9fa3007a44471d47414dd29b3ff07ead6af28ede820a2b4bae0ce84cf2c5a83
#4 CACHED
#5 [3/6] WORKDIR /src/build
#5 sha256:b2ec58a365fdd74c4f9030b0caff2e2225eea33617da306678ad037fce675388
#5 CACHED
#6 [4/6] RUN apk add --no-cache tzdata ca-certificates
#6 sha256:0966097abf956d5781bc2330d49cf715cd52c3807e8fedfff07dec50907ff03b
#6 CACHED
#9 [6/6] COPY main /main
#9 sha256:f4b81960427c014a020361bea0903728f289e1d796892fe0adc6409434f3ca76
#9 ERROR: "/main" not found: not found
#8 [5/6] COPY ./configs /configs
#8 sha256:630f272dd60dd307f40dbbdaef277ee0dfc24b71fa11e10a3b8efd64d3c05086
#8 ERROR: "/configs" not found: not found
#4 [2/6] RUN mkdir -p /src/build
#4 sha256:b9fa3007a44471d47414dd29b3ff07ead6af28ede820a2b4bae0ce84cf2c5a83
#4 DONE 0.2s
------
> [5/6] COPY ./configs /configs:
------
------
> [6/6] COPY main /main:
------
failed to compute cache key: "/main" not found: not found
PS: I am not able to find where is the problem? Help Please
The two folders /main and /configs does not exist.
The COPY command can't copy into this folders.
1. Solution
Create the folders on build
RUN mkdir -p /main
RUN mkdir -p /configs
And than use COPY
2. Solution
Try to build without COPY and CMD
Than run the the new image
exec into running container with bash or sh
Create the folders
Exit exec container
Create a new image of the running container with docker run commit
Stop the container and delete it
Build again with your new image and include COPY and CMD
This is a basic mistake.
COPY ./configs /configs: copy the folder configs from the host to the Docker image.
COPY main /main: copy the executable file main from the host to the Docker image.
The problems are:
The base Docker images do not have these folders /configs, /main. You must create them manually (Docker understood your command this way).
But I have some advice:
Create 2 Docker images for 2 purposes: build, production.
Copy the source code into Docker builder image which is use for building your app.
Copy necessary output files from the Docker builder image into the Docker production image.
In my case, the issue was the connected vpn/proxy network from my machine.
It worked after I disconnecting the vpn/proxy network.
In my case I missed the folder entry in .dockerignore file. Do something like that.
**/*
!docker-images
!configs
!main
It's commonly known that you can run docker commit against a failed build process to take a snapshot of a container for debugging purposes. The container ID is gleaned from the running in <ID> text. However, this text is not emitted during builds that happen with Docker's newer BuildKit buildx functionality.
I tried using --progress plain on the Docker build command, but that hasn't shown me the container IDs. Plus, I cannot run a new container from the image layer IDs (SHA hashes) that are spit out.
Sample BuildKit Output
Using this command:
#1 [internal] load build definition from Dockerfile
#1 sha256:0e70418d547c3ccb20da7b100cf4f69564bddc416652e3e2b9b514e9a732b4aa
#1 transferring dockerfile: 32B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:396b2cfd81ff476a70ecda27bc5d781bd61c859b608537336f8092e155dd38bf
#2 transferring context: 34B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/node:latest
#3 sha256:1c0b05b884068c98f7acad32e4f7fd374eba1122b4adcbb1de68aa72d5a6046f
#3 DONE 0.0s
#4 [1/4] FROM docker.io/library/node
#4 sha256:5045d46e15358f34ea7fff145af304a1fa3a317561e9c609f4ae17c0bd3359df
#4 DONE 0.0s
#5 [internal] load build context
#5 sha256:49d7a085caed3f75e779f05887e53e0bba96452e3a719963993002a3638cb8a3
#5 transferring context: 35.17kB 0.0s done
#5 DONE 0.1s
#6 [2/4] ADD [trevortest/*, /app/]
#6 sha256:6da32965a50f6e13322efb20007ff49fb0546e2ff55799163b3b00d034a62c57
#6 CACHED
Question: How can I obtain the container IDs of the build process, during each step, specifically when using Docker BuildKit?
The BuildKit works differently than the legacy docker build system. At the moment, there is no direct way to spawn a container from a step in the build and troubleshoot it.
To use the BuildKit potential up to the maximum, best approach is to organize the builds in smaller logical stages. Once the build is organized in this way, When running the builds, you can specify that you want to stop at a certain stage by using --target. When the target is specified, Docker creates an image with the results of the build up to that stage. You can use this container to further troubleshoot in the same way that was possible with the old build system.
Take this example. Here I have 4 stages out of which 2 are parallel stages:
FROM debian:9.11 AS stage-01
# Prepare for installation
RUN apt update && \
apt upgrade -y
FROM stage-01 as stage-02
# Install building tools
RUN apt install -y build-essential
FROM stage-02 as stage-02a
RUN echo "Build 0.1" > /version.txt
FROM stage-02 as stage-03
RUN apt install -y cmake gcc g++
Now you can use the --target option to tell Docker that you want to stop at the stage-02 as follows:
$ docker build -f test-docker.Dockerfile -t test . --target stage-02 [+] Building 67.5s (7/7) FINISHED
=> [internal] load build definition from test-docker.Dockerfile 0.0s
=> => transferring dockerfile: 348B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/debian:9.11 0.0s
=> [stage-01 1/2] FROM docker.io/library/debian:9.11 0.0s
=> CACHED [stage-01 2/2] RUN apt update && apt upgrade -y 0.0s
=> [stage-02 1/1] RUN apt install -y build-essential 64.7s
=> exporting to image 2.6s
=> => exporting layers 2.5s
=> => writing image sha256:ac36b95184b79b6cabeda3e4d7913768f6ed73527b76f025262d6e3b68c2a357 0.0s
=> => naming to docker.io/library/test 0.0s
Now you have the image with the name test and you can spawn a container to troubleshoot.
docker run -ti --rm --name troubleshoot test /bin/bash
root#bbdb0d2188c0:/# ls
Using multiple stages facilitates the troubleshooting, however it really speeds up the build process since the parallel branches can be build on different instances. Also, the readability of the build file is significantly improved.
It's commonly known that you can run docker commit against a failed build process to take a snapshot of a container for debugging purposes. The container ID is gleaned from the running in <ID> text. However, this text is not emitted during builds that happen with Docker's newer BuildKit buildx functionality.
I tried using --progress plain on the Docker build command, but that hasn't shown me the container IDs. Plus, I cannot run a new container from the image layer IDs (SHA hashes) that are spit out.
Sample BuildKit Output
Using this command:
#1 [internal] load build definition from Dockerfile
#1 sha256:0e70418d547c3ccb20da7b100cf4f69564bddc416652e3e2b9b514e9a732b4aa
#1 transferring dockerfile: 32B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:396b2cfd81ff476a70ecda27bc5d781bd61c859b608537336f8092e155dd38bf
#2 transferring context: 34B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/node:latest
#3 sha256:1c0b05b884068c98f7acad32e4f7fd374eba1122b4adcbb1de68aa72d5a6046f
#3 DONE 0.0s
#4 [1/4] FROM docker.io/library/node
#4 sha256:5045d46e15358f34ea7fff145af304a1fa3a317561e9c609f4ae17c0bd3359df
#4 DONE 0.0s
#5 [internal] load build context
#5 sha256:49d7a085caed3f75e779f05887e53e0bba96452e3a719963993002a3638cb8a3
#5 transferring context: 35.17kB 0.0s done
#5 DONE 0.1s
#6 [2/4] ADD [trevortest/*, /app/]
#6 sha256:6da32965a50f6e13322efb20007ff49fb0546e2ff55799163b3b00d034a62c57
#6 CACHED
Question: How can I obtain the container IDs of the build process, during each step, specifically when using Docker BuildKit?
The BuildKit works differently than the legacy docker build system. At the moment, there is no direct way to spawn a container from a step in the build and troubleshoot it.
To use the BuildKit potential up to the maximum, best approach is to organize the builds in smaller logical stages. Once the build is organized in this way, When running the builds, you can specify that you want to stop at a certain stage by using --target. When the target is specified, Docker creates an image with the results of the build up to that stage. You can use this container to further troubleshoot in the same way that was possible with the old build system.
Take this example. Here I have 4 stages out of which 2 are parallel stages:
FROM debian:9.11 AS stage-01
# Prepare for installation
RUN apt update && \
apt upgrade -y
FROM stage-01 as stage-02
# Install building tools
RUN apt install -y build-essential
FROM stage-02 as stage-02a
RUN echo "Build 0.1" > /version.txt
FROM stage-02 as stage-03
RUN apt install -y cmake gcc g++
Now you can use the --target option to tell Docker that you want to stop at the stage-02 as follows:
$ docker build -f test-docker.Dockerfile -t test . --target stage-02 [+] Building 67.5s (7/7) FINISHED
=> [internal] load build definition from test-docker.Dockerfile 0.0s
=> => transferring dockerfile: 348B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/debian:9.11 0.0s
=> [stage-01 1/2] FROM docker.io/library/debian:9.11 0.0s
=> CACHED [stage-01 2/2] RUN apt update && apt upgrade -y 0.0s
=> [stage-02 1/1] RUN apt install -y build-essential 64.7s
=> exporting to image 2.6s
=> => exporting layers 2.5s
=> => writing image sha256:ac36b95184b79b6cabeda3e4d7913768f6ed73527b76f025262d6e3b68c2a357 0.0s
=> => naming to docker.io/library/test 0.0s
Now you have the image with the name test and you can spawn a container to troubleshoot.
docker run -ti --rm --name troubleshoot test /bin/bash
root#bbdb0d2188c0:/# ls
Using multiple stages facilitates the troubleshooting, however it really speeds up the build process since the parallel branches can be build on different instances. Also, the readability of the build file is significantly improved.
It's commonly known that you can run docker commit against a failed build process to take a snapshot of a container for debugging purposes. The container ID is gleaned from the running in <ID> text. However, this text is not emitted during builds that happen with Docker's newer BuildKit buildx functionality.
I tried using --progress plain on the Docker build command, but that hasn't shown me the container IDs. Plus, I cannot run a new container from the image layer IDs (SHA hashes) that are spit out.
Sample BuildKit Output
Using this command:
#1 [internal] load build definition from Dockerfile
#1 sha256:0e70418d547c3ccb20da7b100cf4f69564bddc416652e3e2b9b514e9a732b4aa
#1 transferring dockerfile: 32B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:396b2cfd81ff476a70ecda27bc5d781bd61c859b608537336f8092e155dd38bf
#2 transferring context: 34B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/node:latest
#3 sha256:1c0b05b884068c98f7acad32e4f7fd374eba1122b4adcbb1de68aa72d5a6046f
#3 DONE 0.0s
#4 [1/4] FROM docker.io/library/node
#4 sha256:5045d46e15358f34ea7fff145af304a1fa3a317561e9c609f4ae17c0bd3359df
#4 DONE 0.0s
#5 [internal] load build context
#5 sha256:49d7a085caed3f75e779f05887e53e0bba96452e3a719963993002a3638cb8a3
#5 transferring context: 35.17kB 0.0s done
#5 DONE 0.1s
#6 [2/4] ADD [trevortest/*, /app/]
#6 sha256:6da32965a50f6e13322efb20007ff49fb0546e2ff55799163b3b00d034a62c57
#6 CACHED
Question: How can I obtain the container IDs of the build process, during each step, specifically when using Docker BuildKit?
The BuildKit works differently than the legacy docker build system. At the moment, there is no direct way to spawn a container from a step in the build and troubleshoot it.
To use the BuildKit potential up to the maximum, best approach is to organize the builds in smaller logical stages. Once the build is organized in this way, When running the builds, you can specify that you want to stop at a certain stage by using --target. When the target is specified, Docker creates an image with the results of the build up to that stage. You can use this container to further troubleshoot in the same way that was possible with the old build system.
Take this example. Here I have 4 stages out of which 2 are parallel stages:
FROM debian:9.11 AS stage-01
# Prepare for installation
RUN apt update && \
apt upgrade -y
FROM stage-01 as stage-02
# Install building tools
RUN apt install -y build-essential
FROM stage-02 as stage-02a
RUN echo "Build 0.1" > /version.txt
FROM stage-02 as stage-03
RUN apt install -y cmake gcc g++
Now you can use the --target option to tell Docker that you want to stop at the stage-02 as follows:
$ docker build -f test-docker.Dockerfile -t test . --target stage-02 [+] Building 67.5s (7/7) FINISHED
=> [internal] load build definition from test-docker.Dockerfile 0.0s
=> => transferring dockerfile: 348B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/debian:9.11 0.0s
=> [stage-01 1/2] FROM docker.io/library/debian:9.11 0.0s
=> CACHED [stage-01 2/2] RUN apt update && apt upgrade -y 0.0s
=> [stage-02 1/1] RUN apt install -y build-essential 64.7s
=> exporting to image 2.6s
=> => exporting layers 2.5s
=> => writing image sha256:ac36b95184b79b6cabeda3e4d7913768f6ed73527b76f025262d6e3b68c2a357 0.0s
=> => naming to docker.io/library/test 0.0s
Now you have the image with the name test and you can spawn a container to troubleshoot.
docker run -ti --rm --name troubleshoot test /bin/bash
root#bbdb0d2188c0:/# ls
Using multiple stages facilitates the troubleshooting, however it really speeds up the build process since the parallel branches can be build on different instances. Also, the readability of the build file is significantly improved.