Rebuilding Docker image taking time - docker

Dockerfile has the following content,
FROM node:16.4.2-alpine3.14
WORKDIR /app
COPY package.json .
COPY . /app
And ran the following build command,
docker build -t app:0.1 .
It took 28.4 seconds and below is the terminal logs,
[+] Building 28.4s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 124B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 53B 0.0s
=> [internal] load metadata for docker.io/library/node:16.4.2-alpine3.14 17.1s
=> [auth] library/node:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 2.01MB 0.0s
=> [1/4] FROM docker.io/library/node:16.4.2-alpine3.14#sha256:fabfca5e7dcb339097f998d6ef11c53dd80a3f99ed5cecc005e93d0ff6d4bda9 9.9s
=> => resolve docker.io/library/node:16.4.2-alpine3.14#sha256:fabfca5e7dcb339097f998d6ef11c53dd80a3f99ed5cecc005e93d0ff6d4bda9 0.0s
=> => sha256:fabfca5e7dcb339097f998d6ef11c53dd80a3f99ed5cecc005e93d0ff6d4bda9 1.00kB / 1.00kB 0.0s
=> => sha256:75dec02064547a8ec570f2953e8d68a1674ad3f37730160f1570cce077be9ed0 1.16kB / 1.16kB 0.0s
=> => sha256:40cb916373b08a087466d2e72402d0b3a4587fd3e9135169498cf0db4ff42a88 6.53kB / 6.53kB 0.0s
=> => sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b 2.81MB / 2.81MB 0.8s
=> => sha256:c118dce16b0057d713fc98e31606a84e4348fa2c967eaf1bb5fd21ba42825956 35.55MB / 35.55MB 7.1s
=> => sha256:aef8e8137ac43c8199343c96874993063af6584260f22b15e99f735cce5de653 2.35MB / 2.35MB 2.6s
=> => extracting sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b 0.2s
=> => sha256:ad336e0e52b8dfc38c23599663deb060b1ac169d548dec8072ead94712f708be 281B / 281B 2.0s
=> => extracting sha256:c118dce16b0057d713fc98e31606a84e4348fa2c967eaf1bb5fd21ba42825956 2.0s
=> => extracting sha256:aef8e8137ac43c8199343c96874993063af6584260f22b15e99f735cce5de653 0.2s
=> => extracting sha256:ad336e0e52b8dfc38c23599663deb060b1ac169d548dec8072ead94712f708be 0.0s
=> [2/4] WORKDIR /app 0.6s
=> [3/4] COPY package.json . 0.1s
=> [4/4] COPY . /app 0.1s
=> exporting to image 0.2s
=> => exporting layers 0.2s
=> => writing image sha256:91d93eddff55cba6bd8b72144b7320e025de93e9865177ff584c75b94d1bafc1 0.0s
=> => naming to docker.io/library/app:0.1
When I run the same build command again, it is taking 14.6 seconds.
However if I pull the node:16.4.2-alpine3.14 using,
docker pull node:16.4.2-alpine3.14
and then run the build command, then build takes only 0.3 seconds
I think, when we build an image, dependencies also get downloaded and that is why the time taken reduces from 28.4 to 14.6 seconds. But why even 14.6 seconds? It should be as less as 0.3 seconds.
Why is this so? What am I missing?

The following could be the reasons it takes 18.4 seconds:
Docker client takes the entire build context to the docker daemon.The build context is the entire directory the Dockerfile.some files or folders can take up a lot of space e.g node_modules. A remedy to this is to add the file that is not required eg .git, node_module, log files to the .dockerignore file to get Docker to ignore some files.
DNS resolution. Check how long it takes to resolve the docker registry using the dig command.
Enabling the buildkit for your build will help with improving the build time. Setting the DOCKER_BUILDKIT=1 environment variable when invoking the docker build command such as:
DOCKER_BUILDKIT=1 docker build .
please look at this link for more information

Related

BuildKit: RUN mount secret not mounted, but no error message?

This is a MWE where I try to expose a file called abc within RUN as a secret:
Dockerfile:
# syntax = docker/dockerfile:1.3
FROM alpine:latest
RUN --mount=type=secret,id=aaa,target=/tmp/aaa ls -l /tmp/aaa
Then I run echo 123 > aaa (that's the file that will be exposed) and docker build (or DOCKER_BUILDKIT=1 docker build if that is not the default), but the secret is not to be found. I will be grateful for some help with this (I am running the latest Docker release 20.10.21):
[+] Building 3.1s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 157B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.3 0.5s
=> CACHED docker-image://docker.io/docker/dockerfile:1.3#sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load .dockerignore 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 1.5s
=> [auth] library/alpine:pull token for registry-1.docker.io 0.0s
=> [stage-0 1/2] FROM docker.io/library/alpine:latest#sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 0.5s
=> => resolve docker.io/library/alpine:latest#sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 0.0s
=> => sha256:8914eb54f968791faf6a8638949e480fef81e697984fba772b3976835194c6d4 1.64kB / 1.64kB 0.0s
=> => sha256:c0d488a800e4127c334ad20d61d7bc21b4097540327217dfab52262adc02380c 528B / 528B 0.0s
=> => sha256:49176f190c7e9cdb51ac85ab6c6d5e4512352218190cd69b08e6fd803ffbf3da 1.47kB / 1.47kB 0.0s
=> => sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 3.37MB / 3.37MB 0.3s
=> => extracting sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 0.1s
=> ERROR [stage-0 2/2] RUN --mount=type=secret,id=aaa,target=/tmp/aaa ls -l /tmp/aaa 0.4s
------
> [stage-0 2/2] RUN --mount=type=secret,id=aaa,target=/tmp/aaa ls -l /tmp/aaa:
#10 0.314 ls: /tmp/aaa: No such file or directory
------
executor failed running [/bin/sh -c ls -l /tmp/aaa]: exit code: 1

NerdCTL will not use local image when building

I am trying to build a common base image, that many of my other images will use. But Nerdctl/Containerd doesn't seem to check for the image locally first.
Reproduced with the simple example
# Dockerfile.base
FROM python:3
# Dockerfile.child
From mybase:local
OUTPUT:
PS C:\<REDACTED>\local_build_issue> nerdctl build -t mybase:local -f Dockerfile.base .
[+] Building 10.1s (5/5) FINISHED
=> [internal] load build definition from Dockerfile.base 0.1s
=> => transferring dockerfile: 55B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3 0.9s
=> CACHED [1/1] FROM docker.io/library/python:3#sha256:555f5affd32250ca74758b297f262fa8f421eb0102877596b48c0b8b464606ea 0.0s
=> => resolve docker.io/library/python:3#sha256:555f5affd32250ca74758b297f262fa8f421eb0102877596b48c0b8b464606ea 0.0s
=> exporting to oci image format 9.0s
=> => exporting layers 0.0s
=> => exporting manifest sha256:6bfdb5e41f7a1f0faf28f922e48cdd33e95b75cff958709cb7945291d34dffda 0.0s
=> => exporting config sha256:d449529a9f8347e81301101d04645624c6a52d71b0c6b85abfd8da6c65b19e31 0.0s
=> => sending tarball 9.0s
unpacking docker.io/library/mybase:local (sha256:6bfdb5e41f7a1f0faf28f922e48cdd33e95b75cff958709cb7945291d34dffda)...done
PS C:\<REDACTED>\local_build_issue> nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
mybase local 6bfdb5e41f7a About a minute ago linux/amd64 945.8 MiB 334.7 MiB
PS C:\<REDACTED>\local_build_issue> nerdctl build -t mychild:local -f Dockerfile.child .
[+] Building 0.5s (3/3) FINISHED
=> [internal] load build definition from Dockerfile.child 0.1s
=> => transferring dockerfile: 62B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> ERROR [internal] load metadata for docker.io/library/mybase:local 0.4s
------
> [internal] load metadata for docker.io/library/mybase:local:
------
Dockerfile.child:1
--------------------
1 | >>> FROM mybase:local
2 |
--------------------
error: failed to solve: mybase:local: pull access denied, repository does not exist or may require authorization: authorization status: 401: authorization failed
FATA[0000] unrecognized image format
I can't figure out how to get it to NOT look for the image on docker.io. All the recommendations for the docker cli do not seem apply or work.
it actually can't pull image locally first .
the nerdctl build image use the buildkitd tools.it has two types of backends.
you can find the answer from https://github.com/containerd/nerdctl/blob/master/docs/build.md

Docker errors when trying to build in ARM64 Apple M1: "Failed to resolve full path of the current executable [/proc/self/exe]"

I'm having trouble building docker containers on an Apple M1
The project uses sdk 2.2 which is incompatible with arm64 architecture. So I changed the sdk and aspnet cores to 2.2-alpine3.8 and they seem to build ok but the process fails when it needs to publish
The docker file:
FROM mcr.microsoft.com/dotnet/core/sdk:2.2-alpine3.8 AS build-env
WORKDIR /app
# Copy csproj and restore as distinct layers
COPY . .
RUN dotnet publish src/admin/MyContainer.Admin.csproj -c Release -o ./publish
# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2-alpine3.8
WORKDIR /app
COPY --from=build-env /app/src/admin/publish .
# Copy script that allows waiting for the database on docker-compose
# As referenced here https://docs.docker.com/compose/startup-order/
# This was done because admin starts up very quickly on e2e env
# and the sqlserver container wasn't getting ready fast enough
COPY --from=build-env /app/docker-wait-for-it.sh .
EXPOSE 80
CMD ["dotnet", "MyContainer.Admin.dll"]
the build command I'm using is:
docker buildx build --platform linux/arm64/v8 -t MyContainer.azurecr.io/admin-api-web:development ./backend -f ./backend/Admin.Api.Web.Dockerfile
and the log output is:
[+] Building 0.5s (11/13)
=> [internal] load build definition from Admin.Api.Web.Dockerfile 0.0s
=> => transferring dockerfile: 46B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 34B 0.0s
=> [internal] load metadata for mcr.microsoft.com/dotnet/core/aspnet:2.2-alpine3.8 0.1s
=> [internal] load metadata for mcr.microsoft.com/dotnet/core/sdk:2.2-alpine3.8 0.1s
=> [build-env 1/4] FROM mcr.microsoft.com/dotnet/core/sdk:2.2-alpine3.8#sha256:1299ac0379146c1de7b588196727dbd56eace3abfbdd4321c547c9ff4a18a2f7 0.0s
=> => resolve mcr.microsoft.com/dotnet/core/sdk:2.2-alpine3.8#sha256:1299ac0379146c1de7b588196727dbd56eace3abfbdd4321c547c9ff4a18a2f7 0.0s
=> [stage-1 1/4] FROM mcr.microsoft.com/dotnet/core/aspnet:2.2-alpine3.8#sha256:4d6e528f4c09c55804b6032ecc5d60565a3ee16f68bb08d2cf337dff99cdb8c3 0.0s
=> => resolve mcr.microsoft.com/dotnet/core/aspnet:2.2-alpine3.8#sha256:4d6e528f4c09c55804b6032ecc5d60565a3ee16f68bb08d2cf337dff99cdb8c3 0.0s
=> [internal] load build context 0.2s
=> => transferring context: 473.91kB 0.1s
=> CACHED [stage-1 2/4] WORKDIR /app 0.0s
=> CACHED [build-env 2/4] WORKDIR /app 0.0s
=> CACHED [build-env 3/4] COPY . . 0.0s
=> ERROR [build-env 4/4] RUN dotnet publish src/admin/MyContainer.Admin.csproj -c Release -o ./publish 0.1s
------
> [build-env 4/4] RUN dotnet publish src/admin/MyContainer.Admin.csproj -c Release -o ./publish:
#11 0.127 Failed to resolve full path of the current executable [/proc/self/exe]
I've tried with different platforms on the --platform flag and without any specified as well... If I try to build with core 2.2 I get this output:
[+] Building 0.5s (4/4) FINISHED
=> [internal] load build definition from Admin.Api.Web.Dockerfile 0.0s
=> => transferring dockerfile: 745B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 34B 0.0s
=> ERROR [internal] load metadata for mcr.microsoft.com/dotnet/core/aspnet:2.2 0.4s
=> CANCELED [internal] load metadata for mcr.microsoft.com/dotnet/core/sdk:2.2 0.4s
------
> [internal] load metadata for mcr.microsoft.com/dotnet/core/aspnet:2.2:
------
error: failed to solve: failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest sha256:08xxx: not found
Which I've found out to be because core 2.2 has no support form arm64. But this file builds without any errors on an Intel MacBook but with M1 I can't get past the build error. Any ideas?
This is not an answer but a followup to the problem,
which might shed some more light on the issue.
I'm using aspnet:6 and it passes the build.
The problem later on is that the runnable .dll throws
cannot execute binary file: Exec format error
which seems to be a platform issue, using Mac with an M1 Silicon chipset.
I have no issue with other docker images that are made for linux/amd64
The Docker daemon seems to deal with this fine (having installed Rosetta2 and the Latest Docker Desktop for M1)
https://docs.docker.com/desktop/mac/apple-silicon/
Have you managed to resolve the issue and execute the dll?

file not found error when running container

I have a test automation project which gets uses the code built as part of jar file and that jar gets invoked via bat file. All these files are stored within my project folder.
contents of my Docker file:
FROM maven:3.8.1-adoptopenjdk-11
#WORKDIR C:/Work/Kickstart_TEM/Prefs
COPY Prefs /home/Prefs
COPY KickStart.jar /home/Prefs/KickStart.jar
CMD home\prefs\run.bat && cmd
docker build generates following output
[+] Building 0.3s (8/8) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 210B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/maven:3.8.1-adoptopenjdk-11 0.0s
=> [1/3] FROM docker.io/library/maven:3.8.1-adoptopenjdk-11 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 390B 0.0s
=> CACHED [2/3] COPY Prefs /home/Prefs 0.0s
=> CACHED [3/3] COPY KickStart.jar /home/Prefs/KickStart.jar 0.0s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:4c878e8a895b2fad307e00f1b2fb5c9b5df7dc630e87414230d1989b75a5ee17 0.0s
=> => naming to docker.io/library/demo2
Docker run generates following error:
PS C:\Work\Docker_POC> docker run -i -p 4044:4044 demo2
/bin/sh: 1: homeprefsrun.bat: not found
My containers stops right away, so I am not even able to figure out if my files and folders got copied successfully or not. And I am unsure of how to resolve this error.
First of all, you're trying to run a batch script under Linux (the docker image you're using determines this).
In general, your CMD statement should look like CMD ["/bin/sh", "-c", "/home/Prefs/run.sh && cmd"] (although I'm not sure what cmd is and why you want to run it)
You should convert this batch script (run.bat) to a shell script. Also, there is a difference between home and /home and filenames are case-sensitive (thus it's Prefs and not prefs).

Why are ARGS specified in `docker-compose.yml` available in `Dockerfile` for `docker compose run` but not `docker compose build`?

Relatively new to Docker and Compose, but I have read every letter of the Docker Compose documentation, and unsuccessfully bounced around SO for hours, with no resolution to the above question.
I have an (example) directory with the following files:
./Dockerfile:
# syntax=docker/dockerfile:1
ARG CUSTOM_NODE_VERSION
FROM node:$CUSTOM_NODE_VERSION
ARG CUSTOM_NODE_VERSION
ARG HELLO
RUN echo "HELLO: -> $HELLO"
RUN echo "NODE_VERSION -> $NODE_VERSION"
RUN echo "CUSTOM_NODE_VERSION -> $CUSTOM_NODE_VERSION"
./docker-compose.yml:
version: "3.8"
services:
test:
build:
context: .
dockerfile: Dockerfile
args:
CUSTOM_NODE_VERSION: alpine
HELLO: 5
What I want is for docker compose build to use the args specified in the docker-compose.yml file, but it doesn't:
> docker compose build test
[+] Building 0.8s (4/4) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.5s
=> CACHED docker-image://docker.io/docker/dockerfile:1#sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2 0.0s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to create LLB definition: failed to parse stage name "node:": invalid reference format
But run works fine:
docker compose run test
[+] Running 1/1
⠿ Network compose-args_default Created 4.3s
[+] Building 3.1s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.5s
=> CACHED docker-image://docker.io/docker/dockerfile:1#sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2 0.0s
=> [internal] load metadata for docker.io/library/node:alpine 1.0s
=> CACHED [1/4] FROM docker.io/library/node:alpine#sha256:f372a9ffcec27159dc9623bad29997a1b61eafbb145dbf4f7a64568b 0.0s
=> [2/4] RUN echo "HELLO: -> 5" 0.5s
=> [3/4] RUN echo "NODE_VERSION -> 16.3.0" 0.3s
=> [4/4] RUN echo "CUSTOM_NODE_VERSION -> alpine" 0.5s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:e61653277599e3555b67c1a50699dd83d5c1ed1a93fe8a1a16529c6ec20e3e31 0.0s
=> => naming to docker.io/library/compose-args_test
This is all the more baffling to me because, per the Compose docs:
args
Add build arguments, which are environment variables accessible only during the build process.
Any help is much appreciated 🙏🏻
Build args were only recently added to compose-cli. Most likely that change hasn't reached the version of docker compose you're running. You can use docker-compose build (with a -) until this feature reaches your install.

Resources