Docker cache invalidation - docker

I'm having some weird issues with my custom Dockerfile, compiling a .Net core app in alpine containers.
I've tried numerous different configurations to no avail - cache is ALWAYS invalidated when I implement the final FROM instruction (if I comment that and everything below it out, caching works fine). Here's the file:
FROM microsoft/dotnet:2.1-sdk-alpine3.7 AS build
ARG ASPNETCORE_ENVIRONMENT=development
ARG ASPNET_CONFIGURATION=Debug
ARG PROJECT_DIR=src/API/
ARG PROJECT_NAME=MyAPI
ARG SOLUTION_NAME=MySolution
RUN export
WORKDIR /source
COPY ./*.sln ./nuget.config ./
# Copy source project files
COPY src/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done
# # Copy test project files
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done
RUN dotnet restore
COPY . ./
RUN for dir in test/*.Tests/; do (cd "$dir" && dotnet test --filter TestType!=Integration); done
WORKDIR /source/${PROJECT_DIR}
RUN dotnet build ${PROJECT_NAME}.csproj -c $ASPNET_CONFIGURATION -o /app
RUN dotnet publish ${PROJECT_NAME}.csproj -c $ASPNET_CONFIGURATION -o /app --no-restore
FROM microsoft/dotnet:2.1-aspnetcore-runtime-alpine3.7
ARG ASPNETCORE_ENVIRONMENT=development
RUN export
COPY --from=build /app .
WORKDIR /app
EXPOSE 80
VOLUME /app/logs
ENTRYPOINT ["dotnet", "MyAssembly.dll"]
Any ideas? Hints? Tips? Blazingly obvious mistakes? I've checked each layer and the COPY . ./ instruction ONLY copies the files I expect it to - and none of them change between builds.
Its also worth noting that if I remove the last FROM instruction (and other relevant lines) the cache works perfectly - but the final image size is obviously considerably bigger than the base microsoft/dotnet:2.1-aspnetcore-runtime-alpine3.7 (172Mb vs 1.8Gb) image. I have tried just commenting out the COPY instruction after the FROM, but it doesn't affect the cache invalidation. The following works as expected:
FROM microsoft/dotnet:2.1-sdk-alpine3.7 AS build
ARG ASPNETCORE_ENVIRONMENT=development
ARG ASPNET_CONFIGURATION=Debug
ARG PROJECT_DIR=src/API/
ARG PROJECT_NAME=MyAPI
ARG SOLUTION_NAME=MySolution
RUN export
WORKDIR /source
COPY ./*.sln ./nuget.config ./
# Copy source project files
COPY src/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done
# # Copy test project files
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done
RUN dotnet restore
COPY . ./
RUN for dir in test/*.Tests/; do (cd "$dir" && dotnet test --filter TestType!=Integration); done
WORKDIR /source/${PROJECT_DIR}
RUN dotnet build ${PROJECT_NAME}.csproj -c $ASPNET_CONFIGURATION -o /app
RUN dotnet publish ${PROJECT_NAME}.csproj -c $ASPNET_CONFIGURATION -o /app --no-restore
WORKDIR /app
EXPOSE 80
VOLUME /app/logs
ENTRYPOINT ["dotnet", "MyAssembly.dll"]
.dockerignore below:
base-images/
docker-compose.yml
docker-compose.*.yml
VERSION
**/.*
**/*.ps1
**/*.DotSettings
**/*.csproj.user
**/*.md
**/*.log
**/*.sh
**/Dockerfile
**/bin
**/obj
**/node_modules
**/.vs
**/.vscode
**/dist
**/packages/
**/wwwroot/
Last bit of info: I'm building containers using docker-compose - specifically by running docker-compose build myservicename, but building the image with docker build -f src/MyAssembly/Dockerfile -t MyImageName . yields the same results.

If you're building locally and the cache isn't working – then I don't know what the issue is :)
But if you're building as part of CI – then the issue may be that you need to pull, build, and push the intermediate stage explicitly:
> docker pull MyImageName:build || true
> docker pull MyImageName:latest || true
> docker build --target build --tag MyImageName:build .
> docker build --cache-from MyImageName:build --tag MyImageName:latest .
> docker push MyImageName:build
> docker push MyImageName:latest
The || true part is there because the images won't be there on the initial CI build. The "magic sauce" of this recipe is docker build --target <intermediate-stage-name> and docker build --cache-from <intermediate-stage-name>.
I can't explain why building and pushing the intermediate stage explicitly is needed to get the cache to work – other than some handwaving about only the final image gets pushed, and not the intermediate stage and its layers. But it worked for me – I learned this "trick" from here: https://pythonspeed.com/articles/faster-multi-stage-builds/

Related

dockerfile COPY does not copy all the files

I do
git clone https://github.com/openzipkin/zipkin.git
cd zipkin
The create a Dockerfile as below
FROM openjdk
RUN mkdir app
WORKDIR /app
COPY ./ .
ENTRYPOINT ["sleep", "1000000"]
then
docker build -t abc .
docker run abc
I then run docker exec -it CONTAINER_ID bash
pwd returns /app which is expected
but I ls and see that the files are not copied
only the directories and the xml file is copied into the /app directory
What is the reason? how to fix it?
Also I tried
FROM openjdk
RUN mkdir app
WORKDIR /app
COPY . /app
ENTRYPOINT ["sleep", "1000000"]
That repository contains a .dockerignore file which excludes everything except a set of things it selects.
That repository's docker directory also contains several build scripts for official images and you may find it easier to start your custom image FROM openzipkin/zipkin rather than trying to reinvent it.

how to compile golang project with docker volume and download private module from bitbucket?

how to compile golang project with docker container? I want to compile and retrieve the project build, using docker cli
sample docker command
docker run -v $(pwd)/:/app -w /app -v $SSH_AUTH_SOCK:/tmp/ssh_auth.sock -e SSH_AUTH_SOCK=/tmp/ssh_auth.sock --name golanguser golang:1.17 sh -c "export GOPRIVATE=https://user:pass#bitbucket.org/repo/ && go build -o main bitbucket.org/repo/source"
Building a go project is usually done in a multi-stage build
# syntax=docker/dockerfile:1
##
## Build
##
FROM golang:1.16-buster AS build
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /docker-gs-ping
##
## Deploy
##
FROM gcr.io/distroless/base-debian10
WORKDIR /
COPY --from=build /docker-gs-ping /docker-gs-ping
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/docker-gs-ping"]
With:
docker build -t docker-gs-ping:multistage -f Dockerfile.multistage .
That way, you can deploy the built application in the image of your choice, resulting in a markedly smaller image size to run.

Create docker container to run `go test` with all module dependencies downloaded and cached

I want to test my Go code in a CI environment which requires using Docker. How do I create a Docker image that has all the dependencies listed in go.mod already downloaded and compiled so that docker run $IMG go test uses the cached artifacts?
The desired properties of this image:
The image only uses go.mod to compile dependencies. I don't want to use the full source code because then any change to source code would invalidate the Docker layer that hold cached dependencies.
docker run $IMG go test ./... doesn't redownload or recompile dependencies listed in go.mod.
Avoid experimental Docker features.
Existing approaches
Parsing go.mod and using go get
From https://github.com/golang/go/issues/27719#issuecomment-578246826
This approach is close but it doesn't appear to use GOCACHE when I run go test. This also appears to choke on certain module paths, like gopkg.in/DataDog/dd-trace-go.v1:
FROM golang:1.13
WORKDIR /src
COPY go.mod ./
RUN set -eu \
&& go mod graph \
| cut -d '#' -f 1 \
| cut -d ' ' -f 2 \
| sort -u \
| sed -e 's#dd-trace-go.v1#&/ddtrace#' \
| xargs go get -v
docker run --mount /src:/src $IMG go test ./...
Using DOCKER_BUILDKIT with a mount cache
Originally described in https://github.com/golang/go/issues/27719#issuecomment-514747274. This only works for go build. I can't use it for go test because the cache mount is unmounted after the RUN command so it doesn't exist in the created Docker image.
This also depends on experimental docker features.
# syntax = docker/dockerfile:experimental
FROM golang:1.13 as go-builder
ARG VERSION
WORKDIR /src
COPY . /src/
# With a mount cache, Docker will cache the target directories for future
# invocations of this RUN layer. Meaning, once this command is run once, all
# successive calls will use the already downloaded and already compiled assets.
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go build ./server
I often put COPY go.mod on the very begin of a Dockerfile, as it does not change that often.
FROM golang:1.14.3 as builder
WORKDIR /app
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
RUN go build -tags netgo -ldflags '-extldflags "-static"' -o app .
FROM centos:7
WORKDIR /root
COPY --from=builder /app/app .
So, if you go mod does not change, the line RUN go mod download only run for the first time.
In your dockerfile if you run go get ./... it will download all the dependencies to your docker image based on the go.mod and go.sum. You can also add the --insecure flag to the go get command if you are dealing with internal self signed repositories. Very similar to go mod download. Aside from that you can have a shell script that actually initiates the Go test ./... and reports it out to your CI environment if that allows, I know that Gitlab allows this.
FROM golang:1.15-alpine AS builder
RUN apk add --update git gcc musl-dev
RUN apk update && apk add bash
COPY . /app
RUN go version
WORKDIR /app
ENV CGO_ENABLED=0
RUN git config --global http.sslVerify false
RUN go get ./...
WORKDIR /app
RUN chmod +x ./unitTest.sh
RUN ./unitTest.sh
WORKDIR /app/cmd/svr
RUN go build -o app
RUN chmod 700 app
FROM alpine:latest
WORKDIR /root/
ARG build_stamp
ARG git_commit
ARG build_number
ENV BUILD_STAMP=$build_stamp
ENV GIT_COMMIT=$git_commit
ENV BUILD_NUMBER=$build_number
COPY --from=builder /app/cmd/svr .
EXPOSE 8000
CMD ["./app"]
and the script
#!/usr/bin/env bash
TESTS=$(go test -v -covermode=count -coverprofile=count.txt ./...)
echo "$TESTS"
if echo "$TESTS" | grep -q "FAIL" ; then
echo ""
echo "One or more Unit Tests for app have Failed. Build will now fail. Pipeline will also fail..."
echo ""
exit 1
else
echo ""
echo "All Unit Tests for application have passed!"
echo "Running Code Coverage..."
echo ""
COVERAGE=$(go tool cover -func=./count.txt)
echo "$COVERAGE"
exit 0
fi

Docker multistage: how to COPY built files between stages?

I'm beginner with Docker, and I'm trying to build an image in two stages, as explained here: https://docs.docker.com/develop/develop-images/multistage-build/
You can selectively copy artifacts from one stage to another
Looking at the examples given there, I had thought that one could build some files during a first stage, and then make them available for the next one:
FROM golang:1.7.3 AS builder
WORKDIR /go/src/github.com/alexellis/href-counter/
RUN go get -d -v golang.org/x/net/html
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /go/src/github.com/alexellis/href-counter/app .
CMD ["./app"]
(Example taken from the above-linked page)
Isn't that what the COPY app.go . and the COPY --from=builder /go/src/github.com/alexellis/href-counter/app . are supposed to do?
I probably have a complete misunderstanding of what is going on, because when I try to do something similar (see below), it seems that the COPY command from the first stage is not able to see the files that have just been built (I can confirm that they have been actually built using a RUN ls step, but then I get a lstat <the file>: no such file or directory error).
And indeed, most other information I can gather regarding COPY (except the examples in the above link) rather suggest that COPY is actually meant to copy files from the directory where the docker build command was launched, not from within the build environment.
Here is my Dockerfile:
FROM haskell:8.6.5 as haskell
RUN git clone https://gitlab+deploy-token-75:sakyTxfe-PxPHDwqsoGm#gitlab.pasteur.fr/bli/bioinfo_utils.git
WORKDIR bioinfo_utils/remove-duplicates-from-sorted-fastq/Haskell
RUN stack --resolver ghc-8.6.5 build && \
stack --resolver ghc-8.6.5 install --local-bin-path .
RUN pwd; echo "---"; ls
COPY remove-duplicates-from-sorted-fastq .
FROM python:3.7-buster
RUN python3.7 -m pip install snakemake
RUN mkdir -p /opt/bin
COPY --from=haskell /bioinfo_utils/remove-duplicates-from-sorted-fastq/Haskell/remove-duplicates-from-sorted-fastq /opt/bin/remove-duplicates-from-sorted-fastq
CMD ["/bin/bash"]
And here is how the build ends when I run docker build . from the directory containing the Dockerfile:
Step 5/11 : RUN pwd; echo "---"; ls
---> Running in 28ff49fe9150
/bioinfo_utils/remove-duplicates-from-sorted-fastq/Haskell
---
LICENSE
Setup.hs
install.sh
remove-duplicates-from-sorted-fastq
remove-duplicates-from-sorted-fastq.cabal
src
stack.yaml
---> f551efc6bba2
Removing intermediate container 28ff49fe9150
Step 6/11 : COPY remove-duplicates-from-sorted-fastq .
lstat remove-duplicates-from-sorted-fastq: no such file or directory
How am I supposed to proceed to have the built file available for the next stage?
Well, apparently, I was mislead by the COPY step used in the first stage in the doc example. In my case, this is actually useless, and I can just COPY --from=haskell in my second stage, without any COPY in the first stage.
The following Dockerfile builds without issues:
FROM haskell:8.6.5 as haskell
RUN git clone https://gitlab+deploy-token-75:sakyTxfe-PxPHDwqsoGm#gitlab.pasteur.fr/bli/bioinfo_utils.git
WORKDIR bioinfo_utils/remove-duplicates-from-sorted-fastq/Haskell
RUN stack --resolver ghc-8.6.5 build && \
stack --resolver ghc-8.6.5 install --local-bin-path .
FROM python:3.7-buster
RUN python3.7 -m pip install snakemake
RUN mkdir -p /opt/bin
COPY --from=haskell /bioinfo_utils/remove-duplicates-from-sorted-fastq/Haskell/remove-duplicates-from-sorted-fastq /opt/bin/remove-duplicates-from-sorted-fastq
CMD ["/bin/bash"]

Issues with COPY when using multistage Dockerfile builds -- no such file or directory

I'm trying to convert my project to use multi-stage builds. However, the final step always fails with an error:
Step 11/13 : COPY --from=build /bin/grafana-server /bin/grafana-server
COPY failed: stat /var/lib/docker/overlay2/xxxx/merged/bin/grafana-server: no such file or directory
My Dockerfile looks like this:
FROM golang:latest AS build
ENV SRC_DIR=/go/src/github.com/grafana/grafana/
ENV GIT_SSL_NO_VERIFY=1
COPY . $SRC_DIR
WORKDIR $SRC_DIR
# Building of Grafana
RUN \
npm run build && \
go run build.go setup && \
go run build.go build
# Create final stage containing only required artifacts
FROM scratch
COPY --from=build /bin/grafana-server /bin/grafana-server
EXPOSE 3001
CMD ["./bin/grafana-server"]
The build.go build step will output artifacts to ./bin/ -- The error is pretty unhelpful other than telling me the files don't exist where I think they should exist.
My folder structure on my machine is:
--| ~/Documents/dev/grafana/src/grafana/grafana
--------| bin
------------| <grafan-server builds to here>
--------| deploy
------------| docker
----------------| Dockerfile
From ~/Documents/dev/grafana/src/grafana/grafana is where I issue: docker build -t grafana -f deploy/docker/Dockerfile .
To follow-up my comment, the path you set with the WORKDIR is absolute and should be specified in the same way in the COPY --from=build command.
So this could lead to the following Dockerfile:
FROM golang:latest AS build
ENV SRC_DIR=/go/src/github.com/grafana/grafana/
ENV GIT_SSL_NO_VERIFY=1
COPY . $SRC_DIR
WORKDIR $SRC_DIR
# Building of Grafana
RUN \
npm run build && \
go run build.go setup && \
go run build.go build
# Create final stage containing only required artifacts
FROM scratch
ENV SRC_DIR=/go/src/github.com/grafana/grafana/
WORKDIR $SRC_DIR
COPY --from=build ${SRC_DIR}/bin/grafana-server ${SRC_DIR}/bin/grafana-server
EXPOSE 3001
CMD ["./bin/grafana-server"]
(only partially tested)

Resources