I'm trying to make Dockerfile runs faster by copying whole directory (including vendor, because redownloading dependencies took about 10m+ in 3rd world country where I live), but when I tried to run it, it always redownload vendor again and again, unlike when go mod vendor in local:
FROM golang:1.14-alpine AS builder
RUN apk --update add ca-certificates git make g++
ENV GO111MODULE=on
WORKDIR /app
RUN go get github.com/go-delve/delve/cmd/dlv
COPY . .
RUN go mod vendor
ARG COMMIT_HASH
ENV COMMIT_HASH=${COMMIT_HASH}
ARG BUILD_DATE
ENV BUILD_DATE=${BUILD_DATE}
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
go build \
-o app
FROM golang:1.14-alpine
WORKDIR /app
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /go/bin/dlv /
COPY --from=builder /app/app .
COPY --from=builder /app/db ./db
EXPOSE 8080 63342
CMD [ "/dlv", "--listen=:63342", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./app" ]
previously using this (without vendor) also slow:
COPY go.mod .
COPY go.sum .
RUN go mod download -x
COPY . .
trying with this also didn't work:
COPY vendor /go/pkg/mod
COPY vendor /go/pkg/mod/cache/download
COPY go.mod .
COPY go.sum .
RUN go mod download -x
COPY . .
how to force it to use copied vendor directory instead of redownloading again and again?
so the expected behavior are:
when local have vendor (used go mod vendor), the docker build should use it
but when on CI (since vendor/*/* not committed to the repo) or developer that doesn't have vendor/*/* it should probably redownload everything (I don't really care, since they have good bandwidth)
the go mod vendor command is for the CI and devs that haven't use go mod vendor
go mod vendor only download dependency from network if the dependency not ready in local. Otherwise, it will just copy dependency to the folder vendor without access network. So here, your issue comes from the go mod cache not be reused during multiple build.
As a solution, you could use buildkit cache solution, next is a minimal example:
main.go:
package main
import _ "github.com/jeanphorn/log4go"
func main() {
}
Dockerfile:
# syntax = docker/dockerfile:1.3
FROM golang:1.14-alpine AS builder
RUN apk --update add git
ENV GO111MODULE=on
WORKDIR /app
COPY main.go /app
RUN go mod init hello
RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go mod vendor
1st Execution:
$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
#16 [builder 6/7] RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
#16 sha256:ae394bc67787799808175eada48c5f4e09101b6e153d535ddb5e4040fbf74395
#16 1.941 go: downloading github.com/go-delve/delve v1.7.1
#16 4.296 go: found github.com/go-delve/delve/cmd/dlv in github.com/go-delve/delve v1.7.1
......
#16 23.78 go: finding module for package github.com/toolkits/file
#16 23.96 go: downloading github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#16 24.17 go: found github.com/toolkits/file in github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#16 DONE 27.3s
2nd Execution:
$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
#15 [builder 6/7] RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
#15 sha256:bee74f92ceb79cce449b9702c892cb39815461981838f6b63d500414be87c21d
#15 1.467 go: found github.com/go-delve/delve/cmd/dlv in github.com/go-delve/delve v1.7.1
#15 7.511 go: github.com/jeanphorn/log4go upgrade => v0.0.0-20190526082429-7dbb8deb9468
#15 7.533 go: finding module for package github.com/toolkits/file
#15 7.675 go: found github.com/toolkits/file in github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#15 DONE 8.7s
You could see golang mod cache generated by 1st run already be reused by 2nd run without downloading from internet, now it effects as same when you do it on host.
NOTE: I didn't suggest to directly bind any cache on host to container, it's not portable I think.
Related
My Go module, stored in GitHub, successfully compiles locally; however, if I try to do it via docker, even locally in the same folder, I get an error complaining that my local package does not exist, for every local import in a subfolder:
=> ERROR [build 7/7] RUN go build -o myrepo-test . 0.6s
------
> [build 7/7] RUN go build -o myrepo-test .:
#14 0.535 main.go:10:2: no required module provides package github.com/myuser/myrepo-test/common; to add it:
#14 0.535 go get github.com/myuser/myrepo-test/common
#14 0.535 main.go:13:2: no required module provides package github.com/myuser/myrepo-test/scraper/data/process; to add it:
#14 0.535 go get github.com/myuser/myrepo-test/scraper/data/process
(....)
Here is my go.mod:
module github.com/myuser/myrepo-test
go 1.16
And the docker file:
# use alpine due to its small footprint
FROM golang:1.16-buster AS build
WORKDIR /app
# download the required Go dependencies
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
# FAIL
RUN go build -o myrepo-test .
##########
# Deploy #
##########
FROM gcr.io/distroless/base-debian10
WORKDIR /
COPY --from=build /myrepo-test /myrepo-test
USER nonroot:nonroot
ENTRYPOINT ["/myrepo-test"]
CMD ["/myrepo-test"]
#14 0.535 main.go:10:2: no required module provides package github.com/myuser/myrepo-test/common; to add it:
#14 0.535 go get github.com/myuser/myrepo-test/common
Above implies you have a package common in your source. But COPY *.go ./ won't add these folders to docker build container, it will just copy go files in current directory into docker build. As go build can't find the package common in your source in container, it will try to download it from github, so the build fails.
Then, the correct solution is as next:
Dockerfile:
FROM golang:1.16-buster AS build
WORKDIR /app
# download the required Go dependencies
COPY go.mod ./
COPY go.sum ./
RUN go mod download
#COPY *.go ./
COPY . ./
RUN ls
RUN go build -o myrepo-test .
Above will add all your sources to container including the package common etc, then build could be ok.
I've just started to get deep inside dockerfile syntax.
Here is the one I use currently:
FROM node:12-alpine as install
WORKDIR /Backend-graphql
COPY ./src ./src
COPY ./index.js ./index.js
COPY ./schema.graphql ./schema.graphql
COPY ./package.json ./
COPY ./package-lock.json ./package-lock.json
RUN npm install
FROM node:12-alpine as prismawork
WORKDIR /PrismaWork
COPY --from=install /Backend-graphql .
COPY ./datamodel.prisma ./datamodel.prisma
COPY ./prisma.yml ./prisma.yml
RUN npx prisma deploy
RUN npx prisma generate
FROM node:12-alpine
#curl needed for healthcheck
RUN apk --update --no-cache add curl
WORKDIR /app
COPY --from=prismawork /PrismaWork .
ENTRYPOINT ["npm", "start"]
EXPOSE 4000
From personnals tests and documentations founds online, i've respected the following advice :
Use multi-stage builds
But I notice something, docker do not reuse cache after the first COPY layer different in current and followings build stages. And I think its a problem, because I use an automatic bump version git hook based on commit message semantic versionning syntax who modifies my package.json. So, at each commit docker build re-RUN npm install and subsequents layers.
First of all, have I understood docker cache layering system ?
Secondly, should I use an other file for automatic bumping version and COPY it at the very end of my Dockerfile ?
First of all, have I understood docker cache layering system?
Yes, you should.
If any change happens in any step, like changes in package.json, the docker will rebuild the rest of the steps.
No need to copy from the same image multiple times. We are also doing npm install after performing unrelated steps to catching other steps.
FROM node:12-alpine
#curl needed for healthcheck
RUN apk --update --no-cache add curl
WORKDIR /app
COPY ./src ./src
COPY ./index.js ./index.js
COPY ./schema.graphql ./schema.graphql
COPY ./datamodel.prisma ./datamodel.prisma
COPY ./prisma.yml ./prisma.yml
COPY ./package.json ./
COPY ./package-lock.json ./package-lock.json
RUN npm install
RUN npx prisma deploy
RUN npx prisma generate
ENTRYPOINT ["npm", "start"]
EXPOSE 4000
Multiple staging useful when you need to build between multiple images like this example:
FROM node:12-alpine
RUN npm install -g gzipper
WORKDIR /build
ADD . .
RUN npm install
ARG CONFIGURATION
RUN npm run build:${CONFIGURATION}
RUN gzipper --gzip-level=6 ./dist
FROM nginx:latest
WORKDIR /usr/share/nginx/html
COPY --from=0 /build/dist .
COPY nginx/default.conf /etc/nginx/conf.d/default.conf
I’ve the following docker which works ok, I was able to run it and build it successfully!
FROM golang:1.13.6 AS build-env
ENV GO111MODULE=on
ENV GOOS=linux
ENV CGO_ENABLED=0
RUN mkdir -p /go/src/github.company.corp/deng/fst-cl
WORKDIR /go/src/github.company.corp/deng/fsr-clie
COPY ./ ./
# build the code
RUN go build -v -o ./fsr ./src/cmd/main.go
Now I want to change the image to use lighter docker image such as go alpine
So I change the from and added alpine version and also added git ,however the build is failing for
So go lib which doesn’t happen before the change, any idea what could be missing ?
FROM golang:1.13.6-alpine AS build-env
ENV GO111MODULE=on
ENV GOOS=linux
ENV CGO_ENABLED=0
## git is required to fetch go dependencies
RUN apk add --no-cache ca-certificates git
RUN apk add --no-cache gcc musl-dev
RUN mkdir -p /go/src/github.company.corp/deng/fst-cl
WORKDIR /go/src/github.company.corp/deng/fsr-clie
COPY ./ ./
# build the code
RUN go build -v -o ./fsr ./src/cmd/main.go
The error is for specifid repo which resides on our company git repo, but I don’t understand why its happen on golang:1.13.6-alpine and works ok on golang:1.13.6 ????
Btw I try to use different version of go alpine without success…
This is the error:
get "github.company.corp/deng/logger-ut": found meta tag get.metaImport{Prefix:"github.company.corp/deng/logger-ut", VCS:"git", RepoRoot:"https://github.company.corp/deng/logger-ut.git"} at //github.company.corp/deng/logger-ut?go-get=1
go: github.company.corp/deng/logger-ut#v1.0.0: reading github.company.corp/deng/logger-ut/go.mod at revision v1.0.0: unknown revision v1.0.0
If you want a lighter image and wish to use apline, you can use example below. Your final app image should be something like 7MB on scratch. Adjust it as it fits!
# STAGE 1: prepare
FROM golang:1.13.1-alpine3.10 as prepare
WORKDIR /source
COPY go.mod .
COPY go.sum .
RUN go mod download
# STAGE 2: build
FROM prepare AS build
COPY . .
RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o bin/app -v your/app.go
# STAGE 3: run
FROM scratch as run
COPY --from=build /source/bin/app /app
ENTRYPOINT ["/app"]
I'm building a multi-stage Dockerfile for my go project.
FROM golang:latest as builder
COPY ./go.mod /app/go.mod
COPY ./go.sum /app/go.sum
#exporting go1.11 module support variable
ENV GO111MODULE=on
WORKDIR /app/
#create vendor directory
RUN go mod download
COPY . /app/
RUN go mod vendor
#building source code
RUN go build -mod=vendor -o main -v ./src/
FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=builder /app/main /app/main
WORKDIR /app/
ARG port="80"
ENV PORT=$port
EXPOSE $PORT
CMD ["./main"]
When I'm running the image, it throws error:
standard_init_linux.go:207: exec user process caused "no such file or directory"
I've verified that the 'main' file exist in /app/main.
I also tried to give executable permission by adding
chmod +x /app/main
but still it doesn't work.
What can possibly be wrong?
The "latest" version of the golang image is debian based, which uses libc. Alpine uses musl. If you do not compile with CGO_ENABLED=0, networking libraries will link to libc and the no such file or directory error point to a missing library. You can check these shared library links with ldd /app/main. A few solutions I can think of:
compile your program with CGO_ENABLED=0
switch your build image to FROM golang:alpine
change your second stage to be FROM debian
I'm using the build docker image from golang:alpine.
My purpose is just to copy the executed binary file to a new scratch image.
Below its my Dockerfile:
############################
# STEP 1 build executable binary
############################
FROM golang#sha256:d481168873b7516b9f34d322615d589fafb166ff5fd57d93e96f64787a58887c AS builder
RUN apk update && apk add --no-cache git tzdata ca-certificates && update-ca-certificates
ADD . $GOPATH/src/piggybank2go
WORKDIR $GOPATH/src/piggybank2go
COPY . .
# Fetch dependencies.
RUN go get -u github.com/golang/dep/cmd/dep
RUN dep ensure -v
# Build executeable binary
RUN GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o $GOPATH/bin/piggybank2go
# RUN go build -o /go/bin/piggybank2go
############################
# STEP 2 build a small image
############################
FROM scratch
# Copy our static executable
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder $GOPATH/bin/piggybank2go $GOPATH/bin/piggybank2go
# Port on which the service will be exposed.
EXPOSE 8081
ENTRYPOINT ["$GOPATH/bin/piggybank2go"]
But I got this error:
Step 12/14 : COPY --from=builder $GOPATH/bin/piggybank2go $GOPATH/bin/piggybank2go
COPY failed: stat /var/lib/docker/overlay2/b37bbe725b51ba50e3082d162e75d4cdee368499e26887c6921486415c089920/merged/bin/piggybank2go: no such file or directory
I think the problem is that the environment variable $GOPATH only exists in the golang image and not the scratch image. So try change the COPY-line to:
COPY --from=builder /go/bin/piggybank2go /go/bin/piggybank2go
Environment variables from the first stage are not available in the second stage. For this reason "$GOPATH" cannot be resolved correctly in the second stage, hence the error.
In the second stage you should know exactly what and to where you are copying:
COPY --from=builder /go/bin/piggybank2go /go/bin/piggybank2go