My Go module, stored in GitHub, successfully compiles locally; however, if I try to do it via docker, even locally in the same folder, I get an error complaining that my local package does not exist, for every local import in a subfolder:
=> ERROR [build 7/7] RUN go build -o myrepo-test . 0.6s
------
> [build 7/7] RUN go build -o myrepo-test .:
#14 0.535 main.go:10:2: no required module provides package github.com/myuser/myrepo-test/common; to add it:
#14 0.535 go get github.com/myuser/myrepo-test/common
#14 0.535 main.go:13:2: no required module provides package github.com/myuser/myrepo-test/scraper/data/process; to add it:
#14 0.535 go get github.com/myuser/myrepo-test/scraper/data/process
(....)
Here is my go.mod:
module github.com/myuser/myrepo-test
go 1.16
And the docker file:
# use alpine due to its small footprint
FROM golang:1.16-buster AS build
WORKDIR /app
# download the required Go dependencies
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
# FAIL
RUN go build -o myrepo-test .
##########
# Deploy #
##########
FROM gcr.io/distroless/base-debian10
WORKDIR /
COPY --from=build /myrepo-test /myrepo-test
USER nonroot:nonroot
ENTRYPOINT ["/myrepo-test"]
CMD ["/myrepo-test"]
#14 0.535 main.go:10:2: no required module provides package github.com/myuser/myrepo-test/common; to add it:
#14 0.535 go get github.com/myuser/myrepo-test/common
Above implies you have a package common in your source. But COPY *.go ./ won't add these folders to docker build container, it will just copy go files in current directory into docker build. As go build can't find the package common in your source in container, it will try to download it from github, so the build fails.
Then, the correct solution is as next:
Dockerfile:
FROM golang:1.16-buster AS build
WORKDIR /app
# download the required Go dependencies
COPY go.mod ./
COPY go.sum ./
RUN go mod download
#COPY *.go ./
COPY . ./
RUN ls
RUN go build -o myrepo-test .
Above will add all your sources to container including the package common etc, then build could be ok.
Related
My Dockerfile is below:
# syntax=docker/dockerfile:1
FROM golang:1.18-alpine
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /datapuller
EXPOSE 8080
CMD [ "/datapuller" ]
I tried to build with $ docker build --tag datapuller .
But got error:
main.go:13:2: no required module provides package gitlab.com/mycorp/mycompany/data/datapuller/dbutil; to add it:
go get gitlab.com/mycorp/mycompany/data/datapuller/dbutil
main.go:14:2: no required module provides package gitlab.com/mycorp/mycompany/data/datapuller/models; to add it:
go get gitlab.com/mycorp/mycompany/data/datapuller/models
How to solve this, I can run directly with go run main.go just fine.
My main.go's import is below. I think the imports caused this problem:
package main
import (
"encoding/json"
client "github.com/bozd4g/go-http-client"
"github.com/robfig/cron/v3"
"github.com/xuri/excelize/v2"
"gitlab.com/mycorp/mycompany/data/datapuller/dbutil"
"gitlab.com/mycorp/mycompany/data/datapuller/models"
"gorm.io/gorm"
)
func main() {
...
Because the associated package needs to be pulled when building.
Docker may be missing the necessary environment variables to pull these packages.
It is recommended that you use the go mod vendor command,then build image
FROM golang:1.18-alpine
ADD . /go/src/<project name>
WORKDIR /go/src/<project name>
RUN go build -mod=vendor -v -o /go/src/bin/main main.go
RUN rm -rf /go/src/<project name>
WORKDIR /go/src/bin
CMD ["/go/src/bin/main"]
When you copy your source code into the image, you only copy files in the current directory
COPY *.go ./ # just the current directory's *.go, not any subdirectories
It's usually more common to copy in the entire host source tree, maybe using a .dockerignore file to cause some of the source tree to be ignored
COPY ./ ./
Otherwise you need to copy the specific subdirectories you need into the image (each directory needs a separate COPY command)
COPY *.go ./
COPY dbutil/ dbutil/
COPY models/ models/
I have this structure of my project:
https://i.stack.imgur.com/SqqDh.png
And this is my Dockerfile:
FROM golang:1.19
ADD . /go/src/myapp
WORKDIR /go/src/myapp
RUN go mod init cloudmeta
RUN go get github.com/go-sql-driver/mysql
RUN go get -u github.com/gin-gonic/gin
RUN go build -o bin/cloudmeta
CMD [ "bin/cloudmeta" ]
When I trying to build my docker-container I have this error:
package cloudmeta/backend/handlers is not in GOROOT (/usr/local/go/src/cloudmeta/backend/handlers)
When building Go code in docker, you shouldn't use go mod init. Take a look at the following example dockerfile from docker docs:
# syntax=docker/dockerfile:1
FROM golang:1.16-alpine
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /docker-gs-ping
EXPOSE 8080
CMD [ "/docker-gs-ping" ]
The docker docs guide goes into more depth but to summarise things:
You should copy your go.mod and go.sum files into your project directory in the image.
Now you can run the go mod download command to install the go modules required.
Then you need to copy your source code into the image.
Now you can compile your source code with the go build command.
I'm trying to make Dockerfile runs faster by copying whole directory (including vendor, because redownloading dependencies took about 10m+ in 3rd world country where I live), but when I tried to run it, it always redownload vendor again and again, unlike when go mod vendor in local:
FROM golang:1.14-alpine AS builder
RUN apk --update add ca-certificates git make g++
ENV GO111MODULE=on
WORKDIR /app
RUN go get github.com/go-delve/delve/cmd/dlv
COPY . .
RUN go mod vendor
ARG COMMIT_HASH
ENV COMMIT_HASH=${COMMIT_HASH}
ARG BUILD_DATE
ENV BUILD_DATE=${BUILD_DATE}
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
go build \
-o app
FROM golang:1.14-alpine
WORKDIR /app
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /go/bin/dlv /
COPY --from=builder /app/app .
COPY --from=builder /app/db ./db
EXPOSE 8080 63342
CMD [ "/dlv", "--listen=:63342", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./app" ]
previously using this (without vendor) also slow:
COPY go.mod .
COPY go.sum .
RUN go mod download -x
COPY . .
trying with this also didn't work:
COPY vendor /go/pkg/mod
COPY vendor /go/pkg/mod/cache/download
COPY go.mod .
COPY go.sum .
RUN go mod download -x
COPY . .
how to force it to use copied vendor directory instead of redownloading again and again?
so the expected behavior are:
when local have vendor (used go mod vendor), the docker build should use it
but when on CI (since vendor/*/* not committed to the repo) or developer that doesn't have vendor/*/* it should probably redownload everything (I don't really care, since they have good bandwidth)
the go mod vendor command is for the CI and devs that haven't use go mod vendor
go mod vendor only download dependency from network if the dependency not ready in local. Otherwise, it will just copy dependency to the folder vendor without access network. So here, your issue comes from the go mod cache not be reused during multiple build.
As a solution, you could use buildkit cache solution, next is a minimal example:
main.go:
package main
import _ "github.com/jeanphorn/log4go"
func main() {
}
Dockerfile:
# syntax = docker/dockerfile:1.3
FROM golang:1.14-alpine AS builder
RUN apk --update add git
ENV GO111MODULE=on
WORKDIR /app
COPY main.go /app
RUN go mod init hello
RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go mod vendor
1st Execution:
$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
#16 [builder 6/7] RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
#16 sha256:ae394bc67787799808175eada48c5f4e09101b6e153d535ddb5e4040fbf74395
#16 1.941 go: downloading github.com/go-delve/delve v1.7.1
#16 4.296 go: found github.com/go-delve/delve/cmd/dlv in github.com/go-delve/delve v1.7.1
......
#16 23.78 go: finding module for package github.com/toolkits/file
#16 23.96 go: downloading github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#16 24.17 go: found github.com/toolkits/file in github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#16 DONE 27.3s
2nd Execution:
$ export DOCKER_BUILDKIT=1
$ docker build --progress=plain -t abc:1 . --no-cache
#15 [builder 6/7] RUN --mount=type=cache,mode=0755,target=/go/pkg/mod go get github.com/go-delve/delve/cmd/dlv && go get github.com/jeanphorn/log4go
#15 sha256:bee74f92ceb79cce449b9702c892cb39815461981838f6b63d500414be87c21d
#15 1.467 go: found github.com/go-delve/delve/cmd/dlv in github.com/go-delve/delve v1.7.1
#15 7.511 go: github.com/jeanphorn/log4go upgrade => v0.0.0-20190526082429-7dbb8deb9468
#15 7.533 go: finding module for package github.com/toolkits/file
#15 7.675 go: found github.com/toolkits/file in github.com/toolkits/file v0.0.0-20160325033739-a5b3c5147e07
#15 DONE 8.7s
You could see golang mod cache generated by 1st run already be reused by 2nd run without downloading from internet, now it effects as same when you do it on host.
NOTE: I didn't suggest to directly bind any cache on host to container, it's not portable I think.
My dockerfile:
FROM golang:1.14
RUN mkdir /app
ADD . /app
WORKDIR /app
RUN go build -o main .
CMD ["/app/main"]
error:
main.go:11:2: cannot find package "github.com/gorilla/mux" in any of:
/usr/local/go/src/github.com/gorilla/mux (from $GOROOT)
/go/src/github.com/gorilla/mux (from $GOPATH)
My PATH in GOPATH is
GOPATH=/Users/pstrom/go
I'm coming from a javascript background and there you run NPM INSTALL which adds all external packages to directory node_modules in same directory as the project.
Is there any similar command in Go? Can't find any. I don't want add any PATH in docker, because I wanna run it from anywhere.
How do I handle external packages in Docker in Go?
See the comments too.
It's possible you need to create a go.mod file which functions like package.json. If you don't have a go.mod file but just want to get going, you can go mod init x in the directory alongside main.go and Dockerfile. Then, to force packages to be added to go.mod, you can just go run . (or go run main.go).
Then:
FROM golang:1.15
WORKDIR /app
COPY go.mod .
RUN go mod download
COPY . .
RUN go build -o main .
ENTRYPOINT ["/app/main"]
I recommend bumping to Go 1.15
WORKDIR creates the directory if not present so you skip the mkdir
/app is outside of ${GOPATH} which is correct when using modules
COPY >> ADD (my preference)
go mod download gets dependencies defined in go.mod
COPY . . everything else, may just need to be COPY main.go .
ENTRYPOINT >> CMD and the container will default to running your binary
I'm trying to speed up Docker builds of my Go app. Right now, it's spending maybe 60s just building dependencies (it's a k8s controller, so there are a lot).
One very important constraint: my project depends on private GitHub repos. I do go mod vendor outside docker build, where I have creds for the repos set up.
My Dockerfile right now is roughly:
FROM golang:1.12
WORKDIR /src
COPY . .
RUN go build -mod=vendor
...
Even without having to download the deps, that build takes a while because it rebuilds several hundred packages every docker build.
What I'd like to do is something like:
FROM golang:1.12
WORKDIR /src
# these shouldn't change very often
COPY go.mod go.sum vendor ./
RUN go build -mod=vendor <all dependency packages>
COPY . .
RUN go build -mod=vendor
...
I tried parsing go.mod, but of course that lists modules, not packages. I tried go list but never managed to get a working incantation.
I've got a nasty hack that seems to work:
FROM golang:1.12
WORKDIR /src
COPY go.mod go.sum ./
COPY vendor/ ./vendor
RUN go build -mod=vendor $(cat deps|grep -v mypackage | grep -v internal)
COPY . .
RUN go build -mod=vendor
...
go list -f '{{join .Deps "\n"}}' > deps
docker build .
Docker documentation has a guide specific to Go docker images (Build your Go image).
It works as follows:
# Layer for dependency installation
COPY go.mod go.sum ./
RUN go mod download
# Layer for app build
COPY . .
RUN go build -o main .