I have a mono repo with the structure.
mono-repo
- serviceA
- main.go
- Dockerfile
-serviceB
- main.go
- Dockerfile
go.mod
go.sum
The Dockerfile in serviceA contains the following code.
FROM golang
ENV GO111MODULE=on
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build
ENTRYPOINT ["/app/serviceA"]
I want to build the Docker image and include the dependencies from the root of my mono-repo inside the container, I am currently receiving an error saying it can't find any of the dependency packages when I run
docker build -t serviceA .
Unless I place a go.mod inside serviceA I can't see a nice way of achieving what I want. By placing a go.mod inside the service it feels like I'm losing the advantage of services sharing dependencies within the repo.
By placing a go.mod inside the service it feels like I'm losing the advantage of services sharing dependencies within the repo.
Yet, this is an approach seen here or there, where COPY go.mod . (and COPY go.sum .) is followed by RUN go mod download.
#This is the ‘magic’ step that will download all the dependencies that are specified in
# the go.mod and go.sum file.
# Because of how the layer caching system works in Docker, the go mod download
# command will _ only_ be re-run when the go.mod or go.sum file change
# (or when we add another docker instruction this line)
RUN go mod download
Related
My Dockerfile is below:
# syntax=docker/dockerfile:1
FROM golang:1.18-alpine
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /datapuller
EXPOSE 8080
CMD [ "/datapuller" ]
I tried to build with $ docker build --tag datapuller .
But got error:
main.go:13:2: no required module provides package gitlab.com/mycorp/mycompany/data/datapuller/dbutil; to add it:
go get gitlab.com/mycorp/mycompany/data/datapuller/dbutil
main.go:14:2: no required module provides package gitlab.com/mycorp/mycompany/data/datapuller/models; to add it:
go get gitlab.com/mycorp/mycompany/data/datapuller/models
How to solve this, I can run directly with go run main.go just fine.
My main.go's import is below. I think the imports caused this problem:
package main
import (
"encoding/json"
client "github.com/bozd4g/go-http-client"
"github.com/robfig/cron/v3"
"github.com/xuri/excelize/v2"
"gitlab.com/mycorp/mycompany/data/datapuller/dbutil"
"gitlab.com/mycorp/mycompany/data/datapuller/models"
"gorm.io/gorm"
)
func main() {
...
Because the associated package needs to be pulled when building.
Docker may be missing the necessary environment variables to pull these packages.
It is recommended that you use the go mod vendor command,then build image
FROM golang:1.18-alpine
ADD . /go/src/<project name>
WORKDIR /go/src/<project name>
RUN go build -mod=vendor -v -o /go/src/bin/main main.go
RUN rm -rf /go/src/<project name>
WORKDIR /go/src/bin
CMD ["/go/src/bin/main"]
When you copy your source code into the image, you only copy files in the current directory
COPY *.go ./ # just the current directory's *.go, not any subdirectories
It's usually more common to copy in the entire host source tree, maybe using a .dockerignore file to cause some of the source tree to be ignored
COPY ./ ./
Otherwise you need to copy the specific subdirectories you need into the image (each directory needs a separate COPY command)
COPY *.go ./
COPY dbutil/ dbutil/
COPY models/ models/
When I use multi stage in Docker, it cannot find the container config file, but when I use only the first stage, the container works fine. I need this as I'm going to reduce the size for the deployment phase, but I couldn't find a solution. I have no problem reading the config file in my local.
Here is my docker file
FROM golang:1.18.3-alpine AS builder
WORKDIR /app/policy-manangement-service
ENV GO111MODULE=on
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix nocgo -o /policy-manangement-service ./cmd/policy-management
FROM alpine:latest
COPY --from=builder /policy-manangement-service ./
ENTRYPOINT ["./policy-manangement-service"]
and my error:
{"app-name":"policy-management-service","error":"Config File \"local\" Not Found in \"[/configs]\"","level":"fatal","message":"Error while reading configs","time":"2022-07-12T14:27:58.486529597Z"}
my project structure:
cmd
- policy-manangement
-- main
configs
- local.yaml
internal
My dockerfile:
FROM golang:1.14
RUN mkdir /app
ADD . /app
WORKDIR /app
RUN go build -o main .
CMD ["/app/main"]
error:
main.go:11:2: cannot find package "github.com/gorilla/mux" in any of:
/usr/local/go/src/github.com/gorilla/mux (from $GOROOT)
/go/src/github.com/gorilla/mux (from $GOPATH)
My PATH in GOPATH is
GOPATH=/Users/pstrom/go
I'm coming from a javascript background and there you run NPM INSTALL which adds all external packages to directory node_modules in same directory as the project.
Is there any similar command in Go? Can't find any. I don't want add any PATH in docker, because I wanna run it from anywhere.
How do I handle external packages in Docker in Go?
See the comments too.
It's possible you need to create a go.mod file which functions like package.json. If you don't have a go.mod file but just want to get going, you can go mod init x in the directory alongside main.go and Dockerfile. Then, to force packages to be added to go.mod, you can just go run . (or go run main.go).
Then:
FROM golang:1.15
WORKDIR /app
COPY go.mod .
RUN go mod download
COPY . .
RUN go build -o main .
ENTRYPOINT ["/app/main"]
I recommend bumping to Go 1.15
WORKDIR creates the directory if not present so you skip the mkdir
/app is outside of ${GOPATH} which is correct when using modules
COPY >> ADD (my preference)
go mod download gets dependencies defined in go.mod
COPY . . everything else, may just need to be COPY main.go .
ENTRYPOINT >> CMD and the container will default to running your binary
I have two go modules github.com/myuser/mymainrepo and github.com/myuser/commonrepo
Here is how i have the files in my local computer
- allmyrepos
- mymainrepo
- Dockerfile
- go.mod
- commonrepo
- go.mod
mymainrepo/go.mod
...
require (
github.com/myuser/commonrepo
)
replace (
github.com/myuser/commonrepo => ../commonrepo
)
It works well i can do local development with it. Problem happens when i'm building docker image of mymainrepo
mymainrepo/Dockerfile
...
WORKDIR /go/src/mymainrepo
COPY go.mod go.sum ./
RUN go mod download
COPY ./ ./
RUN go build -o appbinary
...
Here replace replaces github.com/myuser/commonrepo with ../commonrepo but in Docker /go/src/commonrepo does not exists.
I'm building the Docker image on CI/CD which needs to fetch directly from remote github url but i also need to do local development on commonrepo. How can i do both ?
I tried to put all my files in GOPATH so it's ~/go/src/github.com/myuser/commonrepo and go/src/github.com/myuser/mymainrepo. And i removed the replace directive. But it looks for commonrepo inside ~/go/pkg/mod/... that's downloaded from github.
Create two go.mod files: one for local development, and one for your build. You can name it go.build.mod for example.
Keep the replace directive in your go.mod file but remove it from go.build.mod.
Finally, in your Dockerfile:
COPY go.build.mod ./go.mod
COPY go.sum ./
I still can't find other better solution even the voted answer doesn't work for me. Here a trick I've done that workaround for me. This is an example structure for doing this:
|---sample
| |---...
| |---go.mod
| |---Dockerfile
|---core
| |---...
| |---go.mod
We know that docker build error when it can't find our local module. Let's make one in the builder process:
# Use the offical golang image to create a binary.
# This is based on Debian and sets the GOPATH to /go.
# https://hub.docker.com/_/golang
FROM golang:1.16.3-buster AS builder
# Copy core library
RUN mkdir /core
COPY core/ /core
# Create and change to the app directory.
WORKDIR /app
# Retrieve application dependencies.
# This allows the container build to reuse cached dependencies.
# Expecting to copy go.mod and if present go.sum.
COPY go.* ./
RUN go mod download
# Copy local code to the container image.
COPY . ./
# Build the binary
RUN go build -o /app/sample cmd/main.go
...
...
Ok, our working dir is /app and our core lib placed next to it /core.
Let's make a trick when build a docker image! Yeah, you know it.
cp -R ../core . && docker build --tag sample-service . && rm -R core/
Update
A way better, create a Makefile next to Dockerfile, with content below:
build:
cp -R ../core .
docker build -t sample-service .
rm -R core/
Then command, make build in the sample directory.
You can create make submit or make deploy commands as you like to.
=> Production ready!
Be aware that if there's an error occurs during docker build process, it won't delete back the core folder we have copied to sample.
Pls let me know if you find any better solution. ;)
I'm trying to speed up Docker builds of my Go app. Right now, it's spending maybe 60s just building dependencies (it's a k8s controller, so there are a lot).
One very important constraint: my project depends on private GitHub repos. I do go mod vendor outside docker build, where I have creds for the repos set up.
My Dockerfile right now is roughly:
FROM golang:1.12
WORKDIR /src
COPY . .
RUN go build -mod=vendor
...
Even without having to download the deps, that build takes a while because it rebuilds several hundred packages every docker build.
What I'd like to do is something like:
FROM golang:1.12
WORKDIR /src
# these shouldn't change very often
COPY go.mod go.sum vendor ./
RUN go build -mod=vendor <all dependency packages>
COPY . .
RUN go build -mod=vendor
...
I tried parsing go.mod, but of course that lists modules, not packages. I tried go list but never managed to get a working incantation.
I've got a nasty hack that seems to work:
FROM golang:1.12
WORKDIR /src
COPY go.mod go.sum ./
COPY vendor/ ./vendor
RUN go build -mod=vendor $(cat deps|grep -v mypackage | grep -v internal)
COPY . .
RUN go build -mod=vendor
...
go list -f '{{join .Deps "\n"}}' > deps
docker build .
Docker documentation has a guide specific to Go docker images (Build your Go image).
It works as follows:
# Layer for dependency installation
COPY go.mod go.sum ./
RUN go mod download
# Layer for app build
COPY . .
RUN go build -o main .