I have two go modules github.com/myuser/mymainrepo and github.com/myuser/commonrepo
Here is how i have the files in my local computer
- allmyrepos
- mymainrepo
- Dockerfile
- go.mod
- commonrepo
- go.mod
mymainrepo/go.mod
...
require (
github.com/myuser/commonrepo
)
replace (
github.com/myuser/commonrepo => ../commonrepo
)
It works well i can do local development with it. Problem happens when i'm building docker image of mymainrepo
mymainrepo/Dockerfile
...
WORKDIR /go/src/mymainrepo
COPY go.mod go.sum ./
RUN go mod download
COPY ./ ./
RUN go build -o appbinary
...
Here replace replaces github.com/myuser/commonrepo with ../commonrepo but in Docker /go/src/commonrepo does not exists.
I'm building the Docker image on CI/CD which needs to fetch directly from remote github url but i also need to do local development on commonrepo. How can i do both ?
I tried to put all my files in GOPATH so it's ~/go/src/github.com/myuser/commonrepo and go/src/github.com/myuser/mymainrepo. And i removed the replace directive. But it looks for commonrepo inside ~/go/pkg/mod/... that's downloaded from github.
Create two go.mod files: one for local development, and one for your build. You can name it go.build.mod for example.
Keep the replace directive in your go.mod file but remove it from go.build.mod.
Finally, in your Dockerfile:
COPY go.build.mod ./go.mod
COPY go.sum ./
I still can't find other better solution even the voted answer doesn't work for me. Here a trick I've done that workaround for me. This is an example structure for doing this:
|---sample
| |---...
| |---go.mod
| |---Dockerfile
|---core
| |---...
| |---go.mod
We know that docker build error when it can't find our local module. Let's make one in the builder process:
# Use the offical golang image to create a binary.
# This is based on Debian and sets the GOPATH to /go.
# https://hub.docker.com/_/golang
FROM golang:1.16.3-buster AS builder
# Copy core library
RUN mkdir /core
COPY core/ /core
# Create and change to the app directory.
WORKDIR /app
# Retrieve application dependencies.
# This allows the container build to reuse cached dependencies.
# Expecting to copy go.mod and if present go.sum.
COPY go.* ./
RUN go mod download
# Copy local code to the container image.
COPY . ./
# Build the binary
RUN go build -o /app/sample cmd/main.go
...
...
Ok, our working dir is /app and our core lib placed next to it /core.
Let's make a trick when build a docker image! Yeah, you know it.
cp -R ../core . && docker build --tag sample-service . && rm -R core/
Update
A way better, create a Makefile next to Dockerfile, with content below:
build:
cp -R ../core .
docker build -t sample-service .
rm -R core/
Then command, make build in the sample directory.
You can create make submit or make deploy commands as you like to.
=> Production ready!
Be aware that if there's an error occurs during docker build process, it won't delete back the core folder we have copied to sample.
Pls let me know if you find any better solution. ;)
Related
My Dockerfile is below:
# syntax=docker/dockerfile:1
FROM golang:1.18-alpine
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /datapuller
EXPOSE 8080
CMD [ "/datapuller" ]
I tried to build with $ docker build --tag datapuller .
But got error:
main.go:13:2: no required module provides package gitlab.com/mycorp/mycompany/data/datapuller/dbutil; to add it:
go get gitlab.com/mycorp/mycompany/data/datapuller/dbutil
main.go:14:2: no required module provides package gitlab.com/mycorp/mycompany/data/datapuller/models; to add it:
go get gitlab.com/mycorp/mycompany/data/datapuller/models
How to solve this, I can run directly with go run main.go just fine.
My main.go's import is below. I think the imports caused this problem:
package main
import (
"encoding/json"
client "github.com/bozd4g/go-http-client"
"github.com/robfig/cron/v3"
"github.com/xuri/excelize/v2"
"gitlab.com/mycorp/mycompany/data/datapuller/dbutil"
"gitlab.com/mycorp/mycompany/data/datapuller/models"
"gorm.io/gorm"
)
func main() {
...
Because the associated package needs to be pulled when building.
Docker may be missing the necessary environment variables to pull these packages.
It is recommended that you use the go mod vendor command,then build image
FROM golang:1.18-alpine
ADD . /go/src/<project name>
WORKDIR /go/src/<project name>
RUN go build -mod=vendor -v -o /go/src/bin/main main.go
RUN rm -rf /go/src/<project name>
WORKDIR /go/src/bin
CMD ["/go/src/bin/main"]
When you copy your source code into the image, you only copy files in the current directory
COPY *.go ./ # just the current directory's *.go, not any subdirectories
It's usually more common to copy in the entire host source tree, maybe using a .dockerignore file to cause some of the source tree to be ignored
COPY ./ ./
Otherwise you need to copy the specific subdirectories you need into the image (each directory needs a separate COPY command)
COPY *.go ./
COPY dbutil/ dbutil/
COPY models/ models/
My dockerfile:
FROM golang:1.14
RUN mkdir /app
ADD . /app
WORKDIR /app
RUN go build -o main .
CMD ["/app/main"]
error:
main.go:11:2: cannot find package "github.com/gorilla/mux" in any of:
/usr/local/go/src/github.com/gorilla/mux (from $GOROOT)
/go/src/github.com/gorilla/mux (from $GOPATH)
My PATH in GOPATH is
GOPATH=/Users/pstrom/go
I'm coming from a javascript background and there you run NPM INSTALL which adds all external packages to directory node_modules in same directory as the project.
Is there any similar command in Go? Can't find any. I don't want add any PATH in docker, because I wanna run it from anywhere.
How do I handle external packages in Docker in Go?
See the comments too.
It's possible you need to create a go.mod file which functions like package.json. If you don't have a go.mod file but just want to get going, you can go mod init x in the directory alongside main.go and Dockerfile. Then, to force packages to be added to go.mod, you can just go run . (or go run main.go).
Then:
FROM golang:1.15
WORKDIR /app
COPY go.mod .
RUN go mod download
COPY . .
RUN go build -o main .
ENTRYPOINT ["/app/main"]
I recommend bumping to Go 1.15
WORKDIR creates the directory if not present so you skip the mkdir
/app is outside of ${GOPATH} which is correct when using modules
COPY >> ADD (my preference)
go mod download gets dependencies defined in go.mod
COPY . . everything else, may just need to be COPY main.go .
ENTRYPOINT >> CMD and the container will default to running your binary
I'm trying to speed up Docker builds of my Go app. Right now, it's spending maybe 60s just building dependencies (it's a k8s controller, so there are a lot).
One very important constraint: my project depends on private GitHub repos. I do go mod vendor outside docker build, where I have creds for the repos set up.
My Dockerfile right now is roughly:
FROM golang:1.12
WORKDIR /src
COPY . .
RUN go build -mod=vendor
...
Even without having to download the deps, that build takes a while because it rebuilds several hundred packages every docker build.
What I'd like to do is something like:
FROM golang:1.12
WORKDIR /src
# these shouldn't change very often
COPY go.mod go.sum vendor ./
RUN go build -mod=vendor <all dependency packages>
COPY . .
RUN go build -mod=vendor
...
I tried parsing go.mod, but of course that lists modules, not packages. I tried go list but never managed to get a working incantation.
I've got a nasty hack that seems to work:
FROM golang:1.12
WORKDIR /src
COPY go.mod go.sum ./
COPY vendor/ ./vendor
RUN go build -mod=vendor $(cat deps|grep -v mypackage | grep -v internal)
COPY . .
RUN go build -mod=vendor
...
go list -f '{{join .Deps "\n"}}' > deps
docker build .
Docker documentation has a guide specific to Go docker images (Build your Go image).
It works as follows:
# Layer for dependency installation
COPY go.mod go.sum ./
RUN go mod download
# Layer for app build
COPY . .
RUN go build -o main .
We are working with two project at the moment:
1 C++ based project
2 Nodejs based project
These two projectes are separated which means they have different codebase(git repoitory) and working directory.
C++ project will produce a node binding file .node which will be used by Nodejs project.
And we try to build an docker image for the Nodejs project with multi-stage like this:
from ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
copy (?) . #1 copy the c++ source codes
RUN make
from node:10
WORKDIR /app
copy (?) . #1 copy the nodejs cource codes
RUN npm install
copy --from=u /app/dist/xx.node ./lib/
node index.js
And I will build the image by docker build -t xx (?) #2.
However as commented in the dockerfile and the command, how to setup the context directory(see comment #2)? Since it will affect the path in the dockerfile (see comment #1).
Also which project should I put inside for the above dockerfile?
You will have two options on this, as the limiting factor is that Docker only allows copying from the same directory as the Dockerfile:
Create a new Repository
You can either create a new repo and use your repos as submodules or just for the Dockerfile (Than you would have to copy both repos into the root folder at build time). In the End what you want to achieve is the following structure:
/ (root)
|-- C-plus-plus-Repo
|-- |-- <Files>
|-- Node-Repo
|-- |-- <Files>
|-- Dockerfile
Than you can build your project with:
from ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
#1 copy the c++ source files
copy ./C-plus-plus-Repo .
RUN make
from node:10
WORKDIR /app
#1 copy the nodejs cource codes
copy ./Node-Repo .
RUN npm install
copy --from=u /app/dist/xx.node ./lib/
node index.js
In the root Directory execute:
docker build -t xx .
Build your staging containers extra
Docker allows to copy from an external container as stage.
So you can build the C++ container in your C++ Repo root
from ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
#1 copy the c++ source files
copy . .
RUN make
and Tag it:
# Build your C++ Container in root of the c++ repo
docker build . -t c-stage
then copy the files from it, using the tag (in your node Repo root):
from node:10
WORKDIR /app
#1 copy the nodejs source files
copy . .
RUN npm install
# Use the Tag-Name of the already build container "c-stage"
copy --from=c-stage /app/dist/xx.node ./lib/
node index.js
Both build steps can be executed from their respective repo roots.
Creating a deploy project with git submodules
How about creating a deploy project using git submodules?
This project would only exist for building the docker image and contains the Dockerfile and both of your projects as git submodules.
Since you not just copy the two projects, but manage them with git, you can always keep them up to date with git submodules update --remote, but note that this leaves your submodule in a detached head state. However this is not a problem as long as you do not try to update your C++ project or the node project from the deploy project.
You can create the project with the following commands:
mkdir deploy_project && cd deploy_project
git init
git submodule add git#your-gitserver.com:YourName/YourCppProject.git cpp_project
git submodule add git#your-gitserver.com:YourName/YourNodeProject.git nodejs_project
Then you can simply add the paths to the subprojects to your dockerfile and build the image in the root directory of the deploy project.
The dockerfile would look like this
FROM ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
COPY cpp_project/ . #1 copy the c++ source codes
RUN make
FROM node:10
WORKDIR /app
COPY nodejs_project/ . #1 copy the nodejs cource codes
RUN npm install
COPY --from=u /app/dist/xx.node ./lib/
You can use ADD command (her context watching to host directory where Dockerfile placed. It will copy everything placed in same directory as Dockerfile in host machine (in this case content of cpp_app directory) into the docker container.
...
ADD cpp_app /place/to/build
WORKDIR /place/to/build
RUN make
RUN mv result_file /place/where/result_file/have/to/be
WORKDIR /place/where/result_file/have/to/be
... execute your nodejs stuff
I have a Go project with a large vendor/ directory which almost never changes.
I am trying to use the new go 1.10 build cache feature to speed up my builds in Docker engine locally.
Avoiding recompilation of my vendor/ directory would be enough optimization. So I'm trying to do Go equivalent of this common Dockerfile pattern for Python:
FROM python
COPY requirements.txt . # <-- copy your dependency list
RUN pip install -r requirements.txt # <-- install dependencies
COPY ./src ... # <-- your actual code (everything above is cached)
Similarly I tried:
FROM golang:1.10-alpine
COPY ./vendor ./src/myproject/vendor
RUN go build -v myproject/vendor/... # <-- pre-build & cache "vendor/"
COPY . ./src/myproject
However this is giving "cannot find package" error (likely because you cannot build stuff in vendor/ directly normally either).
Has anyone been able to figure this out?
Here's something that works for me:
FROM golang:1.10-alpine
WORKDIR /usr/local/go/src/github.com/myorg/myproject/
COPY vendor vendor
RUN find vendor -maxdepth 2 -mindepth 2 -type d -exec sh -c 'go install -i github.com/myorg/myproject/{}/... || true' \;
COPY main.go .
RUN go build main.go
It makes sure the vendored libraries are installed first. As long as you don't change a library, you're good.
Just use go install -i ./vendor/....
Consider the following Dockerfile:
FROM golang:1.10-alpine
ARG APP
ENV PTH $GOPATH/src/$APP
WORKDIR $PTH
# Pre-compile vendors.
COPY vendor/ $PTH/vendor/
RUN go install -i ./vendor/...
ADD . $PTH/
# Display time taken and the list of the packages being compiled.
RUN time go build -v
You can test it doing something like:
docker build -t test --build-arg APP=$(go list .) .
On the project I am working on, without pre-compile, it takes ~12sec with 90+ package each time, after, it take ~1.2s with only 3 (only the local ones).
If you still have "cannot find package", it means there are missing vendors. Re-run dep ensure should fix it.
An other tip, unrelated to Go is to have your .dockerignore start with *. i.e. ignore everything and then whitelist what you need.
As of Go 1.11, you would use go modules to accomplish this;
FROM golang
WORKDIR /src/myproject
COPY go.mod go.sum ./ # <-- copy your dependency list
RUN go mod download # <-- install dependencies
COPY . . # <-- your actual code (everything above is cached)
As long as go.sum doesn't change, the image layer created by go mod download shall be reused from cache.
Using go mod download alone didn't do the trick for me.
What ended up working for me was to leverage buildkit to mount Go's build cache.
This was the article that lead me to this feature.
My Dockerfile looks something like this
FROM golang AS build
WORKDIR /go/src/app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY . ./
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go build -o /my-happy-app
For local development this did quite a different for me, changing buildtimes from 1.5 minutes to 3 seconds.
I'm using colima (to run Docker on my Mac. Just mentioning this since I'm not using Docker for Mac, but it should work just the same) with
colima version 0.4.2
git commit: f112f336d05926d62eb6134ee3d00f206560493b
runtime: docker
arch: x86_64
client: v20.10.9
server: v20.10.11
And golang 1.17, so this is not a 1.10 specific answer.
I took this setup from docker compose's cli dockerfile here.
Your mileage may vary.