Golang cross compiling with CGO inside docker image - docker

Question

Requirement: An application has to be containerised as a docker image and needs to support arm64 and amd64 architectures.
Codebase: It is a golang application that needs to make use of git2go library and must have CGO_ENABLED=1 to build the project. The minimum reproducible example can be found here on github.
Host machine: I am using arm64 M1 mac and docker desktop to build the app but the results are similar on our amd64 Jenkins CI build system.
Dockerfile:
FROM golang:1.17.6-alpine3.15 as builder
WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN apk add --no-cache libgit2 libgit2-dev git gcc g++ pkgconfig
RUN go mod download
COPY main.go main.go
ARG TARGETARCH TARGETOS
RUN CGO_ENABLED=1 GO111MODULE=on GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -tags static,system_libgit2 -a -o gitoperations main.go
FROM alpine:3.15 as runner
WORKDIR /
COPY --from=builder /workspace/gitoperations .
ENTRYPOINT ["/gitoperations"]
Build steps:
docker buildx create --name gitops --use
docker buildx build --platform=linux/amd64,linux/arm64 --pull .
This setup works but the build is taking way too long when building for different arch. The time difference between this specific build step:
RUN CGO_ENABLED=1 GO111MODULE=on GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -tags static,system_libgit2 -a -o gitoperations main.go is always 10x longer when building for different arch:
example:
On arm64 M1 mac (without rossetta): Building arm64 executable takes ~30s and amd64 takes ~300seconds.
On our amd64 Jenkins CI system: Building arm64 executable takes 10x longer than building amd64 executable.
This build times can be seen by looking at the docker buildx build command output.
I believe (and I can most certainly be wrong) its happening because docker is using qemu emulation when building for a cpu architecture thats not the same as host machine's cpu arch. So I want to make use of golang cross-compilation capabilities to speed up the build times.
What I have tried: I thought of having a single builder stage in this dockerfile for arm and amd arch by trying this syntax:
FROM --platform=$BUILDPLATFORM golang:1.17.6-alpine3.15 as builder.
But using the same docker build commands after making this change to dockerfile gives build errors, this is what I get when running on arm64 M1 mac:
> [linux/arm64->amd64 builder 9/9] RUN CGO_ENABLED=1 GO111MODULE=on GOOS=linux GOARCH=amd64 go build -tags static,system_libgit2 -a -o gitoperations main.go:
#0 1.219 # runtime/cgo
#0 1.219 gcc: error: unrecognized command-line option '-m64'
After reading through golang CGO documentation I think this error is happening because go is not selecting the correct c compiler that is able to build for both architectures and I need to set the CC env variable to instruct go which c compiler to use.
Question: Am I right in assuming that qemu is causing the build time difference and it can be reduced by using golang's native cross-compilation functionality?
How can I make go build compile for amd64 and arm64 from any host machine using docker desktop as I dont have any experience working with C code and gcc and I am not sure what value I should set for CC flag in the go build command if I need to support linux/amd64 and linux/arm64?

Answer 1

Btw I want to share a little bit of my experience, I also tried to build a Go application Docker image using GitHub Actions and go build was run inside the Docker, although the application was not too big, but I felt it was quite a long process. Then I tried to build the binary outside the Docker and the process was much faster, especially if we store the cache from the previous build process.

Answer 2

To be able to compile C code on go you need to set CC variable to arm cross compiler. You can see your CC variable by go env. The error you have is related with the native compiler in the host system you use. You should sudo apt-get install gcc-arm-linux-gnueabi in your dockerfile. After you downloaded necessary cross compilation tools. You need to link your gcc command to compiler you have downloaded via command I mentioned. Then you should be able to compile your application for arm64.
Could you also share your go env output. You might have to edit GOGCCFLAGS variable too.

Answer 3

Yes, you are correct in assuming that qemu is causing the build time difference and using golang's native cross-compilation functionality could help reduce the build time.
To compile for both amd64 and arm64 architectures from any host machine using Docker Desktop, you can set the CC environment variable in the go build command to specify the compiler that Go should use.
For example, to cross-compile for arm64, you can set CC=aarch64-linux-gnu-gcc. For amd64, you can set CC=gcc. You can set these values as arguments in your docker buildx build command.
Note that you would need to have the appropriate cross-compilation tools installed on the host machine for this to work, such as aarch64-linux-gnu-gcc for arm64.