I can't remember where I saw it, I thought it was on Datadog or on NewRelic or maybe CloudFlare? but I remember someone mentioning that with Golang, they run release binaries in production (of course), and within their Docker containers, they also include a separate file containing debug symbols in case a crash occurs so as to be able to see what happened.
Background
I'm building and running in Docker with a Dockerfile like this:
# do all of our docker building in one image
FROM golang:latest as build
WORKDIR /usr/src/api
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# build the application with relevant flags to make it completely self-contained for a scratch container
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags "-s" -a -installsuffix cgo -o app
# and then copy the built binary to an empty image
FROM ubuntu:latest
COPY --from=build /usr/src/api/app /
COPY --from=build /usr/src/api/config.defaults.json /config.json
COPY --from=build /usr/src/api/logo.png /
# default to running in a dev environment
ENV ENV=dev
EXPOSE 8080
ENTRYPOINT ["/bin/bash"]
If I don't use the flags above, the binary will fail to execute in alpine and scratch base images:
standard_init_linux.go:219: exec user process caused: no such file or directory
Running this in ubuntu:latest just works, so the above compile flags seem to fix the issue for alpine and scratch.
Question
With this environment in mind, is it possible to have go build emit debug symbols into a separate file to live alongside my static binary in the Docker image?
Use go tool compile using -E flag to Debug symbol export. Is that what you need?
$ go tool compile -E *.go
Type:
go tool compile
for more help regarding how to use it and what are the options available.
Reference:
https://golang.org/cmd/compile/
You don't need to use " -a -installsuffix cgo" flags when building with CGO_ENABLED=0 -- just setting the environment variable will do the trick.
You are building with "-ldflags -s", which is going to strip out all debug symbols and ELF symbol table information. Instead of doing that, do a regular build, archive that executable (in case you need the symbols later) and then remove symbols using strip. E.g.
$ CGO_ENABLED=0 GOOS=linux go build -o app.withsymbols
$ cp app.withsymbols /my/archive/for/debugging/production/issues
$ strip app.withsymbols -o app.stripped
$ cp app.stripped /production/bin
That should give you the behavior you're asking for (e.g. a small production binary, but also a backup binary with symbols for debugging problems in production).
Related
I have go web application and im trying to deploy on Docker but im keep getting this messages. Im Running this on Windows 10 enterprise 11th Gen Intel(R) Core(TM) i7-1185G7 # 3.00GHz 64 bit
Docker file
FROM golang:latest
RUN mkdir /app
COPY bin/main /app/main
WORKDIR /app
CMD ["/app/main"]
MAKEFILE
GOOS=linux
GOARCH=amd64
build:
go build -o bin/main main.go
run:
go run main.go
compile:
echo "Compiling for every OS and Platform"
GOOS=linux GOARCH=arm go build -o bin/main-linux-arm main.go
GOOS=linux GOARCH=arm64 go build -o bin/main-linux-arm64 main.go
GOOS=freebsd GOARCH=386 go build -o bin/main-freebsd-386 main.go
im running this commands
go build -o bin/main main.go
docker build -t tiny .
docker run -p 127.0.0.1:8080:8080 tiny
error im getting: exec /app/main: exec format error
go file: log.Fatal(http.ListenAndServe (":8080",nil))
When your Dockerfile says
COPY bin/main /app/main
you're copying a main binary built on your (Windows) host system into your (Linux) Docker image. Since the operating systems don't match, you're getting that "exec format error" message.
It's very common when you're building a Docker image to do the actual compilation step in the Dockerfile itself. In a compiled language like Go, a further extremely common approach is to use a multi-stage build so that the Go toolchain itself isn't part of the final image; this gives you a much smaller image and doesn't republish your source code to end users.
FROM golang:1.19-bullseye AS build
WORKDIR /app
COPY ./ ./
RUN go build -o main ./
FROM debian:bullseye
COPY --from=build /app/main /usr/local/bin/main
CMD ["main"]
This approach ignores your Makefile, which isn't really using any Make features (for example, there is nowhere one rule depends on another, and there is nowhere you might skip a rule execution if its output file already exists). This particular Dockerfile builds a dynamically-linked binary, and makes sure the build and execution phase are using the same major version of the same Linux distribution, and therefore the same system C library.
I was trying to get a minimal example go app running inside a docker container.
But I kept getting exec /app: no such file or directory when running the container.
I checked and double checked all my paths in the image where I built and copied my application data, even looked inside the container with interactive shell to verify my app was there but nothing worked.
My Dockerfile:
# syntax=docker/dockerfile:1
# BUILD-STAGE
FROM golang:1.17-alpine as build
WORKDIR /go/app
COPY . .
RUN go mod download
RUN go build -o /go/bin/app
# RUN-STAGE
FROM gcr.io/distroless/static-debian11
COPY --from=build /go/bin/app .
EXPOSE 8080
CMD ["./app"]
After several hours of try and error I finally tried to add CGO_ENABLED=0 to my go build command.... and it worked!
My question is now.... why exactly does this happen?
There was no error when I built my image with CGO implicitly enabled and I even verified that the binary was copied into my second stage!
Why does the runtime say there is no such file when it was built using CGO, but can find it easily when it was built with CGO disabled?
The image that you are using does not contain libc 1 which you build your go app against when using CGO_ENABLED=1.
As suggested in 1 you can use gcr.io/distroless/base.
There was no error when building your app because golang:1.17-alpine contains musl libc (something like libc but smaller).
Then you tried running the app that required libc in an environment that does not have it anymore. So the no such file error.
google container tools
I'm trying to use the lilliput library for Go. It is only made to run on Linux and OS X.
On my Linux (Debian 10.3) host machine as well as my WSL2 setup (Ubuntu 20.04.1), I have no problems running and building my code (excerpt below).
// main.go
package main
import (
"github.com/discordapp/lilliput"
)
func main() {
...
decoder, err := lilliput.NewDecoder(data)
...
}
However, when I try to put it in a Docker container, with the following configuration, it fails to build.
# Dockerfile v1
FROM golang:1.14.4-alpine AS build
RUN apk add build-base
WORKDIR /src
ENV CGO_ENABLED=1
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
RUN go build -o /out/api .
ENTRYPOINT ["/out/api"]
EXPOSE 8080
I already tried adjusting the Dockerfile with different approaches, for example:
FROM alpine:edge AS build
RUN apk update
RUN apk upgrade
RUN apk add --update go=1.15.3-r0 gcc=10.2.0-r5 g++=10.2.0-r5
WORKDIR /app
RUN go env
ENV GOPATH /app
ADD . /app/src
WORKDIR /app/src
RUN go get -d -v
RUN CGO_ENABLED=1 GOOS=linux go build -o /app/bin/server
FROM alpine:edge
WORKDIR /app
RUN cd /app
COPY --from=build /app/bin/server /app/bin/server
CMD ["bin/server"]
Both result in the following build log:
https://pastebin.com/zMEbEac3
For completeness, the go env of the host machine.
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/kingofdog/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/kingofdog/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go-1.11"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.11/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/kingofdog/{PROJECT FOLDER}/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build589460337=/tmp/go-build -gno-record-gcc-switches"
I already searched online for this error, but all I could find dealt with errors in the way others imported C libraries in their Go project. Yet, in my case I'm quite sure that it is not a mistake regarding the source code but rather a configuration mistake of the docker container, as the code itself works perfectly fine outside Docker and I couldn't find a similar issue on the lilliput repository.
The alpine docker image is a minimalistic Linux version - using musl-libc instead of glibc - and is typically used for building tiny images.
To get the more featureful glibc - and resolve your missing CGO dependencies - use the non-alpine version of the golang Docker image to build your asset:
#FROM golang:1.14.4-alpine AS build
#RUN apk add build-base
FROM golang:1.14.4 AS build
Did you build the dependencies?
You have to run the script to build the dependencies on Linux.
Script: https://github.com/discord/lilliput/blob/master/deps/build-deps-linux.sh
Their documentation mentions:
Building Dependencies
Go does not provide any mechanism for arbitrary building of dependencies, e.g. invoking make or cmake. In order to make lilliput usable as a standard Go package, prebuilt static libraries have been provided for all of lilliput's dependencies on Linux and OSX. In order to automate this process, lilliput ships with build scripts alongside compressed archives of the sources of its dependencies. These build scripts are provided for OSX and Linux.
In case it still fails, then issue might be linked to glibc-musl because alpine images have musl libc instead of glibc (GNU's libc). So, you can try it with maybe Ubuntu/ CentOS/etc. minimal images or find a way to get glibc on alpine.
I'm having a ripping good time using skaffold to develop some kubernetes services, but one of the longest steps in my cycle is pulling all the dependencies for the container.
Does anyone have recommendations on how I can best cache all the dependencies in a layer? Are there best practices with building go binaries inside docker containers? Should I have a layer where I do a go get? (Also I'm a novice building go binaries, don't know all the bells and whistles yet.)
I encountered exactly the same issue while integrating skaffold with kubebuilder and the idea to completely resolve this issue was:
To install buildkit, for example: brew install buildkit;
To enable skaffold for local build by having something like this:
apiVersion: skaffold/v1beta9
kind: Config
build:
local:
useBuildkit: true
useDockerCLI: true
...
Edit the Dockerfile to enable it:
# syntax=docker/dockerfile:experimental
# Build the manager binary
FROM golang:1.12.5 as builder
...
# Build
#RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -o manager main.go
...
Then at first time it will still download every dependency but afterwards it will use the cache, which dramatically speeds up the build process.
I agree with Grigoriy Mikhalkin. Regarding your performance improvements, I want to name the Docker Build Enhancements which are based on moby/buildkit. At the time of writing, the tools aren't properly documented, but with some trial and error, you might find your solution.
Using buildkit, you can use a cache in your RUN statements in order to reduce the time of subsequent executions. They provide an example of Go in their docs, as well. In order to have it work, you have to enable the experimental features for both the Docker daemon and client (described on the link above).
I found this article after googling some more which covers the process: Using go mod download to speed up Golang Docker builds
The gist of the trick is to copy your go.mod and go.sum files into the container, then run go mod download to download dependencies, and then in another step continue with your build.
This works because your the go.mod and go.sum files do not change unless you add more dependencies. So, when the next RUN statement happens which is the go mod download docker knows that that it can cache this layer. (Source)
FROM golang:1.13.9-buster as builder
# Make Build dir
RUN mkdir /build
WORKDIR /build
# Copy golang dependency manifests
COPY go.mod .
COPY go.sum .
# Cache the downloaded dependency in the layer.
RUN go mod download
# add the source code
COPY . .
# Build
RUN go build -o app
# Run
FROM debian:buster-slim
COPY --from=builder /build
WORKDIR /app
CMD ["./app"]
It's customary to use multi-stage build for go services.
So that all dependencies resolved and executable is built at build stage. And final stage is actually running executable. As a result, your final image will be more slim in size. Although, it's not gonna speed up dependency resolution stage.
I am attempting to create a container with my Go binary in for use as a database migrator. If I run the binary it works perfectly, however, I am struggling to put it into a container and run it in my docker-compose stack.
Below is my Dockerfile.
FROM golang:1.11 AS build_base
WORKDIR /app
ENV GO111MODULE=on
# We want to populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .
RUN go mod download
FROM build_base AS binary_builder
# Here we copy the rest of the source code
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build
#In this last stage, we start from a fresh Alpine image, to reduce the image size and not ship the Go compiler in our production artifacts.
FROM alpine AS database-migrator
# We add the certificates to be able to verify remote instances
RUN apk add ca-certificates
COPY --from=binary_builder /app /app
ENTRYPOINT ["/app/binary-name"]
When I run my docker-compose stack the MySQL database gets setup correctly but I receive this error in the logs for my database migrator container.
data-migrator_1 | standard_init_linux.go:190: exec user process caused "exec format error"
I had the same error message. For me the fix was to cross build the for the right architecture. In my case amd64. Like this:
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o [OUTPUT] .
Check if this is similar to containers/buildah issue 475 :
I think it is because the system does not know how to execute the file.
FYI: What's the appropriate Go shebang line?
Also be aware of the difference between the shell form and exec form of CMD/ENTRYPOINT in Dockerfile.
just adding #!/bin/bash to my entry point file fixed the issue.
Or:
Turns out the #!/bin/bash was in my entry point file, but since I did a copy and paste into that file, the first line was a newline, not the #!/bin/bash, effectively ignoring it.
If this helps anyone as well: Deleted the empty line and all worked fine.
Or:
In case anyone finds this useful, you can get this issue if your shell script uses CRLF for line endings and/or UTF-8 with BOM (e.g. if you created a shell script file in Visual Studio).
Changing to LF only and straight UTF-8 fixed it for me.
Or (probably not your case, but to be complete):
For anyone who got a standard_init_linux.go:190: exec user process caused "no such file or directory" error after applying this fix, you're probably on an alpine base image which does not come with bash.
Replacing #!/bin/bash with #!/bin/sh will do the trick!