Build fails when importing external package with Golang and Docker - docker

Question

I can't build this simple example of confluent kafka using Docker. Probably a trick with go path or an special build parameter, can't find out, tried all the default folders from go, no success.
Dockerfile
FROM golang:alpine AS builder
# Set necessary environmet variables needed for our image
ENV GO111MODULE=on \
CGO_ENABLED=0 \
GOOS=linux \
GOARCH=amd64
ADD . /go/app
# Install librdkafka
RUN apk add librdkafka-dev pkgconf
# Move to working directory /build
WORKDIR /go/app
# Copy and download dependency using go mod
COPY go.mod .
RUN go mod download
# Copy the code into the container
COPY . .
# Build the application
RUN go build -o main .
# Run test
RUN go test ./... -v
# Move to /dist directory as the place for resulting binary folder
WORKDIR /dist
# Copy binary from build to main folder
RUN cp /go/app/main .
############################
# STEP 2 build a small image
############################
FROM scratch
COPY --from=builder /dist/main /
# Command to run the executable
ENTRYPOINT ["/main"]
Source
import (
"fmt"
"github.com/confluentinc/confluent-kafka-go/kafka"
"os"
)
func main() {
if len(os.Args) != 3 {
fmt.Fprintf(os.Stderr, "Usage: %s <broker> <topic>\n",
os.Args[0])
os.Exit(1)
}
broker := os.Args[1]
topic := os.Args[2]
p, err := kafka.NewProducer(&kafka.ConfigMap{"bootstrap.servers": broker})
if err != nil {
fmt.Printf("Failed to create producer: %s\n", err)
os.Exit(1)
}
fmt.Printf("Created Producer %v\n", p)
deliveryChan := make(chan kafka.Event)
value := "Hello Go!"
err = p.Produce(&kafka.Message{
TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
Value: []byte(value),
Headers: []kafka.Header{{Key: "myTestHeader", Value: []byte("header values are binary")}},
}, deliveryChan)
e := <-deliveryChan
m := e.(*kafka.Message)
if m.TopicPartition.Error != nil {
fmt.Printf("Delivery failed: %v\n", m.TopicPartition.Error)
} else {
fmt.Printf("Delivered message to topic %s [%d] at offset %v\n",
*m.TopicPartition.Topic, m.TopicPartition.Partition, m.TopicPartition.Offset)
}
close(deliveryChan)
}
Error
./producer_example.go:37:12: undefined: kafka.NewProducer
./producer_example.go:37:31: undefined: kafka.ConfigMap
./producer_example.go:48:28: undefined: kafka.Event
./producer_example.go:51:19: undefined: kafka.Message

Answer 1

Edit
I can confirm that using the musl build tag works:
FROM golang:alpine as build
WORKDIR /go/src/app
# Set necessary environmet variables needed for our image
ENV GOOS=linux GOARCH=amd64
COPY . .
RUN apk update && apk add gcc librdkafka-dev openssl-libs-static zlib-static zstd-libs libsasl librdkafka-static lz4-dev lz4-static zstd-static libc-dev musl-dev
RUN go build -tags musl -ldflags '-w -extldflags "-static"' -o main
FROM scratch
COPY --from=build /go/src/app/main /
# Command to run the executable
ENTRYPOINT ["/main"]
Works with the test setup as shown below.
Ok, the used version 1.4.0 of github.com/confluentinc/confluent-kafka-go/kafka seems to be generally incompatible with at least the current state of alpine 3.11.
Furthermore, despite my best efforts, I was unable to build a statically compiled binary, fit for the use with FROM scratch.
However, I was able to get you code running against a current version of Kafka. The image is a bit bigger, but I guess working and a bit bigger is better than not working and elegant.
Todos
1. Downgrade to confluent-kafka-go#v1.1.0
As simple as
$ go get -u -v github.com/confluentinc/confluent-kafka-go#v1.1.0
2. Modify your Dockerfile
You were lacking some build dependencies to begin with. And obviously, we need a runtime dependency as well, since we do not use FROM scratch any more. Please note that I also tried to simplify it and left jwilder/dockerize in, which I used so that I do not have to time my test setup:
FROM golang:alpine as build
# The default location is /go/src
WORKDIR /go/src/app
ENV GOOS=linux \
GOARCH=amd64
# We simply copy everything to /go/src/app
COPY . .
# Add the required build libraries
RUN apk update && apk add gcc librdkafka-dev zstd-libs libsasl lz4-dev libc-dev musl-dev
# Run the build
RUN go build -o main
FROM alpine
# We use dockerize to make sure the kafka sever is up and running before the command starts.
ENV DOCKERIZE_VERSION v0.6.1
ENV KAFKA kafka
# Add dockerize
RUN apk --no-cache upgrade && apk --no-cache --virtual .get add curl \
&& curl -L -O https://github.com/jwilder/dockerize/releases/download/${DOCKERIZE_VERSION}/dockerize-linux-amd64-${DOCKERIZE_VERSION}.tar.gz \
&& tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
&& rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
&& apk del .get \
# Add the runtime dependency.
&& apk add --no-cache librdkafka
# Fetch the binary
COPY --from=build /go/src/app/main /
# Wait for kafka to come up, only then start /main
ENTRYPOINT ["sh","-c","/usr/local/bin/dockerize -wait tcp://${KAFKA}:9092 /main kafka test"]
3. Test it
I created a docker-compose.yaml to check wether everything works:
version: "3.7"
services:
zookeeper:
image: 'bitnami/zookeeper:3'
ports:
- '2181:2181'
volumes:
- 'zookeeper_data:/bitnami'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'bitnami/kafka:2'
ports:
- '9092:9092'
volumes:
- 'kafka_data:/bitnami'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper
server:
image: fals/kafka-main
build: .
command: "kafka test"
volumes:
zookeeper_data:
kafka_data:
You can check that the setup works with:
$ docker-compose build && docker-compose up -d && docker-compose logs -f server
[...]
server_1 | 2020/04/18 18:37:33 Problem with dial: dial tcp 172.24.0.4:9092: connect: connection refused. Sleeping 1s
server_1 | 2020/04/18 18:37:34 Connected to tcp://kafka:9092
server_1 | Created Producer rdkafka#producer-1
server_1 | Delivered message to topic test [0] at offset 0
server_1 | 2020/04/18 18:37:36 Command finished successfully.
kfka_server_1 exited with code 0