Multiple Dockerfiles re-using shared library in project - docker

Question

I have the following code structure and I am trying to structure my Dockerfile(s) as to maximize caching and the like.
serverfoo/
Dockerfile
main.go
serverbar/
Dockerfile
main.go
proto/
Dockerfile
sharedproto.proto // Generates a sharedproto.pb.go file to be imported.
Both serverfoo and serverbar import the compiled sharedproto.pb.go file which I manually regenerate on my workstation. This works fine but now I am attempting to containerize my two servers.
The Dockerfiles with my server folders cannot (by default) copy proto/ content. Ideally I pre-compile the protobufs into a sharedproto.pb.go then import a cached version of that file into the two server Dockerfiles. The goal is to cache the compiled protobufs until the underlying protos are modified.
I am new to Docker and need some best practice for this type of thing. I want to avoid a root Dockerfile in my project's directory that just has code to compile a zillion different servers.
I am open to restructuring my project to some degree.

Answer 1

NOTE: I suppose your target is to have on the specific server
container both the compiled go file ( from specific main.go file )
and the compiled protocol buffer file ( from shared
sharedproto.proto file ).
Assuming your files are organized as follow on your workstation:
serverfoo/
Dockerfile
main.go
serverbar/
Dockerfile
main.go
proto/
Dockerfile
sharedproto.proto
You can structure the specific server Dockerfile using the multistage build as follow ( e.g. serverbar Dockerfile ):
#####
# The serverbar Dockerfile
#####
#----
# Compile proto stage
#----
FROM moul/protoc-gen-gotemplate AS protostage
WORKDIR /workspace
# Copy .proto file
COPY proto/sharedproto.proto .
# Compile .pb.go
RUN protoc -I=. --go_out=. sharedproto.proto
#----
# Build stage
#----
FROM golang:1.12.4-alpine3.9 as buildstage
WORKDIR /workspace
COPY serverbar/main.go .
RUN GOOS=linux GOARCH=amd64 go build -o serverbar main.go
#----
# Final stage
#----
FROM alpine:3.7
WORKDIR /home
COPY --from=buildstage workspace/serverbar .
COPY --from=protostage workspace/sharedproto.pb.go .
CMD ["./serverbar"]
Using this approach you basically have the following 3 stages:
proto stage: On the container created on this stage you need to compile the shared protocol buffer source file into the sharedproto.pb.go that then will be included on the third final stage. So here you would need to install on the container the protoc compiler and the related Go plugin. However, as usual with Docker, you'll find a docker image that already includes your needed tools. For this purpose we can start from the moul/protoc-gen-gotemplate docker image.
Specifically the follow Dockerfile instruction generates the workspace/sharedproto.pb.go:
RUN protoc -I=. --go_out=. sharedproto.proto
build stage: Here you need to compile the server source file into the executable one. Also this will be included on the third final stage. To avoid to install Golang we can start from the golang:1.12.4-alpine3.9 docker image that already includes all the needed tools.
Specifically the follow Dockerfile instruction generates the workspace/serverbar executable:
RUN GOOS=linux GOARCH=amd64 go build -o serverbar main.go
final stage: This is the server container that we'll then upload on our Docker registry for test or production where we'll copy the files compiled on the previous two stage with the following commands:
COPY --from=buildstage workspace/serverbar .
COPY --from=protostage workspace/sharedproto.pb.go .
One of the advantages of this solution is that, for each server build, you can cache the compiled protobufs until the underlying protos are modified.
Example:
Building first time the serverbar container we can note that .proto compilation is performed on a new container with id 92ae211bd27d:
> docker build -f serverbar/Dockerfile .
Sending build context to Docker daemon 10.24kB
Step 1/13 : FROM moul/protoc-gen-gotemplate AS protostage
---> 635345fde953
Step 2/13 : WORKDIR /workspace
---> Using cache
---> de8890a5e775
Step 3/13 : COPY proto/sharedproto.proto .
---> 1253fa0576aa
Step 4/13 : RUN protoc -I=. --go_out=. sharedproto.proto
---> Running in 8426f5810b98
Removing intermediate container 8426f5810b98
---> 92ae211bd27d <=========================================
Step 5/13 : FROM golang:1.12.4-alpine3.9 as buildstage
---> b97a72b8e97d
Step 6/13 : WORKDIR /workspace
....
Building then a second time without modifying the sharedproto.proto we can note that container with id 92ae211bd27d is re-used from cache.
> docker build -f serverbar/Dockerfile .
Sending build context to Docker daemon 10.24kB
Step 1/13 : FROM moul/protoc-gen-gotemplate AS protostage
---> 635345fde953
Step 2/13 : WORKDIR /workspace
---> Using cache
---> de8890a5e775
Step 3/13 : COPY proto/sharedproto.proto .
---> Using cache
---> 1253fa0576aa
Step 4/13 : RUN protoc -I=. --go_out=. sharedproto.proto
---> Using cache <=========================================
---> 92ae211bd27d
Step 5/13 : FROM golang:1.12.4-alpine3.9 as buildstage
---> b97a72b8e97d
....