I'm writing a simple app in GO and using postges I have this folder structure
|--- Dockerfile
|--- api.go
|--- vendor/
database/
init.go
and here is my dockerfile
FROM golang:1.9
ARG app_env
ENV APP_ENV $app_env
COPY . .
WORKDIR /project
RUN go get ./vendor/database
RUN go get ./
RUN go build
CMD if [ ${APP_ENV} = production ]; \
then \
api; \
else \
api; \
fi
EXPOSE 8080
when I working docker-compose up I m getting this error:
Error Message
Step 6/10 : RUN go get ./vendor/database
---> Running in 459740ba584c
can't load package: package ./vendor/database: cannot find package "./vendor/database" in:
/project/vendor/database
Service 'api' failed to build: The command '/bin/sh -c go get ./vendor/database' returned a non-zero code: 1
Where am I going wrong with the project structure?
You are copying the source to default directory of the base image with the command COPY . .. Then you are making /project as working directory with WORKDIR /project. So when you run RUN go get ./vendor/database, the command is actually run in /project/vendor/database which does not exist. Switch the order of COPY and WORKDIR as follows
WORKDIR /project
COPY . .
Related
I want to test my Go code in a CI environment which requires using Docker. How do I create a Docker image that has all the dependencies listed in go.mod already downloaded and compiled so that docker run $IMG go test uses the cached artifacts?
The desired properties of this image:
The image only uses go.mod to compile dependencies. I don't want to use the full source code because then any change to source code would invalidate the Docker layer that hold cached dependencies.
docker run $IMG go test ./... doesn't redownload or recompile dependencies listed in go.mod.
Avoid experimental Docker features.
Existing approaches
Parsing go.mod and using go get
From https://github.com/golang/go/issues/27719#issuecomment-578246826
This approach is close but it doesn't appear to use GOCACHE when I run go test. This also appears to choke on certain module paths, like gopkg.in/DataDog/dd-trace-go.v1:
FROM golang:1.13
WORKDIR /src
COPY go.mod ./
RUN set -eu \
&& go mod graph \
| cut -d '#' -f 1 \
| cut -d ' ' -f 2 \
| sort -u \
| sed -e 's#dd-trace-go.v1#&/ddtrace#' \
| xargs go get -v
docker run --mount /src:/src $IMG go test ./...
Using DOCKER_BUILDKIT with a mount cache
Originally described in https://github.com/golang/go/issues/27719#issuecomment-514747274. This only works for go build. I can't use it for go test because the cache mount is unmounted after the RUN command so it doesn't exist in the created Docker image.
This also depends on experimental docker features.
# syntax = docker/dockerfile:experimental
FROM golang:1.13 as go-builder
ARG VERSION
WORKDIR /src
COPY . /src/
# With a mount cache, Docker will cache the target directories for future
# invocations of this RUN layer. Meaning, once this command is run once, all
# successive calls will use the already downloaded and already compiled assets.
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go build ./server
I often put COPY go.mod on the very begin of a Dockerfile, as it does not change that often.
FROM golang:1.14.3 as builder
WORKDIR /app
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
RUN go build -tags netgo -ldflags '-extldflags "-static"' -o app .
FROM centos:7
WORKDIR /root
COPY --from=builder /app/app .
So, if you go mod does not change, the line RUN go mod download only run for the first time.
In your dockerfile if you run go get ./... it will download all the dependencies to your docker image based on the go.mod and go.sum. You can also add the --insecure flag to the go get command if you are dealing with internal self signed repositories. Very similar to go mod download. Aside from that you can have a shell script that actually initiates the Go test ./... and reports it out to your CI environment if that allows, I know that Gitlab allows this.
FROM golang:1.15-alpine AS builder
RUN apk add --update git gcc musl-dev
RUN apk update && apk add bash
COPY . /app
RUN go version
WORKDIR /app
ENV CGO_ENABLED=0
RUN git config --global http.sslVerify false
RUN go get ./...
WORKDIR /app
RUN chmod +x ./unitTest.sh
RUN ./unitTest.sh
WORKDIR /app/cmd/svr
RUN go build -o app
RUN chmod 700 app
FROM alpine:latest
WORKDIR /root/
ARG build_stamp
ARG git_commit
ARG build_number
ENV BUILD_STAMP=$build_stamp
ENV GIT_COMMIT=$git_commit
ENV BUILD_NUMBER=$build_number
COPY --from=builder /app/cmd/svr .
EXPOSE 8000
CMD ["./app"]
and the script
#!/usr/bin/env bash
TESTS=$(go test -v -covermode=count -coverprofile=count.txt ./...)
echo "$TESTS"
if echo "$TESTS" | grep -q "FAIL" ; then
echo ""
echo "One or more Unit Tests for app have Failed. Build will now fail. Pipeline will also fail..."
echo ""
exit 1
else
echo ""
echo "All Unit Tests for application have passed!"
echo "Running Code Coverage..."
echo ""
COVERAGE=$(go tool cover -func=./count.txt)
echo "$COVERAGE"
exit 0
fi
So, I am trying to dockerize a golang application with different directories containing supplementary code for my main file.
I am using gorilla/mux. The directory structure looks like this.
$GOPATH/src/github.com/user/server
|--- Dockerfile
|--- main.go
|--- routes/
handlers.go
|--- public/
index.gohtml
It works on my host machine with no problem. The problem is that when I try to deploy the docker image it does not run and exits shortly after creation. I have tried changing the WORKDIR command in my dockerfile to /go/src and dump all my files there, but still no luck. I have also tried the official documentation on docker hub. Doesn't work either.
My Dockerfile.
FROM golang:latest
WORKDIR /go/src/github.com/user/server
COPY . .
RUN go get -d github.com/gorilla/mux
EXPOSE 8000
CMD ["go","run","main.go"]
My golang main.go
package main
import (
"github.com/gorilla/mux"
"github.com/user/server/routes"
"log"
"net/http"
"time"
)
func main(){
//...
}
I get this error message when I check the logs of my docker image.
Error Message
main.go:5:2: cannot find package "github.com/user/server/routes" in any of:
/usr/local/go/src/github.com/user/server/routes (from $GOROOT)
/go/src/github.com/user/server/routes (from $GOPATH)
Try the following Docker file:
# GO Repo base repo
FROM golang:1.12.0-alpine3.9 as builder
RUN apk add git
# Add Maintainer Info
LABEL maintainer="<>"
RUN mkdir /app
ADD . /app
WORKDIR /app
COPY go.mod go.sum ./
# Download all the dependencies
RUN go mod download
COPY . .
# Build the Go app
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
# GO Repo base repo
FROM alpine:latest
RUN apk --no-cache add ca-certificates curl
RUN mkdir /app
WORKDIR /app/
# Copy the Pre-built binary file from the previous stage
COPY --from=builder /app/main .
# Expose port 8000
EXPOSE 8000
# Run Executable
CMD ["./main"]
Here, we are creating an intermediate docker builder container, copying the code into it, build the code inside the builder container and then copy the binary image to the actual docker.
This will help in both having all the dependencies in the final container and also, the size of the final image will be very small
Been stuck on this for the last 3 days. I'm building an image in a docker and
copy command fails due to not finding the right directory.
FROM python:3.6.7-alpine
WORKDIR /usr/src/app
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip3 install -r requirements.txt
COPY . /usr/src/app
CMD python3 manage.py run -h 0.0.0.0
which is run by this docker-dev file:
version: '3.7'
services:
users:
build:
context: ./services/users
dockerfile: Dockerfile-dev
volumes:
- './services/users:/usr/src/app'
ports:
- 5001:5000
environment:
- FLASK_APP=project/__init__.py
- FLASK_ENV=development
and getting this error:
Building users
Step 1/6 : FROM python:3.6.7-alpine
---> cb04a359db13
Step 2/6 : WORKDIR /usr/src/app
---> Using cache
---> 06bb39a49444
Step 3/6 : COPY ./requirements.txt /usr/src/app/requirements.txt
ERROR: Service 'users' failed to build: COPY failed: stat /var/snap/docker/common/var-lib-docker/tmp/docker-builder353668631/requirements.txt: no such file or directory
I don't even know where to start with debugging this. When I tried to access the directory it gave me permission error. So I tried to run the command with sudo which didn't help. Any thoughts ?
Little late to reply, but second COPY command COPY . /usr/src/app replaces the /usr/src/app content generated by RUN pip3 install -r requirements.txt.
Try
FROM python:3.6.7-alpine
WORKDIR /usr/src/app
# install in temp directory
RUN mkdir /dependencies
COPY ./requirements.txt /dependencies/requirements.txt
RUN cd /dependencies && pip3 install -r requirements.txt
COPY . /usr/src/app
# copy generated dependencies
RUN cp -r /dependencies/* /usr/src/app/
CMD python3 manage.py run -h 0.0.0.0
As larsks suggests in his comment, you need the file in the services/users directory. To understand why, an understanding of the "context" is useful.
Docker does not build on the client, it does not see your current directory, or other files on your filesystem. Instead, the last argument to the build command is passed as the build context. With docker-compose, this context defaults to the current directory, which you will often see as . in a docker build command, but you can override that as you've done here with ./services/users as your context. When you run a build, the very first step is to send that build context from the docker client to the server. Even when the client and server are on the same host (a common default, especially for desktop environments), this same process happens. Files listed in .dockerignore, and files in parent directories to the build context are not sent to the docker server.
When you run a COPY or ADD command, the first argument (or all but the last argument when you have multiple) refer to files from the build context, and the last argument is the destination file or directory inside the image.
Therefore, when you put together this compose file entry:
build:
context: ./services/users
dockerfile: Dockerfile-dev
with this COPY command:
COPY ./requirements.txt /usr/src/app/requirements.txt
the COPY will try to copy the requirements.txt file from the build context generated from ./services/users, meaning ./services/users/requirements.txt needs to exist, and not be excluded by a .dockerignore file in ./services/users.
I had a similar problem building an image with beryllium, and I solved this deleting it into the .dockerignore
$ sudo docker build -t apache .
Sending build context to Docker daemon
10.55MB Step 1/4 : FROM centos ---> 9f38484d220f Step 2/4 :
RUN yum install httpd -y
---> Using cache ---> ccdafc4ae476 Step 3/4 :
**COPY ./**beryllium** /var/www/html COPY failed: stat /var/snap/docker/common/var-lib-docker/tmp/docker-builder04301**
$nano .dockerignore
startbootstrap-freelancer-master
run.sh
pro
fruit
beryllium
Bell.zip
remove beryllium from that file
$ sudo docker build -t apache .
Sending build context to Docker daemon 12.92MB
Step 1/4 : FROM centos
---> 9f38484d220f
Step 2/4 : RUN yum install httpd -y
---> Using cache
---> ccdafc4ae476
Step 3/4 : COPY ./beryllium /var/www/HTML
---> 40ebc02992a9
Step 4/4 : CMD apachectl -DFOREGROUND
---> Running in dab0a406c89e
Removing intermediate container dab0a406c89e
---> 1bea741cfb65
Successfully built 1bea741cfb65
Successfully tagged apache:latest
I'm trying to convert my project to use multi-stage builds. However, the final step always fails with an error:
Step 11/13 : COPY --from=build /bin/grafana-server /bin/grafana-server
COPY failed: stat /var/lib/docker/overlay2/xxxx/merged/bin/grafana-server: no such file or directory
My Dockerfile looks like this:
FROM golang:latest AS build
ENV SRC_DIR=/go/src/github.com/grafana/grafana/
ENV GIT_SSL_NO_VERIFY=1
COPY . $SRC_DIR
WORKDIR $SRC_DIR
# Building of Grafana
RUN \
npm run build && \
go run build.go setup && \
go run build.go build
# Create final stage containing only required artifacts
FROM scratch
COPY --from=build /bin/grafana-server /bin/grafana-server
EXPOSE 3001
CMD ["./bin/grafana-server"]
The build.go build step will output artifacts to ./bin/ -- The error is pretty unhelpful other than telling me the files don't exist where I think they should exist.
My folder structure on my machine is:
--| ~/Documents/dev/grafana/src/grafana/grafana
--------| bin
------------| <grafan-server builds to here>
--------| deploy
------------| docker
----------------| Dockerfile
From ~/Documents/dev/grafana/src/grafana/grafana is where I issue: docker build -t grafana -f deploy/docker/Dockerfile .
To follow-up my comment, the path you set with the WORKDIR is absolute and should be specified in the same way in the COPY --from=build command.
So this could lead to the following Dockerfile:
FROM golang:latest AS build
ENV SRC_DIR=/go/src/github.com/grafana/grafana/
ENV GIT_SSL_NO_VERIFY=1
COPY . $SRC_DIR
WORKDIR $SRC_DIR
# Building of Grafana
RUN \
npm run build && \
go run build.go setup && \
go run build.go build
# Create final stage containing only required artifacts
FROM scratch
ENV SRC_DIR=/go/src/github.com/grafana/grafana/
WORKDIR $SRC_DIR
COPY --from=build ${SRC_DIR}/bin/grafana-server ${SRC_DIR}/bin/grafana-server
EXPOSE 3001
CMD ["./bin/grafana-server"]
(only partially tested)
I'm fairly new to both Docker and go, so this might be something obvious, but my google searches haven't found anything.
I'm trying to build a simple go program with docker, but I'm havign trouble with dependencies.
go file:
package main
import (
"fmt"
"log"
"html"
"net/http"
"github.com/gorilla/mux"
)
func hello(writer http.ResponseWriter, r *http.Request) {
path := mux.Vars(r)["rest"]
fmt.Fprintf(writer, "Hello, %q", html.EscapeString(path))
}
func main() {
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/{rest:.*}", hello)
log.Println("Listening...")
log.Fatal(http.ListenAndServe(":8080", router))
}
Docker file:
FROM golang:latest
RUN mkdir /app
ADD ./HelloWorld.go /app/
WORKDIR /app
RUN go get ./*.go
RUN go build -o main .
CMD ["/app/main"]
Error:
Sending build context to Docker daemon 6.482MB
Step 1/7 : FROM golang:latest
---> 138bd936fa29
...
Step 5/7 : RUN go get ./*.go
---> Running in 1e29844961a2
HelloWorld.go:9:5: cannot find package "github.com/gorilla/mux" in any of:
/usr/local/go/src/github.com/gorilla/mux (from $GOROOT)
/go/src/github.com/gorilla/mux (from $GOPATH)
The command '/bin/sh -c go get ./*.go' returned a non-zero code: 1
You may use my Dockerfile as a base. First stage produces working image. That’s enough for many cases.
https://github.com/lisitsky/go-site-search-string/blob/light-docker/Dockerfile
If you want to shrink image size from ~800MB to about ~10-20MB use second stage too.
Just use $GOPATH everywhere in your path to build your images, e.g. $GOPATH/src/app instead of /app
Full example of multistage build:
FROM golang:alpine as builder
RUN apk add --update git
RUN mkdir -p $GOPATH/src/build
ADD . $GOPATH/src/build/
WORKDIR $GOPATH/src/build
RUN go get ./...
RUN go build -o main .
FROM alpine
RUN adduser -S -D -H -h /app appuser
USER appuser
COPY --from=builder /go/src/build/main /app/
WORKDIR /app
CMD ["./main"]