I have a monorepo that has holds various Go services and libraries. The directory structure is like the following:
monorepo
services
service-a
- Dockerfile
go.mod
go.sum
This go.mod file resides in the root of the monorepo directory and the services use the dependencies stated in that file.
I build the Docker image with this command:
docker build -t some:tag ./services/service-a/
When I try to build my Docker image from the root of monorepo directory with the above docker command I get the following error:
COPY failed: Forbidden path outside the build context: ../../go.mod ()
Below is my Dockerfile
FROM golang:1.14.1-alpine3.11
RUN apk add --no-cache ca-certificates git
# Enable Go Modules
ENV GO111MODULE=on
# Set the Current Working Directory inside the container
WORKDIR /app
# Copy go mod and sum files
COPY ../../go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o service-a
ENTRYPOINT ["/app/service-a"]
Is there something I have to do to be able to add files into my Docker image that aren't in the current directory without having to have a separate go.mod and go.sum in each service within the monorepo?
Docker only allows adding files to the image from the context, which is by default the directory containing the Dockerfile. You can specify a different context when you build, but again, it won't let you include files outside that context:
docker build -f ./services/service-a/Dockerfile .
This should use the current directory as the context.
Alternatively, you can create a temp directory, copy all the artifacts there and use that as the build context. This can be automated by a makefile or build script.
You can build and manage your docker containers using docker-compose, then this problem can be solved with the help context directive, for example:
project_folder
├─── src
│ └── folder1
│ └── folder2
│ └── Dockerfile
├── docker-compose.yaml
└── copied_file.ext
docker-compose.yaml
version: '3'
services:
your_service_name:
build:
context: ./ #project_folder for this case
dockerfile: ./src/folder1/folder2/Dockefile
Dockerfile
FROM xxx
COPY copied_file.ext /target_folder/
build or rebuild services:
docker-compose build
run a one-off command on a service:
docker-compose run your_service_name <command> [arguments]
Related
I tried to copy all files from some package into my WORKDIR in my Dockerfile
My Dockerfile look like this:
FROM python:3.8
WORKDIR /yahoo_finance_app
COPY requirements.txt /yahoo_finance_app/requirements.txt
COPY /manage_db/* /yahoo_finance_app/manage_db/
RUN pip3 install -r requirements.txt
COPY .. .
And I execute it with docker-compose which looks like this:
version: "3.8"
services:
py-api-yahoo-finance:
build: ./api-yahoo-finance/yahoo
ports:
- "5000:5000"
container_name: api_yahoo_finance
command: python manage.py runserver 0.0.0.0:5000
The file tree looks like this:
├── api_yahoo_finance
├── yahoo
├── Dockerfile
├── manage_db
I try to copy all the files from manage_db into my WORKDIR in my Dockerfile, but I got the following error:
=> ERROR [5/7] COPY /manage_db/* /yahoo_finance_app/manage_db/ 0.0s
[5/7] COPY /manage_db/* /yahoo_finance_app/manage_db/:
lstat /var/lib/docker/tmp/buildkit-mount563896323/manage_db: no such file or directory
ERROR: Service 'py-api-yahoo-finance' failed to build : Build failed
And I take the relative path, so the folder is existing.
Thanks to all the helpers.
Dockerfiles are built in a variety of environments.
As a security and practical requirement Dockerfile commands can only access files that are in the context directory, either directly or in one of its children.
To keep Dockerfiles many levels deep, you can do this:
py-api-yahoo-finance:
build:
context: .
file: ./api-yahoo-finance/yahoo/Dockerfile
and then all paths in the Dockerfile will be relative to the project root where the compose.yml is.
I created a docker file.
FROM node:13.6.0-alpine3.10
WORKDIR /src
RUN apk add --no-cache bash
COPY ./package.json .
COPY ./package-lock.json .
RUN npm install
COPY . .
EXPOSE 8081
CMD npm run start:dev
Structure of my project.
.
└── my-app
└── docker-compose.yml
└── ...
└── server
└── docker
└── Dockerfile
└── src
└── ....
├── package.json
├── package-lock.json
└── ...
When I build the container, I get an error.
ERROR: Service 'server' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder791989542/package.json: no such file or directory
I understand that my files are in a different directory. I tried to solve my problems like this.
COPY ./../package.json .
COPY ./../package-lock.json .
But I also got an error
ERROR: Service 'server' failed to build: COPY failed: Forbidden path outside the build context: ../package.json ()
How to tell the docker so that he is looking for files not in the docker folder where the dockerfile file is located, but in the src folder, where my package-lock.json and package.json files are located
I created a docker-compose.yml file.
version: "3.3"
services:
server:
container_name: server
command:
- npm
- run
- start:dev
build:
context: ./server/docker
dockerfile: dockerfile
environment:
PORT: 8081
ports:
- 8081:8081
restart: on-failure
volumes:
- ./server:/src/
As explained earlier in comments and previous answers, you cannot copy files which are outside of the build context. So you either need to change the context up the directory path or to move the needed files inside the current context.
Looking at your layout, I would go for the first solution and by default use your base directory as context (once you understand the solution, you can adapt context and dockerfile to fit your exact needs).
In your Dockerfile, change the following lines:
COPY ./server/package.json .
COPY ./server/package-lock.json .
Note: the line COPY . . should be removed (or you need to explain a littke better what you actually try to achieve with this)
The build section in your docker-compose.yml file should become:
build:
context: .
dockerfile: server/docker/Dockerfile
The equivalent manual build would be
cd /path/to/my-app
docker build -f server/docker/Dockerfile .
You cannot do that, because first step of build, is to build the context dirs. You need to change the context.
Straight from docker docs,
COPY obeys the following rules:
The path must be inside the context of the build;
you cannot COPY ../something /something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon.
I built a web app with echo. Some source in server.go is
package main
import ...
type TemplateRenderer struct {
templates *template.Template
}
func (t *TemplateRenderer) Render(w io.Writer, name string, data interface{}, c echo.Context) error {
if viewContext, isMap := data.(map[string]interface{}); isMap {
viewContext["reverse"] = c.Echo().Reverse
}
return t.templates.ExecuteTemplate(w, name, data)
}
func main() {
e := echo.New()
e.Static("/static", "static")
renderer := &TemplateRenderer{
templates: template.Must(template.ParseGlob("public/views/*.html")),
}
e.Renderer = renderer
e.GET("/", func(c echo.Context) error {
return c.Render(http.StatusOK, "index.html", map[string]interface{}{})
})
e.Logger.Fatal(e.Start(":8080"))
}
Project tree
├── helper
│ └── string.go
└─── site
├── Dockerfile
├── go.mod
├── go.sum
├── server.go
├── public
│ └── views
│ └── index.html
└── static
I can run go run server.go to start server, everything works well. But got error if run it with docker.
Dockerfile
FROM golang:1.15.2-alpine3.12 AS builder
WORKDIR /app
COPY . .
WORKDIR /app/site
RUN CGO_ENABLED=0 GOOS=linux go build -o server
FROM alpine:3.12
COPY --from=builder /app/site /bin/.
ENTRYPOINT [ "server" ]
Built a docker image by
docker build -t gcr.io/${PROJECT_ID}/myapp -f site/Dockerfile .
Run app in docker
docker run --rm -p 8080:8080 gcr.io/${PROJECT_ID}/myapp
panic: html/template: pattern matches no files: `public/views/*.html`
goroutine 1 [running]:
html/template.Must(...)
/usr/local/go/src/html/template/template.go:372
main.main()
/app/site/server.go:76 +0x2af
It seems the public folder didn't been copied into image. But there wasn't any error when built the image. What's wrong?
Firstly, you are using a multi-stage Docker build which is perfect for only copying the compiled binary to the final image. In your Dockerfile, however, you are copying the entire build directory - the binary server as well as all the source.
Secondly, your main problem is when the image is run as a container, the default work directory is / - and thus any paths within your server are not find the html files in /bin/public.
If you ever need to debug a docker image - especially if it is an image based on a linux distro like alpine- simply:
docker run -it myimage /bin/sh
Anyway the 2 simple fixes to your docker:
FROM golang:1.15.2-alpine3.12 AS builder
WORKDIR /app
COPY . .
WORKDIR /app/site
RUN CGO_ENABLED=0 GOOS=linux go build -o server
FROM alpine:3.12
COPY --from=builder /app/site/server /bin
COPY --from=builder /app/site/public /public
ENTRYPOINT [ "/bin/server" ]
In Go 1.16 you can compile these files into the binary itself. So you'll need to upgrade the Go toolchain on your host system, and also the FROM line in your build stage in the Dockerfile. Go 1.16 adds the embed package and a new //go:embed directive to support this.
First, you need to tell the compiler to embed the template files, building a filesystem object:
import "embed"
// templateFiles contains the raw text of the template files.
//go:embed public/views/*.html
var templateFiles embed.FS
Then, when you go to use it, Go 1.16 also adds a corresponding ("html/template").ParseFS function:
renderer := &TemplateRenderer{
templates: template.Must(template.ParseFS(templateFiles)),
}
Now all of the files are embedded in the binary itself, and you shouldn't get "file not found" type errors. You might consider copying only the compiled binary and not anything else into the final image.
# Upgrade to Go 1.16
FROM golang:1.16-alpine3.12 AS builder
# Unchanged from original
WORKDIR /app
COPY . .
WORKDIR /app/site
RUN CGO_ENABLED=0 GOOS=linux go build -o server
FROM alpine:3.12
# Only copy the compiled binary and not the source tree
COPY --from=builder /app/site/server /bin
# Generally prefer CMD to ENTRYPOINT
CMD [ "server" ]
I am following directory structure based on:
https://github.com/golang-standards/project-layout
I have created very simple app and basically I want to containerize it.
Basically I have two files there. server.go which is definition of http endpoint and main file which starts server called main.go under cmd/webserver.
Directory of this project looks like:
./
├── cmd
│ └── webserver
│ └── main.go
├── Dockerfile
├── go.mod
└── server.go
go.mod
module github.com/geborskimateusz/auth
go 1.15
a Dockerfile looks like this
FROM golang:alpine
# Set necessary environmet variables needed for our image
ENV GO111MODULE=on \
CGO_ENABLED=0 \
GOOS=linux \
GOARCH=amd64
# Move to working directory /build
WORKDIR /build
# Copy and download dependency using go mod
COPY go.mod .
RUN go mod download
# Copy the code into the container
COPY . .
# Build the application
RUN go build -o main .
# Move to /dist directory as the place for resulting binary folder
WORKDIR /dist
# Copy binary from build to main folder
RUN cp /build/main .
# Export necessary port
EXPOSE 3000
# Command to run when starting the container
CMD ["/dist/main"]:
Build is successful but the problem is that when I run
docker run -p 3000:3000 geborskimateusz/auth I got:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/dist/main\": permission denied": unknown.
ERRO[0000] error waiting for container: context canceled
What am I missing? I assume that maybe I need to cd in Dockerfile into cmd/webserver where main.go (executable file) is placed.
I actually fixed this by modyfing DockerFile to
FROM golang:alpine
WORKDIR /app
COPY go.mod .
RUN go mod download
COPY . .
RUN cd ./cmd/webserver/ && go build -o main . && cp main ../../ && cd ../../
CMD ["./main"]
I created a docker file.
FROM node:13.6.0-alpine3.10
WORKDIR /src
RUN apk add --no-cache bash
COPY ./package.json .
COPY ./package-lock.json .
RUN npm install
COPY . .
EXPOSE 8081
CMD npm run start:dev
Structure of my project.
.
└── my-app
└── docker-compose.yml
└── ...
└── server
└── docker
└── Dockerfile
└── src
└── ....
├── package.json
├── package-lock.json
└── ...
When I build the container, I get an error.
ERROR: Service 'server' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder791989542/package.json: no such file or directory
I understand that my files are in a different directory. I tried to solve my problems like this.
COPY ./../package.json .
COPY ./../package-lock.json .
But I also got an error
ERROR: Service 'server' failed to build: COPY failed: Forbidden path outside the build context: ../package.json ()
How to tell the docker so that he is looking for files not in the docker folder where the dockerfile file is located, but in the src folder, where my package-lock.json and package.json files are located
I created a docker-compose.yml file.
version: "3.3"
services:
server:
container_name: server
command:
- npm
- run
- start:dev
build:
context: ./server/docker
dockerfile: dockerfile
environment:
PORT: 8081
ports:
- 8081:8081
restart: on-failure
volumes:
- ./server:/src/
As explained earlier in comments and previous answers, you cannot copy files which are outside of the build context. So you either need to change the context up the directory path or to move the needed files inside the current context.
Looking at your layout, I would go for the first solution and by default use your base directory as context (once you understand the solution, you can adapt context and dockerfile to fit your exact needs).
In your Dockerfile, change the following lines:
COPY ./server/package.json .
COPY ./server/package-lock.json .
Note: the line COPY . . should be removed (or you need to explain a littke better what you actually try to achieve with this)
The build section in your docker-compose.yml file should become:
build:
context: .
dockerfile: server/docker/Dockerfile
The equivalent manual build would be
cd /path/to/my-app
docker build -f server/docker/Dockerfile .
You cannot do that, because first step of build, is to build the context dirs. You need to change the context.
Straight from docker docs,
COPY obeys the following rules:
The path must be inside the context of the build;
you cannot COPY ../something /something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon.