I'm running with an issue where I can't build Dockerfile that includes multiple proto files(server and text). The server proto is within the Dockerfile dir, but the text proto is within the Dockerfile parent. So I'm building the Dockerfile in the parent dir to COPY the text proto to the Docker build.
The Docker build complaining about proto/text.proto: File not found. even though I COPY the proto/text.proto to the exact location as server/proto/server.proto.
Here are all my files:
DockerFile
FROM --platform=linux/x86_64 golang:1.19.3-bullseye
# Install grpc
RUN go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#v1.2 && \
go install google.golang.org/protobuf/cmd/protoc-gen-go#v1.28
WORKDIR /app
COPY server/. /app
COPY proto/text.proto /app/proto/text.proto
# Install protoc and zip system library
RUN apt-get update && apt-get install -y zip && \
mkdir /opt/protoc && cd /opt/protoc && wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.0/protoc-3.7.0-linux-x86_64.zip && \
unzip protoc-3.7.0-linux-x86_64.zip
# Copy the grpc proto file and generate the go module
RUN /opt/protoc/bin/protoc --go_out=/app/proto --proto_path=/app/proto --go_opt=paths=source_relative --go-grpc_out=/app/proto --go-grpc_opt=paths=source_relative /app/proto/text.proto /app/proto/server.proto
EXPOSE 5051
RUN go build -o /server
ENTRYPOINT ["/server"]
Dir Tree
1.text
├── admin
│ ├── Dockerfile
│ ├── app.js
│ ├── package.json
│ └── web
│ ├── html
│ │ └── index.html
│ └── resources
├── compose.yaml
├── db
│ ├── Dockerfile
│ ├── main.go
│ ├── proto
│ │ ├── db.pb.go
│ │ ├── db.proto
│ │ └── db_grpc.pb.go
│ └── text.db
├── go.mod
├── go.sum
├── proto
│ ├── text.pb.go
│ └── text.proto
└── server
├── Dockerfile
├── main.go
├── proto
│ ├── server.pb.go
│ ├── server.proto
│ └── server_grpc.pb.go
└── text
├── text.go
└── text_test.go
I'm able to run the following protoc in the root text dir:
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative proto/text.proto db/proto/db.proto server/proto/server.proto
And run the server locally, but I'm not able to build my Docker:
CMD
docker build -f server/Dockerfile -t server .
Error
=> ERROR [7/8] RUN /opt/protoc/bin/protoc --go_out=/app/proto --proto_path=/app/proto --go_opt=paths=source_relative --go-grpc_out=/app/proto --go-grpc_opt=paths=source_relative 0.4s
------
> [7/8] RUN /opt/protoc/bin/protoc --go_out=/app/proto --proto_path=/app/proto --go_opt=paths=source_relative --go-grpc_out=/app/proto --go-grpc_opt=paths=source_relative /app/proto/text.proto /app/proto/server.proto:
#11 0.427 proto/text.proto: File not found.
#11 0.429 server.proto: Import "proto/text.proto" was not found or had errors.
#11 0.431 server.proto:25:5: "text.Status" seems to be defined in "text.proto", which is not imported by "server.proto". To use it here, please add the necessary import.
------
executor failed running [/bin/sh -c /opt/protoc/bin/protoc --go_out=/app/pro
text/server/proto
syntax="proto3";
package server;
import "proto/text.proto";
option go_package = "github.com/amb1s1/text/server/proto/server";
message SendMessageRequest {
string token = 1;
string phone = 2;
string message = 3;
bool dry_run = 4;
};
message SendMessageResponse {
text.Status status = 1;
};
service Text {
// SendMessage sents SMS message.
rpc SendMessage(SendMessageRequest) returns (SendMessageResponse) {}
}
text/proto/
syntax="proto3";
package text;
option go_package = "github.com/amb1s1/text/proto";
enum Status {
UNKNOW = 0;
OK = 1;
TOKENS_EXISTS = 2;
TOKEN_NOT_FOUND = 3;
FAILED_NOT_SENT= 4;
DRY_RUN_OK = 5;
ZERO_BALANCE = 6;
WRONG_TOKEN = 7;
}
As per the comments; within your docker image you have the directory structure:
/app/proto/server.proto
/app/proto/text.proto
server.proto imports text.proto with import "proto/text.proto".
This means that protoc will be looking for a file called proto/text.proto within the import path. You specified --proto_path=/app/proto as an argument to protoc meaning that protoc will check for /app/proto/proto/text.proto which does not exist (hence the issue). To fix this remove the --proto_path=/app/proto (so protoc uses the working folder) or specify --proto_path=/app.
Related
I built a small Golang application and I want to run it on a Docker container.
I wrote the following Dockerfile:
# syntax=docker/dockerfile:1
FROM golang:1.16-alpine
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY ./* .
RUN go env -w GO111MODULE=on
RUN go build -o /docker-gs-ping
EXPOSE 8080
CMD [ "/docker-gs-ping" ]
However, when I run the command:
docker build --tag docker-gs-ping .
I get the errors:
#16 0.560 found packages controllers (controller.go) and repositories (csv_file_repository.go) in /app
#16 0.560 main.go:4:2: package MyExercise/controllers is not in GOROOT (/usr/local/go/src/MyExercise/controllers)
I want to mention that the package controllers exists in my working directory and all files associated with this directory are placed in MyExercise/controllers folder.
Do you know how to resolve this error?
Edit:
This is the directory tree:
.
├── Dockerfile
├── REDAME
├── controllers
│ └── controller.go
├── go.mod
├── go.sum
├── logging
│ └── logger.go
├── main.go
├── models
│ └── location.go
├── output.log
├── repositories
│ ├── csv_file_repository.go
│ ├── csv_file_repository_builder.go
│ ├── csv_file_repository_builder_test.go
│ ├── csv_file_repository_test.go
│ ├── repository_builder_interface.go
│ ├── repository_interface.go
│ └── resources
│ └── ip_address_list.txt
└── services
├── ip_location_service.go
├── ip_location_service_test.go
├── rate_limiter_service.go
├── rate_limiter_service_interface.go
├── rate_limiter_service_test.go
└── time_service.go
import section in main.go:
import (
"MyExercise/controllers"
"MyExercise/logging"
"MyExercise/repositories"
"MyExercise/services"
"errors"
"github.com/gin-gonic/gin"
"os"
"strconv"
"sync"
)
Do go mod vendor in your app directory. Documentaion.
For build the container docker build -t app:v1 .
Dockerfile
FROM golang:1.16-alpine
WORKDIR /app/
ADD . .
RUN go build -o /app/main
EXPOSE 5055
CMD [ "/app/main" ]
There is actually an issue with your Dockerfile.
COPY ./* .
does not actually do what you think. It will copy all files recursively in a flat structure to the /app directory.
Modify your Dockerfile to something like:
# syntax=docker/dockerfile:1
FROM golang:1.16-alpine
WORKDIR /app
ADD . /app
RUN go mod download
RUN go env -w GO111MODULE=on
RUN go build -o /docker-gs-ping
EXPOSE 8080
CMD [ "/docker-gs-ping" ]
Basically, remove all of the COPY directives and replace with a single ADD directive
I have an application in AWS SAM consisting of lambda functions in Golang that use docker image build and deploy, I have a problem with build lambda image along with other local modules
The structure of the project looks like this:
├── src
│ ├── configuration # module1
│ │ ├── go.mod
│ │ └── values.go
│ ├── logger # module2
│ │ ├── go.mod
│ │ ├── go.sum
│ │ └── logger.go
│ └── dockerTestLambda # lambda function that use module#1, module#2
│ ├── Dockerfile
│ ├── go.mod
│ ├── go.sum
│ └── main.go
└── template.yml
Two separate modules configuration and logger which are used in the dockerTest Lambda function.
The go.mod in the dockerTestLambda module looks like this:
module my-private-repo.io/my-app/dockerTest
go 1.18
replace my-private-repo.io/my-app/logger => ../logger
replace my-private-repo.io/my-app/configuration => ../configuration
require (
github.com/aws/aws-lambda-go v1.31.1
my-private-repo.io/my-app/configuration v0.0.0-00010101000000-000000000000
my-private-repo.io/my-app/logger v0.0.0-00010101000000-000000000000
)
require (
github.com/pkg/errors v0.9.1 // indirect
github.com/rs/zerolog v1.26.1 // indirect
)
And the lambda itself (file main.go in dockerTestLambda) looks like this:
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/cfn"
"github.com/aws/aws-lambda-go/lambda"
"my-private-repo.io/my-app/configuration"
"my-private-repo.io/my-app/logger"
)
func HandleRequest(ctx context.Context, event cfn.Event) error {
logger.Initialize()
fmt.Println(configuration.GetRegion())
logger.Log(nil, logger.LOG_LEVEL_INFO, "Hello from external deps, logger")
return nil
}
func main() {
lambda.Start(HandleRequest)
}
With definition in tempalte.yml
DockerTestFunction:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
Metadata:
DockerTag: docker-test-v1
DockerContext: ./src/dockerTestLambda
Dockerfile: Dockerfile
The Dockerfile in the lambda function looks like this:
FROM golang:1.18 AS build
WORKDIR /app
COPY go.mod go.mod
COPY go.sum .
RUN go mod download
COPY main.go .
RUN go build -o /usr/local/bin/lambda .
FROM ubuntu:latest
COPY --from=build /usr/local/bin/lambda /usr/local/bin/lambda
CMD [ "/usr/local/bin/lambda" ]
During the sam build command I get an error:
go: my-private-repo.io/my-app/configuration#v0.0.0-00010101000000-000000000000 (replaced by ../configuration): reading /configuration/go.mod: open /configuration/go.mod: no such file or directory
I have no idea how to build an image so that each lambda has its own Dockerfile and each lambda function has all external dependencies packed into it.
Do you have any ideas how to solve this, or some resources that will allow me to better structure the application code to make this problem easier to solve?
Im building a express app but when i use the command sudo docker build - < Dockerfile i get the error COPY failed: file not found in build context or excluded by .dockerignore: stat package.json: file does not exist.
This is how my proyect structure looks like:
.
├── build
│ ├── server.js
│ └── server.js.map
├── Dockerfile
├── esbuild.js
├── package.json
├── package-lock.json
├── Readme.md
└── src
├── index.ts
├── navigate.ts
├── pages.ts
├── responses
│ ├── Errors.ts
│ └── index.ts
└── server.ts
And this is my Dockerfile content
FROM node:14.0.0
WORKDIR /usr/src/app
RUN ls -all
COPY [ "package.json", \
"./"]
COPY src/ ./src
RUN npm install
RUN node esbuild.js
RUN npx nodemon build/server.js
EXPOSE 3001
CMD ["npm", "run", "serve", ]
At the moment of run the command, im located in the root of the project.
Trying to run mlflow run by specifying MLproject and code which lives in a different location as MLproject file.
I have the following directory structure:
/root/mflow_test
.
├── conda
│ ├── conda.yaml
│ └── MLproject
├── docker
│ ├── Dockerfile
│ └── MLproject
├── README.md
├── requirements.txt
└── trainer
├── __init__.py
├── task.py
└── utils.py
When I'm run from: /root/
mlflow run mlflow_test/docker
I get:
/root/miniconda3/bin/python: Error while finding module specification for 'trainer.task' (ImportError: No module named 'trainer')
Since my MLproject file can't find the Python code.
I moved MLproject to mflow_test and this works fine.
This is my MLproject entry point:
name: mlflow_sample
docker_env:
image: mlflow-docker-sample
entry_points:
main:
parameters:
job_dir:
type: string
default: '/tmp/'
command: |
python -m trainer.task --job-dir {job_dir}
How can I run mlflow run and pass the MLproject and ask it to look in a different folder?
I tried:
"cd .. && python -m trainer.task --job-dir {job_dir}"
and I get:
/entrypoint.sh: line 5: exec: cd: not found
Dockerfile
# docker build -t mlflow-gcp-example -f Dockerfile .
FROM gcr.io/deeplearning-platform-release/tf-cpu
RUN git clone github.com/GoogleCloudPlatform/ml-on-gcp.git
WORKDIR ml-on-gcp/tutorials/tensorflow/mlflow_gcp
RUN pip install -r requirements.txt
I need to compile a Golang application for Linux and I can't cross-compile under Mac, because of another library. So I decided to compile within a Docker container. This is my first time to use Docker.
This is my current directory structure:
.
├── Dockerfile
├── Gopkg.lock
├── Gopkg.toml
├── Vagrantfile
├── bootstrap.sh
├── src
│ ├── cmd
│ │ ├── build.bat
│ │ ├── build.sh
│ │ ├── config.json
│ │ ├── readme.md
│ │ └── server.go
│ ├── consumers.go
│ ├── endpoints
│ │ ├── json.go
│ │ ├── rate.go
│ │ ├── test_payment.go
│ │ └── wallet.go
│ ├── middleware
│ │ └── acl.go
│ ├── models.go
│ ├── network
│ │ └── network.go
│ ├── qr
│ │ └── qr.go
│ ├── router
│ │ └── router.go
│ ├── service
│ │ └── walletService.go
│ ├── services.go
│ ├── setup.sql
│ ├── store
│ │ └── wallet.go
│ ├── stores.go
│ └── wallet
│ ├── coin.go
│ └── ethereum.go
Dockerfile:
FROM golang:latest
WORKDIR /src/cmd
RUN ls
RUN go get github.com/go-sql-driver/mysql
RUN go build -o main ./src/cmd/server.go
CMD ["./main"]
I try to build the Docker image with:
docker build -t outyet .
This is the error it returns:
Sending build context to Docker daemon 5.505MB
Step 1/6 : FROM golang:latest
---> d0e7a411e3da
Step 2/6 : WORKDIR /src/cmd
---> Using cache
---> 0c4c2b99e294
Step 3/6 : run ls
---> Using cache
---> 23d3e491a2e1
Step 4/6 : RUN go get github.com/go-sql-driver/mysql
---> Running in f34447e51f6c
Removing intermediate container f34447e51f6c
---> 5731ab22ee43
Step 5/6 : RUN go build -o main server.go
---> Running in ecc48fcf5488
stat server.go: no such file or directory
The command '/bin/sh -c go build -o main server.go' returned a non-zero code: 1
How i can build my Golang application with docker?
The error you're seeing is:
stat server.go: no such file or directory
The command '/bin/sh -c go build -o main server.go' returned a non-zero code: 1
... and in other words, it's telling you that the Go compiler can't find server.go which you're trying to build. Update your Dockerfile to copy your local files into the Docker image:
FROM golang:latest
COPY . /go/src/workdir
WORKDIR /go/src/workdir
RUN go build -o main ./src/cmd/server.go
CMD ["/go/src/workdir/main"]
In your directory I spotted Gopkg.toml which is a dependency manifest used by dep. dep will use a directory called vendor to include all dependencies for your Go project. Before you build the Docker image, you need to ensure all dependencies are present with dep ensure:
$ dep ensure
$ docker build -t outyet .
You need to add your sources in your build container. Add this line in your Dockerfile (after FROM golang:latest for example) :
ADD src/ /src/cmd
Then, you could access files inside the container (/src/cmd) : RUN ls should now return something.