I'm trying to build an image using docker's Go client. Here is the Go program I'm running:
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
log.Fatal(err, " :unable to init client")
}
buf := new(bytes.Buffer)
tw := tar.NewWriter(buf)
defer tw.Close()
dockerFile := "Dockerfile"
dockerFileReader, err := os.Open("./Dockerfile")
if err != nil {
log.Fatal(err, " :unable to open Dockerfile")
}
readDockerFile, err := ioutil.ReadAll(dockerFileReader)
if err != nil {
log.Fatal(err, " :unable to read dockerfile")
}
tarHeader := &tar.Header{
Name: dockerFile,
Size: int64(len(readDockerFile)),
}
err = tw.WriteHeader(tarHeader)
if err != nil {
log.Fatal(err, " :unable to write tar header")
}
_, err = tw.Write(readDockerFile)
if err != nil {
log.Fatal(err, " :unable to write tar body")
}
dockerFileTarReader := bytes.NewReader(buf.Bytes())
imageBuildResponse, err := cli.ImageBuild(
ctx,
dockerFileTarReader,
types.ImageBuildOptions{
Dockerfile: dockerFile,
Tags: []string{"devbuild"},
Remove: true})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
It puts the Dockerfile (located in the current directory) in a tar file and builds an image using cli.ImageBuild. This solution was taken from this post, and my Dockerfile looks like this:
FROM alpine:latest
WORKDIR /gopath/src/build
COPY ./binary_build/ /usr/local/bin/
I'm constantly getting the error on the last step:
{"stream":"Step 3/3 : COPY /binary_build/ /usr/local/bin/"}
{"stream":"\n"}
{"errorDetail":{"message":"COPY failed: stat /var/lib/docker/tmp/docker-builder389608393/binary_build: no such file or directory"},"error":"COPY failed: stat /var/lib/docker/tmp/docker-builder389608393/binary_build: no such file or directory"}
There seems to be similar issue in the past reported here, but it seems to have been patched. I ran go get github.com/docker/docker#latest and my docker version in go.mod is github.com/docker/docker v1.13.1. The issue still persists.
If you just build a docker image, that code means nothing, because the docker building process won't run the program.
You should put your Dockerfile and executable file "binary_build" in a sole folder,may be the Dockerfile should like:
FROM alpine:latest
WORKDIR /gopath/src/build
COPY binary_build /usr/local/bin/
Related
I'm trying to create some docker images on macOS (with Colima backend) using Go Docker SDK with following code:
package main
import (
"context"
"flag"
"fmt"
"io"
"log"
"os"
"path/filepath"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"github.com/docker/docker/pkg/archive"
)
const dockerfile = `
# syntax=docker/dockerfile:1
FROM golang:1.19-alpine
WORKDIR /app
ENV GO111MODULE=on
ENV CGO_ENABLED=0
COPY . ./
RUN go mod download
RUN apk add --no-cache git
RUN go install github.com/securego/gosec/v2/cmd/gosec#latest
RUN go install golang.org/x/vuln/cmd/govulncheck#latest
RUN go get github.com/stripe/safesql
CMD gosec ./... && govulncheck ./... && safesql ./...`
func main() {
ctx := context.Background()
pathFlag := flag.String("path", ".", "path to the directory containing the go files")
flag.Parse()
if pathFlag == nil {
log.Fatal("path flag is nil")
}
err := os.WriteFile(*pathFlag+"/Dockerfile", []byte(dockerfile), 0777)
if err != nil {
fmt.Printf("Unable to write file: %v", err)
}
cli, err := client.NewClientWithOpts()
if err != nil {
log.Fatal(err, " :unable to init client")
}
files, err := WalkMatch(*pathFlag, "*.go")
if err != nil {
log.Fatal(err, " :unable to walk files")
}
reader, err := archive.TarWithOptions(*pathFlag, &archive.TarOptions{
IncludeFiles: append(files, []string{"go.mod", "go.sum", "Dockerfile"}...),
})
if err != nil {
log.Fatal(err, " :unable to create tar")
}
imageBuildResponse, err := cli.ImageBuild(
ctx,
reader,
types.ImageBuildOptions{
Tags: []string{"newname"},
Context: reader,
Dockerfile: "Dockerfile",
})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
func WalkMatch(root, pattern string) ([]string, error) {
var matches []string
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if matched, err := filepath.Match(pattern, filepath.Base(path)); err != nil {
return err
} else if matched {
r, err := filepath.Rel(root, path)
if err != nil {
return err
}
matches = append(matches, r)
}
return nil
})
if err != nil {
return nil, err
}
return matches, nil
}
The creation seems to be successful:
{"aux":{"ID":"sha256:568d5aed149aa73d38b644f49f68590cc06222467be1e1c3b902792da39f6e24"}}
{"stream":"Successfully built 568d5aed149a\n"}
{"stream":"Successfully tagged newname:latest\n"}
However, I can't find these images anywhere, docker run states there are no images with such tag, docker ls tells me the same. Do I miss something crucial? Are there any macOS specific things I should consider?
You are only building the images not running them so you should use the command
docker images
to get all the images present on your system.
docker ps
only gives back the images which are currently running
I have a gRPC server
type Server struct {
name string
host string
port string
dbUser string
dbPassword string
dbURL string
dbParameters string
}
func main() {
/*Start of config*/
server := Server{
"User service",
"",
os.Getenv("PORT"),
"",
"",
"",
"",
}
/*End of config*/
log.Printf("Starting: %s RPCServer\n", server.name)
lis, err := net.Listen("tcp", server.host+":"+server.port)
if err != nil {
log.Fatalf("Failed to liste: %v\n", err)
}
defer func() {
err = lis.Close()
if err != nil {
log.Fatalf("Failed to close listener: %v\n", err)
}
}()
gRPCServer := grpc.NewServer()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
reg := codecs.Register(bson.NewRegistryBuilder()).Build()
mongoURI := options.Client().ApplyURI(
"mongodb+srv://" + server.dbUser + ":" + server.dbPassword + "#" + server.dbURL + server.dbParameters,
)
mongoClient, err := mongo.NewClient(mongoURI, &options.ClientOptions{
Registry: reg,
})
if err != nil {
log.Fatalf("unable to create new mongo clinet: %s\n", err)
}
err = mongoClient.Connect(ctx)
if err != nil {
log.Fatalf("unable to connect to db: %s\n", err)
}
/*Start of registering service*/
authService, err := AuthService.NewAuthService()
if err != nil {
log.Fatalf("unable to create server: %s\n", err)
}
s := UserService.NewServer(authService, mongoClient)
UserService.RegisterUserServiceServer(gRPCServer, &s)
/*End of registering service*/
go func() {
if err := gRPCServer.Serve(lis); err != nil {
log.Printf("Failed to serve: %v\n", err)
}
}()
log.Printf("server successfully started on port: %s\n\n", server.port)
c := make(chan os.Signal)
signal.Notify(c, os.Interrupt)
<-c
log.Printf("\nstopping server...\n")
gRPCServer.Stop()
err = lis.Close()
if err != nil {
log.Fatalf("failed to close listner: %s\n", err)
}
err = mongoClient.Disconnect(ctx)
if err != nil {
log.Fatalf("failed to disconnect from mongodb: %s\n", err)
}
log.Printf("successfully closed server\n")
}
And my Dockerfile is
FROM golang:alpine AS build-env
WORKDIR /app
ADD . /app
RUN cd /app && go build Main/server.go
FROM alpine
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*
WORKDIR /app
COPY --from=build-env /app/server /app
COPY --from=build-env /app/Config/* /app
ENV GOOGLE_APPLICATION_CREDENTIALS=cred.json
EXPOSE 50051
CMD ["./server"]
I put up the docker image on Google Container Registry, and tried to use Cloud Run to run it using the following command
gcloud run deploy grpc-server-streaming\
--project=project-id\
--platform=managed\
--region=asia-south1\
--image=image-tag\
--allow-unauthenticated
After which I went into Cloud Run application and enabled HTTP/2 connections. But am still not able to connect to the service. I get the following error
2021/04/21 17:04:36 rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination
I've been stuck at this for two days and am not sure what to do.
The same is happening for my ASPNet hosted gRPC service. It was working last Friday (2021-04-30), but on Monday (2021-05-03) I started getting this error message.
I have been searching for days, like you, for an answer. Thank-you to #menghanl for the Envoy pointer, I will open that can of worms.
#Chandraaditya Have a look at your Cloud Run Service's Logs to see if your service is actually being triggered.
I can see my service handling the request at the time I am getting the error. The gRPC service call appears to be 200 OK, but "Envoy" is not sending it back to the client.
I am using the Go Docker Client to attempt to build an image from a Dockerfile whose contents are defined in code.
According to the Docker Daemon API Documentation
The input stream must be a tar archive...
...The archive must include a build instructions file, typically called Dockerfile at the archive’s root.
So I want to create the build context in code, write it to a tar file, then send that to the Docker Daemon to be built. To do this I can use the ImageBuild function and pass in the tar file (build context) as an io.ReadCloser. As long as my Dockerfile is at the root of that compressed archive, it should find it and build it.
However, I get the common error:
Error response from daemon: Cannot locate specified Dockerfile: Dockerfile
Which obviously means that it can't find the Dockerfile at the root of the archive. I am unsure why. I believe the way I am doing this adds a Dockerfile to the root of the tar archive. The daemon should see this. What am I misunderstanding here?
code snippet to reproduce
var buf bytes.Buffer
tarWriter := tar.NewWriter(&buf)
contents := "FROM alpine\nCMD [\"echo\", \"this is from the archive\"]"
if err := tarWriter.WriteHeader(&tar.Header{
Name: "Dockerfile",
Mode: 777,
Size: int64(len(contents)),
Typeflag: tar.TypeReg,
}); err != nil {
panic(err)
}
if _, err := tarWriter.Write([]byte(contents)); err != nil {
panic(err)
}
if err := tarWriter.Close(); err != nil {
panic(err)
}
reader := tar.NewReader(&buf)
c, err := client.NewEnvClient()
if err != nil {
panic(err)
}
_, err = c.ImageBuild(context.Background(), reader, types.ImageBuildOptions{
Context: reader,
Dockerfile: "Dockerfile",
})
if err != nil {
panic(err)
}
go.mod file
module docker-tar
go 1.12
require (
github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/docker v1.13.1
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/opencontainers/go-digest v1.0.0-rc1 // indirect
github.com/pkg/errors v0.8.1 // indirect
golang.org/x/net v0.0.0-20191112182307-2180aed22343 // indirect
)
Add a zero in front of 777 for octal numeral system: 0777, 0o777, or 0O777
Use reader := bytes.NewReader(buf.Bytes()) not tar.NewReader(&buf)
Use client.WithAPIVersionNegotiation() for newer versions too.
Try this working version:
package main
import (
"archive/tar"
"bytes"
"context"
"fmt"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
var buf bytes.Buffer
tarWriter := tar.NewWriter(&buf)
contents := `FROM alpine:3.10.3
CMD ["echo", "this is from the archive"]
`
header := &tar.Header{
Name: "Dockerfile",
Mode: 0o777,
Size: int64(len(contents)),
Typeflag: tar.TypeReg,
}
err := tarWriter.WriteHeader(header)
if err != nil {
panic(err)
}
_, err = tarWriter.Write([]byte(contents))
if err != nil {
panic(err)
}
err = tarWriter.Close()
if err != nil {
panic(err)
}
c, err := client.NewClientWithOpts(client.WithAPIVersionNegotiation())
if err != nil {
panic(err)
}
fmt.Println(c.ClientVersion())
reader := bytes.NewReader(buf.Bytes()) // tar.NewReader(&buf)
ctx := context.Background()
buildOptions := types.ImageBuildOptions{
Context: reader,
Dockerfile: "Dockerfile",
Tags: []string{"alpine-echo:1.2.4"},
}
_, err = c.ImageBuild(ctx, reader, buildOptions)
if err != nil {
panic(err)
}
}
Output for docker image ls after go run .:
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine-echo 1.2.4 d81774f32812 26 seconds ago 5.55MB
alpine 3.10.3 b168ac0e770e 4 days ago 5.55MB
Output for docker run alpine-echo:1.2.4:
this is from the archive
Note: You may need to edit FROM alpine:3.10.3 for your specific version.
So I am trying to build a docker image with the Golang SDK, everything runs except the section in the Dockerfile where I use COPY to copy a file across into the image:
COPY testfile.txt /testfile.txt
My code is as follows:
func buildImage() {
// Run in directory where Dockerfile is found
os.Chdir("build-dir")
cli, err := client.NewEnvClient()
if err != nil {log.Fatal(err, " :unable to init client")}
// Image Build requiresa tar file
tar := new(archivex.TarFile)
tar.Create("dockerfile.tar")
tar.AddAll(".", true)
tar.Close()
// Use tar file as docker context
dockerBuildContext, err := os.Open("dockerfile.tar")
defer dockerBuildContext.Close()
options := types.ImageBuildOptions{
SuppressOutput: false,
Remove: true,
ForceRemove: true,
PullParent: true,
Tags: []string{"latest"},
Dockerfile: "Dockerfile",
}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, options)
defer buildResponse.Body.Close()
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
// Copy out response of stream
_, err = io.Copy(os.Stdout, buildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
The code fails with:
{
"errorDetail": {
"message":"COPY failed: stat /var/lib/docker/tmp/docker-builder264844317/testfile.txt: no such file or directory"
},
"error":"COPY failed: stat /var/lib/docker/tmp/docker-builder264844317/testfile.txt: no such file or directory"
}
So far I have tried copying the files into the tar before building and then I have also tried moving the textfile.txt into the directory I run the command from but I still can not seem to get past this point
Extra information:
The file is in the same directory as the Dockerfile:
-- build-dir
|-- Dockerfile
|-- testfile.txt
From Source
The docker build command builds Docker images from a Dockerfile and a “context”. A build’s context is the set of files located in the specified PATH or URL. The build process can refer to any of the files in the context. For example, your build can use a COPY instruction to reference a file in the context.
Docker build-context is the entire directory you send to the docker engine. While building your image, Docker engine will try to find the files from the root of your build-context.
In your case, the file was not added to the build-context.
So a colleague pointed out to me instead of just running just `tar.AddAll' i also need to specify the files I want to add, see updated code below:
func buildCIImage() {
os.Chdir("ci-cd")
cli, err := client.NewEnvClient()
if err != nil {log.Fatal(err, " :unable to init client")}
// open the file to pass into the tar
file, err := os.OpenFile("testfile.txt", os.O_RDWR, os.ModePerm)
// Used to get the files information
fileInfo, err := os.Stat("testfile.txt")
tar := new(archivex.TarFile)
tar.Create("dockerfile.tar")
tar.AddAll(".", true)
// Add file into tar
tar.Add("testfile.txt", file, fileInfo)
tar.Close()
dockerBuildContext, err := os.Open("dockerfile.tar")
defer dockerBuildContext.Close()
options := types.ImageBuildOptions{
SuppressOutput: false,
Remove: true,
ForceRemove: true,
PullParent: true,
Tags: []string{"bootstrap"},
Dockerfile: "Dockerfile",
}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, options)
defer buildResponse.Body.Close()
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
_, err = io.Copy(os.Stdout, buildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
I am trying to use Docker's tutorial in recreating a docker run. Here is the following code from online tutorial
package main
import (
"io"
"os"
"github.com/docker/docker/client"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"golang.org/x/net/context"
)
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
panic(err)
}
_, err = cli.ImagePull(ctx, "alpine", types.ImagePullOptions{})
if err != nil {
panic(err)
}
resp, err := cli.ContainerCreate(ctx, &container.Config{
Image: "alpine",
Cmd: []string{"echo", "hello world"},
}, nil, nil, "")
if err != nil {
panic(err)
}
if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
panic(err)
}
if _, err = cli.ContainerWait(ctx, resp.ID); err != nil {
panic(err)
}
out, err := cli.ContainerLogs(ctx, resp.ID, types.ContainerLogsOptions{ShowStdout: true})
if err != nil {
panic(err)
}
io.Copy(os.Stdout, out)
}
The problem I see with this is, if 'alpine' docker, is not available locally, it doesn't pull the latest and ends up throwing an error.
e.g
XXXXX$ go run go_docker.go
panic: Error: No such image: alpine
goroutine 1 [running]:
panic(0x27ffa0, 0xc4202afa50)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
main.main()
/Users/rvenkatesh/go_coding/raghu_test_code/go_docker.go:30 +0x592
exit status 2
But when I run the commandline equivalent, I see
XXXX$ docker run alpine echo hello world
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
627beaf3eaaf: Pull complete
Digest:sha256:58e1a1bb75db1b5a24a462dd5e2915277ea06438c3f105138f97eb53149673c4
Status: Downloaded newer image for alpine:latest
hello world
I tried looking through Go client, do I need to tweak anything with ImagePull function? Any help here would be appreciated!
Here is the link to the docs https://docs.docker.com/engine/api/getting-started/
Update: I had tested the same tutorial for python version, and it worked just fine. I wonder if the Golang page needs update.
Was having the same issue, the "Pull" didn't seem to be working. Found a Fix though.
1) Modify your pull line to
pullstat, err = cli.ImagePull(ctx, "alpine", types.ImagePullOptions{})
and add
io.Copy(os.StdOut,pullstat)
after the ImagePull
I haven't tried doing an
io.Copy(nil,pullstat)
but that's on my list of things to try next.
Image.Pull returns an io.Reader which you must read and close; if you do not the connection will be closed before the image is pulled.
You can just discard the contents of it and close it, then the pull will work.
The docker client is open source and written in Go, so you can see exactly how they've implemented their version. I believe the relevant code is in the container/create.go pullImage function.