I am using the Go Docker Client to attempt to build an image from a Dockerfile whose contents are defined in code.
According to the Docker Daemon API Documentation
The input stream must be a tar archive...
...The archive must include a build instructions file, typically called Dockerfile at the archive’s root.
So I want to create the build context in code, write it to a tar file, then send that to the Docker Daemon to be built. To do this I can use the ImageBuild function and pass in the tar file (build context) as an io.ReadCloser. As long as my Dockerfile is at the root of that compressed archive, it should find it and build it.
However, I get the common error:
Error response from daemon: Cannot locate specified Dockerfile: Dockerfile
Which obviously means that it can't find the Dockerfile at the root of the archive. I am unsure why. I believe the way I am doing this adds a Dockerfile to the root of the tar archive. The daemon should see this. What am I misunderstanding here?
code snippet to reproduce
var buf bytes.Buffer
tarWriter := tar.NewWriter(&buf)
contents := "FROM alpine\nCMD [\"echo\", \"this is from the archive\"]"
if err := tarWriter.WriteHeader(&tar.Header{
Name: "Dockerfile",
Mode: 777,
Size: int64(len(contents)),
Typeflag: tar.TypeReg,
}); err != nil {
panic(err)
}
if _, err := tarWriter.Write([]byte(contents)); err != nil {
panic(err)
}
if err := tarWriter.Close(); err != nil {
panic(err)
}
reader := tar.NewReader(&buf)
c, err := client.NewEnvClient()
if err != nil {
panic(err)
}
_, err = c.ImageBuild(context.Background(), reader, types.ImageBuildOptions{
Context: reader,
Dockerfile: "Dockerfile",
})
if err != nil {
panic(err)
}
go.mod file
module docker-tar
go 1.12
require (
github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/docker v1.13.1
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/opencontainers/go-digest v1.0.0-rc1 // indirect
github.com/pkg/errors v0.8.1 // indirect
golang.org/x/net v0.0.0-20191112182307-2180aed22343 // indirect
)
Add a zero in front of 777 for octal numeral system: 0777, 0o777, or 0O777
Use reader := bytes.NewReader(buf.Bytes()) not tar.NewReader(&buf)
Use client.WithAPIVersionNegotiation() for newer versions too.
Try this working version:
package main
import (
"archive/tar"
"bytes"
"context"
"fmt"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
var buf bytes.Buffer
tarWriter := tar.NewWriter(&buf)
contents := `FROM alpine:3.10.3
CMD ["echo", "this is from the archive"]
`
header := &tar.Header{
Name: "Dockerfile",
Mode: 0o777,
Size: int64(len(contents)),
Typeflag: tar.TypeReg,
}
err := tarWriter.WriteHeader(header)
if err != nil {
panic(err)
}
_, err = tarWriter.Write([]byte(contents))
if err != nil {
panic(err)
}
err = tarWriter.Close()
if err != nil {
panic(err)
}
c, err := client.NewClientWithOpts(client.WithAPIVersionNegotiation())
if err != nil {
panic(err)
}
fmt.Println(c.ClientVersion())
reader := bytes.NewReader(buf.Bytes()) // tar.NewReader(&buf)
ctx := context.Background()
buildOptions := types.ImageBuildOptions{
Context: reader,
Dockerfile: "Dockerfile",
Tags: []string{"alpine-echo:1.2.4"},
}
_, err = c.ImageBuild(ctx, reader, buildOptions)
if err != nil {
panic(err)
}
}
Output for docker image ls after go run .:
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine-echo 1.2.4 d81774f32812 26 seconds ago 5.55MB
alpine 3.10.3 b168ac0e770e 4 days ago 5.55MB
Output for docker run alpine-echo:1.2.4:
this is from the archive
Note: You may need to edit FROM alpine:3.10.3 for your specific version.
Related
I'm trying to create some docker images on macOS (with Colima backend) using Go Docker SDK with following code:
package main
import (
"context"
"flag"
"fmt"
"io"
"log"
"os"
"path/filepath"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"github.com/docker/docker/pkg/archive"
)
const dockerfile = `
# syntax=docker/dockerfile:1
FROM golang:1.19-alpine
WORKDIR /app
ENV GO111MODULE=on
ENV CGO_ENABLED=0
COPY . ./
RUN go mod download
RUN apk add --no-cache git
RUN go install github.com/securego/gosec/v2/cmd/gosec#latest
RUN go install golang.org/x/vuln/cmd/govulncheck#latest
RUN go get github.com/stripe/safesql
CMD gosec ./... && govulncheck ./... && safesql ./...`
func main() {
ctx := context.Background()
pathFlag := flag.String("path", ".", "path to the directory containing the go files")
flag.Parse()
if pathFlag == nil {
log.Fatal("path flag is nil")
}
err := os.WriteFile(*pathFlag+"/Dockerfile", []byte(dockerfile), 0777)
if err != nil {
fmt.Printf("Unable to write file: %v", err)
}
cli, err := client.NewClientWithOpts()
if err != nil {
log.Fatal(err, " :unable to init client")
}
files, err := WalkMatch(*pathFlag, "*.go")
if err != nil {
log.Fatal(err, " :unable to walk files")
}
reader, err := archive.TarWithOptions(*pathFlag, &archive.TarOptions{
IncludeFiles: append(files, []string{"go.mod", "go.sum", "Dockerfile"}...),
})
if err != nil {
log.Fatal(err, " :unable to create tar")
}
imageBuildResponse, err := cli.ImageBuild(
ctx,
reader,
types.ImageBuildOptions{
Tags: []string{"newname"},
Context: reader,
Dockerfile: "Dockerfile",
})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
func WalkMatch(root, pattern string) ([]string, error) {
var matches []string
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if matched, err := filepath.Match(pattern, filepath.Base(path)); err != nil {
return err
} else if matched {
r, err := filepath.Rel(root, path)
if err != nil {
return err
}
matches = append(matches, r)
}
return nil
})
if err != nil {
return nil, err
}
return matches, nil
}
The creation seems to be successful:
{"aux":{"ID":"sha256:568d5aed149aa73d38b644f49f68590cc06222467be1e1c3b902792da39f6e24"}}
{"stream":"Successfully built 568d5aed149a\n"}
{"stream":"Successfully tagged newname:latest\n"}
However, I can't find these images anywhere, docker run states there are no images with such tag, docker ls tells me the same. Do I miss something crucial? Are there any macOS specific things I should consider?
You are only building the images not running them so you should use the command
docker images
to get all the images present on your system.
docker ps
only gives back the images which are currently running
I'm trying to build an image using docker's Go client. Here is the Go program I'm running:
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
log.Fatal(err, " :unable to init client")
}
buf := new(bytes.Buffer)
tw := tar.NewWriter(buf)
defer tw.Close()
dockerFile := "Dockerfile"
dockerFileReader, err := os.Open("./Dockerfile")
if err != nil {
log.Fatal(err, " :unable to open Dockerfile")
}
readDockerFile, err := ioutil.ReadAll(dockerFileReader)
if err != nil {
log.Fatal(err, " :unable to read dockerfile")
}
tarHeader := &tar.Header{
Name: dockerFile,
Size: int64(len(readDockerFile)),
}
err = tw.WriteHeader(tarHeader)
if err != nil {
log.Fatal(err, " :unable to write tar header")
}
_, err = tw.Write(readDockerFile)
if err != nil {
log.Fatal(err, " :unable to write tar body")
}
dockerFileTarReader := bytes.NewReader(buf.Bytes())
imageBuildResponse, err := cli.ImageBuild(
ctx,
dockerFileTarReader,
types.ImageBuildOptions{
Dockerfile: dockerFile,
Tags: []string{"devbuild"},
Remove: true})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
It puts the Dockerfile (located in the current directory) in a tar file and builds an image using cli.ImageBuild. This solution was taken from this post, and my Dockerfile looks like this:
FROM alpine:latest
WORKDIR /gopath/src/build
COPY ./binary_build/ /usr/local/bin/
I'm constantly getting the error on the last step:
{"stream":"Step 3/3 : COPY /binary_build/ /usr/local/bin/"}
{"stream":"\n"}
{"errorDetail":{"message":"COPY failed: stat /var/lib/docker/tmp/docker-builder389608393/binary_build: no such file or directory"},"error":"COPY failed: stat /var/lib/docker/tmp/docker-builder389608393/binary_build: no such file or directory"}
There seems to be similar issue in the past reported here, but it seems to have been patched. I ran go get github.com/docker/docker#latest and my docker version in go.mod is github.com/docker/docker v1.13.1. The issue still persists.
If you just build a docker image, that code means nothing, because the docker building process won't run the program.
You should put your Dockerfile and executable file "binary_build" in a sole folder,may be the Dockerfile should like:
FROM alpine:latest
WORKDIR /gopath/src/build
COPY binary_build /usr/local/bin/
I need to know Docker Daemon API version and setup the environment variable of DOCKER_API_VERSION when I have to create Docker client with NewEnvClient(), if not I will get an error about:
Error response from daemon: client version 1.36 is too new. Maximum supported API version is 1.35
If you are executing your code in the same docker host you can use the following approach to get the API version. It execute docker version command and get the API version from that output.
package main
import (
"os/exec"
"bytes"
"os"
"github.com/docker/docker/client"
"golang.org/x/net/context"
"github.com/docker/docker/api/types"
"strings"
)
func main() {
cmd := exec.Command("docker", "version", "--format", "{{.Server.APIVersion}}")
cmdOutput := &bytes.Buffer{}
cmd.Stdout = cmdOutput
err := cmd.Run()
if err != nil {
panic(err)
}
apiVersion := strings.TrimSpace(string(cmdOutput.Bytes()))
// TODO: (optional) verify the api version is in the correct format(a.b)
os.Setenv("DOCKER_API_VERSION", apiVersion)
// execute docker commands
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
panic(err)
}
_, err = cli.ImagePull(ctx, "alpine", types.ImagePullOptions{})
if err != nil {
panic(err)
}
}
I'm using "golang.org/x/net/ipv4" in order to use its SetTTL function. Unfortunately it does not seem to work on Linux, only on Mac, even though the documentation indicates Linux supports all the functions.
Here's a minimal example of the problem, with a Dockerfile:
main.go:
package main
import (
"fmt"
"net"
"bufio"
xnet "golang.org/x/net/ipv4"
)
const Host = "google.com"
func main() {
var err error
conn, err := net.Dial("tcp4", Host + ":80")
if err != nil {
panic(err)
}
defer conn.Close()
xconn := xnet.NewConn(conn)
err = xconn.SetTTL(5)
if err != nil {
panic(err)
}
defer xconn.Close()
fmt.Fprint(conn, "GET / HTTP/1.1\r\nHOST: google.com\r\n\r\n")
firstLine, err := bufio.NewReader(xconn).ReadString('\n')
if err != nil {
panic(err)
}
fmt.Println(firstLine)
}
Dockerfile:
FROM golang:1.8.1-alpine
RUN apk --no-cache add git
RUN go get golang.org/x/net/ipv4
COPY . /go/src/me.com/me/xnetproblem
RUN go install me.com/me/xnetproblem
CMD ["/go/bin/xnetproblem"]
I run this command:
docker build -t xnet .
I get this output:
john xnetproblem > docker build -t xnet .
Sending build context to Docker daemon 90.62 kB
Step 1/6 : FROM golang:1.8.1-alpine
[snip]
Step 5/6 : RUN go install me.com/me/xnetproblem
---> Running in c3802fe61d63
# me.com/me/xnetproblem
src/me.com/me/xnetproblem/main.go:25: xconn.Close undefined (type *ipv4.Conn has no field or method Close)
src/me.com/me/xnetproblem/main.go:28: cannot use xconn (type *ipv4.Conn) as type io.Reader in argument to bufio.NewReader:
*ipv4.Conn does not implement io.Reader (missing Read method)
The command '/bin/sh -c go install me.com/me/xnetproblem' returned a non-zero code: 2
Using go install natively, instead of Docker, the program works on Mac but not on Linux.
Thanks to #JimB's comment I realized my Mac had an old version of the ipv4 package installed. After updating I was able to fix the code.
Here's a complete working version:
package main
import (
"fmt"
"net"
"bufio"
"golang.org/x/net/ipv4"
)
const Host = "google.com"
func main() {
var err error
conn, err := net.Dial("tcp4", Host + ":80")
if err != nil {
panic(err)
}
defer conn.Close()
if err = ipv4.NewConn(conn).SetTTL(5); err != nil {
panic(err)
}
fmt.Fprint(conn, fmt.Sprintf("GET / HTTP/1.1\r\nHost: %v\r\n\r\n", Host))
firstLine, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
panic(err)
}
fmt.Println(firstLine)
}
Here is the output:
HTTP/1.1 301 Moved Permanently
I am trying to use Docker's tutorial in recreating a docker run. Here is the following code from online tutorial
package main
import (
"io"
"os"
"github.com/docker/docker/client"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"golang.org/x/net/context"
)
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
panic(err)
}
_, err = cli.ImagePull(ctx, "alpine", types.ImagePullOptions{})
if err != nil {
panic(err)
}
resp, err := cli.ContainerCreate(ctx, &container.Config{
Image: "alpine",
Cmd: []string{"echo", "hello world"},
}, nil, nil, "")
if err != nil {
panic(err)
}
if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
panic(err)
}
if _, err = cli.ContainerWait(ctx, resp.ID); err != nil {
panic(err)
}
out, err := cli.ContainerLogs(ctx, resp.ID, types.ContainerLogsOptions{ShowStdout: true})
if err != nil {
panic(err)
}
io.Copy(os.Stdout, out)
}
The problem I see with this is, if 'alpine' docker, is not available locally, it doesn't pull the latest and ends up throwing an error.
e.g
XXXXX$ go run go_docker.go
panic: Error: No such image: alpine
goroutine 1 [running]:
panic(0x27ffa0, 0xc4202afa50)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
main.main()
/Users/rvenkatesh/go_coding/raghu_test_code/go_docker.go:30 +0x592
exit status 2
But when I run the commandline equivalent, I see
XXXX$ docker run alpine echo hello world
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
627beaf3eaaf: Pull complete
Digest:sha256:58e1a1bb75db1b5a24a462dd5e2915277ea06438c3f105138f97eb53149673c4
Status: Downloaded newer image for alpine:latest
hello world
I tried looking through Go client, do I need to tweak anything with ImagePull function? Any help here would be appreciated!
Here is the link to the docs https://docs.docker.com/engine/api/getting-started/
Update: I had tested the same tutorial for python version, and it worked just fine. I wonder if the Golang page needs update.
Was having the same issue, the "Pull" didn't seem to be working. Found a Fix though.
1) Modify your pull line to
pullstat, err = cli.ImagePull(ctx, "alpine", types.ImagePullOptions{})
and add
io.Copy(os.StdOut,pullstat)
after the ImagePull
I haven't tried doing an
io.Copy(nil,pullstat)
but that's on my list of things to try next.
Image.Pull returns an io.Reader which you must read and close; if you do not the connection will be closed before the image is pulled.
You can just discard the contents of it and close it, then the pull will work.
The docker client is open source and written in Go, so you can see exactly how they've implemented their version. I believe the relevant code is in the container/create.go pullImage function.