Golang Docker SDK image build failing with COPY - docker

So I am trying to build a docker image with the Golang SDK, everything runs except the section in the Dockerfile where I use COPY to copy a file across into the image:
COPY testfile.txt /testfile.txt
My code is as follows:
func buildImage() {
// Run in directory where Dockerfile is found
os.Chdir("build-dir")
cli, err := client.NewEnvClient()
if err != nil {log.Fatal(err, " :unable to init client")}
// Image Build requiresa tar file
tar := new(archivex.TarFile)
tar.Create("dockerfile.tar")
tar.AddAll(".", true)
tar.Close()
// Use tar file as docker context
dockerBuildContext, err := os.Open("dockerfile.tar")
defer dockerBuildContext.Close()
options := types.ImageBuildOptions{
SuppressOutput: false,
Remove: true,
ForceRemove: true,
PullParent: true,
Tags: []string{"latest"},
Dockerfile: "Dockerfile",
}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, options)
defer buildResponse.Body.Close()
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
// Copy out response of stream
_, err = io.Copy(os.Stdout, buildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
The code fails with:
{
"errorDetail": {
"message":"COPY failed: stat /var/lib/docker/tmp/docker-builder264844317/testfile.txt: no such file or directory"
},
"error":"COPY failed: stat /var/lib/docker/tmp/docker-builder264844317/testfile.txt: no such file or directory"
}
So far I have tried copying the files into the tar before building and then I have also tried moving the textfile.txt into the directory I run the command from but I still can not seem to get past this point
Extra information:
The file is in the same directory as the Dockerfile:
-- build-dir
|-- Dockerfile
|-- testfile.txt

From Source
The docker build command builds Docker images from a Dockerfile and a “context”. A build’s context is the set of files located in the specified PATH or URL. The build process can refer to any of the files in the context. For example, your build can use a COPY instruction to reference a file in the context.
Docker build-context is the entire directory you send to the docker engine. While building your image, Docker engine will try to find the files from the root of your build-context.
In your case, the file was not added to the build-context.

So a colleague pointed out to me instead of just running just `tar.AddAll' i also need to specify the files I want to add, see updated code below:
func buildCIImage() {
os.Chdir("ci-cd")
cli, err := client.NewEnvClient()
if err != nil {log.Fatal(err, " :unable to init client")}
// open the file to pass into the tar
file, err := os.OpenFile("testfile.txt", os.O_RDWR, os.ModePerm)
// Used to get the files information
fileInfo, err := os.Stat("testfile.txt")
tar := new(archivex.TarFile)
tar.Create("dockerfile.tar")
tar.AddAll(".", true)
// Add file into tar
tar.Add("testfile.txt", file, fileInfo)
tar.Close()
dockerBuildContext, err := os.Open("dockerfile.tar")
defer dockerBuildContext.Close()
options := types.ImageBuildOptions{
SuppressOutput: false,
Remove: true,
ForceRemove: true,
PullParent: true,
Tags: []string{"bootstrap"},
Dockerfile: "Dockerfile",
}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, options)
defer buildResponse.Body.Close()
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
_, err = io.Copy(os.Stdout, buildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}

Related

Can't find local Docker images created with Go SDK

I'm trying to create some docker images on macOS (with Colima backend) using Go Docker SDK with following code:
package main
import (
"context"
"flag"
"fmt"
"io"
"log"
"os"
"path/filepath"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"github.com/docker/docker/pkg/archive"
)
const dockerfile = `
# syntax=docker/dockerfile:1
FROM golang:1.19-alpine
WORKDIR /app
ENV GO111MODULE=on
ENV CGO_ENABLED=0
COPY . ./
RUN go mod download
RUN apk add --no-cache git
RUN go install github.com/securego/gosec/v2/cmd/gosec#latest
RUN go install golang.org/x/vuln/cmd/govulncheck#latest
RUN go get github.com/stripe/safesql
CMD gosec ./... && govulncheck ./... && safesql ./...`
func main() {
ctx := context.Background()
pathFlag := flag.String("path", ".", "path to the directory containing the go files")
flag.Parse()
if pathFlag == nil {
log.Fatal("path flag is nil")
}
err := os.WriteFile(*pathFlag+"/Dockerfile", []byte(dockerfile), 0777)
if err != nil {
fmt.Printf("Unable to write file: %v", err)
}
cli, err := client.NewClientWithOpts()
if err != nil {
log.Fatal(err, " :unable to init client")
}
files, err := WalkMatch(*pathFlag, "*.go")
if err != nil {
log.Fatal(err, " :unable to walk files")
}
reader, err := archive.TarWithOptions(*pathFlag, &archive.TarOptions{
IncludeFiles: append(files, []string{"go.mod", "go.sum", "Dockerfile"}...),
})
if err != nil {
log.Fatal(err, " :unable to create tar")
}
imageBuildResponse, err := cli.ImageBuild(
ctx,
reader,
types.ImageBuildOptions{
Tags: []string{"newname"},
Context: reader,
Dockerfile: "Dockerfile",
})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
func WalkMatch(root, pattern string) ([]string, error) {
var matches []string
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if matched, err := filepath.Match(pattern, filepath.Base(path)); err != nil {
return err
} else if matched {
r, err := filepath.Rel(root, path)
if err != nil {
return err
}
matches = append(matches, r)
}
return nil
})
if err != nil {
return nil, err
}
return matches, nil
}
The creation seems to be successful:
{"aux":{"ID":"sha256:568d5aed149aa73d38b644f49f68590cc06222467be1e1c3b902792da39f6e24"}}
{"stream":"Successfully built 568d5aed149a\n"}
{"stream":"Successfully tagged newname:latest\n"}
However, I can't find these images anywhere, docker run states there are no images with such tag, docker ls tells me the same. Do I miss something crucial? Are there any macOS specific things I should consider?
You are only building the images not running them so you should use the command
docker images
to get all the images present on your system.
docker ps
only gives back the images which are currently running

moving a file in a container to a folder that has a mounted volume docker

am trying to run a golang application on docker. But when i try to move created file in the container to the folder the created volume is mounted on,i get an error
:rename /mygo/newt /mygo/store/newt: invalid cross-device link
my golang code
package main
import (
"bufio"
"fmt"
"os"
"path/filepath"
"strings"
)
func main() {
for {
fmt.Println("do you want to create a file,y for yes, n for no")
var ans string
fmt.Scanln(&ans)
if ans == "y" {
var userFile string
fmt.Println("enter name of file")
fmt.Scanln(&userFile)
myfile, err := os.Create(userFile)
if err != nil {
fmt.Printf("error creating file::%v\n", err)
return
}
fmt.Println("enter text to write in file")
reader := bufio.NewReader(os.Stdin)
input, err := reader.ReadString('\t')
if err != nil {
fmt.Println("an error occured while reading::", err)
return
}
input = strings.TrimSuffix(input, "\t")
num, err := myfile.WriteString(input)
if err != nil {
fmt.Println("error while writing to file", err)
}
fmt.Printf("%v characters entered \n", num)
defer myfile.Close()
fmt.Println("created a file", userFile)
fmt.Println("===========")
fmt.Println("moving file to default folder")
pwd, err_pwd := os.Getwd()
if err_pwd != nil {
fmt.Printf("could not get current working directory::%v\n", err_pwd)
}
userFilePath := pwd + "/" + userFile
fmt.Println(pwd)
destinationFilePath := pwd + "/store/" + userFile
//destinationFilePath := pwd + "/default/" + userFile
err_moving := os.Rename(userFilePath, destinationFilePath)
if err_moving != nil {
fmt.Printf("Error occured while moving file::%v\n", err_moving)
return
}
fmt.Println("file moved")
continue
}
pwd, err_pwd := os.Getwd()
if err_pwd != nil {
fmt.Printf("could not get current working directory::%v\n", err_pwd)
}
fmt.Println("enter full path to move to default location")
var userFilePath string
fmt.Scanln(&userFilePath)
userFileName := filepath.Base(userFilePath)
destinationFilePath := pwd + "/" + userFileName
err_move := os.Rename(userFilePath, destinationFilePath)
if err_move != nil {
fmt.Printf("error occured while moving file:::%v", err_move)
return
}
fmt.Println("file moved")
continue
}
}
dockerfile
FROM golang
WORKDIR /mygo
COPY . .
RUN go build -o app
CMD ["./app"]
running the container
the program exits after the error
There are two ways to "rename" a file in Linux.
Move the directory entry to a new place, but keep the file contents unchanged.
This has the advantage of being fast. It has the disadvantage that it doesn't work when moving a file from one filesystem to another.
Create a new file, copy the data to the new file, and delete the old file.
However, it will work if the source and destination are on two different filesystems.
Method #1 will not work in this case. You need method #2.
More resources:
This golang-dev discussion explains why this happens.
This question talks about the same problem, but in the context of C++.
Go uses the renameat() syscall internally. This manual page explains how it works. The specific error you're encountering is the EXDEV error: "oldpath and newpath are not on the same mounted filesystem. (Linux permits a filesystem to be mounted at multiple points, but rename() does not work across different mount points, even if the same filesystem is mounted on both.)"

Send Docker Context as Tar with Go Client can't find Dockerfile

I am using the Go Docker Client to attempt to build an image from a Dockerfile whose contents are defined in code.
According to the Docker Daemon API Documentation
The input stream must be a tar archive...
...The archive must include a build instructions file, typically called Dockerfile at the archive’s root.
So I want to create the build context in code, write it to a tar file, then send that to the Docker Daemon to be built. To do this I can use the ImageBuild function and pass in the tar file (build context) as an io.ReadCloser. As long as my Dockerfile is at the root of that compressed archive, it should find it and build it.
However, I get the common error:
Error response from daemon: Cannot locate specified Dockerfile: Dockerfile
Which obviously means that it can't find the Dockerfile at the root of the archive. I am unsure why. I believe the way I am doing this adds a Dockerfile to the root of the tar archive. The daemon should see this. What am I misunderstanding here?
code snippet to reproduce
var buf bytes.Buffer
tarWriter := tar.NewWriter(&buf)
contents := "FROM alpine\nCMD [\"echo\", \"this is from the archive\"]"
if err := tarWriter.WriteHeader(&tar.Header{
Name: "Dockerfile",
Mode: 777,
Size: int64(len(contents)),
Typeflag: tar.TypeReg,
}); err != nil {
panic(err)
}
if _, err := tarWriter.Write([]byte(contents)); err != nil {
panic(err)
}
if err := tarWriter.Close(); err != nil {
panic(err)
}
reader := tar.NewReader(&buf)
c, err := client.NewEnvClient()
if err != nil {
panic(err)
}
_, err = c.ImageBuild(context.Background(), reader, types.ImageBuildOptions{
Context: reader,
Dockerfile: "Dockerfile",
})
if err != nil {
panic(err)
}
go.mod file
module docker-tar
go 1.12
require (
github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/docker v1.13.1
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/opencontainers/go-digest v1.0.0-rc1 // indirect
github.com/pkg/errors v0.8.1 // indirect
golang.org/x/net v0.0.0-20191112182307-2180aed22343 // indirect
)
Add a zero in front of 777 for octal numeral system: 0777, 0o777, or 0O777
Use reader := bytes.NewReader(buf.Bytes()) not tar.NewReader(&buf)
Use client.WithAPIVersionNegotiation() for newer versions too.
Try this working version:
package main
import (
"archive/tar"
"bytes"
"context"
"fmt"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
var buf bytes.Buffer
tarWriter := tar.NewWriter(&buf)
contents := `FROM alpine:3.10.3
CMD ["echo", "this is from the archive"]
`
header := &tar.Header{
Name: "Dockerfile",
Mode: 0o777,
Size: int64(len(contents)),
Typeflag: tar.TypeReg,
}
err := tarWriter.WriteHeader(header)
if err != nil {
panic(err)
}
_, err = tarWriter.Write([]byte(contents))
if err != nil {
panic(err)
}
err = tarWriter.Close()
if err != nil {
panic(err)
}
c, err := client.NewClientWithOpts(client.WithAPIVersionNegotiation())
if err != nil {
panic(err)
}
fmt.Println(c.ClientVersion())
reader := bytes.NewReader(buf.Bytes()) // tar.NewReader(&buf)
ctx := context.Background()
buildOptions := types.ImageBuildOptions{
Context: reader,
Dockerfile: "Dockerfile",
Tags: []string{"alpine-echo:1.2.4"},
}
_, err = c.ImageBuild(ctx, reader, buildOptions)
if err != nil {
panic(err)
}
}
Output for docker image ls after go run .:
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine-echo 1.2.4 d81774f32812 26 seconds ago 5.55MB
alpine 3.10.3 b168ac0e770e 4 days ago 5.55MB
Output for docker run alpine-echo:1.2.4:
this is from the archive
Note: You may need to edit FROM alpine:3.10.3 for your specific version.

docker build in Go client fail on COPY

I'm trying to build an image using docker's Go client. Here is the Go program I'm running:
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
log.Fatal(err, " :unable to init client")
}
buf := new(bytes.Buffer)
tw := tar.NewWriter(buf)
defer tw.Close()
dockerFile := "Dockerfile"
dockerFileReader, err := os.Open("./Dockerfile")
if err != nil {
log.Fatal(err, " :unable to open Dockerfile")
}
readDockerFile, err := ioutil.ReadAll(dockerFileReader)
if err != nil {
log.Fatal(err, " :unable to read dockerfile")
}
tarHeader := &tar.Header{
Name: dockerFile,
Size: int64(len(readDockerFile)),
}
err = tw.WriteHeader(tarHeader)
if err != nil {
log.Fatal(err, " :unable to write tar header")
}
_, err = tw.Write(readDockerFile)
if err != nil {
log.Fatal(err, " :unable to write tar body")
}
dockerFileTarReader := bytes.NewReader(buf.Bytes())
imageBuildResponse, err := cli.ImageBuild(
ctx,
dockerFileTarReader,
types.ImageBuildOptions{
Dockerfile: dockerFile,
Tags: []string{"devbuild"},
Remove: true})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
It puts the Dockerfile (located in the current directory) in a tar file and builds an image using cli.ImageBuild. This solution was taken from this post, and my Dockerfile looks like this:
FROM alpine:latest
WORKDIR /gopath/src/build
COPY ./binary_build/ /usr/local/bin/
I'm constantly getting the error on the last step:
{"stream":"Step 3/3 : COPY /binary_build/ /usr/local/bin/"}
{"stream":"\n"}
{"errorDetail":{"message":"COPY failed: stat /var/lib/docker/tmp/docker-builder389608393/binary_build: no such file or directory"},"error":"COPY failed: stat /var/lib/docker/tmp/docker-builder389608393/binary_build: no such file or directory"}
There seems to be similar issue in the past reported here, but it seems to have been patched. I ran go get github.com/docker/docker#latest and my docker version in go.mod is github.com/docker/docker v1.13.1. The issue still persists.
If you just build a docker image, that code means nothing, because the docker building process won't run the program.
You should put your Dockerfile and executable file "binary_build" in a sole folder,may be the Dockerfile should like:
FROM alpine:latest
WORKDIR /gopath/src/build
COPY binary_build /usr/local/bin/

Build docker images in parallel using docker Go client

I'm using Docker's Go client to build my projects. This post highlights how to do that with the Go client. I'm calling ImageBuild on three of my Dockerfiles (1.Dockerfile, 2.Dockerfile, and 3.Dockerfile) as a test. Here is my code:
func GetContext(filePath string) io.Reader {
// Use homedir.Expand to resolve paths like '~/repos/myrepo'
filePath, _ = homedir.Expand(filePath)
ctx, err := archive.TarWithOptions(filePath, &archive.TarOptions{})
if err != nil {
panic(err)
}
return ctx
}
func testImageBuild() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
log.Fatal(err, " :unable to init client")
}
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
dockerFile := fmt.Sprintf("%d.Dockerfile", i)
imageBuildResponse, err := cli.ImageBuild(
ctx,
GetContext("."),
types.ImageBuildOptions{
Dockerfile: dockerFile,
Tags: []string{fmt.Sprintf("devbuild_%d", i)},
})
if err != nil {
log.Fatal(err, " :unable to build docker image"+string(1))
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response "+string(1))
}
}(i)
}
wg.Wait()
}
func main() {
testImageBuild()
}
GetContext is used to tar the directory path as a context for Docker. testImageBuild spins off three different goroutines to build the three different images.
My question is: When I run this, the output to stdout is always the same and seems deterministic, which makes me think that the images aren't actually been built in parallel. I'm not familiar with how docker build its images, and it seems entirely possible that this approach is simply sending requests to docker server in parallel rather than actually building in parallel. Is this true? If so, how can I build my projects in parallel?
If I understand your question correctly, you have a docker-machine on which you want to build the images concurrently using your GO program.
I tried to do the same thing with Dockerfiles which have the same image being built, and per my understanding, all of them were build concurrently.
Here is the go package that I used to replicate the scenario - https://github.com/nihanthd/stackoverflow/tree/master/docker
Now in your case if you were using 3 different docker files, then certainly they would have different build times, that means the output would be seem to be deterministic

Resources