Grab output from container in Docker SDK - docker

I'm trying to run a container using Docker SDK for golang and I can't get the output from the container. I'm using the following code for that that actually runs the container, but doesn't sends back stderr and stdout of the application. Can you advice what I'm doing wrong?
type dckr struct {
cli *client.Client
username string
password string
addr string
ctx context.Context
}
func (d *dckr) Run(containername string, image string, command []string, bind []string, stdout io.Writer, stderr io.Writer) error {
log.Printf("[Create] %s -> %s \n", image, containername)
res, err := d.cli.ContainerCreate(
d.ctx,
&container.Config{
User: "root",
AttachStdout: true,
AttachStderr: true,
Image: image,
Cmd: command,
},
&container.HostConfig{
AutoRemove: true,
Binds: bind,
},
&network.NetworkingConfig{},
containername,
)
if err != nil {
log.Println("[Create] Failed. %s", err)
return err
}
defer d.cli.ContainerRemove(d.ctx, res.ID, types.ContainerRemoveOptions{Force: true})
log.Printf("[Create] id: %s \n", res.ID)
for wrn := range res.Warnings {
log.Printf("[Create] %s \n", wrn)
}
rsp, err := d.cli.ContainerAttach(d.ctx, containername, types.ContainerAttachOptions{
Stream: false,
Stdout: true,
Stderr: true,
Logs: true,
})
if err != nil {
log.Printf("[Attach] Fail. %s \n", err)
return err
}
log.Printf("[Attach] %s", res.ID)
defer rsp.Close()
err = d.cli.ContainerStart(d.ctx, res.ID, types.ContainerStartOptions{})
if err != nil {
log.Printf("[Run] Fail. %s \n", err)
return err
}
_, err = stdcopy.StdCopy(stdout, stderr, rsp.Reader)
return err
}

The question was asked in 2017 and I'm answering it in 2022. I understand the APIs might have changed, but I have landed on a similar boat.
Let's not talk about how to start a container as you seem to have already done that. Here is my code to fetch the logs from a given container:
// GetLogs return logs from the container io.ReadCloser. It's the caller duty
// duty to do a stdcopy.StdCopy. Any other method might render unknown
// unicode character as log output has both stdout and stderr. That starting
// has info if that line is stderr or stdout.
func GetLogs(ctx context.Context, cli *client.Client, contName string) (logOutput io.ReadCloser) {
options := types.ContainerLogsOptions{ShowStdout: true}
out, err := cli.ContainerLogs(ctx, contName, options)
if err != nil {
panic(err)
}
return out
}
You can call this GetLogs from another routine. I am saving both of the stream in specific files. But you may want to use os.Stdout and os.Stderr if you just want to see them on your terminal:
func main() {
stdoutLog, _ := os.Create("yourContainerName.log")
defer stdoutLog.Close()
stderrLog, _ := os.Create("yourContainerName.err")
defer stderrLog.Close()
var stdout bytes.Buffer
var stderr bytes.Buffer
containerLog := docker.GetLogs(ctx, dc, "yourContainerName")
stdcopy.StdCopy(&stdout, &stderr, containerLog)
stdoutLog.Write(stdout.Bytes())
stderrLog.Write(stderr.Bytes())
}
Let me know the second part if you still have confusion. I'm happy to help as I had a similar problem.

Related

Running docker builds in parallel in an EC2 instance results in longer build times - why?

We are building multiple docker images in the cloud. This is done using this code -
func BuildM1Image(tags []string, dockerfile string, contextPath string, dockerRepoUrl string) error {
dockerExecutable, _ := exec.LookPath("docker")
awsExecutable, _ := exec.LookPath("aws")
toTag := tags[0]
Log("building image with tag " + toTag)
// to push --> I need the AWS credentials to live in the cloud, pull them
os.Setenv("AWS_ACCESS_KEY_ID", Secrets[ECR_ACCESS_KEY_NAME])
os.Setenv("AWS_SECRET_ACCESS_KEY", Secrets[ECR_SECRET_ACCESS_KEY_NAME])
ecrGetCredentialsCMD := &exec.Cmd{
Path: awsExecutable,
Args: []string{awsExecutable, "ecr", "get-login-password", "--region", GetRegion()},
// Stderr: os.Stderr,
// Stdout: os.Stdout,
}
out, _ := ecrGetCredentialsCMD.CombinedOutput()
// if err != nil {
// errorChannel <- err
// return
// }
dockerEcrLoginCMD := &exec.Cmd{
Path: dockerExecutable,
Args: []string{dockerExecutable, "login", "--username", "AWS", "-p", string(out), dockerRepoUrl},
}
if err := dockerEcrLoginCMD.Run(); err != nil {
fmt.Println("Docker login failed")
fmt.Println("error: ", err)
return err
}
buildDockerImage := &exec.Cmd{
Path: dockerExecutable,
Args: []string{dockerExecutable, "buildx", "build", "--platform", "linux/amd64", "-t", toTag, "-f", dockerfile, contextPath},
}
if err := buildDockerImage.Run(); err != nil {
fmt.Println("docker build failed")
logError(err)
return err
}
return nil
}
I put the this function inside of a goroutine and kicked off multiple builds. However, the time taken on the builds is slower than if I were to do this sequentially. This only happens in an EC2 instance and not locally (locally we're running this with a M1 mac and it's working as expected).
Why would this happen?
On the EC2 instance we've tried -
Increasing the compute/storage
Increasing the IOPS
Thanks!

Docker (Moby) golang image build logs are base64 encoded

I'm looking for help with extracting the image build logs from a dockerd (buildkit/moby) image build request sent by a Golang based client using the docker client libraries.
I can request the image build fine and receive the log stream of json messages then decode them as Jsonmessage instances. But the actual log lines from the builder appear to be base64 encoded in an aux field of each json message.
I can decode the base64 easily enough, but they seem to include odd terminal control characters and possibly mis-encoded data, which makes me wonder if they're actually a base64 encoding of some kind of struct I'm supposed to unpack.
What confuses me is that I can't find anything in the docker-ce or moby code that seems to base64-decode an 'aux' payload when processing logs when displaying build progress for docker buildx build.
As far as I can tell, the buildx code doesn't do anything special to the aux payload: https://github.com/docker/docker-ce/blob/523cf7e71252013fbb6a590be67a54b4a88c1dae/components/cli/cli/command/image/build_buildkit.go#L325
For example, trimmed-down build code like:
image := Image{Name: "test"}
contextreader, err := archive.TarWithOptions(buildConf.Build.Context, &archive.TarOptions{})
if err != nil {
return err
}
imageBuildResponse, err := b.client.ImageBuild(
ctx,
contextreader,
types.ImageBuildOptions{
Version: types.BuilderBuildKit,
Context: contextreader,
Dockerfile: dockerfile,
})
if err != nil {
return err
}
defer imageBuildResponse.Body.Close()
buf := bytes.NewBuffer(nil)
imageID := ""
writeAux := func(msg jsonmessage.JSONMessage) {
if msg.ID == "moby.image.id" {
var result types.BuildResult
if err := json.Unmarshal(*msg.Aux, &result); err != nil {
panic("don't do this in your real code")
}
imageID = result.ID
return
}
return err
}
err := jsonmessage.DisplayJSONMessagesStream(imageBuildResponse.Body, buf, os.Stderr.Fd(), false /* not terminal */, writeAux)
if err != nil {
if jerr, ok := err.(*jsonmessage.JSONError); ok {
// If no error code is set, default to 1
if jerr.Code == 0 {
jerr.Code = 1
}
return fmt.Errorf("error while building image: %s", jerr.Message)
}
}
will write json payloads to stderr like
{"id":"moby.buildkit.trace","aux":"Cn0KR3NoYTI1NjozZThhMzMxYmRkZGFjNWZkYmNjOGVhMDFmYWFhYmM3MjA0MDkwMmYwNjdmYzRhOGY0NDJmMmIzYWVlN2RkNGIyGiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQqDAiYw8KaBhCykpCqAg=="}
{"id":"moby.buildkit.trace","aux":"CokBCkdzaGEyNTY6M2U4YTMzMWJkZGRhYzVmZGJjYzhlYTAxZmFhYWJjNzIwNDA5MDJmMDY3ZmM0YThmNDQyZjJiM2FlZTdkZDRiMhokW2ludGVybmFsXSBsb2FkIHJlbW90ZSBidWlsZCBjb250ZXh0KgwImMPCmgYQspKQqgIyCgiZw8KaBhD08F0="}
The base64 strings here don't decode as valid utf-8, and they don't make sense as ISO-8859-1 either. E.g. with a utf-8 console encoding:
$ base64 -d <<<'Cn0KR3NoYTI1NjozZThhMzMxYmRkZGFjNWZkYmNjOGVhMDFmYWFhYmM3MjA0MDkwMmYwNjdmYzRhOGY0NDJmMmIzYWVlN2RkNGIyGiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQqDAiYw8KaBhCykpCqAg=='
}
Gsha256:3e8a331bdddac5fdbcc8ea01faaabc72040902f067fc4a8f442f2b3aee7dd4b2�$[internal] load remote build context*
������
It looks like it's probably a struct, but for the life of me I can't find what decodes and processes it.
So of course I find the answer while writing up the SO question...
The writeAux function in build_buildkit.go calls the write method of a tracer instance, and that does the real work. I must've been blind.
The messages are serialized instances of StatusResponse from the github.com/moby/buildkit/api/services/control package. They are unmarshalled from base64-decoded byte sequences and inspected. If you want logs and to skip everything else, just look for instances with non-empty Logs member arrays, e.g. something like this within the above writeAux function:
} else if msg.ID == "moby.buildkit.trace" {
// Process the message like
// https://github.com/docker/docker-ce/blob/523cf7e71252013fbb6a590be67a54b4a88c1dae/components/cli/cli/command/image/build_buildkit.go#L386
// the 'tracer.write' method in build_buildkit.go
var resp controlapi.StatusResponse
var dt []byte
// ignoring all messages that are not understood
if err := json.Unmarshal(*msg.Aux, &dt); err != nil {
return
}
if err := (&resp).Unmarshal(dt); err != nil {
return
}
for _, v := range resp.Vertexes {
fmt.Printf("layer: %+v", v)
}
for _, v := range resp.Statuses {
fmt.Printf("status: %+v", v)
}
for _, v := range resp.Logs {
fmt.Printf("log: msg.Msg)
}
}
The json.Unmarshal and controlapi.StatusResponse.Unmarshal do the base64 decoding and unpacking for you.

Server in Docker container connection refused, should I add time.Sleep(100 * time.Millisecond) to my tests?

I am currently working on a server that is designed to be run in a Docker container.
Here is my setup method for my tests:
func TestMain(m *testing.M) {
schedulerName := "scheduler1"
IP, err := container.StartNewScheduler(schedulerName)
if err != nil {
log.Println("Could not create container.")
log.Fatal(err)
}
serverIP = IP
code := m.Run()
cleanupContainer(schedulerName)
os.Exit(code)
}
The line container.StartNewScheduler(schedulername) boots up a new docker container called "scheduler1" and tells it to run the server inside of it.
Next I run my tests with the container running in the background, right now I only have one test.
func TestNewScheduler(t *testing.T) {
testCodeInput := "THIS IS A TEST"
requestBody, err := json.Marshal(map[string]string{
"Code": fmt.Sprintf("print(\"%s\")", testCodeInput),
})
if err != nil {
t.Fatal(err)
}
url := fmt.Sprintf("http://%s:%d/execute/python", serverIP, 3000)
contentType := "application/json"
body := bytes.NewBuffer(requestBody)
response := post(url, contentType, body, t)
actual := parseOutput(response.Body, t)
response.Body.Close()
expected := fmt.Sprintf("{\"Stdout\":\"%s\\n\"}", testCodeInput)
if actual != expected {
t.Fatalf("Expected %s, but got %s", expected, actual)
}
}
The problem that I am running into is sometimes I get a connection refused and sometimes I don't.
server_container_test.go:51: Post http://172.20.0.2:3000/execute/python: dial tcp 172.20.0.2:3000: connect: connection refused
I noticed that whenever I try and debug the issue everything seems to work fine. My running theory is because when I step through my code the container has more time to start up and get the server running inside it.
In order to test my Hypothesis I added a second post call in my post method with a timer set before I call it.
func post(url string, contentType string, body io.Reader, t *testing.T) *http.Response {
t.Helper()
response, err := http.Post(url, contentType, body)
if err != nil {
//There is an error where the container takes a second to boot up and so
//the scheduler isn't running when the first request is sent, so we try
//one more time here and check again.
time.Sleep(100 * time.Millisecond) <--- Right here
response, err = http.Post(url, contentType, body)
if err != nil {
t.Fatal(err)
}
}
return response
}
Does anyone else have any guesses as to what could be causing me this issue?
If my hypothesis is correct is this the best way to fix this? Is it a bad idea to add a time.Sleep to your tests?
Thanks!
Ok so after some more thought I changed up my source code, please let me know if you think this is a good solution to my problem. I am still learning Go and HTTP servers so any input is appreciated.
Here is my fix/idea:
Previously once the container was created I just returned it's IP address and forgot about it.
Now I create a go routine that repeatedly tries to send a POST request to the server. If it doesn't fail then I send true through a channel and close the function.
IP := info.NetworkSettings.Networks[networkName].IPAddress
works := make(chan bool)
ctx, canelRoutine := context.WithCancel(context.Background())
defer canelRoutine()
go func(ctx context.Context) {
requestBody, _ := json.Marshal(map[string]string{
"Code": "print()",
})
select {
case <-ctx.Done():
return
default:
for {
_, err := http.Post(
fmt.Sprintf("http://%s:%d/execute/python", IP, 3000),
"application/json",
bytes.NewBuffer(requestBody),
)
if err == nil {
works <- true
return
}
}
}
}(ctx)
After sending the goroutine off I create a timer and and wait for either the timer to return or the goroutine.
timer := time.After(500 * time.Millisecond)
select {
case <-works:
return IP, nil
case <-timer:
return IP, &UnreachableContainerError{name: schedulerName}
}
The upside to this solution is I have now introduced an UnreachableContainerError which allows me to be more specific about my error messages and it can be checked on the receiving side. I also send the IP address back either way just in case the client needs it for some other reason.
Here is the full StartNewScheduler method in case you wanted to see it.
//StartNewScheduler starts a new scheduler with the given options.
//returns the IP address for the given scheduler.
func StartNewScheduler(schedulerName string) (string, error) {
///Defaults
dockerfile := "Dockerfile_standard"
networkName := "scheduler-cluster"
imageID := "lkelly93/scheduler_image:latest"
cli, err := client.NewEnvClient()
if err != nil {
return "", err
}
err = createDefaultImageIfNeeded(
cli,
imageID,
dockerfile)
if err != nil {
return "", err
}
err = createSchedulerClusterNetworkIfNeeded(cli, networkName)
if err != nil {
return "", err
}
ctx := context.Background()
resp, err := cli.ContainerCreate(
ctx,
&container.Config{Image: imageID},
&container.HostConfig{
NetworkMode: container.NetworkMode(networkName),
Privileged: true,
},
nil,
schedulerName,
)
if err != nil {
return "", err
}
err = cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{})
if err != nil {
return "", err
}
//Get container IP
info, err := cli.ContainerInspect(ctx, resp.ID)
if err != nil {
return "", err
}
IP := info.NetworkSettings.Networks[networkName].IPAddress
works := make(chan bool)
ctx, canelRoutine := context.WithCancel(context.Background())
defer canelRoutine()
go func(ctx context.Context) {
requestBody, _ := json.Marshal(map[string]string{
"Code": "print()",
})
select {
case <-ctx.Done():
return
default:
for {
_, err := http.Post(
fmt.Sprintf("http://%s:%d/execute/python", IP, 3000),
"application/json",
bytes.NewBuffer(requestBody),
)
if err == nil {
works <- true
return
}
}
}
}(ctx)
timer := time.After(500 * time.Millisecond)
select {
case <-works:
return IP, nil
case <-timer:
return IP, &UnreachableContainerError{name: schedulerName}
}
}
You could run the test code inside of a container, started via docker compose with the option depends_on.

Docker Go lang SDK returns nothing from ContainerExecCreate

I am trying to execute a command (let's say "pwd) using Docker Go lang SDK, and I expect it returns the working directory on the container. But it returns nothing back. I am not sure what is the issue.
rst, err := cli.ContainerExecCreate(context.Background(), "0df7c1d9d185b1da627efb983886a12fefc32120d035b34e97c3ad13da6dd9cc", types.ExecConfig{Cmd: []string{"pwd"}})
if err != nil {
panic(err)
}
//res, err := cli.ContainerExecInspect(context.Background(), rst.ID)
//print(res.ExitCode)
response, err := cli.ContainerExecAttach(context.Background(), rst.ID, types.ExecStartCheck{})
if err != nil {
panic(err)
}
defer response.Close()
data, _ := ioutil.ReadAll(response.Reader)
fmt.Println(string(data))
GOROOT=/usr/local/Cellar/go/1.13.5/libexec #gosetup
GOPATH=/Users/pt/go #gosetup
/usr/local/Cellar/go/1.13.5/libexec/bin/go build -o /private/var/folders/yp/hh3_03d541x0r6t7_zwqqhqr0000gn/T/___go_build_main_go /Users/pt/go/src/awesomeProject/main.go #gosetup
/private/var/folders/yp/hh3_03d541x0r6t7_zwqqhqr0000gn/T/___go_build_main_go #gosetup
### It does not print the working directory ###
Process finished with exit code 0
This is solved by the following config:
optionsCreate := types.ExecConfig{
AttachStdout: true,
AttachStderr: true,
Cmd: []string{"ls", "-a"},
}

Golang - Docker API - parse result of ImagePull

I'm developing a Go script that uses the Docker API for the purposes of my project. After I login to my repository, I pull the Docker image I want, but the problem is that the ImagePull function returns an instance of io.ReadCloser, which I'm only able to pass to the system output via:
io.Copy(os.Stdout, pullResp)
It's cool that I can see the response, but I can't find a decent way to parse it and implement a logic depending on it, which will do some things if a new version of the image have been downloaded, and other things if the image was up to date.
I'll be glad if you share your experience, if you have ever faced this problem.
You can import github.com/docker/docker/pkg/jsonmessage and use both JSONMessage and JSONProgress to decode the stream but it's easier to call
DisplayJSONMessagesToStream: it both parses the stream and displays the messages as text. Here's how you can display the messages using stderr:
reader, err := cli.ImagePull(ctx, myImageRef, types.ImagePullOptions{})
if err != nil {
return err
}
defer reader.Close()
termFd, isTerm := term.GetFdInfo(os.Stderr)
jsonmessage.DisplayJSONMessagesStream(reader, os.Stderr, termFd, isTerm, nil)
The nice thing is that it adapts to the output: it updates the lines if this a TTY (the way docker pull does) but it doesn't if the output is redirected to a file.
#radoslav-stoyanov before use my example do
# docker rmi busybox
then run code
package main
import (
"encoding/json"
"fmt"
"github.com/docker/distribution/context"
docker "github.com/docker/engine-api/client"
"github.com/docker/engine-api/types"
"io"
"strings"
)
func main() {
// DOCKER
cli, err := docker.NewClient("unix:///var/run/docker.sock", "v1.28", nil, map[string]string{"User-Agent": "engine-api-cli-1.0"})
if err != nil {
panic(err)
}
imageName := "busybox:latest"
events, err := cli.ImagePull(context.Background(), imageName, types.ImagePullOptions{})
if err != nil {
panic(err)
}
d := json.NewDecoder(events)
type Event struct {
Status string `json:"status"`
Error string `json:"error"`
Progress string `json:"progress"`
ProgressDetail struct {
Current int `json:"current"`
Total int `json:"total"`
} `json:"progressDetail"`
}
var event *Event
for {
if err := d.Decode(&event); err != nil {
if err == io.EOF {
break
}
panic(err)
}
fmt.Printf("EVENT: %+v\n", event)
}
// Latest event for new image
// EVENT: {Status:Status: Downloaded newer image for busybox:latest Error: Progress:[==================================================>] 699.2kB/699.2kB ProgressDetail:{Current:699243 Total:699243}}
// Latest event for up-to-date image
// EVENT: {Status:Status: Image is up to date for busybox:latest Error: Progress: ProgressDetail:{Current:0 Total:0}}
if event != nil {
if strings.Contains(event.Status, fmt.Sprintf("Downloaded newer image for %s", imageName)) {
// new
fmt.Println("new")
}
if strings.Contains(event.Status, fmt.Sprintf("Image is up to date for %s", imageName)) {
// up-to-date
fmt.Println("up-to-date")
}
}
}
You can see API formats to create your structures (like my Event) to read them here https://docs.docker.com/engine/api/v1.27/#operation/ImageCreate
I hope it helps you solve your problem, thanks.
I have used similar approach for my purpose (not a moby client). Typically idea is same for reading stream response. Give it a try and implement yours.
Reading stream response of any response type:
reader := bufio.NewReader(pullResp)
defer pullResp.Close() // pullResp is io.ReadCloser
var resp bytes.Buffer
for {
line, err := reader.ReadBytes('\n')
if err != nil {
// it could be EOF or read error
// handle it
break
}
resp.Write(line)
resp.WriteByte('\n')
}
// print it
fmt.Println(resp.String())
However your sample response in the comment seems valid JSON structure. The json.Decoder is best way to read JSON stream. This is just an idea-
type ImagePullResponse struct {
ID string `json"id"`
Status string `json:"status"`
ProgressDetail struct {
Current int64 `json:"current"`
Total int64 `json:"total"`
} `json:"progressDetail"`
Progress string `json:"progress"`
}
And do
d := json.NewDecoder(pullResp)
for {
var pullResult ImagePullResponse
if err := d.Decode(&pullResult); err != nil {
// handle the error
break
}
fmt.Println(pullResult)
}

Resources