How can I get Container Logs using Golang? (Error) - docker

I am trying to code a Docker Monitoring software in Golang.
my Code looks as followed:
package main
import (
"bytes"
"context"
"fmt"
"time"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
ctx := context.Background()
cli, err := client.NewClientWithOpts(client.FromEnv)
if err != nil {
panic(err)
}
containers, err := cli.ContainerList(ctx, types.ContainerListOptions{})
if err != nil {
panic(err)
}
for _, container := range containers {
out, err := cli.ContainerLogs(ctx, container.ID, types.ContainerLogsOptions{
ShowStderr: true,
ShowStdout: true,
Timestamps: false,
Follow: true,
Tail: "40"})
if err != nil {
panic(err)
}
fmt.Println("The \"" + container.Image + "\" container, with the ID \"" + container.ID + "\" logged: ")
fmt.Println()
buf := new(bytes.Buffer)
fmt.Println(buf.ReadFrom(out))
fmt.Println(buf.String())
}
time.Sleep(time.Second * 3)
}
The problem is that the execution of the above code stops on the fmt.Println(buf.ReadFrom(out)) statement. The code used to work, but it suddenly just doesn't anymore. Either it stops without an error, or it returns an empty String.
The client I am trying to collect the logs from is also coded by myself, and it looks like follows:
package main
import (
"log"
"time"
)
func main() {
for i := 0; i > -1; i++ {
log.Output(1, "Hello World logged!")
time.Sleep(time.Minute)
}
}
I already tried debugging and checking Variables, but I just can't get to the source of the Problem.

I am really not sure as I don't have any error logs to confirm my hypothesis.
But could it be the case that as the ContainerLogs returns a stream (io.ReadCloser), maybe the stream itself hasn't closed yet?
If this is a possibility, you can either do a dry run first to test out this theory by adding a timeout and logging it after every small duration ?
one possible way to do this is
select {
case <-time.After(5 * time.Second):
fmt.Println("Timeout exceeded while reading container logs")
case <-ctx.Done():
fmt.Println("Context cancelled while reading container logs")
case b := <-out:
if b != nil {
buf.Write(b)
}
}

Related

Read docker containers logs in JSON format - Golang

I have a requirement to fetch docker container's logs, I'm using below code to fetch docker logs,
ctx := context.Background()
cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
if err != nil {
panic(err)
}
options := types.ContainerLogsOptions{
ShowStdout: true,
ShowStderr: true,
Since: "",
Until: "",
Timestamps: false,
Follow: true,
Tail: "",
Details: true,
}
out, err := cli.ContainerLogs(ctx, "bcd693465a62", options)
if err != nil {
panic(err)
}
buf := new(strings.Builder)
_, err = io.Copy(buf, out)
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s", buf)
My problem is I'm getting docker logs in two different format, for some containers it's just plain text
exec /entrypoint.sh: no such file or directory
but in containerID-json.log file it's showing in below format
{"log":"exec /entrypoint.sh: no such file or directory\n","stream":"stderr","time":"2022-09-06T12:23:57.741316145Z"}
and for some containers it's showing in dynamic format/schema like
time="2022-09-05T02:13:44Z" level=debug msg="cleanup aborting ingest" ref="buildkit/1/layer-sha256:2774afd0c4d3ded992c58f3b5e5939d091bd26f40e507c6dc21dcbd8b7ff486f"
time="2022-09-05T02:13:44Z" level=debug msg="content garbage collected" d=2.971574ms
How I can collect all docker container's logs in same schema/format so it can be stored in below JSON format?
{
"containerID": "bcd693465a62",
"logs": [
{"log":"exec /entrypoint.sh: no such file or directory\n","stream":"stderr","time":"2022-09-06T12:23:57.741316145Z"},
{"log":"cleanup aborting ingest","stream":"stdout","time":"2022-09-05T02:13:44Z"}
]
}
Any help is appreciated
I wrote below code with the help of code reference by #BrianWagner to fulfill my requirements (It works as expected), please suggest if it can be done better way, or if any improvements.
package main
import (
"context"
"encoding/binary"
"fmt"
"io"
"log"
"strings"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
type DockerContainerLog struct {
ContainerID string `json:"containerID"`
DockerLogs []DockerLog `json:"dockerLogs"`
}
type DockerLog struct {
Time string `json:"time"`
Stream string `json:"stream"`
Log string `json:"log"`
}
func main() {
logs := getContainerLogs("cd3d8362ba45")
fmt.Print(logs)
}
func getContainerLogs(cid string) DockerContainerLog {
cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
if err != nil {
panic(err)
}
options := types.ContainerLogsOptions{
ShowStdout: true,
ShowStderr: true,
Since: "",
Until: "",
Timestamps: true,
Follow: true,
Tail: "",
Details: false,
}
reader, err := cli.ContainerLogs(context.Background(), cid, options)
if err != nil {
log.Fatal(err)
}
defer reader.Close()
dockerLogs := DockerContainerLog{ContainerID: cid}
hdr := make([]byte, 8)
for {
var docLog DockerLog
_, err := reader.Read(hdr)
if err != nil {
if err == io.EOF {
return dockerLogs
}
log.Fatal(err)
}
count := binary.BigEndian.Uint32(hdr[4:])
dat := make([]byte, count)
_, err = reader.Read(dat)
if err != nil && err != io.EOF {
log.Fatal(err)
}
time, log, found := strings.Cut(string(dat), " ")
if found {
docLog.Time = time
docLog.Log = log
switch hdr[0] {
case 1:
docLog.Stream = "Stdout"
default:
docLog.Stream = "Stderr"
}
dockerLogs.DockerLogs = append(dockerLogs.DockerLogs, docLog)
}
}
}

Vuejs client can't communicate with Go backend via gRPC on MacOS

I built a simple web app with a VueJS front (client) and a Go back (server), using gRPC and protobuf.
In order to communicate, an envoy proxy has to be set up between them to convert web client HTTP/1.1 to HTTP/2.
I have no issue deploying this app on Linux, and have dockerized all three services.
However, I cannot manage to get my services to communicate on MacOS.
The app repository is available here: https://github.com/42Projects/nPuzzle
The deploy folder contains the docker-compose file, the envoy.yaml file required for the envoy container, as well as all three dockerfiles.
On MacOS the last line of the envoy.yaml needs to be updated, localhost has to be changed to host.docker.internal.
The client contains a ping function (in client/src/App.vue), the window.location.hostname might need to be changed to the docker-machine IP, unsure about that, my tries made no difference:
created () {
this.client = new NpuzzleClient('http://' + window.location.hostname + ':8080', null, null);
const ping = () => {
let message = new Message();
message.setMessage('ping');
this.client.greets(message, {}, err => this.serverOnline = !err);
};
ping();
window.setInterval(ping, 1000);
},
which is executed every second, and will display that the server is online on the top right corner. I have been so far unsuccessful in reaching my server while deploying on MacOS. It works perfectly fine on Linux.
Here is the code for the go server, I think it is correct to listen on localhost:9090 as the server will be dockerized, but I might be wrong and the address might need to be changed.
package main
import (
"context"
"flag"
"fmt"
pb "github.com/42Projects/nPuzzle/proto"
npuzzle "github.com/42Projects/nPuzzle/src"
"google.golang.org/grpc"
"log"
"net"
"time"
)
var port = flag.Int("port", 9090, "the server port")
type server struct{}
func (s *server) Greets(ctx context.Context, message *pb.Message) (*pb.Message, error) {
return &pb.Message{Message: "pong!"}, nil
}
func (s *server) Parse(ctx context.Context, message *pb.Message) (*pb.Matrix, error) {
log.Printf("received parsing request: %#v", message.Message)
m, err := npuzzle.ParseMatrix(message.Message)
if err != nil {
log.Println(err)
return &pb.Matrix{
Success: false,
Error: err.Error(),
}, nil
}
rows := make([]*pb.Matrix_Row, len(m))
for index, row := range m {
/* We need unsigned 32bits integer for protobuf */
uIntRow := make([]uint32, len(m))
for rowIndex, num := range row {
uIntRow[rowIndex] = uint32(num)
}
rows[index] = &pb.Matrix_Row{Num: uIntRow}
}
return &pb.Matrix{
Success: true,
Rows: rows,
}, nil
}
func (s *server) Solve(ctx context.Context, problem *pb.Problem) (*pb.Result, error) {
/* Choose heuristic function */
var heuristic npuzzle.Heuristic
switch problem.Heuristic {
case "hamming":
heuristic = npuzzle.HammingDistance
case "manhattan":
heuristic = npuzzle.ManhattanDistance
case "manhattan + linear conflicts":
heuristic = npuzzle.ManhattanPlusLinearConflicts
}
/* Choose between greedy search and uniform-cost search */
var goal npuzzle.Goal
var search npuzzle.Search
switch problem.Search {
case "greedy":
goal = npuzzle.GreedyGoalReached
search = npuzzle.GreedySearch
case "uniform-cost":
goal = npuzzle.UniformCostGoalReached
search = npuzzle.UniformCostSearch
}
/* Convert protobuf unsigned 32bits integer to regular integer */
size := len(problem.Rows)
m := make(npuzzle.Matrix, size)
for y, row := range problem.Rows {
m[y] = make([]int, size)
for x, num := range row.Num {
m[y][x] = int(num)
}
}
log.Printf("received problem:\n - heuristic: %v\n - search: %v\n - matrix: %v\n", problem.Heuristic, problem.Search, m)
if npuzzle.IsSolvable(m) == false {
log.Println("failed to solve problem: unsolvable")
return &pb.Result{
Success: false,
Error: "unsolvable",
}, nil
}
begin := time.Now()
log.Printf("starting solve on %v...", m)
res, totalNumberOfStates, maxNumberOfStates, err := m.Solve(heuristic, search, goal, 30*time.Second)
if err != nil {
log.Printf("timed ouf after %v", 30*time.Second)
return &pb.Result{
Success: false,
Error: fmt.Sprintf("timed ouf after %v", 30*time.Second),
}, nil
}
duration := time.Since(begin)
log.Printf("solved %v in %v seconds", m, duration)
var path string
if res.Parent == nil {
path = "already solved!"
} else {
path = npuzzle.StringifyPath(res)
}
return &pb.Result{
Success: true,
Time: duration.String(),
Moves: int32(res.Cost),
TotalStates: int32(totalNumberOfStates),
MaxStates: int32(maxNumberOfStates),
Path: path,
}, nil
}
func main() {
flag.Parse()
lis, err := net.Listen("tcp", fmt.Sprintf("localhost:%d", *port))
if err != nil {
log.Fatalf("failed to listen: %v\n", err)
}
s := grpc.NewServer()
pb.RegisterNpuzzleServer(s, &server{})
log.Printf("starting server on port %v\n", *port)
log.Fatalf("failed to serve: %v\n", s.Serve(lis))
}
I tried to launch the client and the server locally, and only containerize the envoy proxy, while trying different addresses (my docker machine IP address and localhost), but it didn't work. I also tried to launch all three containers but no result here either.
I'm unsure what to change to successfully manage to make my app work properly on MacOS.

Golang Docker API: get events

I want to get all new events from docker via the golang integration.
The problem is that it returns two channels and I couldn't figure out how to subscribe to them.
cli, err := client.NewClientWithOpts(client.WithVersion("1.37"))
if err != nil {
panic(err)
}
ctx, _ := context.WithCancel(context.Background())
msg, err := <- cli.Events(ctx, types.EventsOptions{})
There are many solutions. A solution could be:
msgs, errs := cli.Events(ctx, types.EventsOptions{})
for {
select {
case err := <-errs:print(err)
case msg := <-msgs:print(msg)
}
}

Programmatically check if Docker container process ended with non-zero status

I'm working on a Go application which starts some Docker containers using Go Docker SDK. I need to check if containers' processes exit with zero (success) status code.
Here's the minimal working example:
package main
import (
"context"
"io"
"log"
"os"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/client"
)
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
log.Fatal(err)
}
reader, err := cli.ImagePull(
ctx,
"docker.io/library/alpine",
types.ImagePullOptions{},
)
if err != nil {
log.Fatal(err)
}
io.Copy(os.Stdout, reader)
resp, err := cli.ContainerCreate(ctx, &container.Config{
Image: "alpine",
Cmd: []string{"sh", "-c", "echo hello world; return 1"},
Tty: true,
}, nil, nil, "")
if err != nil {
log.Fatal(err)
}
err = cli.ContainerStart(
ctx,
resp.ID,
types.ContainerStartOptions{},
)
if err != nil {
log.Fatal(err)
}
statusCh, errCh := cli.ContainerWait(
ctx,
resp.ID,
container.WaitConditionNotRunning,
)
select {
case err := <-errCh:
if err != nil {
log.Fatal(err)
}
case <-statusCh:
}
out, err := cli.ContainerLogs(
ctx,
resp.ID,
types.ContainerLogsOptions{ShowStdout: true},
)
if err != nil {
log.Fatal(err)
}
io.Copy(os.Stdout, out)
}
As you can see, the process in the container ends with non-zero status (sh -c "echo hello world; return 1"). However, it doesn't log any fatal errors and simply displays hello world when built and executed:
{"status":"Pulling from library/alpine","id":"latest"}
{"status":"Digest: sha256:7043076348bf5040220df6ad703798fd8593a0918d06d3ce30c6c93be117e430"}
{"status":"Status: Image is up to date for alpine:latest"}
hello world
How can I check that container process exited with non-zero status using Docker Go SDK?
I think you should use the status channel to get the exit code. The error channel seems to be used to signal if there was an error while talking to the docker daemon, see https://godoc.org/github.com/docker/docker/client#Client.ContainerWait.
This works for me:
select {
case err := <-errCh:
if err != nil {
log.Fatal(err)
}
case status := <-statusCh:
log.Printf("status.StatusCode: %#+v\n", status.StatusCode)
}

How can I get Docker container output through a websocket?

I am trying to send output from a docker container to the console using fmt, but when trying to do it i get this.
&{0xc0422a65c0 {0 0} false <nil> 0x6415a0 0x641540}
How do I do this? This is my full code.
func main() {
imageName := "hidden/hidden"
ctx := context.Background()
cli, err := client.NewClient("tcp://0.0.0.0:0000", "v0.00", nil, nil)
if err != nil {
panic(err)
}
fmt.Println("Pulling \"" + imageName + "\"")
_, err = cli.ImagePull(ctx, imageName, types.ImagePullOptions{})
if err != nil {
panic(err)
}
containerConfig := &container.Config{
Image: imageName,
Cmd: []string{"./app/start.sh", "--no-wizard"},
}
resp, err := cli.ContainerCreate(ctx, containerConfig, nil, nil, "")
if err != nil {
panic(err)
}
if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
panic(err)
}
timer := time.NewTimer(time.Minute)
go func() {
<-timer.C
if err := cli.ContainerStop(ctx, resp.ID, nil); err != nil {
panic(err)
}
}()
out, err := cli.ContainerLogs(ctx, resp.ID, types.ContainerLogsOptions{ShowStdout: true, ShowStderr: true, Follow: true})
if err != nil {
panic(err)
}
io.Copy(os.Stdout, out) // This is what I want to change to print with "fmt".
}
Tried: (but does not display until container is done.)
buf := new(bytes.Buffer)
buf.ReadFrom(out)
fmt.Println(buf.String())
Intention: Allow real-time console output to the web.
This seems to be the answer to my question, I did some searching about scanners as Cerise Limón commented. Anyone else who seems to be having the issue that I did can use this code. Thanks to all that helped.
scanner := bufio.NewScanner(out)
scanner.Split(bufio.ScanLines)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
&{0xc0422a65c0 {0 0} false <nil> 0x6415a0 0x641540} is not a bad output. This is a perfectly fine struct output. I think the the main problem is here just your lack of golang experience.
I'm a beginner as well and I can imagine that when you see an output like above you thought that "I made a mistake"
No you didn't. It's default fmt behavior when u try to print out a struct which contains pointers.
Checkout this instead of your fmt.Println:
fmt.Printf("%+v\n", out)
Well this answer stands on my assumptions but if this is the case just ping me for more info.

Resources