I have the following date string: 2017-09-04T04:00:00Z
I need to parse this string into a golang time in order to have uniform data across my application. Here is the code so far:
parsedTime := "2017-09-04T04:00:00Z"
test, err := time.Parse(time.RFC3339, parsedTime)
check(err)
fmt.Println(test)
I get the following error when I try to run the program:
": extra text: 0:00 +0000 UTC parsing time "2017-09-04T04:00:00Z
How can I either add the extra text that it is looking for or get the parser to stop looking after the Z?
I have also tried the following:
parsedTime := "2017-09-04T04:00:00Z"
test, err := time.Parse("2006-01-02T03:04:05Z", parsedTime)
check(err)
fmt.Println(test)
Which returns the following error:
": extra text: 017-09-04T04:00:00Z
Both formats you used work with the current version of go: https://play.golang.org/p/Typyq3Okrd
var formats = []string{
time.RFC3339,
"2006-01-02T03:04:05Z",
}
func main() {
parsedTime := "2017-09-04T04:00:00Z"
for _, format := range formats {
if test, err := time.Parse(format, parsedTime); err != nil {
fmt.Printf("ERROR: format %q resulted in error: %v\n", format, err)
} else {
fmt.Printf("format %q yielded %s\n", format, test)
}
}
}
Can you provide a working example that demonstrates your problem? You can use the go playground for shareable snippets.
Related
I'm looking for help with extracting the image build logs from a dockerd (buildkit/moby) image build request sent by a Golang based client using the docker client libraries.
I can request the image build fine and receive the log stream of json messages then decode them as Jsonmessage instances. But the actual log lines from the builder appear to be base64 encoded in an aux field of each json message.
I can decode the base64 easily enough, but they seem to include odd terminal control characters and possibly mis-encoded data, which makes me wonder if they're actually a base64 encoding of some kind of struct I'm supposed to unpack.
What confuses me is that I can't find anything in the docker-ce or moby code that seems to base64-decode an 'aux' payload when processing logs when displaying build progress for docker buildx build.
As far as I can tell, the buildx code doesn't do anything special to the aux payload: https://github.com/docker/docker-ce/blob/523cf7e71252013fbb6a590be67a54b4a88c1dae/components/cli/cli/command/image/build_buildkit.go#L325
For example, trimmed-down build code like:
image := Image{Name: "test"}
contextreader, err := archive.TarWithOptions(buildConf.Build.Context, &archive.TarOptions{})
if err != nil {
return err
}
imageBuildResponse, err := b.client.ImageBuild(
ctx,
contextreader,
types.ImageBuildOptions{
Version: types.BuilderBuildKit,
Context: contextreader,
Dockerfile: dockerfile,
})
if err != nil {
return err
}
defer imageBuildResponse.Body.Close()
buf := bytes.NewBuffer(nil)
imageID := ""
writeAux := func(msg jsonmessage.JSONMessage) {
if msg.ID == "moby.image.id" {
var result types.BuildResult
if err := json.Unmarshal(*msg.Aux, &result); err != nil {
panic("don't do this in your real code")
}
imageID = result.ID
return
}
return err
}
err := jsonmessage.DisplayJSONMessagesStream(imageBuildResponse.Body, buf, os.Stderr.Fd(), false /* not terminal */, writeAux)
if err != nil {
if jerr, ok := err.(*jsonmessage.JSONError); ok {
// If no error code is set, default to 1
if jerr.Code == 0 {
jerr.Code = 1
}
return fmt.Errorf("error while building image: %s", jerr.Message)
}
}
will write json payloads to stderr like
{"id":"moby.buildkit.trace","aux":"Cn0KR3NoYTI1NjozZThhMzMxYmRkZGFjNWZkYmNjOGVhMDFmYWFhYmM3MjA0MDkwMmYwNjdmYzRhOGY0NDJmMmIzYWVlN2RkNGIyGiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQqDAiYw8KaBhCykpCqAg=="}
{"id":"moby.buildkit.trace","aux":"CokBCkdzaGEyNTY6M2U4YTMzMWJkZGRhYzVmZGJjYzhlYTAxZmFhYWJjNzIwNDA5MDJmMDY3ZmM0YThmNDQyZjJiM2FlZTdkZDRiMhokW2ludGVybmFsXSBsb2FkIHJlbW90ZSBidWlsZCBjb250ZXh0KgwImMPCmgYQspKQqgIyCgiZw8KaBhD08F0="}
The base64 strings here don't decode as valid utf-8, and they don't make sense as ISO-8859-1 either. E.g. with a utf-8 console encoding:
$ base64 -d <<<'Cn0KR3NoYTI1NjozZThhMzMxYmRkZGFjNWZkYmNjOGVhMDFmYWFhYmM3MjA0MDkwMmYwNjdmYzRhOGY0NDJmMmIzYWVlN2RkNGIyGiRbaW50ZXJuYWxdIGxvYWQgcmVtb3RlIGJ1aWxkIGNvbnRleHQqDAiYw8KaBhCykpCqAg=='
}
Gsha256:3e8a331bdddac5fdbcc8ea01faaabc72040902f067fc4a8f442f2b3aee7dd4b2�$[internal] load remote build context*
������
It looks like it's probably a struct, but for the life of me I can't find what decodes and processes it.
So of course I find the answer while writing up the SO question...
The writeAux function in build_buildkit.go calls the write method of a tracer instance, and that does the real work. I must've been blind.
The messages are serialized instances of StatusResponse from the github.com/moby/buildkit/api/services/control package. They are unmarshalled from base64-decoded byte sequences and inspected. If you want logs and to skip everything else, just look for instances with non-empty Logs member arrays, e.g. something like this within the above writeAux function:
} else if msg.ID == "moby.buildkit.trace" {
// Process the message like
// https://github.com/docker/docker-ce/blob/523cf7e71252013fbb6a590be67a54b4a88c1dae/components/cli/cli/command/image/build_buildkit.go#L386
// the 'tracer.write' method in build_buildkit.go
var resp controlapi.StatusResponse
var dt []byte
// ignoring all messages that are not understood
if err := json.Unmarshal(*msg.Aux, &dt); err != nil {
return
}
if err := (&resp).Unmarshal(dt); err != nil {
return
}
for _, v := range resp.Vertexes {
fmt.Printf("layer: %+v", v)
}
for _, v := range resp.Statuses {
fmt.Printf("status: %+v", v)
}
for _, v := range resp.Logs {
fmt.Printf("log: msg.Msg)
}
}
The json.Unmarshal and controlapi.StatusResponse.Unmarshal do the base64 decoding and unpacking for you.
In short, I'm writing a program to collect some kubelets metrics remotely.
But these metrics doesn't contain information about node_name so it will duplicate when prometheus scraps it
So I want to parse their metrics, add the node name label to them, then re-parse them into prometheus metrics so I can host as an endpoint to let prometheus scraps from it
But I met problems in parsing metrics
package main
import (
"flag"
"fmt"
"log"
"os"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
)
func fatal(err error) {
if err != nil {
log.Fatalln(err)
}
}
func parseMF(path string) (map[string]*dto.MetricFamily, error) {
reader, err := os.Open(path)
if err != nil {
return nil, err
}
var parser expfmt.TextParser
mf, err := parser.TextToMetricFamilies(reader)
if err != nil {
return nil, err
}
return mf, nil
}
func main() {
f := flag.String("f", "", "set filepath")
flag.Parse()
mf, err := parseMF(*f)
fatal(err)
for k, v := range mf {
fmt.Println("KEY: ", k)
fmt.Println("VAL: ", v)
}
}
With this simple metrics like this:
# HELP net_conntrack_dialer_conn_attempted_total
# TYPE net_conntrack_dialer_conn_attempted_total untyped
net_conntrack_dialer_conn_attempted_total{dialer_name="federate",instance="localhost:9090",job="prometheus"} 1 1608520832877
the result:
name:"net_conntrack_dialer_conn_attempted_total" type:UNTYPED metric:<label:<name:"dialer_name" value:"federate" > label:<name:"instance" value:"localhost:9090" > label:<name:"job" value:"prometheus" > untyped:<value:1 > timestamp_ms:1608520832877 >
I just want to take out the name,labels,value and timestamp but I can't take it out to process, unlike python
from prometheus_client.parser import text_string_to_metric_families
import requests
metrics = """
# HELP net_conntrack_dialer_conn_attempted_total
# TYPE net_conntrack_dialer_conn_attempted_total untyped
net_conntrack_dialer_conn_attempted_total{dialer_name="federate",instance="localhost:9090",job="prometheus"} 1 1608520832877
"""
for family in text_string_to_metric_families(metrics):
for sample in family.samples:
print("{0}\n{1}\n{2}\n{4}".format(*sample))
Python result:
net_conntrack_dialer_conn_attempted_total
{'dialer_name': 'federate', 'instance': 'localhost:9090', 'job': 'prometheus'}
1.0
1608520832.877
So back to the main question: How can I take out those specific value and modify it?
Using a URL that has worked in the past, I know receive a parsing error from net/url. What's wrong with it?
parse postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require: net/url: invalid userinfo
Sample application
See https://play.golang.com/p/mQZaN5JN3_q to run.
package main
import (
"fmt"
"net/url"
)
func main() {
dsn := "postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require"
u, err := url.Parse(dsn)
fmt.Println(u, err)
}
Well, you can just
url.QueryEscape("your#$%^&*(proper$#$%%^(password")
and use this one to parse your url.
It turns out up until Go v1.9.3 net/url didn't validate the user info when parsing a url. This may break existing applications when compiled using v1.9.4 if the username or password contain special characters.
It now expects the user info to be percent encoded string in order to handle special characters. The new behaviour got introduced in ba1018b.
Fixed sample application
package main
import (
"fmt"
"net/url"
)
func main() {
dsn1 := "postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require" // this works up until 1.9.3 but no longer in 1.9.4
dsn2 := "postgres://user:abc%7BDEf1=ghi#example.com:5432/db?sslmode=require" // this works everywhere, note { is now %7B
u, err := url.Parse(dsn1)
fmt.Println("1st url:\t", u, err)
u, err = url.Parse(dsn2)
fmt.Println("2nd url:\t", u, err)
}
Run the code on https://play.golang.com/p/jGIQgbiKZwz.
Use url.UserPassword func :
package main
import (
"fmt"
"net/url"
)
func main() {
dsn := "postgres://example.com:5432/db?sslmode=require"
u, err := url.Parse(dsn)
if err != nil {
fmt.Printf("ERROR: %v\n", err)
return
}
u.User = url.UserPassword("user", "abc{DEf1=ghi")
fmt.Println("url:\t", u)
}
I'm trying to compress a JPEG image in go using mozjpeg. Since it doesn't have official go binding, I think I'll just invoke its CLI to do the compression.
I try to model the usage after compress/gzip:
c := jpeg.NewCompresser(destFile)
_, err := io.Copy(c, srcFile)
Now the question is, how do I wrap the CLI inside Compresser so it can support this usage?
I tried something like this:
type Compresser struct {
cmd exec.Command
}
func NewCompressor(w io.Writer) *Compresser {
cmd := exec.Command("jpegtran", "-copy", "none")
cmd.Stdout = w
c := &Compresser{cmd}
return c
}
func (c *Compresser) Write(p []byte) (n int, err error) {
if c.cmd.Process == nil {
err = c.cmd.Start()
if err != nil {
return
}
}
// How do I write p into c.cmd.Stdin?
}
But couldn't finish it.
Also, a second question is, when do I shut down the command? How to shut down the command?
You should take a look at the Cmd.StdinPipe. There is an example in the documentation, which suits your case:
package main
import (
"fmt"
"io"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("cat")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
io.WriteString(stdin, "values written to stdin are passed to cmd's standard input")
}()
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
}
In this case, CombinedOutput() executes your command, and the execution is finished, when there are no more bytes to read from out.
As per Kiril's answer, use the cmd.StdInPipe to pass on the data you receive to Write.
However, in terms of closing, I'd be tempted to implement io.Closer. This would make *Compresser automatically implement the io.WriteCloser interface.
I would use Close() as the notification that there is no more data to be sent and that the command should be terminated. Any non-zero exit code returned from the command that indicates failure could be caught and returned as an error.
I would be wary of using CombinedOutput() inside Write() in case you have a slow input stream. The utility could finish processing the input stream and be waiting for more data. This would be incorrectly detected as command completion and would result in an invalid output.
Remember, the Write method can be called an indeterminate number of times during IO operations.
I'm developing a Go script that uses the Docker API for the purposes of my project. After I login to my repository, I pull the Docker image I want, but the problem is that the ImagePull function returns an instance of io.ReadCloser, which I'm only able to pass to the system output via:
io.Copy(os.Stdout, pullResp)
It's cool that I can see the response, but I can't find a decent way to parse it and implement a logic depending on it, which will do some things if a new version of the image have been downloaded, and other things if the image was up to date.
I'll be glad if you share your experience, if you have ever faced this problem.
You can import github.com/docker/docker/pkg/jsonmessage and use both JSONMessage and JSONProgress to decode the stream but it's easier to call
DisplayJSONMessagesToStream: it both parses the stream and displays the messages as text. Here's how you can display the messages using stderr:
reader, err := cli.ImagePull(ctx, myImageRef, types.ImagePullOptions{})
if err != nil {
return err
}
defer reader.Close()
termFd, isTerm := term.GetFdInfo(os.Stderr)
jsonmessage.DisplayJSONMessagesStream(reader, os.Stderr, termFd, isTerm, nil)
The nice thing is that it adapts to the output: it updates the lines if this a TTY (the way docker pull does) but it doesn't if the output is redirected to a file.
#radoslav-stoyanov before use my example do
# docker rmi busybox
then run code
package main
import (
"encoding/json"
"fmt"
"github.com/docker/distribution/context"
docker "github.com/docker/engine-api/client"
"github.com/docker/engine-api/types"
"io"
"strings"
)
func main() {
// DOCKER
cli, err := docker.NewClient("unix:///var/run/docker.sock", "v1.28", nil, map[string]string{"User-Agent": "engine-api-cli-1.0"})
if err != nil {
panic(err)
}
imageName := "busybox:latest"
events, err := cli.ImagePull(context.Background(), imageName, types.ImagePullOptions{})
if err != nil {
panic(err)
}
d := json.NewDecoder(events)
type Event struct {
Status string `json:"status"`
Error string `json:"error"`
Progress string `json:"progress"`
ProgressDetail struct {
Current int `json:"current"`
Total int `json:"total"`
} `json:"progressDetail"`
}
var event *Event
for {
if err := d.Decode(&event); err != nil {
if err == io.EOF {
break
}
panic(err)
}
fmt.Printf("EVENT: %+v\n", event)
}
// Latest event for new image
// EVENT: {Status:Status: Downloaded newer image for busybox:latest Error: Progress:[==================================================>] 699.2kB/699.2kB ProgressDetail:{Current:699243 Total:699243}}
// Latest event for up-to-date image
// EVENT: {Status:Status: Image is up to date for busybox:latest Error: Progress: ProgressDetail:{Current:0 Total:0}}
if event != nil {
if strings.Contains(event.Status, fmt.Sprintf("Downloaded newer image for %s", imageName)) {
// new
fmt.Println("new")
}
if strings.Contains(event.Status, fmt.Sprintf("Image is up to date for %s", imageName)) {
// up-to-date
fmt.Println("up-to-date")
}
}
}
You can see API formats to create your structures (like my Event) to read them here https://docs.docker.com/engine/api/v1.27/#operation/ImageCreate
I hope it helps you solve your problem, thanks.
I have used similar approach for my purpose (not a moby client). Typically idea is same for reading stream response. Give it a try and implement yours.
Reading stream response of any response type:
reader := bufio.NewReader(pullResp)
defer pullResp.Close() // pullResp is io.ReadCloser
var resp bytes.Buffer
for {
line, err := reader.ReadBytes('\n')
if err != nil {
// it could be EOF or read error
// handle it
break
}
resp.Write(line)
resp.WriteByte('\n')
}
// print it
fmt.Println(resp.String())
However your sample response in the comment seems valid JSON structure. The json.Decoder is best way to read JSON stream. This is just an idea-
type ImagePullResponse struct {
ID string `json"id"`
Status string `json:"status"`
ProgressDetail struct {
Current int64 `json:"current"`
Total int64 `json:"total"`
} `json:"progressDetail"`
Progress string `json:"progress"`
}
And do
d := json.NewDecoder(pullResp)
for {
var pullResult ImagePullResponse
if err := d.Decode(&pullResult); err != nil {
// handle the error
break
}
fmt.Println(pullResult)
}