I'm using Go and hitting Docker's API to pull an image, and then create and execute a container. Specifically I'm using the docker-newman image. I can see that it's actually being executed in Kitematic, so I know that everything is setup correctly. However, my Go application isn't attaching and then tailing the logs being output.
I've seen this answer, and it references what appears to be the way to attach to the image and view the log. I can't seem to get it to work regardless of what I try.
attachToContainerOptions := docker.AttachToContainerOptions{
Container: container.ID,
OutputStream: os.Stdout,
ErrorStream: os.Stderr,
Logs: true,
Stdout: true,
Stderr: true,
}
if err := client.AttachToContainer(attachToContainerOptions); err != nil {
panic(err)
}
No error occurs, but this immediatley gets passed over without streaming anything to the console. How do I get this to stream to the console until the docker cmd completes?
Attach only works on a running container and the container lifetime for that image is ephemeral. Try *Client.Logs, instead, to get the resulting logs.
Here is a code sample:
logsOptions := docker.LogsOptions{
Container: container.ID,
OutputStream: os.Stdout,
ErrorStream: os.Stderr,
Follow: true,
Stdout: true,
Stderr: true,
}
if err := client.Logs(logsOptions); err != nil {
panic(err)
}
Related
I have a small Golang program and I'm trying to connect to an FTP server running in a docker container (https://registry.hub.docker.com/r/atmoz/sftp).
My machine is a M1 Pro MacBook.
The container is started with the following command:
docker run -p 22:22 -d atmoz/sftp foo:pass:::upload
The Go version is 1.17.13.
The code code of the program is the following:
package main
import (
"log"
"time"
"github.com/jlaffaye/ftp"
)
func main() {
c, err := ftp.Dial("localhost:22", ftp.DialWithTimeout(5*time.Second))
if err != nil {
log.Fatal(err, " cannot connect")
}
err = c.Login("foo", "pass")
if err != nil {
log.Fatal(err, "cannot login")
}
// Do something with the FTP conn
if err := c.Quit(); err != nil {
log.Fatal(err)
}
}
Somehow, I'm unable to connect to the FTP server executing this code, it results in the following output:
EOF cannot connect
I tried connect to the same FTP server using FileZilla and it works fine, im able to connect to the server with success.
Any ideias on how to fix this or further debug the issue? Thank you
The port 22 is typically SSH/SFTP, not FTP. Note that FileZilla supports both FTP and SFTP. So chances are that you are actually connecting with SFTP using FileZilla. Those two protocols are completely different and incompatible.
There seems to be an "sftp" package for Go:
https://pkg.go.dev/github.com/pkg/sftp
I'm trying to access a docker registry (public or private) using Go. A simple program which can access any registry and verify if an image is present.
I looked at docker client available in Go https://pkg.go.dev/github.com/docker/docker#v20.10.11+incompatible/client
But the problem is, this client needs a docker daemon running in order to work. Is there any way to query a docker registry (ex: hub.docker.com) without any dependency on underlying docker engine?
My idea is to run this program on a docker container and there wont be any docker engine running inside a container. And I don't want to run docker inside docker or any sort of hack. I just want to connect to a registry and query an image. And please don't quote other questions in stack overflow. No one has answered this.
This is what I have done so far
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"github.com/docker/docker/api/types/filters"
"time"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
cli, err := client.NewClientWithOpts(client.WithHost("https://hub.docker.com"), client.WithAPIVersionNegotiation())
if err != nil {
fmt.Println(err.Error())
return
}
err = imagemanifest(cli)
if err != nil {
fmt.Println(err)
}
err = imageSearch(cli)
}
func imagemanifest(dockerClient *client.Client) error {
var authConfig = types.AuthConfig{
Username: "amokkara",
Password: "M#vr1ck2009",
ServerAddress: "https://index.docker.io/v2/",
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*1200)
defer cancel()
authConfigBytes, _ := json.Marshal(authConfig)
authConfigEncoded := base64.URLEncoding.EncodeToString(authConfigBytes)
ctx, cancel = context.WithTimeout(context.Background(), time.Second*1200)
defer cancel()
searchres , err := dockerClient.DistributionInspect(ctx,"amokkara/amokkara:3",authConfigEncoded)
if err != nil {
return err
}
fmt.Println(searchres.Descriptor.Digest.String())
return nil
}
If I initialize client like this
cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
This works because its using underlying docker daemon (in my case docker desktop) to query the registry. But if create client using
client.NewClientWithOpts(client.WithHost("https://hub.docker.com"), client.WithAPIVersionNegotiation())
it fails giving 404 error. Does this client require a docker daemon to work. If so, is there any other way i can query a registry? Please help me with this.
Skopeo is the leader software on dealing with registries without daemon.
It's also written with Go.
You can inspire from inspect.go
Note that, you don't need to use github.com/docker/docker/* modules, but it will be github.com/containers/*, and namely https://github.com/containers/image
I've been stuck on this for awhile. I would like docker to ignore a particular directory when building an image, because my user account does not have permissions to read that directory. I cannot move it, so that's not an alternative.
This is the structure of my project. docker/data is the directory that I do not have permissions to read, and docker/node-express.dockerfile is the image I'm trying to build.
Running docker build --no-cache --tag node-express --file ./docker/node-express.dockerfile . in the root directory outputs the error
error checking context: 'can't stat '/home/anthony/Repositories/Anthony-Monterrosa/aws-postgres-node-stack/docker/data''.
After this error and a bit of googling I learned about .dockerignore files, and made one in the root directory. The following is the file's text.
docker/data
I ran the command again but got an identical error. A bit more googling and I found out about image-specific .dockerignore files, so I set DOCKER_BUILDKIT to 1, created docker/node-express.dockerfile.dockerignore with the following content
data
docker/data
(I am not sure how relative paths work with image-specific .dockerignores, so I added both). Ran the command again, but still with the same error.
So, I don't seem to have ignores working correctly with either .dockerignore file, or both. What am I missing here?
The error is:
error checking context: 'can't stat '/home/anthony/Repositories/Anthony-Monterrosa/aws-postgres-node-stack/docker/data''.
So looks there is some operation before .dockerignore effect.
As there is no context content in your docker folder, I suggest you just add docker in .dockerignore.
This way, although still error, but the build will continue like next:
shubuntu1#shubuntu1:~/trial2020/trial$ docker build -t abcd:1 -f docker/Dockerfile .
ERRO[0000] Tar: Can't stat file /home/shubuntu1/trial2020/trial to tar: open
/home/shubuntu1/trial2020/trial/docker/data: permission denied
Sending build context to Docker daemon 3.072kB
Step 1/1 : FROM ubuntu:18.04
---> 3556258649b2
Successfully built 3556258649b2
Successfully tagged abcd:1
UPDATE why according to your comments:
You may want to have a look for docker-ce source code, build.go & context.go:
build.go:
if err := build.ValidateContextDirectory(contextDir, excludes); err != nil {
return errors.Errorf("error checking context: '%s'.", err)
}
context.go:
func ValidateContextDirectory(srcPath string, excludes []string) error {
contextRoot, err := getContextRoot(srcPath)
if err != nil {
return err
}
pm, err := fileutils.NewPatternMatcher(excludes)
if err != nil {
return err
}
return filepath.Walk(contextRoot, func(filePath string, f os.FileInfo, err error) error {
if err != nil {
if os.IsPermission(err) {
return errors.Errorf("can't stat '%s'", filePath)
}
if os.IsNotExist(err) {
return errors.Errorf("file ('%s') not found or excluded by .dockerignore", filePath)
}
return err
}
// skip this directory/file if it's not in the path, it won't get added to the context
if relFilePath, err := filepath.Rel(contextRoot, filePath); err != nil {
return err
} else if skip, err := filepathMatches(pm, relFilePath); err != nil {
return err
} else if skip {
if f.IsDir() {
return filepath.SkipDir
}
return nil
}
......
})
}
Before docker daemon tar the build context, it will first try to validate the context directory:
docker/data in .dockerignore:
It will use Walk to ergodic all things under docker, when it comes to docker/data, next code finally make the build exit, so you did not get image generated:
if os.IsPermission(err) {
return errors.Errorf("can't stat '%s'", filePath)
}
docker in .dockerignore:
Same as above, difference is next code will effect when comes to the match docker in .dockerignore:
return filepath.SkipDir
This will make the Walk ignore the subfolders of docker, then docker/data no chance to be ergodic, so no permission error there.
The ERRO[0000] Tar: Can't stat file comes from other later steps which won't exit the image build.
I have a golang script that calls an HTTP API. Everything works great when I run it on my development computer. Once I create the container and run it I get an EOF at the end of the URL on panic. I have read a lot of issues like this and have tried everyone I see. I added a ca-certificates.crt from my host machine to the container to /etc/ssl/certs/. I set the request close to true, I have disabled Keep-Alive and Compression. All of these steps worked for others on other posts. Any help appreciated.
I got my certfile from https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
I am running go version go1.9.2 darwin/amd64 and the container is on a ubuntu 16.04 host.
client := &http.Client{Timeout: 30 * time.Second, Transport: &http.Transport{
DisableCompression:true,
DisableKeepAlives: true,
}}
url := "https://myapiurl.com"
req, err := http.NewRequest("POST", url, bytes.NewBuffer(mybodybytes))
req.Close = true
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
log.Panic(err) // Panics Here
}
defer resp.Body.Close()
...
Dockerfile:
FROM scratch
ADD ca-certificates.crt /etc/ssl/certs/
ADD myservice /
CMD ["/myservice"]
Using the docker golang sdk the following method can be used to create a container and bind it's output to stdout.
resp, err := cli.ContainerCreate(ctx, &container.Config{
Image: "alpine",
Cmd: []string{"echo", "Hello World"},
AttachStdout: true,
}, nil, nil, "")
How can I redirect this output to a file using the SDK ?
I'm using the official SDK of docker - github.com/docker/docker/client
You can use something like below
out, err := cli.ContainerLogs(ctx, resp.ID, types.ContainerLogsOptions{ShowStdout: true})
if err != nil {
panic(err)
}
f, err := os.Create("/tmp/clogs")
io.Copy(f, out)
But make sure to to do that after you have started the container, Create will only create the container and not start it
The format of docker logs contains 8 bytes of header for each message, indicating for instance whether the output was on stdout or stderr. So one cannot simply copy the log output to a destination as Tarun Lalwani is mentioning in the other answer, because the header would then be interpreted as characters, garbling the output.
Unfortunately the client docs don't even mention the issue. This article explains it a bit and offers a library to solve the issue:
import (
"github.com/docker/docker/client"
"github.com/ahmetb/dlog"
)
// ---
reader, err := cli.ContainerLogs(ctx, resp.ID, nil)
if err != nil {
panic(err)
}
file, err := os.Create("/path/to/your/file")
io.Copy(file, dlog.NewReader(reader))