I am trying to use the docker go client to connect to the google container registry to list and delete images. My golang application also exists in a docker container.
This is the dockerfile for my golang application:
FROM docker:latest
USER root
RUN apk add --update openssl
ADD ./data /app/data
ADD ./data/docker /app/data/docker
ADD mygolangapp /app
RUN chmod -R a+rwx ./app/data/docker/generate_docker_cert.sh
RUN sh ./app/data/docker/generate_docker_cert.sh
ENV GOOGLE_APPLICATION_CREDENTIALS ./app/data/myserviceaccount.json
ENV DOCKER_CONFIG ./app/data/docker
ENV DOCKER_CERT_PATH .
ENV DOCKER_HOST ????????
ENTRYPOINT ["/app/mygolangapp"]
This is the generate_docker_cert.sh file
(https://gist.github.com/bradrydzewski/a6090115b3fecfc25280)
This is my golang code to create the docker go client and list containers.
jsonBytes, err := ioutil.ReadFile(os.Getenv("GOOGLE_APPLICATION_CREDENTIALS"))
if err != nil {
panic(err)
}
dockercli, err := client.NewEnvClient()
if err != nil {
panic(err)
}
dockercli.RegistryLogin(context.Background(), types.AuthConfig{
Username: "_json_key",
Password: string(jsonBytes),
ServerAddress: "https://eu.gcr.io",
})
containers, err := dockercli.ContainerList(context.Background(), types.ContainerListOptions{})
if err != nil {
panic(err)
}
for _, container := range containers {
fmt.Printf("%s %s\n", container.ID[:10], container.Image)
}
Currently I am getting this error:
error during connect: Get https://%2Fvar%2Frun%2Fdocker.sock/v1.25/containers/json?limit=0: dial tcp: lookup /var/run/docker.sock: no such host
So I added docker.sock as a volume in my docker-compose, but it is not working?
mygolangapp:
build: ./mygolangapp
volumes:
- /var/run/docker.sock:/var/run/docker.sock
My question is: How can I use the docker golang client within a golang application for google container registry. What am I doing wrong or missing here? What should DOCKER_HOST be?
Thank you for any help. Any other approach is more than welcome!
As a potential alternate, you could explore the google/go-containerregistry library: https://github.com/google/go-containerregistry
It looks like you're trying to implement some sort of garbage collection tool. If so, you could also look here for an example of how to use the library: https://github.com/google/go-containerregistry/pull/300
Related
I am using the go-rod library to do some web automation, this service that I am making will live inside a container, and for local debugging I want to be able to connect to the browser I have running locally. This issue is that --remote-debugging-address flag only works with --headless flag. This is a big issue for me as I need to inspect and look at the browser while developing. I've read that SSH tunneling can be done but I am unable to get it working. I tried all combinations of flags, ports and hosts and all result in some kind of error.
Current setup
Running the chromium instance on my host chromium --remote-debugging-port=9222. Which gets me an address like so DevTools listening on ws://0.0.0.0:9222/devtools/browser/f66524d5-eecb-44c2-a48c-5b14d8e6d998
Running my app via this script
#!/bin/bash
docker build -t rod-test .
docker run --add-host=host.docker.internal:host-gateway --rm rod-test
The docker file
FROM golang:1.16-alpine
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /rod
CMD [ "/rod" ]
The main.go
package main
import (
"fmt"
"github.com/go-rod/rod"
)
func main() {
browser := rod.New().ControlURL("ws://host.docker.internal:9222/devtools/browser/f66524d5-eecb-44c2-a48c-5b14d8e6d998")
if err := browser.Connect(); err != nil {
fmt.Printf("err while connecting: %v", err)
return
}
fmt.Println(
browser.MustPage("https://mdn.dev/").MustEval("() => document.title"),
)
}
If I use --headless --remote-debugging-address=0.0.0.0 it works, but if I remove the headless part it refuses the connection. The only solution seems to be to use SSH tunneling like it is mentioned here. But these keep erroring out for me as all of the answers are very vague as to what is what and what IP should go where
$ ssh -L 172.17.0.1:9222:localhost:9222 -N localhost
ssh: connect to host localhost port 22: Connection refused
OR
$ ssh -L 172.17.0.1:9222:localhost:9222
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface]
[-b bind_address] [-c cipher_spec] [-D [bind_address:]port]
[-E log_file] [-e escape_char] [-F configfile] [-I pkcs11]
[-i identity_file] [-J [user#]host[:port]] [-L address]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-Q query_option] [-R address] [-S ctl_path] [-W host:port]
[-w local_tun[:remote_tun]] destination [command]
What I want to happen is to be able to connect from the container to the debugger running on my host machine. Some caveats that I would like to cover are
It works on other platforms not just linux
It doesn't require complex setup from the user
This will be used by other teammates and it would be nice to have an approachable setup
So anyone that might have a similar usecase I'll save you a couple of days of debugging and trial and error. The solution I came upon is the following
Using this docker image with chrome debugger pointing at 9222 port. You can run this as is or put it inside docker compose as I did. Now if you navigate to this url chrome://inspect/#devices while the above mentioned image is running, you'll be able to access the and view the browser inside the container. (If its running on 9222 then you can even do localhost:9222 and it will show you the available pages)
Now the code to connect to it using go-rod
ips, err := net.LookupIP("browser-devtools")
if err != nil {
return nil, fmt.Errorf("failed to get browser service ip: %w", err)
}
if len(ips) == 0 {
return nil, errors.New("ip list empty")
}
ip := ips[0].String()
fmt.Printf("IP is %q\n", ip)
resp, err := http.Get(fmt.Sprintf("http://%s:9222/json/version/", ip))
if err != nil {
return nil, fmt.Errorf("failed to get devtools info: %w", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("failed read response body: %w", err)
}
println(string(body))
responseMapped := make(map[string]interface{})
if err = json.Unmarshal(body, &responseMapped); err != nil {
return nil, fmt.Errorf("failed to unmarshal response: %w", err)
}
debuggerUrl, ok := responseMapped["webSocketDebuggerUrl"].(string)
if !ok {
return nil, errors.New("no 'webSocketDebuggerUrl' entry in response map")
}
browser := rod.New().ControlURL(debuggerUrl)
if err := browser.Connect(); err != nil {
return nil, fmt.Errorf("failed connect to browser: %w", err)
}
return browser, nil
Explanation:
ips, err := net.LookupIP("browser-devtools")
There is a slight issue that only localhost host can access the debugger or any (allowed) numerical IP, so basically you cannot simply use the service name in docker compose to access the browser as it will be denied. Here we are resolving the actual IP of the service with a lookup
ip := ips[0].String()
resp, err := http.Get(fmt.Sprintf("http://%s:9222/json/version/", ip))
Here we get the string representation of the IP (something like 127.1.0.0) and pass it into the request. The request in itself is important as we don't have the actual browser debugger url that go-rod expects so this get request will return
{
"Browser": "Chrome/72.0.3601.0",
"Protocol-Version": "1.3",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3601.0 Safari/537.36",
"V8-Version": "7.2.233",
"WebKit-Version": "537.36 (#cfede9db1d154de0468cb0538479f34c0755a0f4)",
"webSocketDebuggerUrl": "ws://localhost:9222/devtools/browser/b0b8a4fb-bb17-4359-9533-a8d9f3908bd8"
}
Something along these lines. And the webSocketDebuggerUrl is the actual url we're connecting to
So the setup is as followed:
I have a host machine on which i run a docker container. From this i want to open a connection to a ftp server on the web and download a test file. The programm is written in go and works when executed directly on the host machine without the container.
The minimum example script looks as follows and I use the ftp package github.com/jlaffaye/ftp: (This should work out of the box as is - at least it does for me)
package main
import (
"fmt"
"io/ioutil"
"log"
"strings"
"time"
"github.com/jlaffaye/ftp"
)
func main() {
URL := "ftp://speedtest.tele2.net/1MB.zip"
User := "anonymous"
Password := "anonymous"
urlElements := strings.Split(URL, "/")
dialTarget := fmt.Sprintf("%s:%s", urlElements[2], "21")
fmt.Println("Attempting FTP connection to ", dialTarget)
c, err := ftp.Dial(dialTarget, ftp.DialWithTimeout(5*time.Second))
if err != nil {
log.Fatalf("TCP dial failed: %v", err)
}
defer c.Quit()
err = c.Login(User, Password)
if err != nil {
log.Fatalf("FTP login failed: %v", err)
}
defer c.Logout()
data, err := c.Retr(strings.Join(urlElements[3:], "/"))
if err != nil {
log.Fatalf("Failed to get file: %v", err)
}
_, err = ioutil.ReadAll(data)
if err != nil {
log.Fatalf("Failed to read file: %v", err)
}
fmt.Println("success!")
}
Since this code part will be part of a larger container later on running this on the host network with --network=host in docker options is not possible. (Apart this did not work for some reason -> lead to timeouts or EOF errors on dial)
Solution I considered:
Open an ssh-bridge to the target machine. I tried this using the package github.com/elliotchance/sshtunnel which provides a (imo) comprehensive example of how to use it: See here for reference.
The idea was to basically tunnel from the container through the host directly onto the target. However while the tunnel returned a connection success, the ftp client still failed with timeouts or EOF errors.
For this reason: Is there a good way to do this? If so which direction would be the best to go in? I am somewhat lost on this, especially since since connecting an samba client to a remote machine works without any issue from within the container for some reason.
Thx in advance for any leads - please let me know if i missed to provide any information/details.
For reference the docker file looks like this:
FROM golang:alpine AS BUILDER
# install certificates to enable usage for http
RUN apk update \
&& apk upgrade \
&& apk add --no-cache \
ca-certificates \
&& update-ca-certificates 2>/dev/null || true
# all prerequsites should be done before this point.
WORKDIR /src
COPY . .
RUN go get -d -v
RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=5 go build -o ./out/ftp-test .
FROM scratch
COPY --from=BUILDER /src/out/ftp-test /ftp-test
COPY --from=BUILDER /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# Run the executable
ENTRYPOINT ["/ftp-test"]
I am prototyping a Go application that will ultimately talk to a remote Docker host. To this end I am using Docker's Go client package (docs at https://godoc.org/github.com/docker/docker/client).
The environment is Ubuntu 19.10 in VirtualBox 6.1.4, using Docker 19.03.6, and Go 1.14. All Go packages have been installed with go get in the last 72 hours.
For local testing purposes I am trying to connect to a local Docker host at tcp://0.0.0.0:2375. That is, I am running
sudo dockerd -H tcp://0.0.0.0:2375
With this, commands such as
docker -H tcp://0.0.0.0:2375 ps
and
curl -k -v -i http://0.0.0.0:2375/v1.40/containers/json
both work, and I am able to observe the traffic over port 2375 with Wireshark.
However, attempting to do the same thing through the Go client package fails with
Cannot connect to the Docker daemon at tcp://0.0.0.0:2375. Is the docker daemon running?
and nothing shows up in Wireshark.
Here is the example Go code :
package main
import (
"context"
"fmt"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
cli, err := client.NewClientWithOpts(client.WithHost("tcp://0.0.0.0:2375"), client.WithAPIVersionNegotiation())
if err != nil {
fmt.Println("error: could not create docker client handle")
fmt.Println(err)
}
options := types.ContainerListOptions{}
data, err := cli.ContainerList(context.Background(), options)
if err != nil {
fmt.Println("error: could not request containers list")
fmt.Println(err)
} else {
fmt.Println(data)
}
}
Attempts to set the environment variables DOCKER_HOST=tcp://0.0.0.0:2375, DOCKER_CERT_PATH=, and DOCKER_TLS_VERIFY=, then configure the client handle through client.FromEnv() also failed in the exact same way.
What am I doing wrong here ?
I have a very simple application. Here is the code:
package main
import (
"fmt"
"math/rand"
"time"
"net/http"
"encoding/base64"
"encoding/json"
)
type Message struct {
Text string `json:"text"`
}
var cookieQuotes = []string{
// Skipped all the stuff
}
const COOKIE_NAME = "your_cookie"
func main() {
http.HandleFunc("/set_cookie", setCookie)
http.HandleFunc("/get_cookie", getCookie)
http.Handle("/favicon.ico", http.NotFoundHandler())
http.ListenAndServe(":8080", nil)
}
func setCookie(w http.ResponseWriter, r *http.Request) {
quote := getRandomCookieQuote()
encQuote := base64.StdEncoding.EncodeToString([]byte(quote))
http.SetCookie(w, &http.Cookie{
Name: COOKIE_NAME,
Value: encQuote,
})
}
func getCookie(w http.ResponseWriter, r *http.Request) {
cookie, err := r.Cookie(COOKIE_NAME)
if err != nil {
fmt.Fprintln(w, "Cannot get the cookie")
}
message, _ := base64.StdEncoding.DecodeString(cookie.Value)
msg := Message{Text:string(message)}
fmt.Println(msg.Text)
respBody, err := json.Marshal(msg)
fmt.Println(string(respBody))
if err != nil {
fmt.Println("Cannot marshall JSON")
}
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, string(respBody))
}
func getRandomCookieQuote() string {
source := rand.NewSource(time.Now().UnixNano())
random := rand.New(source)
i := random.Intn(len(cookieQuotes))
return cookieQuotes[i]
}
It was tested locally, and, also I've tried to run a docker container with it on my machine (Ubuntu) and it was working perfectly. But I want to run it on Virtual Machine (I use Oracle Virtual Box).
So, I have installed docker-machine:
docker-machine version 0.12.2, build 9371605
After that, I've switched to it, like it was recommended in official documentation like this:
eval "$(docker-machine env default)"
So I can do now from a perspective of that machine.
Also I've tried to run ngnix from the documentation example:
docker run -d -p 8000:80 nginx
curl $(docker-machine ip default):8000
And I get the result, I can get to ngnix welcome page by accessing my docker machine ip-address which could be accessed by command:
docker-machine ip default
But when I try to run my own docker image, I could not do this. When I try to access it, I get:
curl $(docker-machine ip default):8080
curl: (7) Failed to connect to 192.168.99.100 port 8080: Connection refused
Also I've tried to skip a port, to add protocol (http, and even https for the sake of luck) - nothing works.
Maybe, something wrong with my Dockerfile?
# Go experiments with cookies
FROM golang:1.8-onbuild
MAINTAINER vasyania2#gmail.com
Could you help me please?
This command maps port 8080 from your docker host to port 80 of your container:
docker run -d -p 8080:80 cookie-app
This instruction tells your go application to listen on port 8080, inside the container:
http.ListenAndServe(":8080", nil)
You have a port mismatch in those above lines, your application is not listening on the port you are forwarding to.
To connect to port 8080 of your container, you can run the following:
docker run -d -p 8080:8080 cookie-app
I need to use golang to access kafka,so i installed a kafka & zookepper in docker.
1.here is kafka install script:
# pull images
docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka
# run kafka & zookepper
docker run -d --name zookeeper -p 2181 -t wurstmeister/zookeeper
docker run --name kafka -e HOST_IP=localhost -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_BROKER_ID=1 -e ZK=zk -p 9092:9092 --link zookeeper:zk -t wurstmeister/kafka
# enter container
docker exec -it ${CONTAINER ID} /bin/bash
cd opt/kafka_2.11-0.10.1.1/
# make a tpoic
bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic mykafka
# start a producer in terminal-1
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mykafka
# start another terminal-2 and start a consumer
bin/kafka-console-consumer.sh --zookeeper zookeeper:2181 --topic mykafka --from-beginning
when i type some message in producer, the consumer will get it immediately.
so i assumed that the kafka is working fine
2.Now i need to create a consumer with golang to access kafka.
here is my golang demo code:
import "github.com/bsm/sarama-cluster"
func Consumer(){
// init (custom) config, enable errors and notifications
config := cluster.NewConfig()
config.Consumer.Return.Errors = true
config.Group.Return.Notifications = true
// init consumer
brokers := []string{"192.168.9.100:9092"}
topics := []string{"mykafka"}
consumer, err := cluster.NewConsumer(brokers, "my-group-id", topics, config)
if err != nil {
panic(err)
}
defer consumer.Close()
// trap SIGINT to trigger a shutdown.
signals := make(chan os.Signal, 1)
signal.Notify(signals, os.Interrupt)
// consume messages, watch errors and notifications
for {
select {
case msg, more := <-consumer.Messages():
if more {
fmt.Fprintf(os.Stdout, "%s/%d/%d\t%s\t%s\n", msg.Topic, msg.Partition, msg.Offset, msg.Key, msg.Value)
consumer.MarkOffset(msg, "") // mark message as processed
}
case err, more := <-consumer.Errors():
if more {
log.Printf("Error: %s\n", err.Error())
}
case ntf, more := <-consumer.Notifications():
if more {
log.Printf("Rebalanced: %+v\n", ntf)
}
case <-signals:
return
}
}
}
actually this demo code is copied from a github repo's demo:sarama-cluster
When running the code, i got an error:
kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
i did use a port map when start kafka,but just can't access it in golang
is there a way to use curl to access kafka?
i'v tried:
curl http://192.168.99.10:9092
and kafka report an error:
[2017-08-02 06:39:15,232] WARN Unexpected error from /192.168.99.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:95)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:75)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:203)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:167)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:379)
at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
at kafka.network.Processor.poll(SocketServer.scala:499)
at kafka.network.Processor.run(SocketServer.scala:435)
at java.lang.Thread.run(Thread.java:748)
BTW:
i use windows 7
dcoker machine's ip :192.168.99.100
it's drived me crazy
Is there some advice or solution? appreciate!!!
If you want to create a consumer to listen a topic from Kafka, let's try that way.
I used confluent-kafka-go from the tutorial: https://github.com/confluentinc/confluent-kafka-go
This is the code on main.go file:
import (
"fmt"
"gopkg.in/confluentinc/confluent-kafka-go.v1/kafka"
)
func main() {
c, err := kafka.NewConsumer(&kafka.ConfigMap{
"bootstrap.servers": "localhost",
"group.id": "myGroup",
"auto.offset.reset": "earliest",
})
if err != nil {
panic(err)
}
c.SubscribeTopics([]string{"myTopic", "^aRegex.*[Tt]opic"}, nil)
for {
msg, err := c.ReadMessage(-1)
if err == nil {
fmt.Printf("Message on %s: %s\n", msg.TopicPartition, string(msg.Value))
} else {
// The client will automatically try to recover from all errors.
fmt.Printf("Consumer error: %v (%v)\n", err, msg)
}
}
c.Close()
}
If you use docker to build: follow this comment to add suitable packages
For Debian and Ubuntu based distros, install librdkafka-dev from the standard repositories or using Confluent's Deb repository.
For Redhat based distros, install librdkafka-devel using Confluent's YUM repository.
For MacOS X, install librdkafka from Homebrew. You may also need to brew install pkg-config if you don't already have it. brew install librdkafka pkg-config.
For Alpine: apk add librdkafka-dev pkgconf
confluent-kafka-go is not supported on Windows.
With Alpine, please remember that install the community version, because it cannot install librdkafka with max version 1.1.0 (not use Alpine community version)
Good luck!
Not sure, if it is possible to use with curl with kafka. But you can use the kafka-console-consumer.
kafka-console-consumer.bat --bootstrap-server 192.168.9.100:9092 --topic mykafka --from-beginning
I'v found the reason.
because the kafka's settings is not correct
this is server.properties:
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
if the listeners is not set , kafka will only receive request from java.net.InetAddress.getCanonicalHostName() which means localhost
so i shuld set :
listeners = PLAINTEXT://0.0.0.0:9092
this will work