I am prototyping a Go application that will ultimately talk to a remote Docker host. To this end I am using Docker's Go client package (docs at https://godoc.org/github.com/docker/docker/client).
The environment is Ubuntu 19.10 in VirtualBox 6.1.4, using Docker 19.03.6, and Go 1.14. All Go packages have been installed with go get in the last 72 hours.
For local testing purposes I am trying to connect to a local Docker host at tcp://0.0.0.0:2375. That is, I am running
sudo dockerd -H tcp://0.0.0.0:2375
With this, commands such as
docker -H tcp://0.0.0.0:2375 ps
and
curl -k -v -i http://0.0.0.0:2375/v1.40/containers/json
both work, and I am able to observe the traffic over port 2375 with Wireshark.
However, attempting to do the same thing through the Go client package fails with
Cannot connect to the Docker daemon at tcp://0.0.0.0:2375. Is the docker daemon running?
and nothing shows up in Wireshark.
Here is the example Go code :
package main
import (
"context"
"fmt"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
cli, err := client.NewClientWithOpts(client.WithHost("tcp://0.0.0.0:2375"), client.WithAPIVersionNegotiation())
if err != nil {
fmt.Println("error: could not create docker client handle")
fmt.Println(err)
}
options := types.ContainerListOptions{}
data, err := cli.ContainerList(context.Background(), options)
if err != nil {
fmt.Println("error: could not request containers list")
fmt.Println(err)
} else {
fmt.Println(data)
}
}
Attempts to set the environment variables DOCKER_HOST=tcp://0.0.0.0:2375, DOCKER_CERT_PATH=, and DOCKER_TLS_VERIFY=, then configure the client handle through client.FromEnv() also failed in the exact same way.
What am I doing wrong here ?
Related
I am using the go-rod library to do some web automation, this service that I am making will live inside a container, and for local debugging I want to be able to connect to the browser I have running locally. This issue is that --remote-debugging-address flag only works with --headless flag. This is a big issue for me as I need to inspect and look at the browser while developing. I've read that SSH tunneling can be done but I am unable to get it working. I tried all combinations of flags, ports and hosts and all result in some kind of error.
Current setup
Running the chromium instance on my host chromium --remote-debugging-port=9222. Which gets me an address like so DevTools listening on ws://0.0.0.0:9222/devtools/browser/f66524d5-eecb-44c2-a48c-5b14d8e6d998
Running my app via this script
#!/bin/bash
docker build -t rod-test .
docker run --add-host=host.docker.internal:host-gateway --rm rod-test
The docker file
FROM golang:1.16-alpine
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY *.go ./
RUN go build -o /rod
CMD [ "/rod" ]
The main.go
package main
import (
"fmt"
"github.com/go-rod/rod"
)
func main() {
browser := rod.New().ControlURL("ws://host.docker.internal:9222/devtools/browser/f66524d5-eecb-44c2-a48c-5b14d8e6d998")
if err := browser.Connect(); err != nil {
fmt.Printf("err while connecting: %v", err)
return
}
fmt.Println(
browser.MustPage("https://mdn.dev/").MustEval("() => document.title"),
)
}
If I use --headless --remote-debugging-address=0.0.0.0 it works, but if I remove the headless part it refuses the connection. The only solution seems to be to use SSH tunneling like it is mentioned here. But these keep erroring out for me as all of the answers are very vague as to what is what and what IP should go where
$ ssh -L 172.17.0.1:9222:localhost:9222 -N localhost
ssh: connect to host localhost port 22: Connection refused
OR
$ ssh -L 172.17.0.1:9222:localhost:9222
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface]
[-b bind_address] [-c cipher_spec] [-D [bind_address:]port]
[-E log_file] [-e escape_char] [-F configfile] [-I pkcs11]
[-i identity_file] [-J [user#]host[:port]] [-L address]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-Q query_option] [-R address] [-S ctl_path] [-W host:port]
[-w local_tun[:remote_tun]] destination [command]
What I want to happen is to be able to connect from the container to the debugger running on my host machine. Some caveats that I would like to cover are
It works on other platforms not just linux
It doesn't require complex setup from the user
This will be used by other teammates and it would be nice to have an approachable setup
So anyone that might have a similar usecase I'll save you a couple of days of debugging and trial and error. The solution I came upon is the following
Using this docker image with chrome debugger pointing at 9222 port. You can run this as is or put it inside docker compose as I did. Now if you navigate to this url chrome://inspect/#devices while the above mentioned image is running, you'll be able to access the and view the browser inside the container. (If its running on 9222 then you can even do localhost:9222 and it will show you the available pages)
Now the code to connect to it using go-rod
ips, err := net.LookupIP("browser-devtools")
if err != nil {
return nil, fmt.Errorf("failed to get browser service ip: %w", err)
}
if len(ips) == 0 {
return nil, errors.New("ip list empty")
}
ip := ips[0].String()
fmt.Printf("IP is %q\n", ip)
resp, err := http.Get(fmt.Sprintf("http://%s:9222/json/version/", ip))
if err != nil {
return nil, fmt.Errorf("failed to get devtools info: %w", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("failed read response body: %w", err)
}
println(string(body))
responseMapped := make(map[string]interface{})
if err = json.Unmarshal(body, &responseMapped); err != nil {
return nil, fmt.Errorf("failed to unmarshal response: %w", err)
}
debuggerUrl, ok := responseMapped["webSocketDebuggerUrl"].(string)
if !ok {
return nil, errors.New("no 'webSocketDebuggerUrl' entry in response map")
}
browser := rod.New().ControlURL(debuggerUrl)
if err := browser.Connect(); err != nil {
return nil, fmt.Errorf("failed connect to browser: %w", err)
}
return browser, nil
Explanation:
ips, err := net.LookupIP("browser-devtools")
There is a slight issue that only localhost host can access the debugger or any (allowed) numerical IP, so basically you cannot simply use the service name in docker compose to access the browser as it will be denied. Here we are resolving the actual IP of the service with a lookup
ip := ips[0].String()
resp, err := http.Get(fmt.Sprintf("http://%s:9222/json/version/", ip))
Here we get the string representation of the IP (something like 127.1.0.0) and pass it into the request. The request in itself is important as we don't have the actual browser debugger url that go-rod expects so this get request will return
{
"Browser": "Chrome/72.0.3601.0",
"Protocol-Version": "1.3",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3601.0 Safari/537.36",
"V8-Version": "7.2.233",
"WebKit-Version": "537.36 (#cfede9db1d154de0468cb0538479f34c0755a0f4)",
"webSocketDebuggerUrl": "ws://localhost:9222/devtools/browser/b0b8a4fb-bb17-4359-9533-a8d9f3908bd8"
}
Something along these lines. And the webSocketDebuggerUrl is the actual url we're connecting to
I'm trying to create a reverse proxy in golang for my private registry, however the following snippet has unexpected behavior:
package main
import (
"fmt"
"log"
"net/http"
"net/http/httputil"
"net/url"
)
func main() {
http.HandleFunc("/", ServeHTTP)
http.ListenAndServe("0.0.0.0:9090", nil)
}
func ServeHTTP(w http.ResponseWriter, r *http.Request) {
fmt.Printf(r.URL.Path + "\n")
remote, err := url.Parse("http://localhost:10000")
if err != nil {
log.Fatal(err)
}
proxy := httputil.NewSingleHostReverseProxy(remote)
proxy.ServeHTTP(w, r)
}
I used docker run -d -p 10000:5000 -v docker_image:/var/lib/registry\ registry to run a private docker registry container.
After running the code, I used curl 127.0.0.1:9090/v2/_catalog and I got the correct answer, but when I used docker push 127.0.0.1:9090/hello-world:latest (after running docker tag), the docker client threw Get "http://127.0.0.1:9090/v2/": dial tcp 127.0.0.1:9090: connect: connection refused.
I checked the logs and it seems that my code does not even get the http request.
What could I do to achieve the desired result?
Update 04/09/20 (jj/mm/aaaa)
I tried running my couchbase instance on custom ports by playing with couchbase configuration files.
docker run -d --privileged --memory 3200M --name bumblebase \
-v '$PWD/static_config:/opt/couchbase/etc/couchbase/static_config' \
-p 3456-3461:3456-3461 -p 6575-6576:6575-6576 \
couchbase:community-6.6.0
static_config is a file I created with the following content, following this page instructions (bottom section) :
{rest_port, 3456}.
{query_port, 3458}.
{fts_http_port, 3459}.
{cbas_http_port, 3460}.
{eventing_http_port, 3461}.
{memcached_port, 6575}.
But then I cannot access my couchbase instance at all (either web UI or rest api). I tried to point my local ports to both custom and default ports (-p 3456-3461:8091-8096) but none worked, and the problem disappear only if I remove the -v option - which brings me back to the original post scenario.
On a side note, I'm still trying to play with setting-alternate-address without any success so far. For some reason, when I set the alternate hostname (which seems to be required for this to run), accessing the alternate address takes a long time to load to eventually fail with a timeout error.
Original post
Since I develop many applications running different couchbase clusters, I have to run them in containers on different port. I created my container with the following command :
docker run -d --memory 2048M --name "my-database" \
-p 3456-3461:8091-8096 \
-p 6210-6211:11210-11211 \
couchbase
I then added buckets, and going on the web UI at localhost:3456 works fine :
Here is my server info :
In my code I have the following connect function :
var cluster *gocb.Cluster
func Cluster() *gocb.Cluster {
return cluster
}
func init() {]
c, err := gocb.Connect(
"couchbase://127.0.0.1:3456",
gocb.ClusterOptions{
Username: "Administrator",
Password: "password",
TimeoutsConfig: gocb.TimeoutsConfig{
ConnectTimeout: 30 * time.Second,
},
},
)
if err != nil {
panic(err)
}
cluster = c
}
Panic doesn't trigger, but whenever I try to perform a KV operation, it fails with the following error :
ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"Add","Opaque":"0x0","TimeObserved":2501547947,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
And this error only comes when I try to connect to my docker couchbase instance. If I run the Couchbase Server application and connect to appropriate port (8091), it works perfectly fine :
var cluster *gocb.Cluster
func Cluster() *gocb.Cluster {
return cluster
}
func init() {]
c, err := gocb.Connect(
"couchbase://localhost",
gocb.ClusterOptions{
Username: "Administrator",
Password: "password",
TimeoutsConfig: gocb.TimeoutsConfig{
ConnectTimeout: 30 * time.Second,
},
},
)
if err != nil {
panic(err)
}
cluster = c
}
I checked the credentials and they are correct, also replacing localhost with 0.0.0.0 or 127.0.0.1 didn't help at all.
I have a very simple application. Here is the code:
package main
import (
"fmt"
"math/rand"
"time"
"net/http"
"encoding/base64"
"encoding/json"
)
type Message struct {
Text string `json:"text"`
}
var cookieQuotes = []string{
// Skipped all the stuff
}
const COOKIE_NAME = "your_cookie"
func main() {
http.HandleFunc("/set_cookie", setCookie)
http.HandleFunc("/get_cookie", getCookie)
http.Handle("/favicon.ico", http.NotFoundHandler())
http.ListenAndServe(":8080", nil)
}
func setCookie(w http.ResponseWriter, r *http.Request) {
quote := getRandomCookieQuote()
encQuote := base64.StdEncoding.EncodeToString([]byte(quote))
http.SetCookie(w, &http.Cookie{
Name: COOKIE_NAME,
Value: encQuote,
})
}
func getCookie(w http.ResponseWriter, r *http.Request) {
cookie, err := r.Cookie(COOKIE_NAME)
if err != nil {
fmt.Fprintln(w, "Cannot get the cookie")
}
message, _ := base64.StdEncoding.DecodeString(cookie.Value)
msg := Message{Text:string(message)}
fmt.Println(msg.Text)
respBody, err := json.Marshal(msg)
fmt.Println(string(respBody))
if err != nil {
fmt.Println("Cannot marshall JSON")
}
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, string(respBody))
}
func getRandomCookieQuote() string {
source := rand.NewSource(time.Now().UnixNano())
random := rand.New(source)
i := random.Intn(len(cookieQuotes))
return cookieQuotes[i]
}
It was tested locally, and, also I've tried to run a docker container with it on my machine (Ubuntu) and it was working perfectly. But I want to run it on Virtual Machine (I use Oracle Virtual Box).
So, I have installed docker-machine:
docker-machine version 0.12.2, build 9371605
After that, I've switched to it, like it was recommended in official documentation like this:
eval "$(docker-machine env default)"
So I can do now from a perspective of that machine.
Also I've tried to run ngnix from the documentation example:
docker run -d -p 8000:80 nginx
curl $(docker-machine ip default):8000
And I get the result, I can get to ngnix welcome page by accessing my docker machine ip-address which could be accessed by command:
docker-machine ip default
But when I try to run my own docker image, I could not do this. When I try to access it, I get:
curl $(docker-machine ip default):8080
curl: (7) Failed to connect to 192.168.99.100 port 8080: Connection refused
Also I've tried to skip a port, to add protocol (http, and even https for the sake of luck) - nothing works.
Maybe, something wrong with my Dockerfile?
# Go experiments with cookies
FROM golang:1.8-onbuild
MAINTAINER vasyania2#gmail.com
Could you help me please?
This command maps port 8080 from your docker host to port 80 of your container:
docker run -d -p 8080:80 cookie-app
This instruction tells your go application to listen on port 8080, inside the container:
http.ListenAndServe(":8080", nil)
You have a port mismatch in those above lines, your application is not listening on the port you are forwarding to.
To connect to port 8080 of your container, you can run the following:
docker run -d -p 8080:8080 cookie-app
I need to use golang to access kafka,so i installed a kafka & zookepper in docker.
1.here is kafka install script:
# pull images
docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka
# run kafka & zookepper
docker run -d --name zookeeper -p 2181 -t wurstmeister/zookeeper
docker run --name kafka -e HOST_IP=localhost -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_BROKER_ID=1 -e ZK=zk -p 9092:9092 --link zookeeper:zk -t wurstmeister/kafka
# enter container
docker exec -it ${CONTAINER ID} /bin/bash
cd opt/kafka_2.11-0.10.1.1/
# make a tpoic
bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic mykafka
# start a producer in terminal-1
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mykafka
# start another terminal-2 and start a consumer
bin/kafka-console-consumer.sh --zookeeper zookeeper:2181 --topic mykafka --from-beginning
when i type some message in producer, the consumer will get it immediately.
so i assumed that the kafka is working fine
2.Now i need to create a consumer with golang to access kafka.
here is my golang demo code:
import "github.com/bsm/sarama-cluster"
func Consumer(){
// init (custom) config, enable errors and notifications
config := cluster.NewConfig()
config.Consumer.Return.Errors = true
config.Group.Return.Notifications = true
// init consumer
brokers := []string{"192.168.9.100:9092"}
topics := []string{"mykafka"}
consumer, err := cluster.NewConsumer(brokers, "my-group-id", topics, config)
if err != nil {
panic(err)
}
defer consumer.Close()
// trap SIGINT to trigger a shutdown.
signals := make(chan os.Signal, 1)
signal.Notify(signals, os.Interrupt)
// consume messages, watch errors and notifications
for {
select {
case msg, more := <-consumer.Messages():
if more {
fmt.Fprintf(os.Stdout, "%s/%d/%d\t%s\t%s\n", msg.Topic, msg.Partition, msg.Offset, msg.Key, msg.Value)
consumer.MarkOffset(msg, "") // mark message as processed
}
case err, more := <-consumer.Errors():
if more {
log.Printf("Error: %s\n", err.Error())
}
case ntf, more := <-consumer.Notifications():
if more {
log.Printf("Rebalanced: %+v\n", ntf)
}
case <-signals:
return
}
}
}
actually this demo code is copied from a github repo's demo:sarama-cluster
When running the code, i got an error:
kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
i did use a port map when start kafka,but just can't access it in golang
is there a way to use curl to access kafka?
i'v tried:
curl http://192.168.99.10:9092
and kafka report an error:
[2017-08-02 06:39:15,232] WARN Unexpected error from /192.168.99.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:95)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:75)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:203)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:167)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:379)
at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
at kafka.network.Processor.poll(SocketServer.scala:499)
at kafka.network.Processor.run(SocketServer.scala:435)
at java.lang.Thread.run(Thread.java:748)
BTW:
i use windows 7
dcoker machine's ip :192.168.99.100
it's drived me crazy
Is there some advice or solution? appreciate!!!
If you want to create a consumer to listen a topic from Kafka, let's try that way.
I used confluent-kafka-go from the tutorial: https://github.com/confluentinc/confluent-kafka-go
This is the code on main.go file:
import (
"fmt"
"gopkg.in/confluentinc/confluent-kafka-go.v1/kafka"
)
func main() {
c, err := kafka.NewConsumer(&kafka.ConfigMap{
"bootstrap.servers": "localhost",
"group.id": "myGroup",
"auto.offset.reset": "earliest",
})
if err != nil {
panic(err)
}
c.SubscribeTopics([]string{"myTopic", "^aRegex.*[Tt]opic"}, nil)
for {
msg, err := c.ReadMessage(-1)
if err == nil {
fmt.Printf("Message on %s: %s\n", msg.TopicPartition, string(msg.Value))
} else {
// The client will automatically try to recover from all errors.
fmt.Printf("Consumer error: %v (%v)\n", err, msg)
}
}
c.Close()
}
If you use docker to build: follow this comment to add suitable packages
For Debian and Ubuntu based distros, install librdkafka-dev from the standard repositories or using Confluent's Deb repository.
For Redhat based distros, install librdkafka-devel using Confluent's YUM repository.
For MacOS X, install librdkafka from Homebrew. You may also need to brew install pkg-config if you don't already have it. brew install librdkafka pkg-config.
For Alpine: apk add librdkafka-dev pkgconf
confluent-kafka-go is not supported on Windows.
With Alpine, please remember that install the community version, because it cannot install librdkafka with max version 1.1.0 (not use Alpine community version)
Good luck!
Not sure, if it is possible to use with curl with kafka. But you can use the kafka-console-consumer.
kafka-console-consumer.bat --bootstrap-server 192.168.9.100:9092 --topic mykafka --from-beginning
I'v found the reason.
because the kafka's settings is not correct
this is server.properties:
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
if the listeners is not set , kafka will only receive request from java.net.InetAddress.getCanonicalHostName() which means localhost
so i shuld set :
listeners = PLAINTEXT://0.0.0.0:9092
this will work