How to fix a ConsumePartition in Golang Sarama - docker

I'm doing tests with Kafka and Golang
I'm using:
Docker:
https://hub.docker.com/r/bitnami/kafka
Sarama:
https://github.com/Shopify/sarama
The example is very simple is a Consumer that connects to Kafka:
https://godoc.org/github.com/Shopify/sarama#example-Consumer
The code is this:
package main
import (
"log"
"os"
"os/signal"
"github.com/Shopify/sarama"
)
func main() {
consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, nil)
if err != nil {
panic(err)
}
defer func() {
if err := consumer.Close(); err != nil {
log.Fatalln(err)
}
}()
partitionConsumer, err := consumer.ConsumePartition("my_topic", 0, sarama.OffsetNewest)
if err != nil {
panic(err)
}
defer func() {
if err := partitionConsumer.Close(); err != nil {
log.Fatalln(err)
}
}()
// Trap SIGINT to trigger a shutdown.
signals := make(chan os.Signal, 1)
signal.Notify(signals, os.Interrupt)
consumed := 0
ConsumerLoop:
for {
select {
case msg := <-partitionConsumer.Messages():
log.Printf("Consumed message offset %d\n", msg.Offset)
consumed++
case <-signals:
break ConsumerLoop
}
}
log.Printf("Consumed: %d\n", consumed)
}
but when executing:
go run main.go
It shows me the following error:
panic: dial tcp: lookup fd6ee3862a45: no such host
goroutine 1 [running]:
main.main()
/Users/vn0sgkq/go/src/github.com/hectorgool/kafka1/main.go:25 +0x3f1
exit status 2
The repo is here:
https://github.com/hectorgool/kafka1/blob/master/main.go#L25
Yes, I know that I am missing the producer for the messages, but the strange thing is that: consumer.ConsumePartition
is not working

Related

Is it possible to recognize a person only once?

I have a problem. I want to write something similar to a turnstile, with face recognition. I use gocv and kagami/go-face for this. I have a stream from a webcam. The problem is that it recognizes the image too quickly and every time it recognizes a face, it opens a passage for it. That is, it pulls the opening function 10 times per second, for example. And I want him to open it once. So I have 2 functions readStream and opener. Maybe you can somehow not read the stream or stop the stream until another person appears in the frame? The goroutine go readStream() is called first, and then go opener()
func readStream(c chan int) {
recognizer, err := face.NewRecognizer(config.Recognizer.ModelsDir)
if err != nil {
log.Fatalf("Can't init face recognizer: %v", err)
}
webcam, err := gocv.OpenVideoCapture(videoSrc)
if err != nil {
log.Fatalf("Error opening capture device: %v", videoSrc)
}
img := gocv.NewMat()
defer img.Close()
window := gocv.NewWindow("videosourse")
defer window.Close()
for {
if ok := webcam.Read(&img); !ok || img.Empty() {
fmt.Printf("Device closed: %v\n", videoSrc)
return
}
jpgImageBuffer, err := gocv.IMEncode(gocv.JPEGFileExt, img)
if err != nil {
log.Errorf("ERROR: %s", err.Error())
}
f, err := recognizer.RecognizeSingle(jpgImageBuffer.GetBytes())
if err == nil && f != nil {
catID := recognizer.ClassifyThreshold(f.Descriptor, config.Recognizer.Tolerance)
if catID > 0 {
c <- catID
}
}
window.IMShow(img)
window.WaitKey(1)
}
}
func opener(c chan int) {
timeout := time.After(5 * time.Second)
for {
select {
case id := <-c:
fmt.Printf("####### OPENED for %d #######\n", id)
case <-timeout:
<-c
fmt.Println("Time's up! All data read in nothing")
break
}
}
}
i tried to use sync.once , but it stopped the webcam video stream after 1st recognition

sqlx: runtime error: invalid memory address or nil pointer dereference

I'am learning to use sqlx,but now I have a problem.
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7006c8]
goroutine 1 [running]:
github.com/jmoiron/sqlx.(*DB).QueryRowx(0x0, {0x75cb6a, 0x22}, {0xc000090c70, 0x1, 0x1})
This is my code and I don't know why this error occurring.
import (
"fmt"
_ "github.com/go-sql-driver/mysql"
"github.com/jmoiron/sqlx"
)
var DB *sqlx.DB
func initializeDatabases() (err error) {
dsn := "user:password#tcp(127.0.0.1:3306)/test?charset=utf8mb4&parseTime=True"
DB, err := sqlx.Connect("mysql", dsn)
if err != nil {
fmt.Printf("connect 2 dtatbase failed,err:%v\n", err)
return
}
DB.SetMaxOpenConns(10)
DB.SetMaxIdleConns(10)
return err
}
func sqlxQuerySingleRow() {
sqlStr := "select * from student where id = ?"
var u User
if err := DB.Get(&u, sqlStr, 1); err != nil {
return
}
fmt.Println("id:%d,name:%s,age:%d", u.Id, u.Name, u.Age)
}
func main() {
if err := initializeDatabases(); err != nil {
panic(err)
}
fmt.Println("connect success")
sqlxQuerySingleRow()
//sqlxMultiRow()
}
I know why,because I defined the DB as a global variable,but in the initializeDatabases() function,I declared the DB by:= which causes the returned Client connection to only take effect in the initializeDatabases()function,so I should change:= to =

Pull a file from a docker image in Golang to local file system

I’m trying to figure out how I can use the Go client (http://godoc.org/github.com/moby/moby/client) to pull a file from a docker image at a specific version (tag)
I want to download the image and then copy a file from the image onto the local file system. I see a lot of commands for dealing with an image, but not how to access its contents. I do see ways to access a container’s contents.
I’m suspecting I would need to download the image, create a container with the image and finally copy the contents out of the container. Though if I could avoid creating a container, that would be preferred.
Does anyone know exactly how to do this? Code snippet would be appreciated.
Thanks!
You must start the container, copy the file and then remove the container... It's not really an expensive operation since the container isn't started anyway.
Here's a working example that copies a file from the specified image to standard output. The API is straightforward to follow:
https://docs.docker.com/engine/api/get-started/
https://godoc.org/github.com/moby/moby/client
https://gist.github.com/ricardobranco777/a3be772935dfb1a183e0831496925585
package main
import (
"archive/tar"
"fmt"
"io"
"io/ioutil"
"os"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/client"
"golang.org/x/net/context"
)
func main() {
if len(os.Args) != 3 {
fmt.Fprintf(os.Stderr, "Usage: %s IMAGE FILE\n", os.Args[0])
os.Exit(1)
}
imageName := os.Args[1]
filePath := os.Args[2]
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
panic(err)
}
out, err := cli.ImagePull(ctx, imageName, types.ImagePullOptions{})
if err != nil {
panic(err)
}
defer out.Close()
if _, err := ioutil.ReadAll(out); err != nil {
panic(err)
}
info, err := cli.ContainerCreate(ctx, &container.Config{
Image: imageName,
}, nil, nil, "")
if err != nil {
panic(err)
}
tarStream, _, err := cli.CopyFromContainer(ctx, info.ID, filePath)
if err != nil {
panic(err)
}
tr := tar.NewReader(tarStream)
if _, err := tr.Next(); err != nil {
panic(err)
}
io.Copy(os.Stdout, tr)
if err := cli.ContainerRemove(ctx, info.ID, types.ContainerRemoveOptions{}); err != nil {
panic(err)
}
}

docker-proxy getting accept4: bad file descriptor

I am trying to add a port forwarding to docker container using docker-proxy but experiencing this error below,
Here are the details,
A container with IP 172.17.0.2 is already running with --net=none. We are providing our own network and not using docker0 network.
Now we want to expose some ports of container to host, so thought of trying docker-proxy.
we executed the below command,
$ docker-proxy -container-ip 172.17.0.2 -container-port 8000 -host-ip 0.0.0.0 -host-port 8000 -proto tcp
and we are getting,
2017/03/16 10:02:30 Stopping proxy on tcp/[::]:8001 for tcp/172.17.0.2:8001 (accept tcp [::]:8001: accept4: bad file descriptor)
Docker version: Docker version 17.03.0-ce, build 60ccb22
I don't think there's any other way of doing this but stopping the container, removing it, then running it again starting from a Dockerfile or simply with docker run by adding -p 8000:8000.
Docker doesn't seem to let you tinker with docker-proxy directly, you have to use the standard commands.
You could also manually expose the port to outside access by directly changing the iptables, i.e. DOCKER chain in the NAT table and the DOCKER chain in filter.
For instance:
iptables -t nat -A DOCKER ! -i your_bridge0 -p tcp -m tcp --dport 8000 -j DNAT --to-destination 172.17.0.2:8000
And:
iptables -A DOCKER ! -i your_bridge0 -o your_bridge0 -d 172.17.0.2 -p tcp --m tcp --dport 80 -j ACCEPT
Of course then you'd have to make sure that the rules are going to stick, which is quite a different problem altogether. Docker doesn't seem to care much about who manages iptables (ufw, firewalld etc.).
This will work, even if docker proxy isn't running at all. docker-proxy binds to the host's ports, which means you control the traffic on the INPUT chain in the filter table (so the host itself).
I still haven't figured out why docker was built this way, but by default, if you expose a container (using -p) and then you delete the DNAT rule, it will still work, because the request is going to hit the INPUT chain directly. Which is mind-boggling, but never mind.
the docker-proxy binary cannot be called outside dockerd directly, If you want call it outside anyway, try look the moby source code the key code is here
there is an example
package example
import (
"fmt"
"io"
"net"
"os"
"os/exec"
"runtime"
"strconv"
"syscall"
"time"
"github.com/ishidawataru/sctp"
)
const userlandProxyCommandName = "docker-proxy"
func newProxyCommand(proto string, hostIP net.IP, hostPort int, containerIP net.IP, containerPort int, proxyPath string) (userlandProxy, error) {
path := proxyPath
if proxyPath == "" {
cmd, err := exec.LookPath(userlandProxyCommandName)
if err != nil {
return nil, err
}
path = cmd
}
args := []string{
path,
"-proto", proto,
"-host-ip", hostIP.String(),
"-host-port", strconv.Itoa(hostPort),
"-container-ip", containerIP.String(),
"-container-port", strconv.Itoa(containerPort),
}
return &proxyCommand{
cmd: &exec.Cmd{
Path: path,
Args: args,
SysProcAttr: &syscall.SysProcAttr{
Pdeathsig: syscall.SIGTERM, // send a sigterm to the proxy if the creating thread in the daemon process dies (https://go.dev/issue/27505)
},
},
wait: make(chan error, 1),
}, nil
}
// proxyCommand wraps an exec.Cmd to run the userland TCP and UDP
// proxies as separate processes.
type proxyCommand struct {
cmd *exec.Cmd
wait chan error
}
func (p *proxyCommand) Start() error {
r, w, err := os.Pipe()
if err != nil {
return fmt.Errorf("proxy unable to open os.Pipe %s", err)
}
defer r.Close()
p.cmd.ExtraFiles = []*os.File{w}
// As p.cmd.SysProcAttr.Pdeathsig is set, the signal will be sent to the
// process when the OS thread on which p.cmd.Start() was executed dies.
// If the thread is allowed to be released back into the goroutine
// thread pool, the thread could get terminated at any time if a
// goroutine gets scheduled onto it which calls runtime.LockOSThread()
// and exits without a matching number of runtime.UnlockOSThread()
// calls. Ensure that the thread from which Start() is called stays
// alive until the proxy or the daemon process exits to prevent the
// proxy from getting terminated early. See https://go.dev/issue/27505
// for more details.
started := make(chan error)
go func() {
runtime.LockOSThread()
defer runtime.UnlockOSThread()
err := p.cmd.Start()
started <- err
if err != nil {
return
}
p.wait <- p.cmd.Wait()
}()
if err := <-started; err != nil {
return err
}
w.Close()
errchan := make(chan error, 1)
go func() {
buf := make([]byte, 2)
r.Read(buf)
if string(buf) != "0\n" {
errStr, err := io.ReadAll(r)
if err != nil {
errchan <- fmt.Errorf("Error reading exit status from userland proxy: %v", err)
return
}
errchan <- fmt.Errorf("Error starting userland proxy: %s", errStr)
return
}
errchan <- nil
}()
select {
case err := <-errchan:
return err
case <-time.After(16 * time.Second):
return fmt.Errorf("Timed out proxy starting the userland proxy")
}
}
func (p *proxyCommand) Stop() error {
if p.cmd.Process != nil {
if err := p.cmd.Process.Signal(os.Interrupt); err != nil {
return err
}
return <-p.wait
}
return nil
}
type userlandProxy interface {
Start() error
Stop() error
}
// ipVersion refers to IP version - v4 or v6
type ipVersion string
const (
// IPv4 is version 4
ipv4 ipVersion = "4"
// IPv4 is version 6
ipv6 ipVersion = "6"
)
// dummyProxy just listen on some port, it is needed to prevent accidental
// port allocations on bound port, because without userland proxy we using
// iptables rules and not net.Listen
type dummyProxy struct {
listener io.Closer
addr net.Addr
ipVersion ipVersion
}
func NewDummyProxy(proto string, hostIP net.IP, hostPort int) (userlandProxy, error) {
// detect version of hostIP to bind only to correct version
version := ipv4
if hostIP.To4() == nil {
version = ipv6
}
switch proto {
case "tcp":
addr := &net.TCPAddr{IP: hostIP, Port: hostPort}
return &dummyProxy{addr: addr, ipVersion: version}, nil
case "udp":
addr := &net.UDPAddr{IP: hostIP, Port: hostPort}
return &dummyProxy{addr: addr, ipVersion: version}, nil
case "sctp":
addr := &sctp.SCTPAddr{IPAddrs: []net.IPAddr{{IP: hostIP}}, Port: hostPort}
return &dummyProxy{addr: addr, ipVersion: version}, nil
default:
return nil, fmt.Errorf("Unknown addr type: %s", proto)
}
}
func (p *dummyProxy) Start() error {
switch addr := p.addr.(type) {
case *net.TCPAddr:
l, err := net.ListenTCP("tcp"+string(p.ipVersion), addr)
if err != nil {
return err
}
p.listener = l
case *net.UDPAddr:
l, err := net.ListenUDP("udp"+string(p.ipVersion), addr)
if err != nil {
return err
}
p.listener = l
case *sctp.SCTPAddr:
l, err := sctp.ListenSCTP("sctp"+string(p.ipVersion), addr)
if err != nil {
return err
}
p.listener = l
default:
return fmt.Errorf("Unknown addr type: %T", p.addr)
}
return nil
}
func (p *dummyProxy) Stop() error {
if p.listener != nil {
return p.listener.Close()
}
return nil
}
func Proxy(hostip string, hostport int, containerip string, containerport int) error {
// userlandProxy, err := NewDummyProxy("tcp", net.ParseIP("0.0.0.0"), 8888)
userlandProxy, err := newProxyCommand("tcp", net.ParseIP(hostip), hostport, net.ParseIP(containerip), containerport, "/usr/bin/docker-proxy")
if err != nil {
return err
}
cleanup := func() error {
// need to undo the iptables rules before we return
userlandProxy.Stop()
// pm.DeleteForwardingTableEntry(m.proto, hostIP, allocatedHostPort, containerIP.String(), containerPort)
// if err := pm.Allocator.ReleasePort(hostIP, m.proto, allocatedHostPort); err != nil {
// return err
// }
return nil
}
if err := userlandProxy.Start(); err != nil {
if err := cleanup(); err != nil {
return fmt.Errorf("Error during port allocation cleanup: %v", err)
}
return err
}
return nil
}
and call it in main package like:
func main() {
// ne(proto, hostIP, allocatedHostPort, sctpAddr.IPAddrs[0].IP, sctpAddr.Port, pm.proxyPath)
if err := example.Proxy("0.0.0.0", 8080, "172.17.0.10", 8080); err != nil {
fmt.Printf("err.Error(): %v\n", err.Error())
}
time.Sleep(time.Second * 500)
}

How do you use the net functions effectively in Go?

For example, having basic packet protocol, like:
[packetType int][packetId int][data []byte]
And making a client and server doing simple things with it (egx, chatting.)
Here's a client and server with sloppy panic error-handling. They have some limitations:
The server only handles one client connection at a time. You could fix this by using goroutines.
Packets always contain 100-byte payloads. You could fix this by putting a length in the packet somewhere and not using encoding/binary for the entire struct, but I've kept it simple.
Here's the server:
package main
import (
"encoding/binary"
"fmt"
"net"
)
type packet struct {
// Field names must be capitalized for encoding/binary.
// It's also important to use explicitly sized types.
// int32 rather than int, etc.
Type int32
Id int32
// This must be an array rather than a slice.
Data [100]byte
}
func main() {
// set up a listener on port 2000
l, err := net.Listen("tcp", ":2000")
if err != nil {
panic(err.String())
}
for {
// start listening for a connection
conn, err := l.Accept()
if err != nil {
panic(err.String())
}
handleClient(conn)
}
}
func handleClient(conn net.Conn) {
defer conn.Close()
// a client has connected; now wait for a packet
var msg packet
binary.Read(conn, binary.BigEndian, &msg)
fmt.Printf("Received a packet: %s\n", msg.Data)
// send the response
response := packet{Type: 1, Id: 1}
copy(response.Data[:], "Hello, client")
binary.Write(conn, binary.BigEndian, &response)
}
Here's the client. It sends one packet with packet type 0, id 0, and the contents "Hello, server". Then it waits for a response, prints it, and exits.
package main
import (
"encoding/binary"
"fmt"
"net"
)
type packet struct {
Type int32
Id int32
Data [100]byte
}
func main() {
// connect to localhost on port 2000
conn, err := net.Dial("tcp", ":2000")
if err != nil {
panic(err.String())
}
defer conn.Close()
// send a packet
msg := packet{}
copy(msg.Data[:], "Hello, server")
err = binary.Write(conn, binary.BigEndian, &msg)
if err != nil {
panic(err.String())
}
// receive the response
var response packet
err = binary.Read(conn, binary.BigEndian, &response)
if err != nil {
panic(err.String())
}
fmt.Printf("Response: %s\n", response.Data)
}
Check out Jan Newmarch's "Network programming with Go".

Resources