How do you use the net functions effectively in Go? - network-programming

For example, having basic packet protocol, like:
[packetType int][packetId int][data []byte]
And making a client and server doing simple things with it (egx, chatting.)

Here's a client and server with sloppy panic error-handling. They have some limitations:
The server only handles one client connection at a time. You could fix this by using goroutines.
Packets always contain 100-byte payloads. You could fix this by putting a length in the packet somewhere and not using encoding/binary for the entire struct, but I've kept it simple.
Here's the server:
package main
import (
"encoding/binary"
"fmt"
"net"
)
type packet struct {
// Field names must be capitalized for encoding/binary.
// It's also important to use explicitly sized types.
// int32 rather than int, etc.
Type int32
Id int32
// This must be an array rather than a slice.
Data [100]byte
}
func main() {
// set up a listener on port 2000
l, err := net.Listen("tcp", ":2000")
if err != nil {
panic(err.String())
}
for {
// start listening for a connection
conn, err := l.Accept()
if err != nil {
panic(err.String())
}
handleClient(conn)
}
}
func handleClient(conn net.Conn) {
defer conn.Close()
// a client has connected; now wait for a packet
var msg packet
binary.Read(conn, binary.BigEndian, &msg)
fmt.Printf("Received a packet: %s\n", msg.Data)
// send the response
response := packet{Type: 1, Id: 1}
copy(response.Data[:], "Hello, client")
binary.Write(conn, binary.BigEndian, &response)
}
Here's the client. It sends one packet with packet type 0, id 0, and the contents "Hello, server". Then it waits for a response, prints it, and exits.
package main
import (
"encoding/binary"
"fmt"
"net"
)
type packet struct {
Type int32
Id int32
Data [100]byte
}
func main() {
// connect to localhost on port 2000
conn, err := net.Dial("tcp", ":2000")
if err != nil {
panic(err.String())
}
defer conn.Close()
// send a packet
msg := packet{}
copy(msg.Data[:], "Hello, server")
err = binary.Write(conn, binary.BigEndian, &msg)
if err != nil {
panic(err.String())
}
// receive the response
var response packet
err = binary.Read(conn, binary.BigEndian, &response)
if err != nil {
panic(err.String())
}
fmt.Printf("Response: %s\n", response.Data)
}

Check out Jan Newmarch's "Network programming with Go".

Related

Is it possible to recognize a person only once?

I have a problem. I want to write something similar to a turnstile, with face recognition. I use gocv and kagami/go-face for this. I have a stream from a webcam. The problem is that it recognizes the image too quickly and every time it recognizes a face, it opens a passage for it. That is, it pulls the opening function 10 times per second, for example. And I want him to open it once. So I have 2 functions readStream and opener. Maybe you can somehow not read the stream or stop the stream until another person appears in the frame? The goroutine go readStream() is called first, and then go opener()
func readStream(c chan int) {
recognizer, err := face.NewRecognizer(config.Recognizer.ModelsDir)
if err != nil {
log.Fatalf("Can't init face recognizer: %v", err)
}
webcam, err := gocv.OpenVideoCapture(videoSrc)
if err != nil {
log.Fatalf("Error opening capture device: %v", videoSrc)
}
img := gocv.NewMat()
defer img.Close()
window := gocv.NewWindow("videosourse")
defer window.Close()
for {
if ok := webcam.Read(&img); !ok || img.Empty() {
fmt.Printf("Device closed: %v\n", videoSrc)
return
}
jpgImageBuffer, err := gocv.IMEncode(gocv.JPEGFileExt, img)
if err != nil {
log.Errorf("ERROR: %s", err.Error())
}
f, err := recognizer.RecognizeSingle(jpgImageBuffer.GetBytes())
if err == nil && f != nil {
catID := recognizer.ClassifyThreshold(f.Descriptor, config.Recognizer.Tolerance)
if catID > 0 {
c <- catID
}
}
window.IMShow(img)
window.WaitKey(1)
}
}
func opener(c chan int) {
timeout := time.After(5 * time.Second)
for {
select {
case id := <-c:
fmt.Printf("####### OPENED for %d #######\n", id)
case <-timeout:
<-c
fmt.Println("Time's up! All data read in nothing")
break
}
}
}
i tried to use sync.once , but it stopped the webcam video stream after 1st recognition

How to fix a ConsumePartition in Golang Sarama

I'm doing tests with Kafka and Golang
I'm using:
Docker:
https://hub.docker.com/r/bitnami/kafka
Sarama:
https://github.com/Shopify/sarama
The example is very simple is a Consumer that connects to Kafka:
https://godoc.org/github.com/Shopify/sarama#example-Consumer
The code is this:
package main
import (
"log"
"os"
"os/signal"
"github.com/Shopify/sarama"
)
func main() {
consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, nil)
if err != nil {
panic(err)
}
defer func() {
if err := consumer.Close(); err != nil {
log.Fatalln(err)
}
}()
partitionConsumer, err := consumer.ConsumePartition("my_topic", 0, sarama.OffsetNewest)
if err != nil {
panic(err)
}
defer func() {
if err := partitionConsumer.Close(); err != nil {
log.Fatalln(err)
}
}()
// Trap SIGINT to trigger a shutdown.
signals := make(chan os.Signal, 1)
signal.Notify(signals, os.Interrupt)
consumed := 0
ConsumerLoop:
for {
select {
case msg := <-partitionConsumer.Messages():
log.Printf("Consumed message offset %d\n", msg.Offset)
consumed++
case <-signals:
break ConsumerLoop
}
}
log.Printf("Consumed: %d\n", consumed)
}
but when executing:
go run main.go
It shows me the following error:
panic: dial tcp: lookup fd6ee3862a45: no such host
goroutine 1 [running]:
main.main()
/Users/vn0sgkq/go/src/github.com/hectorgool/kafka1/main.go:25 +0x3f1
exit status 2
The repo is here:
https://github.com/hectorgool/kafka1/blob/master/main.go#L25
Yes, I know that I am missing the producer for the messages, but the strange thing is that: consumer.ConsumePartition
is not working

docker-proxy getting accept4: bad file descriptor

I am trying to add a port forwarding to docker container using docker-proxy but experiencing this error below,
Here are the details,
A container with IP 172.17.0.2 is already running with --net=none. We are providing our own network and not using docker0 network.
Now we want to expose some ports of container to host, so thought of trying docker-proxy.
we executed the below command,
$ docker-proxy -container-ip 172.17.0.2 -container-port 8000 -host-ip 0.0.0.0 -host-port 8000 -proto tcp
and we are getting,
2017/03/16 10:02:30 Stopping proxy on tcp/[::]:8001 for tcp/172.17.0.2:8001 (accept tcp [::]:8001: accept4: bad file descriptor)
Docker version: Docker version 17.03.0-ce, build 60ccb22
I don't think there's any other way of doing this but stopping the container, removing it, then running it again starting from a Dockerfile or simply with docker run by adding -p 8000:8000.
Docker doesn't seem to let you tinker with docker-proxy directly, you have to use the standard commands.
You could also manually expose the port to outside access by directly changing the iptables, i.e. DOCKER chain in the NAT table and the DOCKER chain in filter.
For instance:
iptables -t nat -A DOCKER ! -i your_bridge0 -p tcp -m tcp --dport 8000 -j DNAT --to-destination 172.17.0.2:8000
And:
iptables -A DOCKER ! -i your_bridge0 -o your_bridge0 -d 172.17.0.2 -p tcp --m tcp --dport 80 -j ACCEPT
Of course then you'd have to make sure that the rules are going to stick, which is quite a different problem altogether. Docker doesn't seem to care much about who manages iptables (ufw, firewalld etc.).
This will work, even if docker proxy isn't running at all. docker-proxy binds to the host's ports, which means you control the traffic on the INPUT chain in the filter table (so the host itself).
I still haven't figured out why docker was built this way, but by default, if you expose a container (using -p) and then you delete the DNAT rule, it will still work, because the request is going to hit the INPUT chain directly. Which is mind-boggling, but never mind.
the docker-proxy binary cannot be called outside dockerd directly, If you want call it outside anyway, try look the moby source code the key code is here
there is an example
package example
import (
"fmt"
"io"
"net"
"os"
"os/exec"
"runtime"
"strconv"
"syscall"
"time"
"github.com/ishidawataru/sctp"
)
const userlandProxyCommandName = "docker-proxy"
func newProxyCommand(proto string, hostIP net.IP, hostPort int, containerIP net.IP, containerPort int, proxyPath string) (userlandProxy, error) {
path := proxyPath
if proxyPath == "" {
cmd, err := exec.LookPath(userlandProxyCommandName)
if err != nil {
return nil, err
}
path = cmd
}
args := []string{
path,
"-proto", proto,
"-host-ip", hostIP.String(),
"-host-port", strconv.Itoa(hostPort),
"-container-ip", containerIP.String(),
"-container-port", strconv.Itoa(containerPort),
}
return &proxyCommand{
cmd: &exec.Cmd{
Path: path,
Args: args,
SysProcAttr: &syscall.SysProcAttr{
Pdeathsig: syscall.SIGTERM, // send a sigterm to the proxy if the creating thread in the daemon process dies (https://go.dev/issue/27505)
},
},
wait: make(chan error, 1),
}, nil
}
// proxyCommand wraps an exec.Cmd to run the userland TCP and UDP
// proxies as separate processes.
type proxyCommand struct {
cmd *exec.Cmd
wait chan error
}
func (p *proxyCommand) Start() error {
r, w, err := os.Pipe()
if err != nil {
return fmt.Errorf("proxy unable to open os.Pipe %s", err)
}
defer r.Close()
p.cmd.ExtraFiles = []*os.File{w}
// As p.cmd.SysProcAttr.Pdeathsig is set, the signal will be sent to the
// process when the OS thread on which p.cmd.Start() was executed dies.
// If the thread is allowed to be released back into the goroutine
// thread pool, the thread could get terminated at any time if a
// goroutine gets scheduled onto it which calls runtime.LockOSThread()
// and exits without a matching number of runtime.UnlockOSThread()
// calls. Ensure that the thread from which Start() is called stays
// alive until the proxy or the daemon process exits to prevent the
// proxy from getting terminated early. See https://go.dev/issue/27505
// for more details.
started := make(chan error)
go func() {
runtime.LockOSThread()
defer runtime.UnlockOSThread()
err := p.cmd.Start()
started <- err
if err != nil {
return
}
p.wait <- p.cmd.Wait()
}()
if err := <-started; err != nil {
return err
}
w.Close()
errchan := make(chan error, 1)
go func() {
buf := make([]byte, 2)
r.Read(buf)
if string(buf) != "0\n" {
errStr, err := io.ReadAll(r)
if err != nil {
errchan <- fmt.Errorf("Error reading exit status from userland proxy: %v", err)
return
}
errchan <- fmt.Errorf("Error starting userland proxy: %s", errStr)
return
}
errchan <- nil
}()
select {
case err := <-errchan:
return err
case <-time.After(16 * time.Second):
return fmt.Errorf("Timed out proxy starting the userland proxy")
}
}
func (p *proxyCommand) Stop() error {
if p.cmd.Process != nil {
if err := p.cmd.Process.Signal(os.Interrupt); err != nil {
return err
}
return <-p.wait
}
return nil
}
type userlandProxy interface {
Start() error
Stop() error
}
// ipVersion refers to IP version - v4 or v6
type ipVersion string
const (
// IPv4 is version 4
ipv4 ipVersion = "4"
// IPv4 is version 6
ipv6 ipVersion = "6"
)
// dummyProxy just listen on some port, it is needed to prevent accidental
// port allocations on bound port, because without userland proxy we using
// iptables rules and not net.Listen
type dummyProxy struct {
listener io.Closer
addr net.Addr
ipVersion ipVersion
}
func NewDummyProxy(proto string, hostIP net.IP, hostPort int) (userlandProxy, error) {
// detect version of hostIP to bind only to correct version
version := ipv4
if hostIP.To4() == nil {
version = ipv6
}
switch proto {
case "tcp":
addr := &net.TCPAddr{IP: hostIP, Port: hostPort}
return &dummyProxy{addr: addr, ipVersion: version}, nil
case "udp":
addr := &net.UDPAddr{IP: hostIP, Port: hostPort}
return &dummyProxy{addr: addr, ipVersion: version}, nil
case "sctp":
addr := &sctp.SCTPAddr{IPAddrs: []net.IPAddr{{IP: hostIP}}, Port: hostPort}
return &dummyProxy{addr: addr, ipVersion: version}, nil
default:
return nil, fmt.Errorf("Unknown addr type: %s", proto)
}
}
func (p *dummyProxy) Start() error {
switch addr := p.addr.(type) {
case *net.TCPAddr:
l, err := net.ListenTCP("tcp"+string(p.ipVersion), addr)
if err != nil {
return err
}
p.listener = l
case *net.UDPAddr:
l, err := net.ListenUDP("udp"+string(p.ipVersion), addr)
if err != nil {
return err
}
p.listener = l
case *sctp.SCTPAddr:
l, err := sctp.ListenSCTP("sctp"+string(p.ipVersion), addr)
if err != nil {
return err
}
p.listener = l
default:
return fmt.Errorf("Unknown addr type: %T", p.addr)
}
return nil
}
func (p *dummyProxy) Stop() error {
if p.listener != nil {
return p.listener.Close()
}
return nil
}
func Proxy(hostip string, hostport int, containerip string, containerport int) error {
// userlandProxy, err := NewDummyProxy("tcp", net.ParseIP("0.0.0.0"), 8888)
userlandProxy, err := newProxyCommand("tcp", net.ParseIP(hostip), hostport, net.ParseIP(containerip), containerport, "/usr/bin/docker-proxy")
if err != nil {
return err
}
cleanup := func() error {
// need to undo the iptables rules before we return
userlandProxy.Stop()
// pm.DeleteForwardingTableEntry(m.proto, hostIP, allocatedHostPort, containerIP.String(), containerPort)
// if err := pm.Allocator.ReleasePort(hostIP, m.proto, allocatedHostPort); err != nil {
// return err
// }
return nil
}
if err := userlandProxy.Start(); err != nil {
if err := cleanup(); err != nil {
return fmt.Errorf("Error during port allocation cleanup: %v", err)
}
return err
}
return nil
}
and call it in main package like:
func main() {
// ne(proto, hostIP, allocatedHostPort, sctpAddr.IPAddrs[0].IP, sctpAddr.Port, pm.proxyPath)
if err := example.Proxy("0.0.0.0", 8080, "172.17.0.10", 8080); err != nil {
fmt.Printf("err.Error(): %v\n", err.Error())
}
time.Sleep(time.Second * 500)
}

How to have multiple consumer from one io.Reader?

I am working on a small script which uses bufio.Scanner and http.Request as well as go routines to count words and lines in parallel.
package main
import (
"bufio"
"fmt"
"io"
"log"
"net/http"
"time"
)
func main() {
err := request("http://www.google.com")
if err != nil {
log.Fatal(err)
}
// just keep main alive with sleep for now
time.Sleep(2 * time.Second)
}
func request(url string) error {
res, err := http.Get(url)
if err != nil {
return err
}
go scanLineWise(res.Body)
go scanWordWise(res.Body)
return err
}
func scanLineWise(r io.Reader) {
s := bufio.NewScanner(r)
s.Split(bufio.ScanLines)
i := 0
for s.Scan() {
i++
}
fmt.Printf("Counted %d lines.\n", i)
}
func scanWordWise(r io.Reader) {
s := bufio.NewScanner(r)
s.Split(bufio.ScanWords)
i := 0
for s.Scan() {
i++
}
fmt.Printf("Counted %d words.\n", i)
}
Source
As more or less expected from streams scanLineWise will count a number while scalWordWise will count zero. This is because scanLineWise already reads everything from req.Body.
I would know like to know: How to solve this elegantly?
My first thought was to build a struct which implements io.Reader and io.Writer. We could use io.Copy to read from req.Body and write it to the writer. When the scanners read from this writer then writer will copy the data instead of reading it. Unfortunately this will just collect memory over time and break the whole idea of streams...
The options are pretty straightforward -- you either maintain the "stream" of data, or you buffer the body.
If you really do need to read over the body more then once sequentially, you need to buffer it somewhere. There's no way around that.
There's a number of way you could stream the data, like having the line counter output lines into the word counter (preferably through channels). You could also build a pipeline using io.TeeReader and io.Pipe, and supply a unique reader for each function.
...
pipeReader, pipeWriter := io.Pipe()
bodyReader := io.TeeReader(res.Body, pipeWriter)
go scanLineWise(bodyReader)
go scanWordWise(pipeReader)
...
That can get unwieldy with more consumers though, so you could use io.MultiWriter to multiplex to more io.Readers.
...
pipeOneR, pipeOneW := io.Pipe()
pipeTwoR, pipeTwoW := io.Pipe()
pipeThreeR, pipeThreeW := io.Pipe()
go scanLineWise(pipeOneR)
go scanWordWise(pipeTwoR)
go scanSomething(pipeThreeR)
// of course, this should probably have some error handling
io.Copy(io.MultiWriter(pipeOneW, pipeTwoW, pipeThreeW), res.Body)
...
You could use channels, do the actual reading in your scanLineWise then pass the lines to scanWordWise, for example:
func countLines(r io.Reader) (ch chan string) {
ch = make(chan string)
go func() {
s := bufio.NewScanner(r)
s.Split(bufio.ScanLines)
cnt := 0
for s.Scan() {
ch <- s.Text()
cnt++
}
close(ch)
fmt.Printf("Counted %d lines.\n", cnt)
}()
return
}
func countWords(ch <-chan string) {
cnt := 0
for line := range ch {
s := bufio.NewScanner(strings.NewReader(line))
s.Split(bufio.ScanWords)
for s.Scan() {
cnt++
}
}
fmt.Printf("Counted %d words.\n", cnt)
}
func main() {
r := strings.NewReader(body)
ch := countLines(r)
go countWords(ch)
time.Sleep(1 * time.Second)
}

iOS Game Center identity verification in Go

I am trying to write a service in Go that takes the parameters given by GameCenter in
//GKLocalPlayer
- (void)generateIdentityVerificationSignatureWithCompletionHandler:(void (^)(NSURL *publicKeyUrl, NSData *signature, NSData *salt, uint64_t timestamp, NSError *error))completionHandler
Inside the completionHandler of the method, I am sending the public key URL, base 64 encoded signature, base 64 encoded salt, timestamp and the user's game center ID to my Go service. Inside my Go (in Google App Engine), this is what I am doing:
Get the certificate from the public key URL
Decode signature and salt
Form the payload based on player ID, bundle ID, timestamp and
salt
Use X509.CheckSignature to verify that the payload matches the
signature when it's hased with the public key
*I know that I still need to verify with the certificate authority but I am skipping that for now (if you know how to do that in Go for this case, please please please share!)
Problem: CheckSignature is returning crypto/rsa: verification error and I really think that I am doing everything as instructed by Apple
https://developer.apple.com/library/ios/documentation/GameKit/Reference/GKLocalPlayer_Ref/Reference/Reference.html#//apple_ref/doc/uid/TP40009587-CH1-SW25
And the code that I have so far:
func (v *ValidationRequest) ValidateGameCenter(publicKeyUrl string, playerId string, bundleId string, signature string, salt string, timestamp uint64) error {
client := urlfetch.Client(v.Context)
resp, err := client.Get(publicKeyUrl)
if err != nil {
v.Context.Errorf("%v", err.Error())
return err
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
v.Context.Errorf("%v", err.Error())
return err
}
cert, err := x509.ParseCertificate(body)
if err != nil {
v.Context.Errorf("%v", err.Error())
return err
}
signatureBytes, err := base64.StdEncoding.DecodeString(signature)
saltBytes, err:= base64.StdEncoding.DecodeString(salt)
payload, err := formPayload(v, playerId, bundleId, timestamp, saltBytes)
if err != nil {
v.Context.Errorf("%v", err.Error())
return err
}
err = cert.CheckSignature(cert.SignatureAlgorithm, payload, signatureBytes)
if err != nil {
v.Context.Errorf("%v", err.Error())
return err
}
return nil
}
func formPayload(v *ValidationRequest, playerId string, bundleId string, timestamp uint64, salt []byte) ([]byte, error) {
bundleIdBytes := []byte(bundleId)
playerIdBytes := []byte(playerId)
payloadBuffer := new(bytes.Buffer)
written, err := payloadBuffer.Write(playerIdBytes)
if err != nil {
return nil, err
}
written, err = payloadBuffer.Write(bundleIdBytes)
if err != nil {
return nil, err
}
var bigEndianTimestamp []byte = make([]byte, 8)
binary.BigEndian.PutUint64(bigEndianTimestamp, timestamp)
if written != len(bundleIdBytes) {
return nil, errors.New(fmt.Sprintf("Failed writing all bytes. Written: %d Length: %d", written, len(bundleIdBytes)))
}
written, err = payloadBuffer.Write(bigEndianTimestamp)
if err != nil {
return nil, err
}
if written != len(bigEndianTimestamp) {
return nil, errors.New(fmt.Sprintf("Failed writing all bytes. Written: %d Length: %d", written, len(bigEndianTimestamp)))
}
written, err = payloadBuffer.Write(salt)
if err != nil {
return nil, err
}
if written != len(salt) {
return nil, errors.New(fmt.Sprintf("Failed writing all bytes. Written: %d Length: %d", written, len(salt)))
}
return payloadBuffer.Bytes(), nil
}

Resources