package main
import (
"fmt"
"log"
"net/http"
"os"
"github.com/steven-ferrer/gonsole"
)
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hi there!\n")
file, err := os.Open("ItemLog.txt")
if err != nil {
log.Fatal(err)
}
reader := gonsole.NewReader(file)
counter := 0
for {
foo, _ := reader.NextWord()
if foo == "<Kept>" {
counter++
fmt.Fprintf(w, "%d"+": ", counter)
foo, _ = reader.NextWord()
for foo != "|" {
fmt.Fprintf(w, foo+" ")
foo, _ = reader.NextWord()
}
if foo == "|" { // need to reader.NewLine instead but this will work for now.
fmt.Fprintf(w, "\n")
}
}
}
}
func main() {
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
My local CLI works but when I try to wrap it up in a server only so many lines are printed. It's like it times out or something. Help?
I need more text so: I'm parsing a text file.
Edit: here's a test file... https://pastebin.com/ZNbut51X
You are not helping yourself ignoring errors:
foo, _ := reader.NextWord()
This is very bad practice. Check the error and it will tell you what is going on.
Update:
You have infinite loop in your code.
for {
...
}
for{} works until you call continue or return inside that loop.
https://tour.golang.org/flowcontrol/4
In your case, it cannot run forever because go-routine that runs it is terminated by timeout.
Update2:
Infinite loop is not compatible with HTTP. With web service, you are getting request and should return response before go-routine is terminated by timeout.
You have two options:
send requests by timer every x seconds and return recent data to
caller in handler.
implement technology that supports bidirectional communications
between client and server -
https://godoc.org/golang.org/x/net/websocket
Both options unfortunately more complicated than console app :(
Related
I am currently working on a server that is designed to be run in a Docker container.
Here is my setup method for my tests:
func TestMain(m *testing.M) {
schedulerName := "scheduler1"
IP, err := container.StartNewScheduler(schedulerName)
if err != nil {
log.Println("Could not create container.")
log.Fatal(err)
}
serverIP = IP
code := m.Run()
cleanupContainer(schedulerName)
os.Exit(code)
}
The line container.StartNewScheduler(schedulername) boots up a new docker container called "scheduler1" and tells it to run the server inside of it.
Next I run my tests with the container running in the background, right now I only have one test.
func TestNewScheduler(t *testing.T) {
testCodeInput := "THIS IS A TEST"
requestBody, err := json.Marshal(map[string]string{
"Code": fmt.Sprintf("print(\"%s\")", testCodeInput),
})
if err != nil {
t.Fatal(err)
}
url := fmt.Sprintf("http://%s:%d/execute/python", serverIP, 3000)
contentType := "application/json"
body := bytes.NewBuffer(requestBody)
response := post(url, contentType, body, t)
actual := parseOutput(response.Body, t)
response.Body.Close()
expected := fmt.Sprintf("{\"Stdout\":\"%s\\n\"}", testCodeInput)
if actual != expected {
t.Fatalf("Expected %s, but got %s", expected, actual)
}
}
The problem that I am running into is sometimes I get a connection refused and sometimes I don't.
server_container_test.go:51: Post http://172.20.0.2:3000/execute/python: dial tcp 172.20.0.2:3000: connect: connection refused
I noticed that whenever I try and debug the issue everything seems to work fine. My running theory is because when I step through my code the container has more time to start up and get the server running inside it.
In order to test my Hypothesis I added a second post call in my post method with a timer set before I call it.
func post(url string, contentType string, body io.Reader, t *testing.T) *http.Response {
t.Helper()
response, err := http.Post(url, contentType, body)
if err != nil {
//There is an error where the container takes a second to boot up and so
//the scheduler isn't running when the first request is sent, so we try
//one more time here and check again.
time.Sleep(100 * time.Millisecond) <--- Right here
response, err = http.Post(url, contentType, body)
if err != nil {
t.Fatal(err)
}
}
return response
}
Does anyone else have any guesses as to what could be causing me this issue?
If my hypothesis is correct is this the best way to fix this? Is it a bad idea to add a time.Sleep to your tests?
Thanks!
Ok so after some more thought I changed up my source code, please let me know if you think this is a good solution to my problem. I am still learning Go and HTTP servers so any input is appreciated.
Here is my fix/idea:
Previously once the container was created I just returned it's IP address and forgot about it.
Now I create a go routine that repeatedly tries to send a POST request to the server. If it doesn't fail then I send true through a channel and close the function.
IP := info.NetworkSettings.Networks[networkName].IPAddress
works := make(chan bool)
ctx, canelRoutine := context.WithCancel(context.Background())
defer canelRoutine()
go func(ctx context.Context) {
requestBody, _ := json.Marshal(map[string]string{
"Code": "print()",
})
select {
case <-ctx.Done():
return
default:
for {
_, err := http.Post(
fmt.Sprintf("http://%s:%d/execute/python", IP, 3000),
"application/json",
bytes.NewBuffer(requestBody),
)
if err == nil {
works <- true
return
}
}
}
}(ctx)
After sending the goroutine off I create a timer and and wait for either the timer to return or the goroutine.
timer := time.After(500 * time.Millisecond)
select {
case <-works:
return IP, nil
case <-timer:
return IP, &UnreachableContainerError{name: schedulerName}
}
The upside to this solution is I have now introduced an UnreachableContainerError which allows me to be more specific about my error messages and it can be checked on the receiving side. I also send the IP address back either way just in case the client needs it for some other reason.
Here is the full StartNewScheduler method in case you wanted to see it.
//StartNewScheduler starts a new scheduler with the given options.
//returns the IP address for the given scheduler.
func StartNewScheduler(schedulerName string) (string, error) {
///Defaults
dockerfile := "Dockerfile_standard"
networkName := "scheduler-cluster"
imageID := "lkelly93/scheduler_image:latest"
cli, err := client.NewEnvClient()
if err != nil {
return "", err
}
err = createDefaultImageIfNeeded(
cli,
imageID,
dockerfile)
if err != nil {
return "", err
}
err = createSchedulerClusterNetworkIfNeeded(cli, networkName)
if err != nil {
return "", err
}
ctx := context.Background()
resp, err := cli.ContainerCreate(
ctx,
&container.Config{Image: imageID},
&container.HostConfig{
NetworkMode: container.NetworkMode(networkName),
Privileged: true,
},
nil,
schedulerName,
)
if err != nil {
return "", err
}
err = cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{})
if err != nil {
return "", err
}
//Get container IP
info, err := cli.ContainerInspect(ctx, resp.ID)
if err != nil {
return "", err
}
IP := info.NetworkSettings.Networks[networkName].IPAddress
works := make(chan bool)
ctx, canelRoutine := context.WithCancel(context.Background())
defer canelRoutine()
go func(ctx context.Context) {
requestBody, _ := json.Marshal(map[string]string{
"Code": "print()",
})
select {
case <-ctx.Done():
return
default:
for {
_, err := http.Post(
fmt.Sprintf("http://%s:%d/execute/python", IP, 3000),
"application/json",
bytes.NewBuffer(requestBody),
)
if err == nil {
works <- true
return
}
}
}
}(ctx)
timer := time.After(500 * time.Millisecond)
select {
case <-works:
return IP, nil
case <-timer:
return IP, &UnreachableContainerError{name: schedulerName}
}
}
You could run the test code inside of a container, started via docker compose with the option depends_on.
I need my application to be able to load policies (rego) and then evaluate an input JSON based on defined policies.
I mistakenly used PrepareForEval in my evaluation API instead of load policy API. The result of this surprised me as the response time kept on increasing exponentially after each evaluation, while the policies were kept unchanged. Although, after this, I've realized and changed my logic to call PrepareForEval method at the time of loading policy and then store the prepared query instance in my struct instance. But I'm still concerned if prepare method gets executed multiple times while loading policies then it'd still become an expensive operation.
So, it'd be great to be pointed towards the correct way of using the prepare method.
Sample code:
// My provider
func init() {
cachedRego := rego.New(rego.Query("data.report"))
}
// My load policy method
func loadPolicy(ctx context.Context, filename, regoPolicy string) {
mod, err := ast.ParseModule(filename, regoPolicy)
rego.ParsedModule(mod)(cachedRego)
}
// My evaluate method
func eval(ctx context.Context, input interface{}) {
// after loading my policies, the following call took 1s, 2s, 5s, 10s,... respectively on eval calls
preparedQuery, _ := cachedRego.PrepareForEval(ctx) // <- I've moved this to my load policy method and cached preparedQuery
// this doesn’t take much time
rs, _ := preparedQuery.Eval(ctx, rego.EvalInput(input))
}
// My use case
func main() {
// load policies and evaluate inputs
for _, p := range policySet1 {
loadPolicy(context.Background(), p.filename, p.regoPolicy)
}
for _, inp := range inputSet1 {
eval(context.Background(), inp)
}
// load more policies to the earlier set and evaluate another input set
for _, p := range policySet2 {
loadPolicy(context.Background(), p.filename, p.regoPolicy)
}
for _, inp := range inputSet2 {
eval(context.Background(), inp)
}
}
TLDR; this is probably not the right place for this question. File an issue on GitHub if you are seeing behaviour you think is wrong.
To answer your question about how to prepare queries properly, your main function should (minimally) look like this:
func main() {
pq1 := loadPolicies(policySet1)
for _, inp := range inputSet1 {
eval(pq1, inp)
}
pq2 := loadPolicies(policySet2)
for _, inp := range inputSet2 {
eval(pq2, inp)
}
}
The easiest way to implement loadPolicies for the above example would be something like:
func loadPolicies(policySet) {
opts := []func(*rego.Rego){}
// setup prepared query. load all modules.
for _, p := range policySet {
opts = append(opts, rego.Module(p.filename, p.regoPolicy))
}
// create new prepared query
return rego.New(rego.Query("data.report"), opts...).PrepareForEval()
}
Then your eval function becomes:
func eval(pq, inp) {
rs, err := pq.Eval(rego.EvalInput(inp))
// handle err
// interpret rs
}
Using a URL that has worked in the past, I know receive a parsing error from net/url. What's wrong with it?
parse postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require: net/url: invalid userinfo
Sample application
See https://play.golang.com/p/mQZaN5JN3_q to run.
package main
import (
"fmt"
"net/url"
)
func main() {
dsn := "postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require"
u, err := url.Parse(dsn)
fmt.Println(u, err)
}
Well, you can just
url.QueryEscape("your#$%^&*(proper$#$%%^(password")
and use this one to parse your url.
It turns out up until Go v1.9.3 net/url didn't validate the user info when parsing a url. This may break existing applications when compiled using v1.9.4 if the username or password contain special characters.
It now expects the user info to be percent encoded string in order to handle special characters. The new behaviour got introduced in ba1018b.
Fixed sample application
package main
import (
"fmt"
"net/url"
)
func main() {
dsn1 := "postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require" // this works up until 1.9.3 but no longer in 1.9.4
dsn2 := "postgres://user:abc%7BDEf1=ghi#example.com:5432/db?sslmode=require" // this works everywhere, note { is now %7B
u, err := url.Parse(dsn1)
fmt.Println("1st url:\t", u, err)
u, err = url.Parse(dsn2)
fmt.Println("2nd url:\t", u, err)
}
Run the code on https://play.golang.com/p/jGIQgbiKZwz.
Use url.UserPassword func :
package main
import (
"fmt"
"net/url"
)
func main() {
dsn := "postgres://example.com:5432/db?sslmode=require"
u, err := url.Parse(dsn)
if err != nil {
fmt.Printf("ERROR: %v\n", err)
return
}
u.User = url.UserPassword("user", "abc{DEf1=ghi")
fmt.Println("url:\t", u)
}
I'm trying to compress a JPEG image in go using mozjpeg. Since it doesn't have official go binding, I think I'll just invoke its CLI to do the compression.
I try to model the usage after compress/gzip:
c := jpeg.NewCompresser(destFile)
_, err := io.Copy(c, srcFile)
Now the question is, how do I wrap the CLI inside Compresser so it can support this usage?
I tried something like this:
type Compresser struct {
cmd exec.Command
}
func NewCompressor(w io.Writer) *Compresser {
cmd := exec.Command("jpegtran", "-copy", "none")
cmd.Stdout = w
c := &Compresser{cmd}
return c
}
func (c *Compresser) Write(p []byte) (n int, err error) {
if c.cmd.Process == nil {
err = c.cmd.Start()
if err != nil {
return
}
}
// How do I write p into c.cmd.Stdin?
}
But couldn't finish it.
Also, a second question is, when do I shut down the command? How to shut down the command?
You should take a look at the Cmd.StdinPipe. There is an example in the documentation, which suits your case:
package main
import (
"fmt"
"io"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("cat")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
io.WriteString(stdin, "values written to stdin are passed to cmd's standard input")
}()
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
}
In this case, CombinedOutput() executes your command, and the execution is finished, when there are no more bytes to read from out.
As per Kiril's answer, use the cmd.StdInPipe to pass on the data you receive to Write.
However, in terms of closing, I'd be tempted to implement io.Closer. This would make *Compresser automatically implement the io.WriteCloser interface.
I would use Close() as the notification that there is no more data to be sent and that the command should be terminated. Any non-zero exit code returned from the command that indicates failure could be caught and returned as an error.
I would be wary of using CombinedOutput() inside Write() in case you have a slow input stream. The utility could finish processing the input stream and be waiting for more data. This would be incorrectly detected as command completion and would result in an invalid output.
Remember, the Write method can be called an indeterminate number of times during IO operations.
I have written a simple client/server in Go that will do an HTTP GET over TLS, but I'm trying to also make it capable of doing an HTTP POST over TLS.
In the example below index.html just contains the text hello, and the HTTP GET is working fine. I want the client to get the HTTP GET and write back, hello world to the server.
client
package main
import (
"crypto/tls"
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
link := "https://10.0.0.1/static/index.html"
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: tr}
response, err := client.Get(link)
if err != nil {
fmt.Println(err)
}
defer response.Body.Close()
content, _ := ioutil.ReadAll(response.Body)
s := strings.TrimSpace(string(content))
fmt.Println(s)
// out := s + " world"
// Not working POST...
// resp, err := client.Post(link, "text/plain", &out)
}
server
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
http.HandleFunc("/static/", func (w http.ResponseWriter, r *http.Request) {
fmt.Println("Got connection!")
http.ServeFile(w, r, r.URL.Path[1:])
})
log.Fatal(http.ListenAndServeTLS(":443", "server.crt", "server.key", nil))
}
I also currently have nothing to handle the POST on the server side, but I just want it to print it out to the screen so when I run the client I will see the server print hello world.
How should I fix my client code to do a proper POST? And what should the corresponding server code look like to accept the POST? Any help would be appreciated, I'm having trouble finding HTTPS/TLS POST examples.
You didn't share the error message, but I assume the client.Post call wasn't allowing a string as its third parameter, because it requires an io.Reader. Try this instead:
out := s + " world"
resp, err := client.Post(link, "text/plain", bytes.NewBufferString(out))
On the server side, you already have the right code set up to handle the POST request. Just check the method:
http.HandleFunc("/static/", func (w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" {
// handle POST requests
} else {
// handle all other requests
}
})
I noticed one other issue. Using index.html probably won't work here. http.ServeFile will redirect that path. See https://golang.org/pkg/net/http/#ServeFile:
As a special case, ServeFile redirects any request where r.URL.Path
ends in "/index.html" to the same path, without the final
"index.html". To avoid such redirects either modify the path or use
ServeContent.
I'd suggest just using a different file name to avoid that issue.