golang : trouble with memory - memory

I have trouble with memory. I don't understand why Go uses more and more memory (never freeing it) when my program runs for a long time.
After the first allocation, program uses nearly 9 MB of memory. Then after 12 hours it starts to use more memory exponentially, until 800 MB.
//.....code.....
if bol {
// Assignment Struct.Var
Struct_VastScript.TxtNoticeTop = JsonStruct_S.Options.TxtNoticeTop
Struct_VastScript.TxtNoticeBottom = JsonStruct_S.Options.TxtNoticeBottom
Struct_VastScript.Loop = JsonStruct_S.Options.Loop
Struct_Image, err := getImage(Struct_VastScript.Video)
if err == nil {
if mobile == "true" {
Struct_VastScript.Image = Struct_Image.URL360
}
}
//open and parse a template file
fi = path.Join("templates/VastPlayer", "TempVastPlayer.txt")
tmpl, err := template.ParseFiles(fi)
if err != nil {
job_1.Complete(health.Panic)
return false, err
}
//substitute fields in the template 'tmpl', with values from 'XmlStruct_V' and write it out to 'buf'
var buf bytes.Buffer
if err := tmpl.Execute(&buf, Struct_VastScript); err != nil {
//if err := tmpl.Execute(w, XmlStruct_V); err != nil {
job_1.Complete(health.Panic)
return false, err
}
// Call Func randString() : return alphanum random
dir := randString(12)
fpath := "http://creative2.xxx.io/api/html/" + dir
// Create a new EndPoint to write the generated 'template' on 'w' http.ResponseWriter
routeHtml := "/api/html/" + dir
http.HandleFunc(routeHtml, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
//writes Template to 'w' http.ResponseWriter
fmt.Fprintf(w, buf.String())
fmt.Println("successfull Operation 2 !!")
fmt.Println("")
job_2.Complete(health.Success)
}))
//Call Func JsonReply(): return the finale Json response
str := JsonReply(fpath, JsonStruct_S.Options.Animated, JsonStruct_S.Options.Responsive, JsonStruct_S.Options.Clickurl, JsonStruct_S.Options.Width, JsonStruct_S.Options.Height, adid, campaignid, JsonStruct_S.Type, JsonStruct_S.Options.Aspectratio, mobile)
w.Header().Set("Content-Type", "application/json")
//writes FinaleJson to 'w' http.ResponseWriter(it contains the link of the second endpoint "/api/html/")
fmt.Fprint(w, str)
fmt.Println("successfull Operation !!")
fmt.Println("")
job_1.Complete(health.Success)
return true, nil
} else {
return false, nil
}
For each call,my service need to generate a new template with the params that I receive,as you see I create a new endpoint for each call, I don't know if it's a good idea, I think the problem comes from this part of code but Im not sure because I don't know how GO manage it.

Obviously, you should not create handler every time request appears. They never free the memory so you will end up having out of memory exception.
Instead, put the handler endpoint into array (slice) and use ONE handler that responds to the request by looking the URL in this slice and then removing the item from the slice with it is not needed any longer.
So basically, instead of
routeHtml := "/api/html/" + dir
http.HandleFunc(routeHtml, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
//writes Template to 'w' http.ResponseWriter
fmt.Fprintf(w, buf.String())
fmt.Println("successfull Operation 2 !!")
fmt.Println("")
job_2.Complete(health.Success)
}))
do
type JobInfo struct {
Path string
// some data here
}
// maybe global context
var jobs []JobInfo
// initialisation
jobs = make([]JobInfo, 0)
http.HandleFunc("/api/html/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path
var job *JobInfo
for _, j := range jobs {
if j.Path == path {
job = &j
break
}
}
if job != nil {
// handle job request here
}
}))
// and then in the jobs' loop
handlers = append(handlers, JobInfo{"/api/html/" + dir, ...})
It will work because:
Patterns name fixed, rooted paths, like "/favicon.ico", or rooted subtrees, like "/images/" (note the trailing slash). Longer patterns take precedence over shorter ones, so that if there are handlers registered for both "/images/" and "/images/thumbnails/", the latter handler will be called for paths beginning "/images/thumbnails/" and the former will receive requests for any other paths in the "/images/" subtree.
Do not forget to clean the array jobs, of course.

Instead of using slice it's better to use map
type JobInfo struct {
Path string
// some data here
}
// global context
var jobs map[string]JobInfo
// initialisation
jobs = make(map[string]JobInfoStruct)
http.HandleFunc("/api/html/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path
var job JobInfoStruct
var ok bool
job, ok = jobs[path]
if ok {
// handle job request here
//then after delete the job
delete(jobs, path)
}
}))
// and then in the jobs' loop
pathVideo := "/api/html/" + dir
jobs[pathVideo] = JobInfoStruct{pathVideo, ...}

Related

Vuejs client can't communicate with Go backend via gRPC on MacOS

I built a simple web app with a VueJS front (client) and a Go back (server), using gRPC and protobuf.
In order to communicate, an envoy proxy has to be set up between them to convert web client HTTP/1.1 to HTTP/2.
I have no issue deploying this app on Linux, and have dockerized all three services.
However, I cannot manage to get my services to communicate on MacOS.
The app repository is available here: https://github.com/42Projects/nPuzzle
The deploy folder contains the docker-compose file, the envoy.yaml file required for the envoy container, as well as all three dockerfiles.
On MacOS the last line of the envoy.yaml needs to be updated, localhost has to be changed to host.docker.internal.
The client contains a ping function (in client/src/App.vue), the window.location.hostname might need to be changed to the docker-machine IP, unsure about that, my tries made no difference:
created () {
this.client = new NpuzzleClient('http://' + window.location.hostname + ':8080', null, null);
const ping = () => {
let message = new Message();
message.setMessage('ping');
this.client.greets(message, {}, err => this.serverOnline = !err);
};
ping();
window.setInterval(ping, 1000);
},
which is executed every second, and will display that the server is online on the top right corner. I have been so far unsuccessful in reaching my server while deploying on MacOS. It works perfectly fine on Linux.
Here is the code for the go server, I think it is correct to listen on localhost:9090 as the server will be dockerized, but I might be wrong and the address might need to be changed.
package main
import (
"context"
"flag"
"fmt"
pb "github.com/42Projects/nPuzzle/proto"
npuzzle "github.com/42Projects/nPuzzle/src"
"google.golang.org/grpc"
"log"
"net"
"time"
)
var port = flag.Int("port", 9090, "the server port")
type server struct{}
func (s *server) Greets(ctx context.Context, message *pb.Message) (*pb.Message, error) {
return &pb.Message{Message: "pong!"}, nil
}
func (s *server) Parse(ctx context.Context, message *pb.Message) (*pb.Matrix, error) {
log.Printf("received parsing request: %#v", message.Message)
m, err := npuzzle.ParseMatrix(message.Message)
if err != nil {
log.Println(err)
return &pb.Matrix{
Success: false,
Error: err.Error(),
}, nil
}
rows := make([]*pb.Matrix_Row, len(m))
for index, row := range m {
/* We need unsigned 32bits integer for protobuf */
uIntRow := make([]uint32, len(m))
for rowIndex, num := range row {
uIntRow[rowIndex] = uint32(num)
}
rows[index] = &pb.Matrix_Row{Num: uIntRow}
}
return &pb.Matrix{
Success: true,
Rows: rows,
}, nil
}
func (s *server) Solve(ctx context.Context, problem *pb.Problem) (*pb.Result, error) {
/* Choose heuristic function */
var heuristic npuzzle.Heuristic
switch problem.Heuristic {
case "hamming":
heuristic = npuzzle.HammingDistance
case "manhattan":
heuristic = npuzzle.ManhattanDistance
case "manhattan + linear conflicts":
heuristic = npuzzle.ManhattanPlusLinearConflicts
}
/* Choose between greedy search and uniform-cost search */
var goal npuzzle.Goal
var search npuzzle.Search
switch problem.Search {
case "greedy":
goal = npuzzle.GreedyGoalReached
search = npuzzle.GreedySearch
case "uniform-cost":
goal = npuzzle.UniformCostGoalReached
search = npuzzle.UniformCostSearch
}
/* Convert protobuf unsigned 32bits integer to regular integer */
size := len(problem.Rows)
m := make(npuzzle.Matrix, size)
for y, row := range problem.Rows {
m[y] = make([]int, size)
for x, num := range row.Num {
m[y][x] = int(num)
}
}
log.Printf("received problem:\n - heuristic: %v\n - search: %v\n - matrix: %v\n", problem.Heuristic, problem.Search, m)
if npuzzle.IsSolvable(m) == false {
log.Println("failed to solve problem: unsolvable")
return &pb.Result{
Success: false,
Error: "unsolvable",
}, nil
}
begin := time.Now()
log.Printf("starting solve on %v...", m)
res, totalNumberOfStates, maxNumberOfStates, err := m.Solve(heuristic, search, goal, 30*time.Second)
if err != nil {
log.Printf("timed ouf after %v", 30*time.Second)
return &pb.Result{
Success: false,
Error: fmt.Sprintf("timed ouf after %v", 30*time.Second),
}, nil
}
duration := time.Since(begin)
log.Printf("solved %v in %v seconds", m, duration)
var path string
if res.Parent == nil {
path = "already solved!"
} else {
path = npuzzle.StringifyPath(res)
}
return &pb.Result{
Success: true,
Time: duration.String(),
Moves: int32(res.Cost),
TotalStates: int32(totalNumberOfStates),
MaxStates: int32(maxNumberOfStates),
Path: path,
}, nil
}
func main() {
flag.Parse()
lis, err := net.Listen("tcp", fmt.Sprintf("localhost:%d", *port))
if err != nil {
log.Fatalf("failed to listen: %v\n", err)
}
s := grpc.NewServer()
pb.RegisterNpuzzleServer(s, &server{})
log.Printf("starting server on port %v\n", *port)
log.Fatalf("failed to serve: %v\n", s.Serve(lis))
}
I tried to launch the client and the server locally, and only containerize the envoy proxy, while trying different addresses (my docker machine IP address and localhost), but it didn't work. I also tried to launch all three containers but no result here either.
I'm unsure what to change to successfully manage to make my app work properly on MacOS.

Combine URL paths with path.Join()

Is there a way in Go to combine URL paths similarly as we can do with filepaths using path.Join()?
For example see e.g. Combine absolute path and relative path to get a new absolute path.
When I use path.Join("http://foo", "bar"), I get http:/foo/bar.
See in Golang Playground.
The function path.Join expects a path, not a URL. Parse the URL to get a path and join with that path:
u, err := url.Parse("http://foo")
if err != nil { log.Fatal(err) }
u.Path = path.Join(u.Path, "bar.html")
s := u.String()
fmt.Println(s) // prints http://foo/bar.html
Use the url.JoinPath function in Go 1.19 or later:
s, err := url.JoinPath("http://foo", "bar.html")
if err != nil { log.Fatal(err) }
fmt.Println(s) // prints http://foo/bar.html
Use ResolveReference if you are resolving a URI reference from a base URL. This operation is different from a simple path join: an absolute path in the reference replaces the entire base path; the base path is trimmed back to the last slash before the join operation.
base, err := url.Parse("http://foo/quux.html")
if err != nil {
log.Fatal(err)
}
ref, err := url.Parse("bar.html")
if err != nil {
log.Fatal(err)
}
u := base.ResolveReference(ref)
fmt.Println(u.String()) // prints http://foo/bar.html
Notice how quux.html in the base URL does not appear in the resolved URL.
ResolveReference() in net/url package
The accepted answer will not work for relative url paths containing file endings like .html or .img. The ResolveReference() function is the correct way to join url paths in go.
package main
import (
"fmt"
"log"
"net/url"
)
func main() {
u, err := url.Parse("../../..//search?q=dotnet")
if err != nil {
log.Fatal(err)
}
base, err := url.Parse("http://example.com/directory/")
if err != nil {
log.Fatal(err)
}
fmt.Println(base.ResolveReference(u))
}
A simple approach to this would be to trim the /'s you don't want and join. Here is an example func
func JoinURL(base string, paths ...string) string {
p := path.Join(paths...)
return fmt.Sprintf("%s/%s", strings.TrimRight(base, "/"), strings.TrimLeft(p, "/"))
}
Usage would be
b := "http://my.domain.com/api/"
u := JoinURL(b, "/foo", "bar/", "baz")
fmt.Println(u)
This removes the need for checking/returning errors
In 1.19 there will be a new function in the standard library that solves this very neatly.
u, err := url.JoinPath("http://host/foo", "bar/")
https://go.dev/play/p/g422ockBq0q?v=gotip
To join a URL with another URL or a path, there is URL.Parse():
func (u *URL) Parse(ref string) (*URL, error)
Parse parses a URL in the context of the receiver. The provided URL
may be relative or absolute. Parse returns nil, err on parse failure,
otherwise its return value is the same as ResolveReference.
func TestURLParse(t *testing.T) {
baseURL, _ := url.Parse("http://foo/a/b/c")
url1, _ := baseURL.Parse("d/e")
require.Equal(t, "http://foo/a/b/d/e", url1.String())
url2, _ := baseURL.Parse("../d/e")
require.Equal(t, "http://foo/a/d/e", url2.String())
url3, _ := baseURL.Parse("/d/e")
require.Equal(t, "http://foo/d/e", url3.String())
}
I wrote this utility function that works for my purposes:
func Join(basePath string, paths ...string) (*url.URL, error) {
u, err := url.Parse(basePath)
if err != nil {
return nil, fmt.Errorf("invalid url")
}
p2 := append([]string{u.Path}, paths...)
result := path.Join(p2...)
u.Path = result
return u, nil
}
https://play.golang.org/p/-QNVvyzacMM

Using a struct with map data

I'm relatively new to Go, I'm struggling to get my head around using POST data with structs. What I essentially want to do is submit a form, then submit that form to MongoDB (haven't got that far yet). I can't work out how to use this form data with a struct.
package main
import "net/http"
type Paste struct {
Title string
Content string
Language string
Expires int
Created int64
}
func index(w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" {
r.ParseForm()
// r.Form = map[title:[Wadup] content:[Brother]]
}
}
func main() {
http.HandleFunc("/", index)
http.ListenAndServe(":1234", nil)
}
What I basically want to do is insert the map values into the struct, without manually assigning all of them, like you can see here: p := Paste{Title: r.Form["title"]}
Use gorilla/schema, which was built for this use-case:
package main
import(
"net/http"
"github.com/gorilla/schema"
)
type Paste struct {
Title string
Content string
Language string
Expires int
Created int64
}
var decoder = schema.NewDecoder()
func index(w http.ResponseWriter, r *http.Request) {
err := r.ParseForm()
// handle error
var paste = &Paste{}
err := decoder.Decode(paste, r.PostForm)
// handle error
}
After you called r.ParseForm() you can access r.Form which holds a map[string][]string of the parsed form data. This map can be accessed by using keys (your form field names):
r.ParseForm()
form := r.Form
someField := form["someField"]
anotherField := form["anotherField"]
or loop through all available keys/ values:
r.ParseForm()
form := r.Form
for key, value := range form {
fmt.Println("Key:", key, "Value:", value)
}
Update
In case you want a more generic solution take a look at the reflect package. Just to give you a rough example it could look like this:
v := url.Values{}
v.Set("Title", "Your Title")
v.Set("Content", "Your Content")
v.Set("Language", "English")
v.Set("Expires", "2015")
v.Set("Created", "2014")
paste := Paste{}
ps := reflect.ValueOf(&paste)
s := ps.Elem()
for key, value := range v {
f := s.FieldByName(key)
if f.IsValid() && f.CanSet() {
switch f.Kind() {
case reflect.String:
f.SetString(value[0])
case reflect.Int64:
i, _ := strconv.ParseInt(value[0], 0, 64)
f.SetInt(i)
}
}
}
Play

Parsing URL with matrix paramers using net/url built-in package

It seems that URL does not support matrix parameters
// From net/url
type URL struct {
Scheme string
Opaque string // encoded opaque data
User *Userinfo // username and password information
Host string // host or host:port
Path string
RawQuery string // encoded query values, without '?'
Fragment string // fragment for references, without '#'
}
Why ?
How can I extract matrix parameters from an URL ? and when should I use them instead of using requests parameters embedded in the request.URL.RawQuery part of the URL ?
The parameters end up getting put in url.Path. Here's a function which can put them in the Query for you:
func ParseWithMatrix(u string) (*url.URL, error) {
parsed, err := url.Parse(u)
if err != nil {
return nil, err
}
if strings.Contains(parsed.Path, ";") {
q := parsed.Path[strings.Index(parsed.Path, ";")+1:]
m, err := url.ParseQuery(q)
if err != nil {
return nil, err
}
for k, vs := range parsed.Query() {
for _, v := range vs {
m.Add(k, v)
}
}
parsed.Path = parsed.Path[:strings.Index(parsed.Path, ";")]
parsed.RawQuery = m.Encode()
}
return parsed, nil
}

Why is this iteration of an Outlook Folder only processing a maximum of half the number of items in the folder?

Outlook rules are putting all Facebook originating mail into a Facebook folder, an external process is running as detailed here to separate the contents of that folder in a way that was not feasible through Outlook rules process, originally I had this process running in VBA in outlook but it was a pig choking outlook resources. So I decided to throw it out externally and as I want to improve my c# skill set, this would be a conversion at the same time. Anyway the mail processing is working as it should items are going to correct sub-folders but for some reason the temporary constraint to exit after i number of iterations is not doing as it should. If there are 800 mails in the Facebook folder ( I am a member of many groups) it only runs through 400 iterations, if there are 30 it only processes 15 etc.
I cant for the life of me see why - can anyone put me right?
Thanks
private void PassFBMail()
{
//do something
// result = MsgBox("Are you sure you wish to run the 'Allocate to FB Recipient' process", vbOKCancel, "Hold up")
//If result = Cancel Then Exit Sub
var result = MessageBox.Show("Are you sure you wish to run the Are you sure you wish to run the 'Allocate to SubFolders' process","Sure?",MessageBoxButtons.OKCancel,MessageBoxIcon.Question,MessageBoxDefaultButton.Button2);
if (result == DialogResult.Cancel)
{
return;
}
try
{
OutLook._Application outlookObj = new OutLook.Application();
OutLook.MAPIFolder inbox = (OutLook.MAPIFolder)
outlookObj.Session.GetDefaultFolder(OutLook.OlDefaultFolders.olFolderInbox);
OutLook.MAPIFolder fdr = inbox.Folders["facebook"];
OutLook.MAPIFolder fdrForDeletion = inbox.Folders["_ForDeletion"];
// foreach (OutLook.MAPIFolder fdr in inbox.Folders)
// {
// if (fdr.Name == "facebook")
// {
// break;
// }
// }
//openFacebookFolder Loop through mail
//LOOPING THROUGH MAIL ITEMS IN THAT FOLDER.
Redemption.SafeMailItem sMailItem = new Redemption.SafeMailItem();
int i = 0;
foreach ( Microsoft.Office.Interop.Outlook._MailItem mailItem in fdr.Items.Restrict("[MessageClass] = 'IPM.Note'"))
{
//temp only process 500 mails
i++;
if (i == 501)
{
break;
}
// eml.Item = em
// If eml.To <> "" And eml.ReceivedByName <> "" Then
// strNewFolder = DeriveMailFolder(eml.To, eml.ReceivedByName)
// End If
sMailItem.Item = mailItem;
string strTgtFdr = null;
if (sMailItem.To != null && sMailItem.ReceivedByName != null)
{
strTgtFdr = GetTargetFolder(sMailItem.To, sMailItem.ReceivedByName );
}
// If fdr.Name <> strNewFolder Then
// If dDebug Then DebugPrint "c", "fdr.Name <> strNewFolder"
// eml.Move myInbox.Folders(strNewFolder)
// If dDebug Then DebugPrint "w", "myInbox.Folders(strNewFolder) = " & myInbox.Folders(strNewFolder)
// Else
// eml.Move myInbox.Folders("_ForDeletion")
// End If
if (fdr.Name != strTgtFdr)
{
OutLook.MAPIFolder destFolder = inbox.Folders[strTgtFdr];
mailItem.Move(destFolder);
}
else
{
mailItem.Move(fdrForDeletion);
}
}
//allocate to subfolders
//Process othersGroups
//Likes Max 3 per day per user, max 20% of group posts
//Comments one per day per user, max 10% of group posts
//Shares one per day per user, max 10% of group posts
}
catch(System.Exception crap)
{
OutCrap(crap);
MessageBox.Show("MailCamp experienced an issues while processing the run request and aborted - please review the error log","Errors during the process",MessageBoxButtons.OK,MessageBoxIcon.Error,MessageBoxDefaultButton.Button1);
}
}
Do not use a foreach loop it you are modifying the number of items in the collection.
Loop from MAPIFolder.Items.Count down to 1.

Resources