Accessing to a comment within a function in Go - parsing

I'm currently working on documentation generator that will parse Go code to produce a custom documentation. I need to access to the comments written inside a function. But I can't retrieve these comments in the AST, neither with the go/doc. Here is an example :
package main
import (
"fmt"
"go/doc"
"go/parser"
"go/token"
)
// GetFoo comments I can find easely
func GetFoo() {
// Comment I would like to access
test := 1
fmt.Println(test)
}
func main() {
fset := token.NewFileSet() // positions are relative to fset
d, err := parser.ParseDir(fset, "./", nil, parser.ParseComments)
if err != nil {
fmt.Println(err)
return
}
for k, f := range d {
fmt.Println("package", k)
p := doc.New(f, "./", 2)
for _, t := range p.Types {
fmt.Println("type", t.Name)
fmt.Println("docs:", t.Doc)
}
for _, v := range p.Vars {
fmt.Println("type", v.Names)
fmt.Println("docs:", v.Doc)
}
for _, f := range p.Funcs {
fmt.Println("type", f.Name)
fmt.Println("docs:", f.Doc)
}
for _, n := range p.Notes {
fmt.Println("body", n[0].Body)
}
}
}
It's easy to find GetFoo comments I can find easely but not Comment I would like to access
I have seen this post quite similar question Go parser not detecting Doc comments on struct type but for exported types
Is there any way to do that ? Thank you !

The problem is that the doc.New functionality is only parsing for documentation strings, and the comment inside the function is not part of the "documentation".
You'll want to directly iterate the ast of the files in the package.
package main
import (
"fmt"
"go/parser"
"go/token"
)
// GetFoo comments I can find easely
func GetFoo() {
// Comment I would like to access
test := 1
fmt.Println(test)
}
func main() {
fset := token.NewFileSet() // positions are relative to fset
d, err := parser.ParseDir(fset, "./", nil, parser.ParseComments)
if err != nil {
fmt.Println(err)
return
}
for k, f := range d {
fmt.Println("package", k)
for n, f := range f.Files {
fmt.Printf("File name: %q\n", n)
for i, c := range f.Comments {
fmt.Printf("Comment Group %d\n", i)
for i2, c1 := range c.List {
fmt.Printf("Comment %d: Position: %d, Text: %q\n", i2, c1.Slash, c1.Text)
}
}
}
}
}

Related

Mapping items in an array

Hoping this is a nice easy one, but I just can't see how to do it.
I am wanting to with rego map items in an array to a cleaner version. For example from the data below
data = [
{
"some": "value",
"another": "mvalue",
"dont": "want"
},
{
"some": "value1",
"another": "mvalue1",
"dont": "want1"
},
{
"some": "value2",
"another": "mvalue2",
"dont": "want2"
}
]
I want to turn data into
result = [
{
"some": "value",
"another": "mvalue"
},
{
"some": "value1",
"another": "mvalue1"
},
{
"some": "value2",
"another": "mvalue2"
}
]
The two closest I think I've got is
result1 = cleaned {
cleaned := {d |
d := {
"some": data[_].some,
"another": data[_].another
}
}
}
result2 = cleaned {
d := data[_]
cleaned := {
"some": p.some,
"another": p.another
}
}
TLDR; If the fields are static and you can easily enumerate them, both of your solutions is almost correct (see below for explanation of why they are incorrect.) Here's the right way to do that:
result = [
mapped |
original := data[_]
mapped := {"some": original["some"], "another": original.another}
]
A slightly more elegant option is to define the fields to include or exclude like in #eephillip's example. For instance:
result = [
mapped |
original := data[_]
mapped := {k: v |
some k
v := original[k]
not exclude[k]}
]
exclude = {"dont", "dont2"} # defines a SET of keys to exclude
Of course you could generalize it even more by making the inner comprehension invoke a function that implements other filters.
Here's an interactive example: https://play.openpolicyagent.org/p/P6wPd3rudJ
Two notes about the original solution.
1. result1 does not iterate over data correctly
{d |
d := {
"some": data[_].some, # problem: _ is a different variable in each expression
"another": data[_].another
}
}
Conceptually each occurrence of _ is a unique variable. If you explicitly declare the variables, the problem is more obvious:
# note: this is still wrong
{d |
some i, j
d := {
"some": data[i]["some"],
"another": data[j].another
}
}
If you run this, you'll discover that it produces a cross-product (which is not what you want). You want the "some" and "another" fields to be selected from the same object like this:
{d |
some i
d := {
"some": data[i]["some"],
"another": data[i].another
}
}
Of course, coming up with unique variable names can be a pain, so you can use _. Just do not mistake multiple _ variables as referring to the same value. We can rewrite the statement to use _ as follows:
{d |
obj := data[_]
d := {
"some": obj["some"],
"another": obj.another
}
}
result2 is close but may assign multiple values (which should be avoided)
result2 = cleaned {
d := data[_]
cleaned := { # problem: there could be multiple values for 'cleaned'
"some": d["some"],
"another": d.another
}
}
Rules of the form NAME = VALUE { BODY } assign VALUE to NAME if the statements in BODY are satisfied. If you omit BODY, i.e., you write NAME = VALUE, then BODY defaults to true (which is always satisfied.)
In your above example:
NAME is result2
VALUE is cleaned
BODY is d := data[_]; cleaned := {...}
In Rego, we call these rules "complete rules". Complete rules are just IF-THEN statements that assign a single value to a variable. The "IF" portion is the rule body and the "THEN" portion is the assignment. You should avoid writing rules that may assign MULTIPLE values to the same variable because that may result in an evaluation time error. For example:
# do not do this
result = v {
v := data[_] # if 'data' is [1,2,3] then what is the value of 'result'? Hint: There is more than one answer.
}
If you want to assign MULTIPLE values to a variable then you can define a "partial rule" For example:
result[v] { # result is a SET.
v := data[_]
}
What about performing a rejection of the key name during the comprehensions?
Probably a more elegant way to do this, but might be helpful.
package play
reject(key) = result {
remove := [ "dont", "someotherthing" ]
result := key == remove[_]
}
result = d {
d := [ obj |
val := input.data[_];
obj := { k: v |
v := val[k]
not reject(k)
}
]
}
https://play.openpolicyagent.org/p/1A3DNLiNfj

Mask sensitive url query params

Say I have this url
https://example.com:8080?private-token=foo&authenticity_token=bar
And I have a function to determine whether to mask a param.
How can I mask the url, but maintaining the order of params.
Currently I have
u, err := url.Parse(originalURL)
if err != nil {
panic(err)
}
m, _ := url.ParseQuery(u.RawQuery)
for key := range m {
if toMask(key) {
m.Set(key, "FILTERED")
}
}
u.RawQuery = m.Encode()
return u.String()
But this would return url with the params being switched around.
https://example.com:8080?authenticity_token=FILTERED&private-token=FILTERED
First, the order of the params should not be of any importance.
But I can see some situation where this rule does not apply (eg when you hash an URL). In this case, you should normalize the URL before using it.
Finally to respond to your question, you cannot keep the order if using Query, as Values is a map, and map don't bother with ordering. You should thus work on the query using u.RawQuery.
u, err := url.Parse(originalURL)
if err != nil {
panic(err)
}
newQuery := ""
for i, queryPart := range strings.Split(u.RawQuery, ";") {
// you now have a slice of string ["private-token=foo", "authenticity_token=bar"]
splitParam := strings.Split(queryPart, "=")
if toMask(splitParam[0]) {
splitParam[1] = "FILTERED"
}
if i != 0 {
newQuery = newQuery + ";"
}
newQuery = splitParam[0] + "=" + splitParam[1]
}
u.RawQuery = newQuery
return u.String()
This code is just example. You have to better check for special cases or errors. You can also use regexp if you want to.

How to parse result from ovs-vsctl get interface <interface> statistics

Result example:
{collisions=0, rx_bytes=258, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=3, tx_bytes=648, tx_dropped=0, tx_errors=0, tx_packets=8}
This format is like JSON, but not JSON.
Is there an easy way to parse this into map[string]int? Like json.Unmarshal(data, &value).
If that transport format is not recursively defined, i.e. a key cannot start a sub-structure, then its language is regular. As such, you can soundly parse it with Go's standard regexp package:
Playground link.
package main
import (
"fmt"
"regexp"
"strconv"
)
const data = `{collisions=0, rx_bytes=258, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=3, tx_bytes=648, tx_dropped=0, tx_errors=0, tx_packets=8}`
const regex = `([a-z_]+)=([0-9]+)`
func main() {
ms := regexp.MustCompile(regex).FindAllStringSubmatch(data, -1)
vs := make(map[string]int)
for _, m := range ms {
v, _ := strconv.Atoi(m[2])
vs[m[1]] = v
}
fmt.Printf("%#v\n", vs)
}
Regexp by #thwd is an elegant solution.
You can get a more efficient solution by using strings.Split() to split by comma-space (", ") to get the pairs, and then split again by the equal sign ("=") to get the key-value pairs. After that you just need to put these into a map:
func Parse(s string) (m map[string]int, err error) {
if len(s) < 2 || s[0] != '{' || s[len(s)-1] != '}' {
return nil, fmt.Errorf("Invalid input, no wrapping brackets!")
}
m = make(map[string]int)
for _, v := range strings.Split(s[1:len(s)-1], ", ") {
parts := strings.Split(v, "=")
if len(parts) != 2 {
return nil, fmt.Errorf("Equal sign not found in: %s", v)
}
if m[parts[0]], err = strconv.Atoi(parts[1]); err != nil {
return nil, err
}
}
return
}
Using it:
s := "{collisions=0, rx_bytes=258, ...}"
fmt.Println(Parse(s))
Try it on the Go Playground.
Note: If performance is important, this can be improved by not using strings.Split() in the outer loop, but instead searching for the comma "manually" and maintaining indices, and only "take out" substrings that represent actual keys and values (but this solution would be more complex).
Another solution...
...but this option is much slower so it is only viable if performance is not a key requirement: you can turn your input string into a valid JSON format and after that you can use json.Unmarshal(). Error checks omitted:
s := "{collisions=0, rx_bytes=258, ...}"
// Turn into valid JSON:
s = strings.Replace(s, `=`, `":`, -1)
s = strings.Replace(s, `, `, `, "`, -1)
s = strings.Replace(s, `{`, `{"`, -1)
// And now simply unmarshal:
m := make(map[string]int)
json.Unmarshal([]byte(s), &m)
fmt.Println(m)
Advantage of this solution is that this also works if the destination value you unmarhsal into is a struct:
// Unmarshal into a struct (you don't have to care about all fields)
st := struct {
Collisions int `json:"collisions"`
Rx_bytes int `json:"rx_bytes"`
}{}
json.Unmarshal([]byte(s), &st)
fmt.Printf("%+v\n", st)
Try these on the Go Playground.

time.Parse with custom layout

I'm trying to parse this string pattern "4-JAN-12 9:30:14" into a time.Time.
Tried time.Parse("2-JAN-06 15:04:05", inputString) and many others but cannot get it working.
I've read http://golang.org/pkg/time/#Parse and https://gobyexample.com/time-formatting-parsing but it seems there aren't any examples like this.
Thanks!
Edit:
full code:
type CustomTime time.Time
func (t *CustomTime) UnmarshalJSON(b []byte) error {
auxTime, err := time.Parse("2-JAN-06 15:04:05", string(b))
*t = CustomTime(auxTime)
return err
}
parsing time ""10-JAN-12 11:20:41"" as "2-JAN-06 15:04:05": cannot
parse ""24-JAN-15 10:27:44"" as "2"
Don't know what you did wrong (should post your code), but it is really just a simple function call:
s := "4-JAN-12 9:30:14"
t, err := time.Parse("2-JAN-06 15:04:05", s)
fmt.Println(t, err)
Outputs:
2012-01-04 09:30:14 +0000 UTC <nil>
Try it on the Go Playground.
Note that time.Parse() returns 2 values: the parsed time.Time value (if parsing succeeds) and an optional error value (if parsing fails).
See the following example where I intentionally specify a wrong input string:
s := "34-JAN-12 9:30:14"
if t, err := time.Parse("2-JAN-06 15:04:05", s); err == nil {
fmt.Println("Success:", t)
} else {
fmt.Println("Failure:", err)
}
Output:
Failure: parsing time "34-JAN-12 9:30:14": day out of range
Try it on the Go Playground.
EDIT:
Now that you posted code and error message, your problem is that your input string contains a leading and trailing quotation mark!
Remove the leading and trailing quotation mark and it will work. This is your case:
s := `"4-JAN-12 9:30:14"`
s = s[1 : len(s)-1]
if t, err := time.Parse("2-JAN-06 15:04:05", s); err == nil {
fmt.Println("Success:", t)
} else {
fmt.Println("Failure:", err)
}
Output (try it on the Go Playground):
Success: 2012-01-04 09:30:14 +0000 UTC

How can i get the content of an html.Node

I would like to get data from a URL using the GO 3rd party library from http://godoc.org/code.google.com/p/go.net/html . But I came across a problem, that is I couldn't get the content of an html.Node.
There's an example code in the reference document, and here's the code.
s := `<p>Links:</p><ul><li>Foo<li>BarBaz</ul>`
doc, err := html.Parse(strings.NewReader(s))
if err != nil {
log.Fatal(err)
}
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
for _, a := range n.Attr {
if a.Key == "href" {
fmt.Println(a.Val)
break
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}
f(doc)
The output is:
foo
/bar/baz
If I want to get
Foo
BarBaz
What should I do?
The tree of <strong>Foo</strong>Bar looks basically like this:
ElementNode "a" (this node also includes a list off attributes)
ElementNode "strong"
TextNode "Foo"
TextNode "Bar"
So, assuming that you want to get the plain text of the link (e.g. FooBar) you would have to walk trough the tree and collect all text nodes. For example:
func collectText(n *html.Node, buf *bytes.Buffer) {
if n.Type == html.TextNode {
buf.WriteString(n.Data)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
collectText(c, buf)
}
}
And the changes in your function:
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
text := &bytes.Buffer{}
collectText(n, text)
fmt.Println(text)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}

Resources