Mask sensitive url query params - url

Say I have this url
https://example.com:8080?private-token=foo&authenticity_token=bar
And I have a function to determine whether to mask a param.
How can I mask the url, but maintaining the order of params.
Currently I have
u, err := url.Parse(originalURL)
if err != nil {
panic(err)
}
m, _ := url.ParseQuery(u.RawQuery)
for key := range m {
if toMask(key) {
m.Set(key, "FILTERED")
}
}
u.RawQuery = m.Encode()
return u.String()
But this would return url with the params being switched around.
https://example.com:8080?authenticity_token=FILTERED&private-token=FILTERED

First, the order of the params should not be of any importance.
But I can see some situation where this rule does not apply (eg when you hash an URL). In this case, you should normalize the URL before using it.
Finally to respond to your question, you cannot keep the order if using Query, as Values is a map, and map don't bother with ordering. You should thus work on the query using u.RawQuery.
u, err := url.Parse(originalURL)
if err != nil {
panic(err)
}
newQuery := ""
for i, queryPart := range strings.Split(u.RawQuery, ";") {
// you now have a slice of string ["private-token=foo", "authenticity_token=bar"]
splitParam := strings.Split(queryPart, "=")
if toMask(splitParam[0]) {
splitParam[1] = "FILTERED"
}
if i != 0 {
newQuery = newQuery + ";"
}
newQuery = splitParam[0] + "=" + splitParam[1]
}
u.RawQuery = newQuery
return u.String()
This code is just example. You have to better check for special cases or errors. You can also use regexp if you want to.

Related

Accessing to a comment within a function in Go

I'm currently working on documentation generator that will parse Go code to produce a custom documentation. I need to access to the comments written inside a function. But I can't retrieve these comments in the AST, neither with the go/doc. Here is an example :
package main
import (
"fmt"
"go/doc"
"go/parser"
"go/token"
)
// GetFoo comments I can find easely
func GetFoo() {
// Comment I would like to access
test := 1
fmt.Println(test)
}
func main() {
fset := token.NewFileSet() // positions are relative to fset
d, err := parser.ParseDir(fset, "./", nil, parser.ParseComments)
if err != nil {
fmt.Println(err)
return
}
for k, f := range d {
fmt.Println("package", k)
p := doc.New(f, "./", 2)
for _, t := range p.Types {
fmt.Println("type", t.Name)
fmt.Println("docs:", t.Doc)
}
for _, v := range p.Vars {
fmt.Println("type", v.Names)
fmt.Println("docs:", v.Doc)
}
for _, f := range p.Funcs {
fmt.Println("type", f.Name)
fmt.Println("docs:", f.Doc)
}
for _, n := range p.Notes {
fmt.Println("body", n[0].Body)
}
}
}
It's easy to find GetFoo comments I can find easely but not Comment I would like to access
I have seen this post quite similar question Go parser not detecting Doc comments on struct type but for exported types
Is there any way to do that ? Thank you !
The problem is that the doc.New functionality is only parsing for documentation strings, and the comment inside the function is not part of the "documentation".
You'll want to directly iterate the ast of the files in the package.
package main
import (
"fmt"
"go/parser"
"go/token"
)
// GetFoo comments I can find easely
func GetFoo() {
// Comment I would like to access
test := 1
fmt.Println(test)
}
func main() {
fset := token.NewFileSet() // positions are relative to fset
d, err := parser.ParseDir(fset, "./", nil, parser.ParseComments)
if err != nil {
fmt.Println(err)
return
}
for k, f := range d {
fmt.Println("package", k)
for n, f := range f.Files {
fmt.Printf("File name: %q\n", n)
for i, c := range f.Comments {
fmt.Printf("Comment Group %d\n", i)
for i2, c1 := range c.List {
fmt.Printf("Comment %d: Position: %d, Text: %q\n", i2, c1.Slash, c1.Text)
}
}
}
}
}

How to parse result from ovs-vsctl get interface <interface> statistics

Result example:
{collisions=0, rx_bytes=258, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=3, tx_bytes=648, tx_dropped=0, tx_errors=0, tx_packets=8}
This format is like JSON, but not JSON.
Is there an easy way to parse this into map[string]int? Like json.Unmarshal(data, &value).
If that transport format is not recursively defined, i.e. a key cannot start a sub-structure, then its language is regular. As such, you can soundly parse it with Go's standard regexp package:
Playground link.
package main
import (
"fmt"
"regexp"
"strconv"
)
const data = `{collisions=0, rx_bytes=258, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=3, tx_bytes=648, tx_dropped=0, tx_errors=0, tx_packets=8}`
const regex = `([a-z_]+)=([0-9]+)`
func main() {
ms := regexp.MustCompile(regex).FindAllStringSubmatch(data, -1)
vs := make(map[string]int)
for _, m := range ms {
v, _ := strconv.Atoi(m[2])
vs[m[1]] = v
}
fmt.Printf("%#v\n", vs)
}
Regexp by #thwd is an elegant solution.
You can get a more efficient solution by using strings.Split() to split by comma-space (", ") to get the pairs, and then split again by the equal sign ("=") to get the key-value pairs. After that you just need to put these into a map:
func Parse(s string) (m map[string]int, err error) {
if len(s) < 2 || s[0] != '{' || s[len(s)-1] != '}' {
return nil, fmt.Errorf("Invalid input, no wrapping brackets!")
}
m = make(map[string]int)
for _, v := range strings.Split(s[1:len(s)-1], ", ") {
parts := strings.Split(v, "=")
if len(parts) != 2 {
return nil, fmt.Errorf("Equal sign not found in: %s", v)
}
if m[parts[0]], err = strconv.Atoi(parts[1]); err != nil {
return nil, err
}
}
return
}
Using it:
s := "{collisions=0, rx_bytes=258, ...}"
fmt.Println(Parse(s))
Try it on the Go Playground.
Note: If performance is important, this can be improved by not using strings.Split() in the outer loop, but instead searching for the comma "manually" and maintaining indices, and only "take out" substrings that represent actual keys and values (but this solution would be more complex).
Another solution...
...but this option is much slower so it is only viable if performance is not a key requirement: you can turn your input string into a valid JSON format and after that you can use json.Unmarshal(). Error checks omitted:
s := "{collisions=0, rx_bytes=258, ...}"
// Turn into valid JSON:
s = strings.Replace(s, `=`, `":`, -1)
s = strings.Replace(s, `, `, `, "`, -1)
s = strings.Replace(s, `{`, `{"`, -1)
// And now simply unmarshal:
m := make(map[string]int)
json.Unmarshal([]byte(s), &m)
fmt.Println(m)
Advantage of this solution is that this also works if the destination value you unmarhsal into is a struct:
// Unmarshal into a struct (you don't have to care about all fields)
st := struct {
Collisions int `json:"collisions"`
Rx_bytes int `json:"rx_bytes"`
}{}
json.Unmarshal([]byte(s), &st)
fmt.Printf("%+v\n", st)
Try these on the Go Playground.

time.Parse with custom layout

I'm trying to parse this string pattern "4-JAN-12 9:30:14" into a time.Time.
Tried time.Parse("2-JAN-06 15:04:05", inputString) and many others but cannot get it working.
I've read http://golang.org/pkg/time/#Parse and https://gobyexample.com/time-formatting-parsing but it seems there aren't any examples like this.
Thanks!
Edit:
full code:
type CustomTime time.Time
func (t *CustomTime) UnmarshalJSON(b []byte) error {
auxTime, err := time.Parse("2-JAN-06 15:04:05", string(b))
*t = CustomTime(auxTime)
return err
}
parsing time ""10-JAN-12 11:20:41"" as "2-JAN-06 15:04:05": cannot
parse ""24-JAN-15 10:27:44"" as "2"
Don't know what you did wrong (should post your code), but it is really just a simple function call:
s := "4-JAN-12 9:30:14"
t, err := time.Parse("2-JAN-06 15:04:05", s)
fmt.Println(t, err)
Outputs:
2012-01-04 09:30:14 +0000 UTC <nil>
Try it on the Go Playground.
Note that time.Parse() returns 2 values: the parsed time.Time value (if parsing succeeds) and an optional error value (if parsing fails).
See the following example where I intentionally specify a wrong input string:
s := "34-JAN-12 9:30:14"
if t, err := time.Parse("2-JAN-06 15:04:05", s); err == nil {
fmt.Println("Success:", t)
} else {
fmt.Println("Failure:", err)
}
Output:
Failure: parsing time "34-JAN-12 9:30:14": day out of range
Try it on the Go Playground.
EDIT:
Now that you posted code and error message, your problem is that your input string contains a leading and trailing quotation mark!
Remove the leading and trailing quotation mark and it will work. This is your case:
s := `"4-JAN-12 9:30:14"`
s = s[1 : len(s)-1]
if t, err := time.Parse("2-JAN-06 15:04:05", s); err == nil {
fmt.Println("Success:", t)
} else {
fmt.Println("Failure:", err)
}
Output (try it on the Go Playground):
Success: 2012-01-04 09:30:14 +0000 UTC

How can i get the content of an html.Node

I would like to get data from a URL using the GO 3rd party library from http://godoc.org/code.google.com/p/go.net/html . But I came across a problem, that is I couldn't get the content of an html.Node.
There's an example code in the reference document, and here's the code.
s := `<p>Links:</p><ul><li>Foo<li>BarBaz</ul>`
doc, err := html.Parse(strings.NewReader(s))
if err != nil {
log.Fatal(err)
}
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
for _, a := range n.Attr {
if a.Key == "href" {
fmt.Println(a.Val)
break
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}
f(doc)
The output is:
foo
/bar/baz
If I want to get
Foo
BarBaz
What should I do?
The tree of <strong>Foo</strong>Bar looks basically like this:
ElementNode "a" (this node also includes a list off attributes)
ElementNode "strong"
TextNode "Foo"
TextNode "Bar"
So, assuming that you want to get the plain text of the link (e.g. FooBar) you would have to walk trough the tree and collect all text nodes. For example:
func collectText(n *html.Node, buf *bytes.Buffer) {
if n.Type == html.TextNode {
buf.WriteString(n.Data)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
collectText(c, buf)
}
}
And the changes in your function:
var f func(*html.Node)
f = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
text := &bytes.Buffer{}
collectText(n, text)
fmt.Println(text)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
}

JSON2 Error / Conflict with another script

I am using the JSON2 library in order to use JSON.stringify to send some JSON data to my MVC controller.
When I include another script in my view (Telerik MVC) I start to get script conflicts when using IE7.
When I click the refresh button in the grid, I get the following error:
Line: 191
Error: Object doesn't support this property or method
String.prototype.toJSON =
Number.prototype.toJSON =
Boolean.prototype.toJSON = function (key) {
return this.valueOf();
};
The error occurs on the following line specifically:
return this.valueOf();
Does anyone have any insight into why this conflict is occurring and how to resolve it? Specifically, why would this work in IE8/Chrome but fail in IE7. What would cause the error? Are both scripts trying to define the same method and that's why it is failing or is it impossible to tell without digging through tons of code?
Edit:
This is the json2.js library I am speaking of: https://github.com/douglascrockford/JSON-js
Probably the reply is too late, but I thought it's worth replying as this might save some valuable lives ;)
The JSON2 script won't initialize/extend the JSON object if there is an existing implementation(Native or Included). However if the JSON object does not exist, the script will create that object and attach few methods to it (JSON.stringify and JSON.parse to be precise). However in order to make those methods to work, there are other objects (like Date, String, Number and Boolean objects) which need to be extended to support certain methods (like toJSON method). The JSON2 script takes care of extending the required objects as well.
Now coming to the specific issue here (Telerik MVC). I faced the same problem while working with Telerik for one of the Projects. However I was able to trace it. The probable cause is the conflict between Telerik scripts and the current JSON2 script. The Date and Boolean Objects' toJSON method somehow conflicts with Telerik's implmentation of the same method for those two objects which breaks the Telerik script at some places. I have modified the JSON2 library for a more robust check which doesn't fail in any scenario (even on use of Telerik MVC on the page). I have tested the script and it works fine for me, however in case someone finds any further conflicts, please reply back.
var JSON;
if (!JSON) {
JSON = {};
}
(function () {
'use strict';
function f(n) {
// Format integers to have at least two digits.
return n < 10 ? '0' + n : n;
}
if (typeof Date.prototype.toJSON !== 'function') {
Date.prototype.toJSON = function (key) {
return isFinite(this.valueOf())
? this.getUTCFullYear() + '-' +
f(this.getUTCMonth() + 1) + '-' +
f(this.getUTCDate()) + 'T' +
f(this.getUTCHours()) + ':' +
f(this.getUTCMinutes()) + ':' +
f(this.getUTCSeconds()) + /*added - start*/ '.'+
f(this.getUTCMilliseconds()) + /*added - end*/ 'Z'
: null;
};
//pushed the below code outside current if block
// String.prototype.toJSON =
// Number.prototype.toJSON =
// Boolean.prototype.toJSON = function (key) {
// return this.valueOf();
// };
}
/*added - start*/
if (typeof String.prototype.toJSON !== 'function') {
String.prototype.toJSON = function (key) {
return ((typeof this.valueOf === 'function') ? this.valueOf(): this.toString());
};
}
if (typeof Number.prototype.toJSON !== 'function') {
Number.prototype.toJSON = function (key) {
return ((typeof this.valueOf === 'function') ? this.valueOf(): this.toString());
};
}
if (typeof Boolean.prototype.toJSON !== 'function') {
Boolean.prototype.toJSON = function (key) {
return ((typeof this.valueOf === 'function') ? this.valueOf(): this.toString());
};
}
/*added - end*/
var cx = /[\u0000\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]/g,
escapable = /[\\\"\x00-\x1f\x7f-\x9f\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]/g,
gap,
indent,
meta = { // table of character substitutions
'\b': '\\b',
'\t': '\\t',
'\n': '\\n',
'\f': '\\f',
'\r': '\\r',
'"' : '\\"',
'\\': '\\\\'
},
rep;
function quote(string) {
// If the string contains no control characters, no quote characters, and no
// backslash characters, then we can safely slap some quotes around it.
// Otherwise we must also replace the offending characters with safe escape
// sequences.
escapable.lastIndex = 0;
return escapable.test(string) ? '"' + string.replace(escapable, function (a) {
var c = meta[a];
return typeof c === 'string'
? c
: '\\u' + ('0000' + a.charCodeAt(0).toString(16)).slice(-4);
}) + '"' : '"' + string + '"';
}
function str(key, holder) {
// Produce a string from holder[key].
var i, // The loop counter.
k, // The member key.
v, // The member value.
length,
mind = gap,
partial,
value = holder[key];
// If the value has a toJSON method, call it to obtain a replacement value.
if (value && typeof value === 'object' &&
typeof value.toJSON === 'function') {
value = value.toJSON(key);
}
// If we were called with a replacer function, then call the replacer to
// obtain a replacement value.
if (typeof rep === 'function') {
value = rep.call(holder, key, value);
}
// What happens next depends on the value's type.
switch (typeof value) {
case 'string':
return quote(value);
case 'number':
// JSON numbers must be finite. Encode non-finite numbers as null.
return isFinite(value) ? String(value) : 'null';
case 'boolean':
case 'null':
// If the value is a boolean or null, convert it to a string. Note:
// typeof null does not produce 'null'. The case is included here in
// the remote chance that this gets fixed someday.
return String(value);
// If the type is 'object', we might be dealing with an object or an array or
// null.
case 'object':
// Due to a specification blunder in ECMAScript, typeof null is 'object',
// so watch out for that case.
if (!value) {
return 'null';
}
// Make an array to hold the partial results of stringifying this object value.
gap += indent;
partial = [];
// Is the value an array?
if (Object.prototype.toString.apply(value) === '[object Array]') {
// The value is an array. Stringify every element. Use null as a placeholder
// for non-JSON values.
length = value.length;
for (i = 0; i < length; i += 1) {
partial[i] = str(i, value) || 'null';
}
// Join all of the elements together, separated with commas, and wrap them in
// brackets.
v = partial.length === 0
? '[]'
: gap
? '[\n' + gap + partial.join(',\n' + gap) + '\n' + mind + ']'
: '[' + partial.join(',') + ']';
gap = mind;
return v;
}
// If the replacer is an array, use it to select the members to be stringified.
if (rep && typeof rep === 'object') {
length = rep.length;
for (i = 0; i < length; i += 1) {
if (typeof rep[i] === 'string') {
k = rep[i];
v = str(k, value);
if (v) {
partial.push(quote(k) + (gap ? ': ' : ':') + v);
}
}
}
} else {
// Otherwise, iterate through all of the keys in the object.
for (k in value) {
if (Object.prototype.hasOwnProperty.call(value, k)) {
v = str(k, value);
if (v) {
partial.push(quote(k) + (gap ? ': ' : ':') + v);
}
}
}
}
// Join all of the member texts together, separated with commas,
// and wrap them in braces.
v = partial.length === 0
? '{}'
: gap
? '{\n' + gap + partial.join(',\n' + gap) + '\n' + mind + '}'
: '{' + partial.join(',') + '}';
gap = mind;
return v;
}
}
// If the JSON object does not yet have a stringify method, give it one.
if (typeof JSON.stringify !== 'function') {
JSON.stringify = function (value, replacer, space) {
// The stringify method takes a value and an optional replacer, and an optional
// space parameter, and returns a JSON text. The replacer can be a function
// that can replace values, or an array of strings that will select the keys.
// A default replacer method can be provided. Use of the space parameter can
// produce text that is more easily readable.
var i;
gap = '';
indent = '';
// If the space parameter is a number, make an indent string containing that
// many spaces.
if (typeof space === 'number') {
for (i = 0; i < space; i += 1) {
indent += ' ';
}
// If the space parameter is a string, it will be used as the indent string.
} else if (typeof space === 'string') {
indent = space;
}
// If there is a replacer, it must be a function or an array.
// Otherwise, throw an error.
rep = replacer;
if (replacer && typeof replacer !== 'function' &&
(typeof replacer !== 'object' ||
typeof replacer.length !== 'number')) {
throw new Error('JSON.stringify');
}
// Make a fake root object containing our value under the key of ''.
// Return the result of stringifying the value.
return str('', {'': value});
};
}
// If the JSON object does not yet have a parse method, give it one.
if (typeof JSON.parse !== 'function') {
JSON.parse = function (text, reviver) {
// The parse method takes a text and an optional reviver function, and returns
// a JavaScript value if the text is a valid JSON text.
var j;
function walk(holder, key) {
// The walk method is used to recursively walk the resulting structure so
// that modifications can be made.
var k, v, value = holder[key];
if (value && typeof value === 'object') {
for (k in value) {
if (Object.prototype.hasOwnProperty.call(value, k)) {
v = walk(value, k);
if (v !== undefined) {
value[k] = v;
} else {
delete value[k];
}
}
}
}
return reviver.call(holder, key, value);
}
// Parsing happens in four stages. In the first stage, we replace certain
// Unicode characters with escape sequences. JavaScript handles many characters
// incorrectly, either silently deleting them, or treating them as line endings.
text = String(text);
cx.lastIndex = 0;
if (cx.test(text)) {
text = text.replace(cx, function (a) {
return '\\u' +
('0000' + a.charCodeAt(0).toString(16)).slice(-4);
});
}
// In the second stage, we run the text against regular expressions that look
// for non-JSON patterns. We are especially concerned with '()' and 'new'
// because they can cause invocation, and '=' because it can cause mutation.
// But just to be safe, we want to reject all unexpected forms.
// We split the second stage into 4 regexp operations in order to work around
// crippling inefficiencies in IE's and Safari's regexp engines. First we
// replace the JSON backslash pairs with '#' (a non-JSON character). Second, we
// replace all simple value tokens with ']' characters. Third, we delete all
// open brackets that follow a colon or comma or that begin the text. Finally,
// we look to see that the remaining characters are only whitespace or ']' or
// ',' or ':' or '{' or '}'. If that is so, then the text is safe for eval.
if (/^[\],:{}\s]*$/
.test(text.replace(/\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4})/g, '#')
.replace(/"[^"\\\n\r]*"|true|false|null|-?\d+(?:\.\d*)?(?:[eE][+\-]?\d+)?/g, ']')
.replace(/(?:^|:|,)(?:\s*\[)+/g, ''))) {
// In the third stage we use the eval function to compile the text into a
// JavaScript structure. The '{' operator is subject to a syntactic ambiguity
// in JavaScript: it can begin a block or an object literal. We wrap the text
// in parens to eliminate the ambiguity.
j = eval('(' + text + ')');
// In the optional fourth stage, we recursively walk the new structure, passing
// each name/value pair to a reviver function for possible transformation.
return typeof reviver === 'function'
? walk({'': j}, '')
: j;
}
// If the text is not JSON parseable, then a SyntaxError is thrown.
throw new SyntaxError('JSON.parse');
};
}
}());
Note: The above code is not my implementation. It is from the source https://github.com/douglascrockford/JSON-js I have just modified it a little to avoid any conflicts with Telerik or otherwise.
I have had exactly the same problem.
I couldn't find any other solution than to edit the json2.js file like you suggested, thanks for that.
However, I found that this would fix the issue for IE7 and still work in IE8/9 as well as firefox, but it now stopped working in Chrome ("this" for [this.valueOf === 'function'] is undefined).
Have you run into that issue, too, or did yours work in Chrome? I'm trying to figure out if this is related to my data or telerik-internal.
Thanks for your post!
Edit:
For now I have just returned null if "this" is undefined/null (in all three functions). Seems to work in all browsers and allows the Telerik grid to rebind without problems.
I don't know how correct this is in the global context of json2.js .toJSON method, though.

Resources