go version go1.8.1 windows/amd64
"net/http" package used to build http request.
req, err := http.NewRequest("GET",`http://domain/_api/Web/GetFolderByServerRelativeUrl('` + root_folder_url + `')?$expand=Folders,Files`, nil)
Here if I print url it shows
http://domain/_api/Web/GetFolderByServerRelativeUrl%28%27rooturl%27%29?$expand=Folders,Files
Not understanding why url parser is replacing ' to %27 here.
Whereas I need ' to be sent as is while requesting.
The http.NewRequest function calls url.Parse to set the Request.URL. The URL.RequestURI method is called to get the request URI written to the network.
An application can override any transformation made by Parse/RequestURI by setting the request URL Opaque field:
req, err := http.NewRequest("GET", "http://domain/", nil)
if err != nil {
// handle error
}
req.URL.Opaque = `/_api/Web/GetFolderByServerRelativeUrl('` + root_folder_url + `')?$expand=Folders,Files`
In this snippet, the argument to NewRequest specifies the protocol and host for the request. The opaque value specifies the request URI written to the network. The request URI does not include host or protocol.
Related
I have a gRPC server (server) written in Go that a Python gRPC client (client) talks to. The server occasionally sends http post requests to a Go based http server (sigsvc). All of these instances run as docker instances spun up through docker-compose sharing the same docker network.
This is the section of code on server that creates and sends the http request:
b := new(bytes.Buffer)
txbytes, err := json.Marshal(tx)
if err != nil {
log.WithError(err).Error("failed to marshal transaction")
return nil, err
}
b.Write(txbytes)
resp, err := http.Post(sigsvc.signerURL, "application/json; charset=utf-8", b)
if err != nil {
log.WithError(err).Errorf("error signing transaction with signer %s", sigsvc.signerURL)
return nil, err
}
defer resp.Body.Close()
var signedTx types.Transaction
err = json.NewDecoder(resp.Body).Decode(&signedTx)
if err != nil {
log.WithError(err).Error("couldn't decode signed transaction")
return nil, err
}
sigsvc.signerURL maps to something like http://signer:6666/sign which is the endpoint on the http signer service that handles the request.
signer refers to the service name listed on a docker-compose.yml specification.
This is how the handler looks like on sigsvc:
func (sv *SignerSv) handleSignTx() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
log.Info("request received to sign transaction")
dump, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, fmt.Sprint(err), http.StatusInternalServerError)
}
log.Debugf("%q", dump)
if r.Body == nil {
log.Error("request body missing")
http.Error(w, "Please send a request body", 400)
return
}
log.Debugf("request body: %v", r.Body)
var tx types.Transaction
err = json.NewDecoder(r.Body).Decode(&tx)
if err != nil {
log.WithError(err).Error("failed to unmarshal transaction")
http.Error(w, err.Error(), 400)
return
}
log.WithFields(log.Fields{
"txhash": tx.Hash().Hex(),
"nonce": tx.Nonce(),
"to": tx.To().Hex(),
"data": tx.Data(),
"gasLimit": tx.Gas(),
"gasPrice": tx.GasPrice(),
"value": tx.Value(),
}).Debug("Decoded transaction from request body")
Both the request and request body are dumped successfully by the debug logs. However, apparently the line decoding the request body to the transaction type is never executed, since no error or decoded transaction logs are logged.
On server, I keep getting the following error:
error="Post http://signer:6666/sign: EOF"
This is how the request is logged on sigsvc:
msg="\"POST /sign HTTP/1.1\\r\\nHost: signer:6666\\r\\nConnection: close\\r\\nAccept-Encoding: gzip\\r\\nConnection: close\\r\\nContent-Length: 10708\\r\\nUser-Agent: Go-http-client/1.1\\r\\n\\r\\n{\\\"nonce\\\":\\\"0x0\\\",\\\"gasPrice\\\":\\\"0x2540be400\\\",\\\"gas\\\":\\\"0x15c285\\\",\\\"to\\\":null,\\\"value\\\":\\\"0x0\\\",\\\"input\\\":\\\"0x6080604055",\\\"v\\\":\\\"0x0\\\",\\\"r\\\":\\\"0x0\\\",\\\"s\\\":\\\"0x0\\\",\\\"hash\\\":\\\"0xab55920fb3d490fc55ccd76a29dfb380f4f8a9e5d0bda4155a3b114fca26da0a\\\"}\"
I have tried reproducing this error on similar but simplified docker setups, but I have failed at that.
I'm trying to understand the following:
If there is anything wrong with this code that is being exposed due
to a particular setup on docker?
Or, do I need to look at some docker setup specifics to debug the instances.
The problem was in the way the http handler code was logging the to field in this logrus call.
log.WithFields(log.Fields{
"txhash": tx.Hash().Hex(),
"nonce": tx.Nonce(),
"to": tx.To().Hex(),
"data": tx.Data(),
"gasLimit": tx.Gas(),
"gasPrice": tx.GasPrice(),
"value": tx.Value(),
}).Debug("Decoded transaction from request body")
Under specific circumstances, the tx.To() call returns nil, which implies calling tx.To().Hex() would lead to an error on account of trying to make a method call on a nil pointer. On the face of it, one would expect the log.WithFields() call to error out or panic, but instead the handler silently closes connection with the client side getting an EOF response.
I want to intercept requests in my application and one of the scenarios I have come across is that - due to nature of the application some of the services have same request endpoint different payload though and I want to send back different responses at different occurrence. Is there a way to assign order to the intercepted request and send a specific response as per the sequence? or any other mechanism to tell browser-mob that example - for this url with this payload send this response? Because if it is the same url then responses are overridden by the last intercepted request.
I have already tried request with url and contents contains. Check the if condition in code below.
proxy.addRequestFilter((request, contents, messageInfo) -> {
String content = null;
try {
content = URLDecoder.decode(contents.getTextContents(), "UTF-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
if (request.getUri().contains(endpoint) && (content != null && content.contains(contentContains))) {
//Use DefaultFullHttpResponse for posting the json body
final HttpResponse httpResponse = new DefaultFullHttpResponse(request.getProtocolVersion(),
statusCode);
//Close the connection so it doesn't pass through
httpResponse.headers().add("CONNECTION", "Close");
//Specify the content-type and charset
httpResponse.headers().add("Content-Type", "application/json; charset=UTF-8");
//replace the body
HttpObjectUtil.replaceTextHttpEntityBody((FullHttpMessage) httpResponse, response);
return httpResponse;
}
return null;
});
Actual - If the two request have same endpoint the last intercepted request overrides the response I want to send back.
Expected- Some mechanism where Bowsermob is able to register different responses for the request with same end point and send back different response.
I have written a simple client/server in Go that will do an HTTP GET over TLS, but I'm trying to also make it capable of doing an HTTP POST over TLS.
In the example below index.html just contains the text hello, and the HTTP GET is working fine. I want the client to get the HTTP GET and write back, hello world to the server.
client
package main
import (
"crypto/tls"
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
link := "https://10.0.0.1/static/index.html"
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: tr}
response, err := client.Get(link)
if err != nil {
fmt.Println(err)
}
defer response.Body.Close()
content, _ := ioutil.ReadAll(response.Body)
s := strings.TrimSpace(string(content))
fmt.Println(s)
// out := s + " world"
// Not working POST...
// resp, err := client.Post(link, "text/plain", &out)
}
server
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
http.HandleFunc("/static/", func (w http.ResponseWriter, r *http.Request) {
fmt.Println("Got connection!")
http.ServeFile(w, r, r.URL.Path[1:])
})
log.Fatal(http.ListenAndServeTLS(":443", "server.crt", "server.key", nil))
}
I also currently have nothing to handle the POST on the server side, but I just want it to print it out to the screen so when I run the client I will see the server print hello world.
How should I fix my client code to do a proper POST? And what should the corresponding server code look like to accept the POST? Any help would be appreciated, I'm having trouble finding HTTPS/TLS POST examples.
You didn't share the error message, but I assume the client.Post call wasn't allowing a string as its third parameter, because it requires an io.Reader. Try this instead:
out := s + " world"
resp, err := client.Post(link, "text/plain", bytes.NewBufferString(out))
On the server side, you already have the right code set up to handle the POST request. Just check the method:
http.HandleFunc("/static/", func (w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" {
// handle POST requests
} else {
// handle all other requests
}
})
I noticed one other issue. Using index.html probably won't work here. http.ServeFile will redirect that path. See https://golang.org/pkg/net/http/#ServeFile:
As a special case, ServeFile redirects any request where r.URL.Path
ends in "/index.html" to the same path, without the final
"index.html". To avoid such redirects either modify the path or use
ServeContent.
I'd suggest just using a different file name to avoid that issue.
So, I'm using the net/http package. I'm GETting a URL that I know for certain is redirecting. It may even redirect a couple of times before landing on the final URL. Redirection is handled automatically behind the scenes.
Is there an easy way to figure out what the final URL was without a hackish workaround that involves setting the CheckRedirect field on a http.Client object?
I guess I should mention that I think I came up with a workaround, but it's kind of hackish, as it involves using a global variable and setting the CheckRedirect field on a custom http.Client.
There's got to be a cleaner way to do it. I'm hoping for something like this:
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
// Try to GET some URL that redirects. Could be 5 or 6 unseen redirections here.
resp, err := http.Get("http://some-server.com/a/url/that/redirects.html")
if err != nil {
log.Fatalf("http.Get => %v", err.Error())
}
// Find out what URL we ended up at
finalURL := magicFunctionThatTellsMeTheFinalURL(resp)
fmt.Printf("The URL you ended up at is: %v", finalURL)
}
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
resp, err := http.Get("http://stackoverflow.com/q/16784419/727643")
if err != nil {
log.Fatalf("http.Get => %v", err.Error())
}
// Your magic function. The Request in the Response is the last URL the
// client tried to access.
finalURL := resp.Request.URL.String()
fmt.Printf("The URL you ended up at is: %v\n", finalURL)
}
Output:
The URL you ended up at is: http://stackoverflow.com/questions/16784419/in-golang-how-to-determine-the-final-url-after-a-series-of-redirects
I would add a note that http.Head method should be enough to retrieve the final URL. Theoretically it should be faster comparing to http.Get as a server is expected to send back just a header:
resp, err := http.Head("http://stackoverflow.com/q/16784419/727643")
...
finalURL := resp.Request.URL.String()
...
I'm receiving a standard request from an API. It looks something like this :
It's content type and length is :
But when this hits my Rails server, Rails responds with
The reason I'm bringing this up, is because the same request seems to work on SCORM Cloud's server. If I upload the exact same content to them, and watch it in the debugger, I see it send out an application/json statement with the same Request payload, but with no unexpected token error.
Does a Rails application/json request have to be written a certain way that differs from other servers? Is there a proper way to rewrite this line in Rack Middleware to prevent this error?
Update
The javascript :
function _TCDriver_XHR_request (lrs, url, method, data, callback, ignore404, extraHeaders) {
_TCDriver_Log("_TCDriver_XHR_request: " + url);
var xhr,
finished = false,
xDomainRequest = false,
ieXDomain = false,
ieModeRequest,
title,
ticks = ['/', '-', '\\', '|'],
location = window.location,
urlParts,
urlPort,
result,
extended,
until,
fullUrl = lrs.endpoint + url
;
urlParts = fullUrl.toLowerCase().match(/^(.+):\/\/([^:\/]*):?(\d+)?(\/.*)?$/);
// add extended LMS-specified values to the URL
if (lrs.extended !== undefined) {
extended = [];
for (var prop in lrs.extended) {
if(lrs.extended[prop] != null && lrs.extended[prop].length > 0){
extended.push(prop + "=" + encodeURIComponent(lrs.extended[prop]));
}
}
if (extended.length > 0) {
fullUrl += (fullUrl.indexOf("?") > -1 ? "&" : "?") + extended.join("&");
}
}
//Consolidate headers
var headers = {};
headers["Content-Type"] = "application/json";
headers["Authorization"] = lrs.auth;
if (extraHeaders !== null) {
for (var headerName in extraHeaders) {
headers[headerName] = extraHeaders[headerName];
}
}
//See if this really is a cross domain
xDomainRequest = (location.protocol.toLowerCase() !== urlParts[1] || location.hostname.toLowerCase() !== urlParts[2]);
if (! xDomainRequest) {
urlPort = (urlParts[3] === null ? ( urlParts[1] === 'http' ? '80' : '443') : urlParts[3]);
xDomainRequest = (urlPort === location.port);
}
//If it's not cross domain or we're not using IE, use the usual XmlHttpRequest
if (! xDomainRequest || typeof XDomainRequest === 'undefined') {
_TCDriver_Log("_TCDriver_XHR_request using XMLHttpRequest");
xhr = new XMLHttpRequest();
xhr.open(method, fullUrl, callback != null);
for (var headerName in headers) {
xhr.setRequestHeader(headerName, headers[headerName]);
}
}
//Otherwise, use IE's XDomainRequest object
else {
_TCDriver_Log("_TCDriver_XHR_request using XDomainRequest");
ieXDomain = true;
ieModeRequest = _TCDriver_GetIEModeRequest(method, fullUrl, headers, data);
xhr = new XDomainRequest ();
xhr.open(ieModeRequest.method, ieModeRequest.url);
}
Rails is being "helpful" here and assuming that the client is correctly using "Content-Type" and passing a value that actually matches that content type. In other words, the payload in the request has to be parseable JSON, and the value being passed is not valid JSON.
Which is an entirely reasonable thing for it to do when you are implementing an in house API that isn't intended for maximum interoperability. What Rails doesn't know is that an LRS' document storage is supposed to be "dumb" and basically allow the client to shove whatever it wants in and get whatever it wants out, which is why SCORM Cloud accepts the request, basically it just stores the content type and the contents, and then regurgitates them as is on request.
The code you pasted is from a very old library that has poor implementation of Content-Type headers. If this code is found anywhere other than in a relatively old version of a piece of content from one of the major e-learning authoring tools then it should be updated to use a recent version of TinCanJS and improve the content type handling.
As far as making this work on Rails, sorry I don't have that much experience with it. Presumably there is a switch or something to turn off automatic request body parsing, at least that's what most other frameworks I've used have.
Does a Rails application/json request have to be written a certain way that differs from other servers?
Not that I know of no.
Is there a proper way to rewrite this line in Rack Middleware to prevent this error?
There might a way yes, maybe even without rack middlewares, although it's quite hard to help you without an actual request to work with.