How to capture http POST requests with browsermob-proxy and selenium - post

Hello trying to capture the actual POST data in an HTTP POST request using browsermob proxy + selenium test framework. So basically i'm running an automated test using selenium and I want to capture the key/value pairs and the actual POST data of a HTTP POST request during the test. Using the following logic I can only capture the key/value pairs of the POST header but not the actual POST data (aka the form field id values). Is there a way to actually capture the POSTDATA (like sniffing applications do such as tamper/live headers in firefox)?
ProxyServer proxyServer = null;
proxyServer = new ProxyServer(9101);
proxyServer.start();
proxyServer.setCaptureContent(true);
proxyServer.setCaptureHeaders(true);
Proxy proxy = proxyServer.seleniumProxy();
proxy.setHttpProxy("localhost:9101");
//selenium test config code, omitted for brevity
proxyServer.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(HttpRequest request, HttpContext context) throws HttpException, IOException {
Header[] headers = request.getAllHeaders();
System.out.println("\nRequest Headers\n\n");
for(Header h : headers) {
System.out.println("Key: " + h.getName() + " | Value: " + h.getValue());
}
}
});
An alternate way I read about but could not get to work was to configure the following flags in the browsermob proxy server to true:
proxyServer.setCaptureContent(true);
proxyServer.setCaptureHeaders(true);
Then output the actual HAR file:
Har har = proxyServer.getHar();
Date date = new Date();
har.writeTo(new File("c:\\tmp\\har_" + date.getTime()));
To see the key/value pairs, POST Data, and actual content of the response... but when I parse the HAR file... I only see the key/value pairs of the POST header again... no POST data... no response content. I am only interested in the actual POST data though.

I also had same problem. As a solution, I captured all the data, converted the HAR file into JSON, and then filtered out only POST requests from the JSON file.

Related

RestAssured - how to send a request without Content-Type?

I am using RestAssured to send a request:
Map<String, Object> headers = new HashMap<>();
headers.put("Accept", "*/*");
headers.put("Accept-Encoding", "gzip, deflate, br");
headers.put("Connection", "keep-alive");
Response response = RestAssured.given().baseUri(BASE_URL)
.headers(headers)
.log().all()
.post(URL_PREFIX + "/documents/request/" + username);
However, in the log I see that 1 more header was automatically added:
Content-Type=application/x-www-form-urlencoded; charset=ISO-8859-1
And I get 415 error.
Is it possible to send a request without Content-Type? I mean, without this header at all; if the request is sent with Content-Type equal to empty line, there is still a 400 error; the only way to make it work is to send the request without this header.
Seems like there is a bug in the RestAssured framework that is still open (I verified that in 4.3.3).
// https://mvnrepository.com/artifact/io.rest-assured/rest-assured
testImplementation group: 'io.rest-assured', name: 'rest-assured', version: '4.3.3'
Founded out, when creating negative tests for a API. Content type below is automatically generated when trying to send request.
Content-Type=application/x-www-form-urlencoded; charset=ISO-8859-1
Bug defined here:
https://github.com/rest-assured/rest-assured/issues/656
https://github.com/rest-assured/rest-assured/issues/986

Http post request with Content-Type: application/x-www-form-urlencoded

How to sent POST request with content-type = application/x-www-form-urlencoded.
Given an access code, I am trying to get the AccessToken using POST request where I have all the information set in POST URL so i don't know what to pass as Http content in the postAysnc method.
According to another post For application/x-www-form-urlencoded, the body of the HTTP message sent to the server is essentially one giant query string -- name/value pairs are separated by the ampersand (&), and names are separated from values by the equals symbol (=). An example of this would be:
MyVariableOne=ValueOne&MyVariableTwo=ValueTwo.
So I have similar case where my POST url has all the information as querystring, in that case I don't know what to pass as HttpContent in the postAysnc method since its a mandatory parameter
HttpClient client = new HttpClient();
StringContent queryString = new StringContent(data);
HttpResponseMessage response = await client.PostAsync(new Uri(url), queryString );
Two options:
Continue to use StringContent, but set the content type: new StringContent(data, Encoding.UTF8, "application/x-www-form-urlencoded")
Instead of using StringContent, use FormUrlEncodedContent; note that this takes in a sequence of KeyValuePair<string, string> describing the values (so for example create a dictionary containing { {"MyVariableOne", "ValueOne"}, {"MyVariableTwo", "ValueTwo"} }

HttpRequest GET with sendData as query parameters?

This question is kind of a duplicate of HTTPRequest.request with sendData, can't seem to get this to work, but I've got some more information now. The goal here is to send a GET request with query parameters attached. I initially tried to send my request as such:
HttpRequest request = new HttpRequest();
request.open("GET", _url, async:true);
request.onError.listen(_onLoadError, onError: _onLoadError);
request.send(sendData);
Where sendData is a String following the normal format for query parameters (?myVariable=2&myOtherVariable=a, etc), since this is the ultimate goal here. The request gets sent, but I never see any additional data (sendData) go with it in any monitoring tools (I'm using Charles). I then tried:
HttpRequest request = new HttpRequest();
request.open("GET", _url + sendData, async:true);
request.onError.listen(_onLoadError, onError: _onLoadError);
request.send();
So now I'm just attaching the query string to the url itself. This works as intended, but is far from elegant. Is there a better solution?
On a GET request you always add the query string to the URL?
When you create the Uri you can pass in a map with the query parameters if you find this more elegant.
Map query = {'xx':'yy', 'zz' : 'ss'};
String url = "http://localhost:8080/myapp/signinService";
Uri uri = new Uri(path: url, queryParameters : query);
HttpRequest.getString(uri.toString().then((HttpRequest req) ...
According to the W3 XMLHttpRequest Specification:
(In the send() method) The argument is ignored if request method is GET or HEAD.
So simple answer to this question is no. sendData cannot be appended to a GET request, this is by XMLHttpRequest specification and not a limitation of Dart.
That said, for requests like this it may be more readable and idiomatic to use HttpRequest.getString
HttpRequest.getString(_url + sendData).then((HttpRequest req) {
// ... Code here
}).catchError(_onLoadError);
If you want to generate a valid URL from url (String) + query parameters (Map) , you can do the following :
Uri uri = Uri.parse(url).replace(queryParameters: parameters);
String finalURL = uri.toString();
url is a String and parameters is a Map

How to (simulate) a POST in jUnit

I am trying to have a server respond to a request which needs a XML structure. The easiest way I thought would be to create a POST with a string containing the XML, using the Play Framework.
However, I cannot seem to get it to work. I am calling the test with the following code:
Map<String,String> map = new HashMap<String,String>();
map.put("data", xmlString);
Http.Response response = POST("/server/", map);
When on the server checking the parameters it is not in it as it returns false:
params._contains("data")

I am sending data over HTTPS but the server side says it is not receiving it

I create an application that sends some data to a secured network.
At the server side they need the data as JSON object. For that am creating the data as JSON object and writing that data in the OutputStream of the connection.
But the response from the server side telling it is not getting the data that I am passing.
The code snippet that am using is something like given below:
HttpsConnection _connection = (HttpsConnection)Connector.open("https://gmail.com/",Connector.READ_WRITE, true); _connection.setRequestMethod(HttpsConnection.POST);
_connection.setRequestProperty("If-Modified-Since", "29 Oct 1999 19:43:31 GMT");
_connection.setRequestProperty("User-Agent","Profile/MIDP-2.0 Configuration/CLDC-1.0");
_connection.setRequestProperty("Content-Language", "en-US");
_connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
byte[] postData = jsonObject.toString().getBytes("UTF-8");
_connection.setRequestProperty("Content-Length", Integer.toString(postData.length));
_connection.setRequestProperty("jsondata",jsonObject.toString());
OutputStream os = _connection.openOutputStream();
os.write(postData);
os.flush();
Please help me to solve the issue.
I guess the reason is "Content-Type" => "application/x-www-form-urlencoded". This type of a POST exists for sending a list of key=value pairs. So the server on its side will parse the post data in terms of key=value pairs. I believe in your case it just fails to parse the got post data, because you don't send the data in the key=value pairs form (instead you just pour the entire json string jsonObject.toString().getBytes("UTF-8") in it).
So basically you need to form a key value pair "json=YOUR_JSON_HERE". Then on the server you will get your data as the json parameter value:
URLEncodedPostData encPostData = new URLEncodedPostData("UTF-8", false);
encPostData.append("json", jsonObject.toString());
byte[] postData = encPostData.toString().getBytes("UTF-8");
Another option (and BTW it would be the most proper way to do this particular task) would be to use "multipart/form-data" POST type. However it will be a bit harder to implement it if you've never done that before on BB.
You have to append appropriate suffix to to your url
eg: If you use simulator use:https://gmail.com/;deviceside=true etc
I have same this problem but finally find solution:
HttpConnection c = (HttpConnection)Connector.open(url);
c.setRequestMethod(HttpConnection.POST);
c.setRequestProperty(
HttpProtocolConstants.HEADER_CONTENT_TYPE, PostData.getContentType());
c.setRequestProperty(
HttpProtocolConstants.HEADER_CONTENT_LENGTH,String.valueOf(oPostData.size()));
c.setRequestProperty("Content-Length", Integer.toString(oPostData.size()));
c.setRequestProperty("Content-Type","application/json");
byte [] postDataBytes = jobj.toString().getBytes("UTF-8");
os = c.openOutputStream();
os.write(postDataBytes);
os.flush();

Resources