Play / Run captured HTTP traffic from file - parsing

I have a raw HTTP traffic file with following format :
---------------------- dataset.txt ----------------------------------
GET http://localhost:8080/tienda1/index.jsp HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.8 (like Gecko)
Pragma: no-cache
Cache-control: no-cache
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: x-gzip, x-deflate, gzip, deflate
Accept-Charset: utf-8, utf-8;q=0.5, *;q=0.5
Accept-Language: en
Host: localhost:8080
Cookie: JSESSIONID=1F767F17239C9B670A39E9B10C3825F4
Connection: close
POST http://localhost:8080/tienda1/publico/anadir.jsp HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.8 (like Gecko)
Pragma: no-cache
Cache-control: no-cache
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: x-gzip, x-deflate, gzip, deflate
Accept-Charset: utf-8, utf-8;q=0.5, *;q=0.5
Accept-Language: en
Host: localhost:8080
Cookie: JSESSIONID=933185092E0B668B90676E0A2B0767AF
Content-Type: application/x-www-form-urlencoded
Connection: close
Content-Length: 68
id=3&nombre=Vino+Rioja&precio=100&cantidad=55&B1=A%F1adir+al+carrito
...
...
Is there any utility to read this file and submit to my local web server?

You have two requests here.
First is GET, second is POST.
Provided that you are having such format as above, you could write simple program that will ( in order)
Divide the sheet into separate HTTP requests
Parse requests and divide them into variables like : type of request (GET or POST), User-Agent and "headers" in general, request DATA, request submit URL
Create and maintain session with specific server (cookies etc)
Iterate through the loop and submit the data
It would be perfect to solve the problem in python-requests. Parsing may be done in python basic lib.

Related

How to print Rack::Request as raw HTTP request?

I interested in the standard way to print an instance of a Rack::Request class as a text HTTP request. Example below.
POST /cgi-bin/process.cgi HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE5.01; Windows NT)
Host: www.tutorialspoint.com
Content-Type: text/xml; charset=utf-8
Content-Length: length
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://clearforest.com/">string</string>
It would be nice for debugging and share with users. I have a team with non-rails stack using my API and raw requests will be useful.

SignalR security warning : Missing Cross-Frame Scripting Defence

A security scanning app picked up a risk against a signalR link in my asp.net MVC 5 website.
the X-Frame-Options response header is missing, which may allow
Cross-Frame Scripting attacks
Any one can tell me what's this about?
And How to solve it?
ASP.NET SignalR Input Validation Flaw Permits Cross-Site Scripting Attacks
Should not be the problem since I am using SignalR 2.1.x
The request is :
POST ***/signalr/send?transport=serverSentEvents&clientProtocol=1.4&connectionToken=bla**bla** HTTP/1.1
Host: ****
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0.1) Gecko/20100101 Firefox/8.0.1
Accept: text/plain, */*; q=0.01
Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: myhost
Pragma: no-cache
Cache-Control: no-cache
Cookie: authentication token
Content-Length: 113
data=********
The Response is :
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: application/json; charset=UTF-8
Expires: -1
Server: Microsoft-IIS/8.0
X-Content-Type-Options: nosniff
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Some html body
I think this warning can safely be ignored. The X-Frame-Options header is used to prevent clickjacking. SignalR responses don't have any links or any other clickable content.
However, it might be a good idea to set an X-Frame-Options header on every response to be extra safe. You can do this via IIS manager or web.config. If you are not using IIS, you can use OWIN middleware instead.

Yandex tank post data

I want to use yandex tank to test my web app, I want to test registration, so i need to send request like this
Header
POST /registration HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 30
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://localhost:8080/registration
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,ru;q=0.6
Cookie: csrftoken=XJ3oheJb0SndHfNAH2lSV2AtKNxxuXdv; JSESSIONID=igq9ejgl10jirr4t73mpjblp
Form Data
login=abracadbra&password=brar
Is it possible to send that kind of requests with different login fields?
Yes, it's possible, for POST requests your should use request-style ammo file, see http://yandextank.readthedocs.org/en/latest/tutorial.html#request-style
You may specify ammo file as command line parameter or put it in tank .ini file in [phantom] section.
Remember that with phantom as load-generator it's not possible to perform scenario-based testing, so you should generate all necessary data for requests in advance. Yandex-tank will just send it according to desired load-scheme.
I'm not sure how CSRF protection works in your case, and if it does not permit re-use of session and csrftoken for multiple requests (and this parameters are mandatory for registration requests), you'll need to somehow obtain valid Cookies for each request to generate ammo. In this case I'd recommend you to switch to some scenario-based tool, i.e. jmeter. You may use jmeter as load-generator for yandex-tank as well, see http://yandextank.readthedocs.org/en/latest/configuration.html#jmeter
In case you may reuse same Cookie for multiple registrations, ammo file will be like that:
649 tag1
POST /registration HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 30
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://localhost:8080/registration
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,ru;q=0.6
Cookie: csrftoken=XJ3oheJb0SndHfNAH2lSV2AtKNxxuXdv; JSESSIONID=igq9ejgl10jirr4t73mpjblp
login=abracadbra&password=brar
646 tag2
POST /registration HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 27
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://localhost:8080/registration
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,ru;q=0.6
Cookie: csrftoken=XJ3oheJb0SndHfNAH2lSV2AtKNxxuXdv; JSESSIONID=igq9ejgl10jirr4t73mpjblp
login=sample2&password=brar
...
There is also URI+POST format similar to uri-style, but with post bodies:
load.ini:
ammo_type=uripost
ammo.uripost:
[Host: example.org]
[Connection: close]
[User-Agent: Tank]
5 /route/?rll=50.262025%2C53.276083~50.056015%2C53.495561&origin=1&simplify=1
class
10 /route/?rll=50.262025%2C53.276083~50.056015%2C53.495561&origin=1&simplify=1
hello!clas
7 /route/?rll=37.565147%2C55.695758~37.412796%2C55.691454&origin=1&simplify=1
uripost
Numbers at the beginnings of lines with uris are sizes of post bodies.

Siege does not send POST data

I'm trying to test my site with siege 3.0.1. But it seems that siege does not send POST data. Here is a request I get from web-browser
POST / HTTP/1.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cache-Control: no-cache
Content-Length: 130
Content-Type: application/x-www-form-urlencoded
Host: rsc
Origin: http://rsc
Pragma: no-cache
Referer: http://rsc/
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.52 Safari/537.36
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: http
X-Real-Ip: 127.0.0.1
came_from=http%3A%2F%2Frsc%2F&login=***&password=***&form.login=%D0%9E%D1%82%D0%BF%D1%80%D0%B0%D0%B2%D0%B8%D1%82%D1%8C
And here a request from siege
siege -g "http://rsc/ POST came_from=http%3A%2F%2Frsc%2F&login=***&password=***&form.login=%D0%9E%D1%82%D0%BF%D1%80%D0%B0%D0%B2%D0%B8%D1%82%D1%8C"
POST / HTTP/1.0
Accept: */*
Accept-Encoding:
Content-Length: 111
Content-Type: application/x-www-form-urlencoded
Host: rsc
User-Agent: JoeDog/1.00 [en] (X11; I; Siege 3.0.1)
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: http
X-Real-Ip: 127.0.0.1
Am I doing something wrong? I checked siege documentation and it says nothing about POST except I should write POST my=params after url.
According to the Siege home page, 3.0.1 has a major bug with HTTP POST and the author suggests upgrading to 3.0.2
I had a similar problem - siege would not post, however the result was a "200" and then a "General Error". I tried everything, the end result: I rolled back to a previous version of 2.7.2 from 3.0.1 and it mysteriously works. Fyi.

How can I get the Kendo upload control to reauthenticate in IE.

I have a form with a kendo upload control and a save button along with a kendo grid. When I open the form using Chrome or Firefox, the upload control works every time. When I open the form in Internet Explorer 10 and leave the form open for more than 1 minute without doing anything, the upload stops working.
I am working on an ASP.net MVC 4 web application.
In Fiddler I see 2 entries.
**POST /PRODUCT/FOLDER/FOLDER2/ProcessSubmit HTTP/1.1**
Accept: */*
Content-Type: multipart/form-data; boundary=---------------------------7dd36d1cb0ce0
Referer: http://localhost:82/PRODUCT/Admin/?contentUrl=/PRODUCT/FOLDER/FOLDER2/Page?cbuId=42aff2e4-33bb-4f73-8500-596f6d73ce3a
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
Connection: Keep-Alive
DNT: 1
Host: localhost:82
Pragma: no-cache
Cookie: .MVCRETURNURL=/PRODUCT/Areas/FOLDER/Views/FOLDER2/Page.cshtml; .MVCAUTH=; .ASPXAUTH=F39BF9AA8D958D1783D6A939F3A6B499B81089434C09AC65C168BEF1E298B480935446E77B12A2044F4A7622C82B91DE7C1C6D8E09C4408A9D7C519E6EBB7ACD481735D851AA1A9E65A05CE34AFAE77FCB5E86DDD2EA2F05988C36BEBF5B30994ACEE33557FF3027D875A14ADE2128D2C9C8F20E
Authorization: Negotiate YHcGBisGAQUFAqBtMGugMDAuBgorBgEEAYI3AgIKBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICHqI3BDVOVExNU1NQAAEAAACXsgjiBQAFADAAAAAIAAgAKAAAAAYBsR0AAAAPQ0tEUDA1MTRFQUVEMQ==
Content-Length: 0
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
WWW-Authenticate: Negotiate oYIBHDCCARigAwoBAaEMBgorBgEEAYI3AgIKooIBAQSB/k5UTE1TU1AAAgAAAAoACgA4AAAAFcKJ4tPeOOS7ZmVeEKH7AQAAAAC8ALwAQgAAAAYBsR0AAAAPRQBBAEUARAAxAAIACgBFAEEARQBEADEAAQAQAEMASwBEAFAAMAA1ADEANAAEACQAcABlAHQAcgBvAGwAZQB1AG0AcABsAGEAYwBlAC4AYwBvAG0AAwA2AEMASwBEAFAAMAA1ADEANAAuAHAAZQB0AHIAbwBsAGUAdQBtAHAAbABhAGMAZQAuAGMAbwBtAAUAJABwAGUAdAByAG8AbABlAHUAbQBwAGwAYQBjAGUALgBjAG8AbQAHAAgAiqpF+oprzgEAAAAA
Date: Mon, 17 Jun 2013 18:46:28 GMT
Content-Length: 341
Proxy-Support: Session-Based-Authentication
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Authorized</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Authorized</h2>
<hr><p>HTTP Error 401. The requested resource requires user authentication.</p>
</BODY></HTML>
------------------------------------------------------------------
**POST /PRODUCT/FOLDER/FOLDER2/ProcessSubmit HTTP/1.1**
Accept: */*
Content-Type: multipart/form-data; boundary=---------------------------7dd36d1cb0ce0
Referer: http://localhost:82/PRODUCT/Admin/?contentUrl=/PRODUCT/FOLDER/FOLDER2/Page?cbuId=42aff2e4-33bb-4f73-8500-596f6d73ce3a
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
Proxy-Connection: Keep-Alive
Content-Length: 12435
DNT: 1
Host: localhost:82
Pragma: no-cache
Cookie: .MVCRETURNURL=/PRODUCT/Areas/FOLDER/Views/FOLDER2/Page.cshtml; .MVCAUTH=; .ASPXAUTH=F39BF9AA8D958D1783D6A939F3A6B499B81089434C09AC65C168BEF1E298B480935446E77B12A2044F4A7622C82B91DE7C1C6D8E09C4408A9D7C519E6EBB7ACD481735D851AA1A9E65A05CE34AFAE77FCB5E86DDD2EA2F05988C36BEBF5B30994ACEE33557FF3027D875A14ADE2128D2C9C8F20E
Authorization: Negotiate oXcwdaADCgEBoloEWE5UTE1TU1AAAwAAAAAAAABYAAAAAAAAAFgAAAAAAAAAWAAAAAAAAABYAAAAAAAAAFgAAAAAAAAAWAAAABXCiOIGAbEdAAAADwjVcbbDNKZpdbLdjWcqj76jEgQQAQAAAPUXp1AtIpqEAAAAAA==
------------------------------------------------------------------
If I press the save button on the same page, I see almost identical fiddler entries.
POST /PRODUCT/FOLDER/FOLDER2/UpdateMajorCodes HTTP/1.1
Accept: */*
Content-Type: application/json; charset=utf-8
X-Requested-With: XMLHttpRequest
Referer: http://localhost:82/PRODUCT/Admin/?contentUrl=/PRODUCT/FOLDER/FOLDER2/Page?cbuId=42aff2e4-33bb-4f73-8500-596f6d73ce3a
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
Host: localhost:82
Content-Length: 0
DNT: 1
Connection: Keep-Alive
Pragma: no-cache
Cookie: .MVCRETURNURL=/PRODUCT/; .MVCAUTH=; .ASPXAUTH=9079B8150CB469C41BA155173534C32ABF863F0369000D0DCE3DB0A2AFF6124AB0878537E11EC70C59B118F8957B657E1E0D6D9C4233F39D2435040ADAFF397EEF703D4535AAA153753A4888BB6FF4DD053A303FA65C506ECB2F2F01A17E1A27F9C9EA2D468F5E77BD2F971698427989BB5372E7
Authorization: Negotiate YHcGBisGAQUFAqBtMGugMDAuBgorBgEEAYI3AgIKBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICHqI3BDVOVExNU1NQAAEAAACXsgjiBQAFADAAAAAIAAgAKAAAAAYBsR0AAAAPQ0tEUDA1MTRFQUVEMQ==
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
WWW-Authenticate: Negotiate oYIBHDCCARigAwoBAaEMBgorBgEEAYI3AgIKooIBAQSB/k5UTE1TU1AAAgAAAAoACgA4AAAAFcKJ4pCCkvtzwTQ5oJ/7AQAAAAC8ALwAQgAAAAYBsR0AAAAPRQBBAEUARAAxAAIACgBFAEEARQBEADEAAQAQAEMASwBEAFAAMAA1ADEANAAEACQAcABlAHQAcgBvAGwAZQB1AG0AcABsAGEAYwBlAC4AYwBvAG0AAwA2AEMASwBEAFAAMAA1ADEANAAuAHAAZQB0AHIAbwBsAGUAdQBtAHAAbABhAGMAZQAuAGMAbwBtAAUAJABwAGUAdAByAG8AbABlAHUAbQBwAGwAYQBjAGUALgBjAG8AbQAHAAgA6r5c6ZdrzgEAAAAA
Date: Mon, 17 Jun 2013 20:19:03 GMT
Content-Length: 341
Proxy-Support: Session-Based-Authentication
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Authorized</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Authorized</h2>
<hr><p>HTTP Error 401. The requested resource requires user authentication.</p>
</BODY></HTML>
------------------------------------------------------------------
POST /PRODUCT/FOLDER/FOLDER2/UpdateMajorCodes HTTP/1.1
Accept: */*
Content-Type: application/json; charset=utf-8
X-Requested-With: XMLHttpRequest
Referer: http://localhost:82/PRODUCT/Admin/?contentUrl=/PRODUCT/FOLDER/FOLDER2/Page?cbuId=42aff2e4-33bb-4f73-8500-596f6d73ce3a
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
Host: localhost:82
Content-Length: 13798
DNT: 1
Connection: Keep-Alive
Pragma: no-cache
Cookie: .MVCRETURNURL=/PRODUCT/; .MVCAUTH=; .ASPXAUTH=9079B8150CB469C41BA155173534C32ABF863F0369000D0DCE3DB0A2AFF6124AB0878537E11EC70C59B118F8957B657E1E0D6D9C4233F39D2435040ADAFF397EEF703D4535AAA153753A4888BB6FF4DD053A303FA65C506ECB2F2F01A17E1A27F9C9EA2D468F5E77BD2F971698427989BB5372E7
Authorization: Negotiate oXcwdaADCgEBoloEWE5UTE1TU1AAAwAAAAAAAABYAAAAAAAAAFgAAAAAAAAAWAAAAAAAAABYAAAAAAAAAFgAAAAAAAAAWAAAABXCiOIGAbEdAAAAD93bE7wOkB2UOLRe3+N5TJ6jEgQQAQAAAPUXp1AtIpqEAAAAAA==
The save works and makes a call in to the server to save the data. The upload server call never happens. I am trying to figure out how to get the upload to work in IE 10 after the authentication timeout.
thanks
Kent Pigott

Resources