Pyshark does not show \r\n\r\n in the HTTP header and instead shows \r\n - wireshark

I am using pyshark to parse .pcap files specifically with HTTP packets. Unlike as in Wireshark, where it shows the \r\n\r\n bytes at the end of the HTTP header, pyshark does not show them and instead shows a single \r\n.
Is there any way to properly parse the HTTP layer of the packet to display the \r\n\r\n's?
If so, how? I have done a fair amount of searching through the web but the sources are limited and does not answer my question.
Also, with pyshark, the headers do not come in the same order as seen on Wireshark. Is there any reason to that as well?
Python code
#!/bin/env python3
import pyshark
packets = []
with pyshark.FileCapture('testing-mutillidae1.pcap') as capture:
for pkt in capture: # storing packets in list
packets.append(pkt)
print(packets[3]) # printing packet details of packet no. 4
HTTP header
I have included the full output of the packet on pastebin: https://pastebin.com/qxjxY6Hw . Since it is too long, I have added only the HTTP layer in this question
Layer HTTP:
GET /mutillidae/index.php?page=add-to-your-blog.php HTTP/1.1\r\n
Expert Info (Chat/Sequence): GET /mutillidae/index.php?page=add-to-your-blog.php HTTP/1.1\r\n
GET /mutillidae/index.php?page=add-to-your-blog.php HTTP/1.1\r\n
Severity level: Chat
Group: Sequence
Request Method: GET
Request URI: /mutillidae/index.php?page=add-to-your-blog.php
Request URI Path: /mutillidae/index.php
Request URI Query: page=add-to-your-blog.php
Request URI Query Parameter: page=add-to-your-blog.php
Request Version: HTTP/1.1
Host: 10.0.2.13\r\n
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
Accept-Language: en-US,en;q=0.5\r\n
Accept-Encoding: gzip, deflate\r\n
Referer: http://10.0.2.13/mutillidae/index.php\r\n
Connection: keep-alive\r\n
Cookie: showhints=0; PHPSESSID=511be46cfd6922ff8sqqhtqmbn\r\n
Cookie pair: showhints=0
Cache-Control: max-age=0\r\n
Full request URI: http://10.0.2.13/mutillidae/index.php?page=add-to-your-blog.php
HTTP request 1/1
\r\n
Upgrade-Insecure-Requests: 1\r\n
Cookie pair: PHPSESSID=511be46cfd6922ff8sqqhtqmbn
Here is the screenshot on my wireshark (I cannot post pictures yet)

Related

Delphi & Indy httpServer how stop sending 400 bad Request

I get data from a device that expects him back ACK (HTTP1/1 200 OK).
My httpserver after receiving header is automatically returned 400 Bad Request (i see on WareShark).
Perhaps the device is not properly built his request.
How do I stop server does not return an error? So I will be able to continue communication with device.
thanks
LOG: 192.168.1.141:57565 Stat Connected.
HTTP Connect 192.168.1.141
LOG: 192.168.1.141:57565 Recv 10.4.2017 г. 00:18:43: POST / HTTP/1.1<EOL>
LOG: 192.168.1.141:57565 Recv 10.4.2017 г. 00:18:43: Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/msword, application/x-shockwave-flash, text/*, */*<EOL>Accept-Language: en-us<EOL>Content-Type: application/x-www-form-urlencoded<EOL>Accept-Encoding: gzip, deflate<EOL>User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)<EOL>Content-Length: 579<EOL>Connection: Keep-Alive<EOL><EOL><?xml version="1.0"?><EOL><Metrics SiteId="BG-001" Sitename="office Pazardjik"><EOL><Properties><EOL><MacAddress>00:b0:9d:7f:b7:b2</MacAddress><EOL><IpAddress>0.0.0.0</IpAddress><EOL><Timezone>2</Timezone><EOL><DST>1</DST><EOL><DeviceType>0</DeviceType><EOL><SerialNumber>8370098</SerialNumber><EOL></Properties><EOL><ReportData Interval="1"><EOL><Report Date="2017-04-06"><EOL><Object Id="0" DeviceId="BG-001-01" Devicename="Main Entrance" ObjectType="0" Name="Main Entrance"><EOL><Count StartTime="05:31:00" EndTime="05:32:00" Enters="0" Exits="0" Status="0"/><EOL></Object><EOL></Report><EOL></ReportData><EOL></Metrics><EOL>
HTTP Header
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/msword, application/x-shockwave-flash, text/*, */*
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)
Content-Length: 579
Connection: Keep-Alive
LOG: 192.168.1.141:57565 Sent 10.4.2017 г. 00:18:43: HTTP/1.1 400 Bad Request<EOL>Connection: close<EOL>Content-Length: 0<EOL>Date: Sun, 09 Apr 2017 21:18:43 GMT<EOL><EOL>
LOG: 192.168.1.141:57565 Stat Disconnected.
LOG: 0.0.0.0:0 Stat Disconnected.
The HTTP client is sending an HTTP 1.1 request, but is not sending a required Host request header. Per RFC 2616 Section 14.23:
A client MUST include a Host header field in all HTTP/1.1 request messages. If the requested URI does not include an Internet host name for the service being requested, then the Host header field MUST be given with an empty value. An HTTP/1.1 proxy MUST ensure that any request message it forwards does contain an appropriate Host header field that identifies the service being requested by the proxy. All Internet-based HTTP/1.1 servers MUST respond with a 400 (Bad Request) status code to any HTTP/1.1 request message which lacks a Host header field.
TIdHTTPServer returns a 400 response if it receives an HTTP 1.1 request without a Host header. As you can see above, the 400 response is mandatory by the HTTP 1.1 spec. I suggest you contact the device manufacturer and report a bug about the missing Host header, maybe they can release a firmware update.
In the meantime, you can use the TIdHTTPServer.OnHeadersAvailable event to insert a dummy Host header if it is missing:
procedure httpServerHeadersAvailable(AContext: TIdContext; const AUri: string; AHeaders: TIdHeaderList; var VContinueProcessing: Boolean);
begin
if AHeaders.Values['Host'] = '' then
AHeaders.Values['Host'] = 'myserver';
VContinueProcessing := true;
end;
As for your log, it shows an Access Violation occurring in your OnCommandGet event handler, due to a nil pointer being accessed. That event is not triggered in the case when the client Host header is missing. So you clearly have a second issue in your code, not related to this issue.

Extract useful data from wireshark/tcpdump

I'm trying to extract data from gathered packages (tcpdump/wireshark). If I go to a website, all I can capture are the headers, but not the content of the webpage. Example:
Tcpdump:
17:34:51.861910 IP HackMachine-G51J.47928 > 50.6.246.185.http: Flags
[P.], seq 511:1032, ack 181, win 237, options [nop,nop,TS val 9134579
ecr 2921721692], length 521
E..=.8#.#.....V2....8.PiI................. ..a..%.\GET /default.css
HTTP/1.1 Host: www.rationallyparanoid.com User-Agent: Mozilla/5.0
(X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0 Accept:
text/css,/;q=0.1 Accept-Language: en-US,en;q=0.5 Accept-Encoding:
gzip, deflate DNT: 1 Referer:
http://www.rationallyparanoid.com/articles/tcpdump.html x-pzi27:
kill+911+warfare x-khy3445: Dear%20NSA%2C%0Afuck%20you%21 Connection:
keep-alive If-Modified-Since: Sat, 20 Apr 2013 23:47:10 GMT
If-None-Match: "3660064-14dd-517328fe" Cache-Control: max-age=0
All I get are headers. Does someone have any idea how to extract the content?
You can't get it from the response that packet, because it's not delivered!
HTTP supports an "If-Modified-Since" header; as the RFC says:
The "If-Modified-Since" header field makes a GET or HEAD request
method conditional on the selected representation's modification date
being more recent than the date provided in the field-value.
Transfer of the selected representation's data is avoided if that
data has not changed.
As you can see, the reply, in the next packet, has a reply of "304 Not Modified", meaning that the page in question hasn't changed since the time specified in the If-Modified-Since header, so any copy the machine already fetched at that time is Good Enough.
If you want the content of the page to show up in a network trace, you would have to convince your browser to discard any copy it's saved, so that it doesn't use If-Modified-Since; I don't know how that's done with Firefox (I assume, from the headers, that you're using Firefox), but repeatedly trying to fetch the page might be treated as a "discard the cached copy" indication, and there might also be ways in the Firefox UI to discard cached copies of pages.

Wireshark POST data

This is the captured data from wireshark
POST /r HTTP/1.1
Content-Type: application/x-www-form-urlencoded
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.0.4; GT-I9100 Build/IMM76L)(en-us)
Cache-Control: no-transform
Host: xx.xx.xx.xx
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 77
WLL202GUI#00000058$CuII4425339CnsI4425339CzsXT3BQnVOa1ZR0OL0+0hLWwgCksHiqQ0V5HTTP/1.1 200 OK
Server: piled
Keep-Alive: timeout=30, max=300
Connection: keep-alive
Content-Type: application/octet-stream
Content-Length: 103
WLL202GUI#00000084$ChsN989338254856CcsD98Cvsb90ccdc057d52d0e53d906f963aabcfa7CqsI4425339CmsHPedr#mCgIC1
What I know is that this is the POST data:
WLL202GUI#00000058$CuII4425339CnsI4425339CzsXT3BQnVOa1ZR0OL0+0hLWwgCksHiqQ0V5
and this the response:
WLL202GUI#00000084$ChsN989338254856CcsD98Cvsb90ccdc057d52d0e53d906f963aabcfa7CqsI4425339CmsHPedr#mCgIC1
(correct me if i'm wrong)
what is the full URI path for this? is it :
http://xx.xx.xx.xx/r
followed by the above data?
i mean how can i send the same post data and recieve the same response? or change some of the data ?
this packets was sent by an app from an android OS (using BlueStacks to be exact)
The post data immediately follows the headers you pasted and should be visible in the tree.
It is not secured by SSL. If it were, you wouldn't be able to read the headers like you have.

make HTTP Post request from blackberry java application

In my Blackberry java native application, i am making a call to a dotnet web service whose Request and Response format is defined as below. Here In response XML is being returned. I have been searching the forums and trying for 2-3 days but no success...
Request format :
POST /cinews.asmx/GetSeatLayout HTTP/1.1
Host: webservices.mclients.com
Content-Type: application/x-www-form-urlencoded
Content-Length: length
strCinews=string&strTransId1=string&lngSessionId=string&blnScreenOnTop=string&strMergeOption=string&strAreaCats=string
Response format :
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Content-Length: length
xml

How can I make a request with both GET and POST parameters?

Here's an excerpt from Live HTTP headers, I've replaced several values for anonymity.
POST blah/admin.php?module_id=1&action=update&id=129&pageNum=17&&eid=362 HTTP/1.1
Host: blah
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Firefox/3.6.12
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: blah
Cookie: blah
Content-Type: multipart/form-data; boundary=---------------------------21278813472729408841849703914
Content-Length: 5110
-----------------------------21278813472729408841849703914
Content-Disposition: form-data; name="MAX_FILE_SIZE"
300000000
This request has both GET and POST values. The script on the other end of this is PHP and expects certain values to be in the GET and others to be in the POST.
I know how to issue a GET
curl -G -d "key=val" "http://yadayadayada"
And I understand how to do a POST
curl -d "key=val" "http://yadayadayada"
curl -F "key=val" "http://yadayadayada"
But how do I mix the two in a single request? Every attempt I've made so far has ended in an error.
GET variables can be included in the URL. You just include the GET variables in the query string. For example, if you wanted to send a GET request with "username=fred" to www.example.com/index.php, you would send a simple GET request to "http://www.example.com/index.php?username=fred". So to answer your question, just use the POST method, but have the URL contain your GET data.
To clarify, GET and POST are HTTP request methods, not value types.
GET variables are called query string parameters. They are part of the URL, and can be included in any request.
POST variables are the contents of a urlencoded message body. These might also be sent with a PUT request.
Therefore, if you want to send both types of values, send the POST data as normal while explicitly writing your query string.
curl -d "key=val" "http://example.com?query_var=1"

Resources