I'm trying to extract data from gathered packages (tcpdump/wireshark). If I go to a website, all I can capture are the headers, but not the content of the webpage. Example:
Tcpdump:
17:34:51.861910 IP HackMachine-G51J.47928 > 50.6.246.185.http: Flags
[P.], seq 511:1032, ack 181, win 237, options [nop,nop,TS val 9134579
ecr 2921721692], length 521
E..=.8#.#.....V2....8.PiI................. ..a..%.\GET /default.css
HTTP/1.1 Host: www.rationallyparanoid.com User-Agent: Mozilla/5.0
(X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0 Accept:
text/css,/;q=0.1 Accept-Language: en-US,en;q=0.5 Accept-Encoding:
gzip, deflate DNT: 1 Referer:
http://www.rationallyparanoid.com/articles/tcpdump.html x-pzi27:
kill+911+warfare x-khy3445: Dear%20NSA%2C%0Afuck%20you%21 Connection:
keep-alive If-Modified-Since: Sat, 20 Apr 2013 23:47:10 GMT
If-None-Match: "3660064-14dd-517328fe" Cache-Control: max-age=0
All I get are headers. Does someone have any idea how to extract the content?
You can't get it from the response that packet, because it's not delivered!
HTTP supports an "If-Modified-Since" header; as the RFC says:
The "If-Modified-Since" header field makes a GET or HEAD request
method conditional on the selected representation's modification date
being more recent than the date provided in the field-value.
Transfer of the selected representation's data is avoided if that
data has not changed.
As you can see, the reply, in the next packet, has a reply of "304 Not Modified", meaning that the page in question hasn't changed since the time specified in the If-Modified-Since header, so any copy the machine already fetched at that time is Good Enough.
If you want the content of the page to show up in a network trace, you would have to convince your browser to discard any copy it's saved, so that it doesn't use If-Modified-Since; I don't know how that's done with Firefox (I assume, from the headers, that you're using Firefox), but repeatedly trying to fetch the page might be treated as a "discard the cached copy" indication, and there might also be ways in the Firefox UI to discard cached copies of pages.
Related
I am using pyshark to parse .pcap files specifically with HTTP packets. Unlike as in Wireshark, where it shows the \r\n\r\n bytes at the end of the HTTP header, pyshark does not show them and instead shows a single \r\n.
Is there any way to properly parse the HTTP layer of the packet to display the \r\n\r\n's?
If so, how? I have done a fair amount of searching through the web but the sources are limited and does not answer my question.
Also, with pyshark, the headers do not come in the same order as seen on Wireshark. Is there any reason to that as well?
Python code
#!/bin/env python3
import pyshark
packets = []
with pyshark.FileCapture('testing-mutillidae1.pcap') as capture:
for pkt in capture: # storing packets in list
packets.append(pkt)
print(packets[3]) # printing packet details of packet no. 4
HTTP header
I have included the full output of the packet on pastebin: https://pastebin.com/qxjxY6Hw . Since it is too long, I have added only the HTTP layer in this question
Layer HTTP:
GET /mutillidae/index.php?page=add-to-your-blog.php HTTP/1.1\r\n
Expert Info (Chat/Sequence): GET /mutillidae/index.php?page=add-to-your-blog.php HTTP/1.1\r\n
GET /mutillidae/index.php?page=add-to-your-blog.php HTTP/1.1\r\n
Severity level: Chat
Group: Sequence
Request Method: GET
Request URI: /mutillidae/index.php?page=add-to-your-blog.php
Request URI Path: /mutillidae/index.php
Request URI Query: page=add-to-your-blog.php
Request URI Query Parameter: page=add-to-your-blog.php
Request Version: HTTP/1.1
Host: 10.0.2.13\r\n
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
Accept-Language: en-US,en;q=0.5\r\n
Accept-Encoding: gzip, deflate\r\n
Referer: http://10.0.2.13/mutillidae/index.php\r\n
Connection: keep-alive\r\n
Cookie: showhints=0; PHPSESSID=511be46cfd6922ff8sqqhtqmbn\r\n
Cookie pair: showhints=0
Cache-Control: max-age=0\r\n
Full request URI: http://10.0.2.13/mutillidae/index.php?page=add-to-your-blog.php
HTTP request 1/1
\r\n
Upgrade-Insecure-Requests: 1\r\n
Cookie pair: PHPSESSID=511be46cfd6922ff8sqqhtqmbn
Here is the screenshot on my wireshark (I cannot post pictures yet)
This is the captured data from wireshark
POST /r HTTP/1.1
Content-Type: application/x-www-form-urlencoded
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.0.4; GT-I9100 Build/IMM76L)(en-us)
Cache-Control: no-transform
Host: xx.xx.xx.xx
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 77
WLL202GUI#00000058$CuII4425339CnsI4425339CzsXT3BQnVOa1ZR0OL0+0hLWwgCksHiqQ0V5HTTP/1.1 200 OK
Server: piled
Keep-Alive: timeout=30, max=300
Connection: keep-alive
Content-Type: application/octet-stream
Content-Length: 103
WLL202GUI#00000084$ChsN989338254856CcsD98Cvsb90ccdc057d52d0e53d906f963aabcfa7CqsI4425339CmsHPedr#mCgIC1
What I know is that this is the POST data:
WLL202GUI#00000058$CuII4425339CnsI4425339CzsXT3BQnVOa1ZR0OL0+0hLWwgCksHiqQ0V5
and this the response:
WLL202GUI#00000084$ChsN989338254856CcsD98Cvsb90ccdc057d52d0e53d906f963aabcfa7CqsI4425339CmsHPedr#mCgIC1
(correct me if i'm wrong)
what is the full URI path for this? is it :
http://xx.xx.xx.xx/r
followed by the above data?
i mean how can i send the same post data and recieve the same response? or change some of the data ?
this packets was sent by an app from an android OS (using BlueStacks to be exact)
The post data immediately follows the headers you pasted and should be visible in the tree.
It is not secured by SSL. If it were, you wouldn't be able to read the headers like you have.
I have an issue similar to Heroku & Rails - Varnish HTTP Cache Not Working, but the solution (wait for a while, then everything works) doesn't seem to apply - I've had the setup below for several days.
This thread on the Heroku Google group has some users with the same problem. They mention that it takes a while for everything to be cached, but my understanding is that after a while, everything should get cached, no? Or does that only apply if there is a Lot of traffic?
I need some advice on where I should be looking/what I can try changing in order to get caching working properly.
My setup:
I have http://www.swingoutlondon.co.uk running on Heroku (Rails 3.0.3, Ruby 1.9.2, bamboo-mri-1.9.2) and the main index page performs a lot of database queries to return what is essentially a static page - usually taking about 2-3 seconds (yes, that's something I really do need to address, but I figure varnish caching is a quick win).
I've set the Cache-Control response header as described here, and indeed that does seem to have been set on the page:
>> curl -I http://swingoutlondon.co.uk
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 13 May 2012 00:01:05 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: public, max-age=300
Etag: "2565201f3ae39c6a9a1f6b1fb8bbbe0a"
X-Ua-Compatible: IE=Edge,chrome=1
X-Runtime: 1.699667
Content-Length: 44224
Accept-Ranges: bytes
X-Varnish: 681634826
Age: 0
Via: 1.1 varnish
Note: Cache-Control: public, max-age=300
I assume that Age: 0 indicates that it hasn't retrieved a cached copy, and indeed the command returns in the normal slow 2-3 seconds.
If keep repeatedly trying that curl, I can occasionally a cached copy (the page loads in under half a second and Age is greater than 0).
I must confess to not fully understanding HTTP headers, but one clue might be: when Age is greater than 0, I get two lots of digits in X-Varnish (in all other cases I only get one set):
X-Varnish: 848670407 848650521
Here's what I've checked:
the source of is identical each time.
I have one before_filter on that page, which sets the time the page was last updated as an instance variable.
there are a number of cookies - as far as I can see they are all set by either Google Analytics or the Twitter or Facebook buttons.
For good measure, here are my Request headers:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-GB,en-US;q=0.8,en;q=0.6
Cache-Control:max-age=0
Connection:keep-alive
Cookie:__utma=264326157.189257391.1336869624.1336869624.1336869624.1; __utmb=264326157.2.10.1336869624; __utmc=264326157; __utmz=264326157.1336869624.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Host:www.swingoutlondon.co.uk
If-None-Match:"2565201f3ae39c6a9a1f6b1fb8bbbe0a"
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19
Ah well - turns out that because Heroku uses multiple independent Varnish servers, and because traffic to Swing Out London is relatively low, I shouldn't expect to have many pages served by the caches if my max-age is only 5 minutes. Setting it to 20 or 30 minutes results in much more caching.
I've written a detailed blog post collecting my learnings. Thanks to Garry Shulter for helping me out with this one.
I am using Ajax binding with the Grid and ran into a problem where ASP.NET MVC was throwing a HttpRequestValidationException when I attempted an operation on the grid that invoked the Ajax call (like sorting).
Using Fiddler I was able to determine my browser was attempting to post back my entire model in the query string and there were some characters that would have triggered ASP.NET's request validation.
I was able to work around this by simply adding
[ValidateInput(false)]
to my controller action.
I'm wondering why was the grid sending back so much data? And is this a symptom of a bug in my code or Telerik's?
Here is what I saw in Fiddler (note the extremely long query string, even though the HTTP verb is "POST").
POST http://127.0.0.1:52601/MyController/DatabindGrid?Items=System.Collections.ObjectModel.Collection%601%5BPMyNamespace.Data.ViewEntities.Alert%5D&DetailsClientUrl=&GetJsonUrl=%2FMonitor%2FGetGridJson&ViewName=&CurrentItem=&PageTitle=My%20Web%20Page&CopyrightText=Copyright%20%26%23x00A9%3B%202004-2010%20My%20Company%20Corporation&BrowserCapabilities=&OemName=&UiVersion=20101123 HTTP/1.1
Referer: http://ipv4.fiddler:52601/MyController/MyAction
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Accept: text/plain, */*; q=0.01
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: 127.0.0.1:52601
Content-Length: 56
Connection: Keep-Alive
Pragma: no-cache
Cookie: ASP.NET_SessionId=bvgwuno4bounqio2werepkw4; .ASPXAUTH=612E18BB916F720D13A0F0D1695A86079B94DFE94A3BF5A9A8F19E35A37AE987282B7B684201C112CEC6081181E3D1C52C5517A66D9158E4CF83C1C3F523EE32FF783BD2E3B6E0A42A35E1874E63BA76C7735F9E8ABBA4E58BF61EB29DA03789E07A201A1BA9E7B85F941516ED7EA26E3E8E1E65D0836F39A109201E357EE97478D1A359B3FB4B4AD4C64A02A0CE7BBB39DC8FE1F73B179F284A14CF55D9C67D
page=1&size=10&orderBy=LocationName-asc&groupBy=&filter=
Rick,
I think you might be interested in this...
http://tv.telerik.com/watch/aspnet-mvc/video/building-a-responsive-site-with-mvc
very detailed and informative.
~AZee
I have some static content on my web site that I have set up caching for (using Asp.NET MVC). According to Firebug, the first time I open the page, Firefox sends this request:
GET /CoreContent/Core.css?asm=0.7.3614.34951
Host: 127.0.0.1:3916
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Accept: text/css,*/*;q=0.1
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://127.0.0.1:3916/Edit/1/101
Cookie: .ASPXAUTH=52312E5A802C1A079E2BA29AA2BFBC5A38058977B84452D62ED52855D4164659B4307661EC73A307BFFB2ED3871C67CB3A9AAFDB3A75A99AC0A21C63A6AADE9A11A7138C672E75125D9FF3EFFBD9BF62
Pragma: no-cache
Cache-Control: no-cache
Which my server replies to with this:
Server: ASP.NET Development Server/9.0.0.0
Date: Mon, 23 Nov 2009 18:44:41 GMT
X-AspNet-Version: 2.0.50727
X-AspNetMvc-Version: 1.0
Cache-Control: public, max-age=31535671
Expires: Tue, 23 Nov 2010 18:39:12 GMT
Last-Modified: Mon, 23 Nov 2009 18:39:12 GMT
Vary: *
Content-Type: text/css
Content-Length: 15006
Connection: Close
So far, so good. However, if I refresh Firefox (not a cache-clearing refresh, just a normal one), during that refresh cycle Firefox will once again go to the server with this request:
GET /CoreContent/Core.css?asm=0.7.3614.34951
Host: 127.0.0.1:3916
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Accept: text/css,*/*;q=0.1
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://127.0.0.1:3916/Edit/1/101
Cookie: .ASPXAUTH=52312E5A802C1A079E2BA29AA2BFBC5A38058977B84452D62ED52855D4164659B4307661EC73A307BFFB2ED3871C67CB3A9AAFDB3A75A99AC0A21C63A6AADE9A11A7138C672E75125D9FF3EFFBD9BF62
If-Modified-Since: Mon, 23 Nov 2009 18:39:20 GMT
Cache-Control: max-age=0
to which my server responds 304 Not Modified.
Why does Firefox issue this second request? In the first response, I said that the cache does not expire for a year (I intend to use query parameters whenever things change). Do I have to add another response header to prevent this extra roundtrip?
Edit: It does not matter whether I press refresh, or whether I go to the page again (or a different URL, which references the same external files). Firefox does the same again. Also, I don't claim this to be a bug in FF, I just wonder if there is another header I can set which means "This document will never change, don't bother me again".
This behavior seems to be "by design" for all modern browsers. In IE you can investigate the same situation. Hitting F5 always causes browser to check whether content was modified. During cache-clearing request browser do not pass Last-Modified header and server must return HTTP 200 (not HTTP 304), so 304 in your situation is not so bad.
Apparently, the issue was due to a Vary: * header being added to the response. To remove this in Asp.Net, add this to web.config inside the <system.web> section:
<caching>
<outputCache omitVaryStar="true"/>
</caching>
To me it seems this switch is "yes, I want a working behaviour rather than a non-working one", but once you find it, flipping it is trivial.