can I view HTTP POST plaintext values when Content-Encoding: snappy is used for compression, I currently get "Decompression failed" in wireshark, you can recommend other tools, scripts etc. to get the job done. Context: debugging VictoriaMetrics time series values being sent here but could apply for other scenarios as well.
Attaching pcap dump here: https://paste.c-net.org/DesignerWoody
Related
I am using tshark to output the human-readable text from TCP packets. I do this with the data.text field. For example, my tshark.exe switches include:
-o data.show_as_text:TRUE -T fields -e data.text
This works just fine for most packets. The trouble comes when a packet is retransmitted by the sender. In these cases, data.text is always blank.
When analyzing the traffic in Wireshark, I see that retransmitted TCP frames (these are labeled as [TCP Retransmission] and include the note "This frame is a (suspected) retransmission") do not include the Data field at all - Wireshark does not reconstruct it (even though I have the parse data as text option enabled). Instead, these frames have a "Retransmitted TCP segment data" field (tcp.segment_data) which does not include any parsed text.
Is this a bug with tshark/wireshark? Why is the Data field not being constructed for these packets? Is there some other way for me to get the text from retransmitted TCP packets, or is my only option to manually decode the hex of tcp.segment_data for retransmitted packets?
The TCP connection I am monitoring frequently has retransmitted packets, so I am missing critical data at the moment.
Do you have TCP reassembly enabled? Maybe try to disable it with -o tcp.desegment_tcp_streams:FALSE? If that doesn't help, maybe there's a bug with the version of Wireshark you're using. You could try upgrading to the latest stable version (2.6.1 as of this writing) and try again. You could even try a recent automated installer, if one is available for your OS or you could try building Wireshark from sources. If the problem still persists, you could file a Wireshark bug report along with a sample capture file to illustrate the problem.
You may also want to add a Wireshark display filter to your command-line to limit the output to only those packets that contain data.text, e.g. -Y data.text.
Refer to the tshark man page for more information about the -Y and other options.
A website usually consists of multiple objects (e.g. text file, a few png files etc.), I would like to know if there's a tool that can capture the individual requests/responses in different pcap files?
So for example if I browse to http://somewebsite.com , and some http://somewebsite.com consists of say {index.html, image1.png, somestylefile.css, image2.png}, the tool would capture the entire load of http://somewebsite.com but generate {index.html.pcap, image1.png.pcap, somestylefile.css.pcap, image2.png.pcap}
I don't know of any tool that can do this, or its possible using scapy or something similar?
A HTTP connection can have multiple requests inside the same TCP connection and browsers make heavy use of this HTTP keep alive. With HTTP pipelining the requests/responses don't even need to be fully separated in time, i.e. a client can send another request even though the response for the previous one is not there. And with HTTP/2 the data can also be interleaved, i.e. several responses transferred at the same time inside the same connection.
Insofar it is not always possible to capture the data as separate pcap file because they might not be separable at the packet level. But if you don't need the original packet boundaries it would be possible to create separate pcap files for each request which not necessarily reflect the original packets but which reflect the application layer, i.e. the response matching the request.
One tool which makes this is httpflow.pl which can extract HTTP/1.x requests/response pairs from an existing pcap (or sniff directly) and writes each request/response into a separate pcap file, as if it would have been a separate TCP connection. It can also clean up the data for easier analysis, i.e. unchunk and uncompress the HTTP body.
I need to make tcpdump using wireshark or tcpdump
For some tcp streams in log I need full packet info.
But for tcp streams in log I need to remove all info except full uri info.
For this I'am going to use regular expressions.
If URI, http.request or http.response match some of regular expressions
then I need to store full tcp stream info.
If doesn't match --- then I need to remove all tcp stream info except full uri info.
I need it to reduce log size.
What tool do you advise for this ?
Or may be some php/python library will be helpful to create script that will filter log ?
tcpick and tcpxtract are two tools available to filter and extract information from a tcpdump file. They are both open-source and available at sourceforge.net. You will have to write your own routine for how and what information you would like to extract. tcpick is fairly flexible regarding what parts of a packet you can view/extract.
I use TraceWrangler in my projects, which despite being beta, works very well. It allows, in addition to editing the L2/L3 headers, to sanitize the packets.
Is it possible to modify users' HTTP request to
www.example.com/options
instead of
www.example.com/options_and_params
My scenario is that about 30000 users connect to my company's network backbone and I want to add one or more server (with the code I'm current working on) between the backbone switches and Radware LoadProof to achieve this work.
After googling all the night, I have no lead but some more questions:
I don't need to intercept every packet through the network. With some helps like iptables, I can filter out the package I want. I have done it before using iptables. However, packet is not equal to HTTP stream. Do I need to do HTTP re-construct?
If I successfully find a way to modify HTTP request URL content, I still should put it back to network stream. As I know TCP packets have a checksum and after I modify the content it must be wrong. How do I calculate a new checksum and put the packet back to network?
It's my first time to do network programming or packet processing develop. Any suggestion is appreciate.
This depends on whether you are doing HTTP/1.0 or HTTP/1.1 and whether its an initial request you need to modify or all requests in a single HTTP 1.1 session.
If you have the packet and can modify it before it is sent on and you are trying to modify just the request then given the length of a typical packet and the location of the URL in the HTTP request stream (very near the beginning) and the fact that it will be the first thing sent in the TCP stream I think you can fairly safely assume that it will be present in the first N bytes of the first packet sent and therefore won't be split over multiple packets.
However, if this is an HTTP/1.1 stream then multiple requests will be being sent via the same TCP connection in which case in future requests the URL may well be split over two TCP packets.
If you can maybe force HTTP/1.0 or possibly if you modify the initial or all requests to be HTTP/1.0 then you can be pretty sure that the first packet will correspond to the first packet of the TCP stream and that you are very unlikely to see the URL split over multiple packets, meaning no reconstruction and the ability to just do a replace.
However this will come at a cost of new TCP connections which is pretty inefficient.
If you don't and you leave it as HTTP/1.1 then the URL could be at any random point in any future request and therefore split over multiple TCP packets (two realistically given the size of the URL).
If I got your question right, then this could be probably done with some fast reverse-proxy like nginx.
I'm looking for a way to find out how long a server has been up based only on the page it sends back. Like, if I went to www.google.com, is there some sort of response header variable that tells how long the server I connected to has been up? I'm doubting there is, but never hurts to ask...
No, because HTTP, as a protocol, doesn't really care. In any case, 50 different requests to google.com may end up at 50 different servers.
If you want that information, you need to build it into the application, something like "http://google.com/uptime" which will deliver Google's view of how long it's been up - Google will probably do this as a static page showing the date of their inception :-).
Not from HTTP.
It is possible, however, to discover uptime on many OSs by interrogating the the TCP packets received. Look to RFC 1323 for more information. I believe that a timestamp header is incremented by some value with every transaction, and reset to zero on reboot.
Caveats: it doesn't work with all OSs, and you've got to track servers over time to get accurate uptime data.
Netcraft does this: see here for a vague description:
The 'uptime' as presented in these
reports is the "time since last
reboot" of the front end computer or
computers that are hosting a site. We
can detect this by looking at the data
that we record when we sample a site.
We can detect how long the responding
computer(s) hosting a web site has
been running, and by recording these
samples over a long period of time we
can plot graphs that show this as a
line. Note that this is not the same
as the availability of a site.
Unfortunately there really isn't. You can check this for yourself by requesting the HTTP headers from the server in question. For example, from google.com you will get:
HTTP/1.0 200 OK
Cache-Control: private, max-age=0
Date: Mon, 08 Jun 2009 03:08:11 GMT
Expires: -1
Content-Type: text/html; charset=UTF-8
Server: gws
Online tool to check HTTP headers:
http://network-tools.com/default.asp?prog=httphead&host=www.google.com
Now if it's your own server, you can create a script that will report the uptime, but I don't think that's what you were looking for.
To add to what Pax said, there are a number of third party services that monitor site up-time by attempting to access server resources at specific intervals. They maintain some amount of history in their own databases, and then report back to you when those resources are inaccessible.
I use Uptime Party for monitoring a server.