How do I use tshark to print request-response pairs from a pcap file? - wireshark

Given a pcap file, I'm able to extract a lot of information from the reconstructed HTTP request and responses using the neat filters provided by Wireshark. I've also been able to split the pcap file into each TCP stream.
Trouble I'm running into now is that of all the cool filters I'm able to use with tshark, I can't find one that will let me print out full request/response bodies. I'm calling something like this:
tshark -r dump.pcap -R "tcp.stream==123 and http.request" -T fields -e http.request.uri
Is there some filter name I can pass to -e to get the request/response body? The closest I've come is to use the -V flag, but it also prints out a bunch of information I don't necessary want and want to avoid having to kludge out with a "dumb" filter.

If you are willing to switch to another tool, tcptrace can do this with the -e option. It also has an HTTP analysis extension (xHTTP option) that generates the HTTP request/repsonse pairs for each TCP stream.
Here is a usage example:
tcptrace --csv -xHTTP -f'port=80' -lten capturefile.pcap
--csv to format output as comma sperated variable
-xHTTP for HTTP request/response written to 'http.times' this also switches on -e to dump the TCP stream payloads, so you really don't need -e as well
-f'port=80' to filter out non-web traffic
-l for long output form
-t to give me progress indication
-n to turn off hostname resolution (much faster without this)

If you captured a pcap file, you can do the following to show all requests+responses.
filename="capture_file.pcap"
for stream in `tshark -r "$filename" -2 -R "tcp and (http.request or http.response)" -T fields -e tcp.stream | sort -n | uniq`; do
echo "==========BEGIN REQUEST=========="
tshark -q -r "$filename" -z follow,tcp,ascii,$stream;
echo "==========END REQUEST=========="
done;
I just made diyism answer a bit easier to understand (you don't need sudo, and multiline script is imo simple to look at)

This probably wasn't an option when the question was asked but newer versions of tshark can "follow" conversations.
tshark -nr dump.pcap -qz follow,tcp,ascii,123
I know this is a super old question. I'm just adding this for anyone that ends up here looking for a current solution.

I use this line to show last 10 seconds request body and response body(https://gist.github.com/diyism/eaa7297cbf2caff7b851):
sudo tshark -a duration:10 -w /tmp/input.pcap;for stream in `sudo tshark -r /tmp/input.pcap -R "tcp and (http.request or http.response) and !(ip.addr==192.168.0.241)" -T fields -e tcp.stream | sort -n | uniq`; do sudo tshark -q -r /tmp/input.pcap -z follow,tcp,ascii,$stream; done;sudo rm /tmp/input.pcap

Related

How do I check if an IP address appears in a PCAP file?

Is it possible to use tshark to check whether one or more ip addresses appear in a pcap file? I know that I can display all occurrences with tshark -r infile -w outfile ip.addr==172.26.29.2 || ip.addr==172.26.31.21, but is there an option to not display all (maybe only the first occurrence.)?
You should be able to achieve this, either taking these few steps:
Get the list of all source IP addresses:
tshark -r infile -Y "ip" -T fields -e ip.src > infile_ips.txt
Append with the list of all destination IP addresses:
tshark -r infile -Y "ip" -T fields -e ip.dst >> infile_ips.txt
Ensure all IPs are listed on separate lines, which they might not be in the case of tunneled IPs or for ICMP errors, then sort the list and eliminate duplicates:
cat infile_ips.txt | tr , '\n' | sort -u
Or a bit more easily using this single step:
tshark -r infile -Y "ip" -T fields -e ip.addr | tr , '\n' | sort -u
Once you have the list of all unique IP addresses present in the capture file, you can search that list for whatever IP address you're interested in by using grep or by whatever other means you want.
Refer to the tshark man page for more information regarding the options I used above.
Another command that would work would be
tshark -q -z endpoints,ip -z endpoints,ipv6 -r infile
which would produce tables of IPv4 and IPv6 addresses, as well as, for each address, information about the amount of data transmitted from and to that address.

Exporting encrypted SNMPv3 traps to JSON with TShark

I have a pcap file with recordings of encrypted SNMPv3 traps from Wireshark (Version 3.2.2). For analyzing the traps, I want to export the protocol data to json using tshark.
tshark.exe -T ek -Y "snmp" -P -V -x -r input.pcap > output.json
Currently, I supply the infos to decrypt the packages via the "snmp_users" file in C:\Users\developer\AppData\Roaming\Wireshark.
# This file is automatically generated, DO NOT MODIFY.
,"snmp_user","SHA1","xxxxxx","AES","yyyyyyy"
Is it possible to supply the options via commandline?
I have tried:
tshark.exe -T ek -Y "snmp" -P -V -x -o "snmp.users_table.username:snmp_user" ...
But that causes an error:
tshark: -o flag "snmp.users_table.username:snmp_user" specifies unknown preference
Update 16.09.2020:
Option -Y used instead of -J:
-Y|--display-filter
Cause the specified filter (which uses the syntax of read/display
filters, rather than that of capture filters) to be applied before
printing a decoded form of packets or writing packets to a file.
You need to specify the option as a User Access Table or uat, with the specific table being the name of the file, namely snmp_users. So, for example:
On Windows:
tshark.exe -o "uat:snmp_users:\"\",\"snmp_user\",\"SHA1\",\"xxxxxx\",\"AES\",\"yyyyyyy\"" -T ek -J "snmp" -P -V -x -r input.pcap > output.json
And on *nix:
tshark -o 'uat:snmp_users:"","snmp_user","SHA1","xxxxxx","AES","yyyyyyy"' -T ek -J "snmp" -P -V -x -r input.pcap > output.json
Unfortunately, the Wireshark documentation is currently lacking in describing the uat option. There is a Google Summer of Code project underway though, of which Wireshark is participating, so perhaps documentation will be improved here.

Converting a pcap file to csv: Tshark displays multiple src,dst IP addresses in a single line for some packets

I want to convert a pcap file to csv/tsv with "Tshark" where each line corresponds to a packet and have the following format:
timestamp src_ip dst_ip protocol
I use this command:
tshark -r <file_name.pcap> -T fields -e frame.time_epoch -e ip.src -e ip.dst -e ip.proto
However, in the displayed output I see some lines to have two src,dst IP addresses and protocol number like this:
1525794300.3842 92.153.107.1,203.46.108.229 203.46.108.229,85.50.172.78 1,1
While for the rest, each line has one src IP, one dst IP and one protocol like the following:
1525794300.3843 185.61.46.124 163.218.99.213 6
Is there any reason that tshark is displaying multiple src and dst ip addresses in a single line? Can we do something so tshark does not do this?
Thanks!
The reason tshark displays multiple src and dst IP addresses as well as multiple protocol numbers is because there are multiple IP headers in the packet. In this case, it's ICMP packet carrying information about another ICMP packet, perhaps a "Time to live exceeded in transit" or some other such error. If you open the file in Wireshark or run tshark -r <file_name.pcap> -Y "icmp", you will be able to see this for yourself.
If you're only interested in the first (outer) IP src and dst addresses and protocol number, then you can limit the output to the first occurrence of each field as follows:
tshark -r <file_name.pcap> -T fields -E occurrence=f -e frame.time_epoch -e ip.src -e ip.dst -e ip.proto
Alternatively, you can specify columns while limiting the field occurrences this way:
On *nix:
tshark -r <file_name.pcap> -o 'gui.column.format:"Epoch Time","%Cus:frame.time_epoch","Src","%Cus:ip.src:1","Dst","%Cus:ip.dst:1","Proto","%Cus:ip.proto:1"'
On Windows:
tshark.exe -r <file_name.pcap> -o "gui.column.format:\"Epoch Time\",\"%Cus:frame.time_epoch\",\"Src\",\"%Cus:ip.src:1\",\"Dst\",\"%Cus:ip.dst:1\",\"Proto\",\"%Cus:ip.proto:1\""
Run tshark -G column-formats for additional help with the column formats.

Extract specific byte offset using tshark

I have a pcap of ICMP packets. I am trying to use tshark to extract the payload data so that I can extract a specific byte offset.
The tshark documentation is highly convoluted, especially for me, a beginner.
I've been searching around a lot and I'm trying to piece together a command for the purpose of my goal.
I can run the following command:
shark -r test.pcapng -Y icmp -z flow,icmp,network > output.bin
But it only outputs the packet list as it were shown in Wireshark.
For example, I am trying to extract the following byte offset from each packet (offset 22):
How would I go about extracting a specific byte offset with tshark?
EDIT:
Issuing the following command only returns a portion of the payload data, how can I get all of it?
tshark -r test.pcapng -Y "frame.number == 13" -T fields -e data -w output.bin
I've provided an answer over at https://ask.wireshark.org/question/14795/extract-specific-byte-offset-using-tshark/, but for convenience, I'll summarize the 2 possible solutions I provided here. In a nutshell:
The highlighted byte in the image appears to be the TTL field of the IP header. If that's the field you're interested in, you can obtain it via:
tshark -r test.pcapng -Y "frame.number == 13" -T fields -e ip.ttl -w output.bin
If you're looking for a more general solution to print the 22nd byte of the packet, regardless of whether it's the ip.ttl field or not, then you can use a solution such as:
tshark -r test.pcapng -Y "frame.number == 13" -x -w output.bin | grep "^0010" | cut -d ' ' -f 9
The 2nd solution above also illustrates how you can dump all the bytes; it's done using tshark's -x option.

Spider a Website and Return URLs Only

I'm looking for a way to pseudo-spider a website. The key is that I don't actually want the content, but rather a simple list of URIs. I can get reasonably close to this idea with Wget using the --spider option, but when piping that output through a grep, I can't seem to find the right magic to make it work:
wget --spider --force-html -r -l1 http://somesite.com | grep 'Saving to:'
The grep filter seems to have absolutely no affect on the wget output. Have I got something wrong or is there another tool I should try that's more geared towards providing this kind of limited result set?
UPDATE
So I just found out offline that, by default, wget writes to stderr. I missed that in the man pages (in fact, I still haven't found it if it's in there). Once I piped the return to stdout, I got closer to what I need:
wget --spider --force-html -r -l1 http://somesite.com 2>&1 | grep 'Saving to:'
I'd still be interested in other/better means for doing this kind of thing, if any exist.
The absolute last thing I want to do is download and parse all of the content myself (i.e. create my own spider). Once I learned that Wget writes to stderr by default, I was able to redirect it to stdout and filter the output appropriately.
wget --spider --force-html -r -l2 $url 2>&1 \
| grep '^--' | awk '{ print $3 }' \
| grep -v '\.\(css\|js\|png\|gif\|jpg\)$' \
> urls.m3u
This gives me a list of the content resource (resources that aren't images, CSS or JS source files) URIs that are spidered. From there, I can send the URIs off to a third party tool for processing to meet my needs.
The output still needs to be streamlined slightly (it produces duplicates as it's shown above), but it's almost there and I haven't had to do any parsing myself.
Create a few regular expressions to extract the addresses from all
<a href="(ADDRESS_IS_HERE)">.
Here is the solution I would use:
wget -q http://example.com -O - | \
tr "\t\r\n'" ' "' | \
grep -i -o '<a[^>]\+href[ ]*=[ \t]*"\(ht\|f\)tps\?:[^"]\+"' | \
sed -e 's/^.*"\([^"]\+\)".*$/\1/g'
This will output all http, https, ftp, and ftps links from a webpage. It will not give you relative urls, only full urls.
Explanation regarding the options used in the series of piped commands:
wget -q makes it not have excessive output (quiet mode).
wget -O - makes it so that the downloaded file is echoed to stdout, rather than saved to disk.
tr is the unix character translator, used in this example to translate newlines and tabs to spaces, as well as convert single quotes into double quotes so we can simplify our regular expressions.
grep -i makes the search case-insensitive
grep -o makes it output only the matching portions.
sed is the Stream EDitor unix utility which allows for filtering and transformation operations.
sed -e just lets you feed it an expression.
Running this little script on "http://craigslist.org" yielded quite a long list of links:
http://blog.craigslist.org/
http://24hoursoncraigslist.com/subs/nowplaying.html
http://craigslistfoundation.org/
http://atlanta.craigslist.org/
http://austin.craigslist.org/
http://boston.craigslist.org/
http://chicago.craigslist.org/
http://cleveland.craigslist.org/
...
I've used a tool called xidel
xidel http://server -e '//a/#href' |
grep -v "http" |
sort -u |
xargs -L1 -I {} xidel http://server/{} -e '//a/#href' |
grep -v "http" | sort -u
A little hackish but gets you closer! This is only the first level. Imagine packing this up into a self recursive script!

Resources