I run the following
sudo tshark -c 100 -f "tcp port 80" -T json -e tcp.payload
and one of the fields I am interested in is tcp.payload and I get it in the below format
{"tcp.payload": "7b:22:6e:61:6d:65:22:3a:20:22:68:65:6c:6c:6f:22:7d"}
How can I get it in the two formats below?
{"tcp.payload": "7b226e616d65223a202268656c6c6f227d"}
{"tcp.payload": {"name": "hello"}}
Related
Is it possible to use tshark to check whether one or more ip addresses appear in a pcap file? I know that I can display all occurrences with tshark -r infile -w outfile ip.addr==172.26.29.2 || ip.addr==172.26.31.21, but is there an option to not display all (maybe only the first occurrence.)?
You should be able to achieve this, either taking these few steps:
Get the list of all source IP addresses:
tshark -r infile -Y "ip" -T fields -e ip.src > infile_ips.txt
Append with the list of all destination IP addresses:
tshark -r infile -Y "ip" -T fields -e ip.dst >> infile_ips.txt
Ensure all IPs are listed on separate lines, which they might not be in the case of tunneled IPs or for ICMP errors, then sort the list and eliminate duplicates:
cat infile_ips.txt | tr , '\n' | sort -u
Or a bit more easily using this single step:
tshark -r infile -Y "ip" -T fields -e ip.addr | tr , '\n' | sort -u
Once you have the list of all unique IP addresses present in the capture file, you can search that list for whatever IP address you're interested in by using grep or by whatever other means you want.
Refer to the tshark man page for more information regarding the options I used above.
Another command that would work would be
tshark -q -z endpoints,ip -z endpoints,ipv6 -r infile
which would produce tables of IPv4 and IPv6 addresses, as well as, for each address, information about the amount of data transmitted from and to that address.
I have a pcap file with recordings of encrypted SNMPv3 traps from Wireshark (Version 3.2.2). For analyzing the traps, I want to export the protocol data to json using tshark.
tshark.exe -T ek -Y "snmp" -P -V -x -r input.pcap > output.json
Currently, I supply the infos to decrypt the packages via the "snmp_users" file in C:\Users\developer\AppData\Roaming\Wireshark.
# This file is automatically generated, DO NOT MODIFY.
,"snmp_user","SHA1","xxxxxx","AES","yyyyyyy"
Is it possible to supply the options via commandline?
I have tried:
tshark.exe -T ek -Y "snmp" -P -V -x -o "snmp.users_table.username:snmp_user" ...
But that causes an error:
tshark: -o flag "snmp.users_table.username:snmp_user" specifies unknown preference
Update 16.09.2020:
Option -Y used instead of -J:
-Y|--display-filter
Cause the specified filter (which uses the syntax of read/display
filters, rather than that of capture filters) to be applied before
printing a decoded form of packets or writing packets to a file.
You need to specify the option as a User Access Table or uat, with the specific table being the name of the file, namely snmp_users. So, for example:
On Windows:
tshark.exe -o "uat:snmp_users:\"\",\"snmp_user\",\"SHA1\",\"xxxxxx\",\"AES\",\"yyyyyyy\"" -T ek -J "snmp" -P -V -x -r input.pcap > output.json
And on *nix:
tshark -o 'uat:snmp_users:"","snmp_user","SHA1","xxxxxx","AES","yyyyyyy"' -T ek -J "snmp" -P -V -x -r input.pcap > output.json
Unfortunately, the Wireshark documentation is currently lacking in describing the uat option. There is a Google Summer of Code project underway though, of which Wireshark is participating, so perhaps documentation will be improved here.
I want to convert a pcap file to csv/tsv with "Tshark" where each line corresponds to a packet and have the following format:
timestamp src_ip dst_ip protocol
I use this command:
tshark -r <file_name.pcap> -T fields -e frame.time_epoch -e ip.src -e ip.dst -e ip.proto
However, in the displayed output I see some lines to have two src,dst IP addresses and protocol number like this:
1525794300.3842 92.153.107.1,203.46.108.229 203.46.108.229,85.50.172.78 1,1
While for the rest, each line has one src IP, one dst IP and one protocol like the following:
1525794300.3843 185.61.46.124 163.218.99.213 6
Is there any reason that tshark is displaying multiple src and dst ip addresses in a single line? Can we do something so tshark does not do this?
Thanks!
The reason tshark displays multiple src and dst IP addresses as well as multiple protocol numbers is because there are multiple IP headers in the packet. In this case, it's ICMP packet carrying information about another ICMP packet, perhaps a "Time to live exceeded in transit" or some other such error. If you open the file in Wireshark or run tshark -r <file_name.pcap> -Y "icmp", you will be able to see this for yourself.
If you're only interested in the first (outer) IP src and dst addresses and protocol number, then you can limit the output to the first occurrence of each field as follows:
tshark -r <file_name.pcap> -T fields -E occurrence=f -e frame.time_epoch -e ip.src -e ip.dst -e ip.proto
Alternatively, you can specify columns while limiting the field occurrences this way:
On *nix:
tshark -r <file_name.pcap> -o 'gui.column.format:"Epoch Time","%Cus:frame.time_epoch","Src","%Cus:ip.src:1","Dst","%Cus:ip.dst:1","Proto","%Cus:ip.proto:1"'
On Windows:
tshark.exe -r <file_name.pcap> -o "gui.column.format:\"Epoch Time\",\"%Cus:frame.time_epoch\",\"Src\",\"%Cus:ip.src:1\",\"Dst\",\"%Cus:ip.dst:1\",\"Proto\",\"%Cus:ip.proto:1\""
Run tshark -G column-formats for additional help with the column formats.
Using the below, I'm able to get both the raw predictions and the final predictions as a file:
cat train.vw.txt | vw -c -k --passes 30 --ngram 5 -b 28 --l1 0.00000001 --l2 0.0000001 --loss_function=logistic -f model.vw --compressed --oaa 3
cat test.vw.txt | vw -t -i model.vw --link=logistic -r raw.txt -p predictions.txt
However, I'm unable to get the raw predictions when I run VW as a daemon:
vw -t -i model.vw --daemon --port 26542 --link=logistic
Do I have a pass in a specific argument or parameter to get the raw predictions? I prefer the raw predictions, not the final predictions. Thanks
On systems supporting /dev/stdout (and /dev/stderr), you may try this:
vw -t -i model.vw --daemon --port 26542 --link=logistic -r /dev/stdout
The daemon will write raw predictions into standard output which in this case end up in the same place as localhost port 26542.
The relative order of lines is guaranteed because the code dealing with different prints within each example (e.g non-raw vs raw) is always serial.
Since November 2015, the easiest way how to obtain probabilities is to use --oaa=N --loss_function=logistic --probabilities -p probs.txt. (Or if you need label-dependent features: --csoaa_ldf=mc --loss_function=logistic --probabilities -p probs.txt.)
--probabilities work with --daemon as well. There should be no more need for using --raw_predictions.
--raw_predictions is a kind of hack (the semantic depends on the reductions used) and it is not supported in --daemon mode. (Something like --output_probabilities would be useful and not difficult to implement and it would work in daemon mode, but so far no one had time to implement it.)
As a workaround, you can run VW in a pipe, so it reads stdin and writes the probabilities to stdout:
cat test.data | vw -t -i model.vw --link=logistic -r /dev/stdout | script.sh
According to https://github.com/VowpalWabbit/vowpal_wabbit/issues/1118 you can try adding --scores option in command line:
vw --scores -t -i model.vw --daemon --port 26542
It helped me with my oaa model.
Given a pcap file, I'm able to extract a lot of information from the reconstructed HTTP request and responses using the neat filters provided by Wireshark. I've also been able to split the pcap file into each TCP stream.
Trouble I'm running into now is that of all the cool filters I'm able to use with tshark, I can't find one that will let me print out full request/response bodies. I'm calling something like this:
tshark -r dump.pcap -R "tcp.stream==123 and http.request" -T fields -e http.request.uri
Is there some filter name I can pass to -e to get the request/response body? The closest I've come is to use the -V flag, but it also prints out a bunch of information I don't necessary want and want to avoid having to kludge out with a "dumb" filter.
If you are willing to switch to another tool, tcptrace can do this with the -e option. It also has an HTTP analysis extension (xHTTP option) that generates the HTTP request/repsonse pairs for each TCP stream.
Here is a usage example:
tcptrace --csv -xHTTP -f'port=80' -lten capturefile.pcap
--csv to format output as comma sperated variable
-xHTTP for HTTP request/response written to 'http.times' this also switches on -e to dump the TCP stream payloads, so you really don't need -e as well
-f'port=80' to filter out non-web traffic
-l for long output form
-t to give me progress indication
-n to turn off hostname resolution (much faster without this)
If you captured a pcap file, you can do the following to show all requests+responses.
filename="capture_file.pcap"
for stream in `tshark -r "$filename" -2 -R "tcp and (http.request or http.response)" -T fields -e tcp.stream | sort -n | uniq`; do
echo "==========BEGIN REQUEST=========="
tshark -q -r "$filename" -z follow,tcp,ascii,$stream;
echo "==========END REQUEST=========="
done;
I just made diyism answer a bit easier to understand (you don't need sudo, and multiline script is imo simple to look at)
This probably wasn't an option when the question was asked but newer versions of tshark can "follow" conversations.
tshark -nr dump.pcap -qz follow,tcp,ascii,123
I know this is a super old question. I'm just adding this for anyone that ends up here looking for a current solution.
I use this line to show last 10 seconds request body and response body(https://gist.github.com/diyism/eaa7297cbf2caff7b851):
sudo tshark -a duration:10 -w /tmp/input.pcap;for stream in `sudo tshark -r /tmp/input.pcap -R "tcp and (http.request or http.response) and !(ip.addr==192.168.0.241)" -T fields -e tcp.stream | sort -n | uniq`; do sudo tshark -q -r /tmp/input.pcap -z follow,tcp,ascii,$stream; done;sudo rm /tmp/input.pcap