The option -Y, -2 and -R in tshark confuse me a long time.
After I read the manual, I know that -Y is used in single-pass filter and -2 in 2-pass filter (in case where we can not get some info until 1st pass filter is over)
But I still can not understand what is the difference between -2 -Y 'blabla' and -2 -R 'balabala' and -2 -Y 'blalal' -R 'blala'
And I also did an experiment that drive me crazy:
tshark -n -r test.pcap -2 -R 'frame.number > 0'
1 0.000000 10.140.28.17 -> 10.74.68.58 TCP 80 62276 > 8989 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1460 WS=32 TSval=330325315 TSecr=0 SACK_PERM=1
2 0.000056 10.74.68.58 -> 10.140.28.17 TCP 76 8989 > 62276 [SYN, ACK, ECN] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=2078759468 TSecr=330325315 WS=128
3 0.000678 10.140.28.17 -> 10.74.68.58 TCP 68 62276 > 8989 [ACK] Seq=1 Ack=1 Win=131744 Len=0 TSval=330325316 TSecr=2078759468
4 0.000756 10.140.28.17 -> 10.74.68.58 HTTP 158 GET /index.html HTTP/1.1
5 0.000770 10.74.68.58 -> 10.140.28.17 TCP 68 8989 > 62276 [ACK] Seq=1 Ack=91 Win=29056 Len=0 TSval=2078759468 TSecr=330325316
But when I execute tshark -n -r test.pcap -2 -R 'frame.number > 1', there is nothing printed. How to explain this?
My tshark version is: TShark 1.10.6 (v1.10.6 from master-1.10)
Can you help me with this problem? Thank you in advance!
I answered this question on http://ask.wireshark.org, but I'll paste my answer here as well in case anyone looks here for an answer instead of there.
-R specifies a read filter, so only matching packets from a file are read and processed; unmatched packets are essentially treated as if the file didn't contain them at all. Contrast this with -Y, which specifies a display filter, so only matching packets are displayed, but all packets are still read and processed.
The problem you're seeing with frame.number is a known bug that was determined not worth fixing long ago. See Bug 380, "wireshark -R doesn't support 'frame.number' as a read filter ".
You can also follow along some of the history behind -R vs. -Y here:
Bug 8223
Bug 8316
Bug 8529
Bug 9048
Wireshark-dev mailing list discussion:
Wireshark's Local archive
MARC archives
The Mail Archive
Related
I am very new to wireshark. in my day to day job i need to remove the packet bytes zero records from captured PCAP file. please help me in this process. attached image is for reference
wireshark packets zero.png
Since you have 47 TCP Streams and 28 that you want to remove, it might be a bit faster to filter for all the TCP streams that you do want to keep since there are only 19 of those.
For the 19 streams you want:
Right-click on the first TCP conversation and choose "Prepare a Filter -> Selected -> A<-->B".
For the next 17 TCP conversations, right-click on each one and choose "Prepare a Filter -> ... And Selected -> A<-->B".
Finally, for the last TCP stream, right-click on the TCP conversation and choose "Apply as Filter -> ... And Selected -> A<-->B".
You may wish to export the resulting filtered packets to a new file via "File -> Export Specified Packets... -> All packets:Displayed" so you won't have to keep filtering for those streams anymore.
If you have a large number of streams to filter, then you are better off scripting something. Here's a script you can use that seems to work well in my testing on my Linux machine. If you're using Windows, you will need to write an equivalent batch file, or you may be able to use it as is if you have Cygwin installed.
#!/bin/sh
# Check usage
if [ ${#} -lt 2 ] ; then
echo "Usage: $0 <infile> <outfile>"
exit 0
fi
infile=${1}
outfile=${2}
# TODO: Could also pass the filter on the command-line too.
filter="ip.dst eq 192.168.10.44 and tcp.len > 0"
stream_filter=
for stream in $(tshark -r ${infile} -Y "${filter}" -T fields -e tcp.stream | sort -u | tr -d '\r')
do
if [[ -z ${stream_filter} ]] ; then
stream_filter="tcp.stream eq ${stream}"
else
stream_filter+=" or tcp.stream eq ${stream}"
fi
done
tshark -r ${infile} -Y "${stream_filter}" -w ${outfile}
echo "Wrote ${outfile}"
I am trying to get the latest version of NodeMCU firmware working on my ESP8266 ES-12 module. I have followed the following steps with v1.5.4.x and it seems to be working just fine. Here are the steps:
Erase the flash esptool.py --port /dev/cu.usbXXXX erase_flash
Flash init data esptool.py --port /dev/cu.usbXXXX write_flash -fm dio -fs 32m 0x3fc000 esp_iot_sdk_v2.1.0/bin/esp_init_data_default.bin
Flash the actual build files esptool.py --port /dev/cu.usbXXXX --baud 115200 write_flash -fm dio -fs 32m 0x00000 0x00000.bin 0x10000 0x10000.bin
When I boot, I encounter the following after a file system format message
Got answer! Communication with MCU established.
AutoDetect firmware...
Can't autodetect firmware, because proper answer not received (may be unknown firmware).
Please, reset module or continue.
l��r��c�n�����p�|����x��ǒ��p
�nn��;�n�����b�$rrp�n�������l���b�n��n���쎟
�p��nn�����l`���#�n��{nr���;����rp�n����r���pp��<��
���p��nn���r��#�n�$�l`�8`rn|��n����l`9~����#�n�$���nn���l`nn����r��l�pp��<����b��<~�n����l`�n��Ìrnr���;����{r�ےn����l`�����x�n��>r���n�b��lrb�������l�|��n���l`�<r�x��l`#�r����8��� #�r����8���0��lrr��;l�Ğb����bp�b��l��p����l�x~�n�nܒ��r���l���l��������l�{�n��ܒ��b�l���8�b�>��ܒ��b�|�rr�
Formatting file system. Please wait...
ets Jan 8 2013,rst cause:4, boot mode:(3,6)
wdt reset
load 0x40100000, len 26772, room 16
tail 4
chksum 0x93
load 0x3ffe8000, len 2432, room 4
tail 12
chksum 0x74
ho 0 tail 12 room 4
load 0x3ffe8980, len 136, room 12
tail 12
chksum 0x67
csum 0x67
csum err
ets_main.c rl��r��c�n�����p�|����x��ǒ��p�nn��;�n�����b�cl`$`nn�������l���b�n��n���쎟�p��nn�����l`���#�n��{nr���;��?��rp�n����r���pp��<�����p��nn���r��#�n�$�l`�8`rn|��n����l`;~����#�n�$���nn���l`nn����r��l�pp��<����b��>~�n����l`�n��Ìrnr���;����{r�ےn����l`�8���;�b�Ē`$��r���wn��bp�
Like I mentioned before, following the exact same steps for v1.5.4 worked absolutely fine and I get a proper lua prompt after the file system is formatted. What am I missing here? Any help would be appreciated.
I am interested in getting a count of all LDAP/Kerberos/DNS packets.
I tried the following, but this captures the full packet.
tcpdump -i any -Z root "tcp port 389 or tcp port 88 or udp port 53" -w ~/ldap_kerberos_dns.cap
Is there a way I can just capture how many ldap/Kerberos/DNS packets were exchanged
without actually capturing the full packet.
Expected output should be something like:
LDAP: 100
Kerberos: 200
UDP: 300
Take a look at tshark statistics:
$ tshark -r 04.pcap -q -z io,phs
===================================================================
Protocol Hierarchy Statistics
Filter:
eth frames:649 bytes:124780
ipv6 frames:605 bytes:116558
udp frames:212 bytes:33686
dhcpv6 frames:171 bytes:28044
dns frames:25 bytes:2914
ntp frames:10 bytes:1300
cldap frames:6 bytes:1428
icmpv6 frames:80 bytes:7008
tcp frames:313 bytes:75864
nbss frames:108 bytes:24063
smb frames:7 bytes:1554
smb2 frames:101 bytes:22509
tcp.segments frames:1 bytes:103
dcerpc frames:16 bytes:4264
epm frames:2 bytes:544
tcp.segments frames:1 bytes:214
drsuapi frames:8 bytes:2352
kerberos frames:16 bytes:9358
tcp.segments frames:8 bytes:2130
dcerpc.cn_deseg_req frames:1 bytes:1514
ldap frames:16 bytes:5945
tcp.segments frames:3 bytes:1101
ldap frames:1 bytes:803
ip frames:40 bytes:8018
udp frames:40 bytes:8018
nbdgm frames:30 bytes:7300
smb frames:30 bytes:7300
mailslot frames:30 bytes:7300
browser frames:30 bytes:7300
dns frames:10 bytes:718
arp frames:4 bytes:204
===================================================================
See the man-page for more information.
tshark approach
If you have Wireshark (based on question tags, not the actual question) then tshark along the lines of #joke's comment is one way to go, if you don't mind its verbose stats output:
tshark -i any -n -q -z 'io,stat,0,FRAMES()tcp.port==389,FRAMES()tcp.port==88,FRAMES()udp.port==53'
Capturing on Pseudo-device that captures on all interfaces
^C142 packets captured
=============================================
| IO Statistics |
| |
| Interval size: 4.319 secs (dur) |
| Col 1: FRAMES()tcp.port==389 |
| 2: FRAMES()tcp.port==88 |
| 3: FRAMES()udp.port==53 |
|-------------------------------------------|
| |1 |2 |3 |
| Interval
| Interval | FRAMES | FRAMES | FRAMES |
|-------------------------------------------|
| 0.000 <> 4.319 | 100 | 200 | 300 |
=============================================
Though the output from that is verbose, I don't think you can get closer with tshark alone. Another approach, still more verbose, would be:
tshark -q -z io,phs "tcp port 389 or tcp port 88 or udp port 53"
These two commands do not write a capture file. Use Ctrl+C when you're ready to quit, or see the comment below about -a and autostop conditions. One caveat is that those statistics rely on the protocol dissectors, not the source/destination ports, so there can be discrepancies (e.g. connections with no data, or content non-compliant with protocol), "malformed" packets will be reported instead. It also means you won't get reliable results if you optimize and only capture 40 bytes per packet ("short" TCP or UDP packets will be reported instead).
tcpdump approach
A simple, but inelegant way is to run multiple tcpdump instances (assume bash as a shell) --
for pp in "tcp port 88" "tcp port 389" "udp port 53"; do
tcpdump -i any -Z root $pp -w /dev/null 2> ${pp// /-}.stats &
done
Packets are not written to a capture file (discarded via /dev/null).
Then wait as required, kill the tcpdump processes (by PID as listed, or kill %1 %2 %3 if no other background jobs), and inspect the .stats files:
grep captured *.stats
tcp-port-389.stats:0 packets captured
tcp-port-88.stats:0 packets captured
udp-port-53.stats:4 packets captured
perl approach
Since this is stackoverflow, here's a quick and dirty perl/libpcap solution (just add error handling):
#!/usr/bin/perl
use Net::Pcap;
use NetPacket::Ethernet;
use NetPacket::IP;
use NetPacket::TCP;
use strict;
use warnings;
my ($dev,$err,$address, $netmask,$pcap,$filter,$cfilter,$fd,%stats);
my $bail=0;
my ($rin,$rout)=('','');
$SIG{INT} = sub { print "quit...\n"; $bail=1; };
$filter='(tcp port 88 or tcp port 389 or udp port 53)';
$dev=Net::Pcap::lookupdev(\$err);
Net::Pcap::lookupnet($dev, \$address, \$netmask, \$err);
$pcap = Net::Pcap::open_live($dev, 60, 1, 0, \$err);
Net::Pcap::pcap_setnonblock($pcap, 1, \$err);
$fd=Net::Pcap::pcap_get_selectable_fd($pcap);
vec($rin,$fd,1)=1;
Net::Pcap::compile( $pcap, \$cfilter, $filter, 0, $netmask);
Net::Pcap::setfilter($pcap, $cfilter);
while (Net::Pcap::dispatch($pcap,0, \&handlepackets, '')>=0) {
select($rout=$rin,undef,undef,0.250);
printf("."); $|=1;
$bail && last;
};
Net::Pcap::freecode($cfilter);
Net::Pcap::close($pcap);
sub handlepackets() {
my ($user_data, $header, $packet) = #_;
my $ether = NetPacket::Ethernet::strip($packet);
my $ip = NetPacket::IP->decode($ether);
my $tcp = NetPacket::TCP->decode($ip->{'data'});
if ($tcp->{'src_port'} < 1024) {
$stats{$tcp->{'src_port'}}++;
} elsif ($tcp->{'dest_port'} < 1024) {
$stats{$tcp->{'dest_port'}}++;
}
}
END {
if (keys %stats) {
for my $kk (sort keys %stats) {
my ($name, $aliases, $port, $prot)=getservbyport($kk,"tcp");
printf("%12s %4i\n",$name,$stats{$kk});
}
} else { printf("No stats...\n"); }
}
i have a large pcap with more than 1000 tcp flows. i want to filter major flows say with packets greater than 100. if i go to conversations and right click on those flows, i can filter those flows, but then i have to do it several times and since i have huge pcap, it may exceed 100. is there any other quick display filter i can use which will give me flows having number of packets > n (n being any +ve integer).
say some filter like :
flow.num_pkt > 100
which can give me all such flows.
thanks a lot,
any help will be greatly appreciated.
Bro is an apt tool for connection-oriented analysis. To find the number of packets per flow, you run simply run Bro on the trace and extract the value from the logs:
bro -r trace.pcap
bro-cut id.orig_h id.orig_p id.resp_h id.resp_p orig_pkts resp_pkts < conn.log \
| awk '$5+$6 > 100 {print $1,$2,$3,$4,$5,$6}' \
| sort -rn -k 5 \
| head
This gives the following output:
192.168.1.105 49325 137.226.34.227 80 73568 146244
192.168.1.105 49547 198.189.255.74 80 16764 57098
192.168.1.105 49531 198.189.255.74 80 5186 14843
192.168.1.105 49255 198.189.255.73 80 4749 32164
192.168.1.104 1422 69.147.86.184 80 2657 2656
192.168.1.105 49251 198.189.255.74 80 2254 13854
192.168.1.1 626 224.0.0.1 626 2175 0
192.168.1.105 49513 198.189.255.82 80 2010 3852
192.168.1.103 2026 151.207.243.129 80 1953 2570
192.168.1.105 49330 143.166.11.10 64334 1514 3101
The tool bro-cut ships with Bro and provides a convenient way to extract certain named columns from the logs. For this task, you want:
id.orig_h: IP of the connection originator (source)
id.orig_p: Transport-layer port of the connection originator (source)
id.resp_h: IP of the connection responder (destination)
id.resp_p: Transport-layer port of the connection responder (source)
orig_pkts: Number of packets sent by the originator
resp_pkts: Number of packets sent by the responder
Note the awk filter expression:
awk '$5+$6 > 100 {print ...}'
It restricts the output to those connections that have a total number of packets greater than 100.
Unless you have fixed-size packets, I encourage you to also investigate other metrics, such as packet size (IP or TCP payload). These are readily in the connection logs via the orig_bytes and resp_bytes columns.
I have a little office network and I'm experiencing a huge internet link latency. We have a simple network topology: a computer configured as router running ubuntu server 10.10, 2 network cards (one to internet link, other to office network) and a switch connecting 20 computers. I have a huge tcpdump log collected at the router and I would like to plot a histogram with the RTT time of all TCP streams to try to find out the best solution to this latency problem. So, could somebody tell me how to do it using wireshark or other tool?
Wireshark or tshark can give you the TCP RTT for each received ACK packet using tcp.analysis.ack_rtt which measures the time delta between capturing a TCP packet and the ACK for that packet.
You need to be careful with this as most of your ACK packets will be from your office machines ACKing packets received from the internet, so you will be measuring the RTT between your router seeing the packet from the internet and seeing the ACK from your office machine.
To measure your internet RTT you need to look for ACKS from the internet (ACKing data sent from your network). Assuming your office machines have IP addresses like 192.168.1.x and you have logged all the data on the LAN port of your router you could use a display filter like so:
tcp.analysis.ack_rtt and ip.dst==192.168.1.255/24
To dump the RTTs into a .csv for analysis you could use a tshark command like so;
tshark -r router.pcap -Y "tcp.analysis.ack_rtt and ip.dst==192.168.1.255/24" -e tcp.analysis.ack_rtt -T fields -E separator=, -E quote=d > rtt.csv
The -r option tells tshark to read from your .pcap file
The -Y option specifies the display filter to use (-R without -2 is deprecated)
The -e option specifies the field to output
The -T options specify the output formatting
You can use the mergecap utility to merge all your pcap files into one one file before running this command. Turning this output into a histogram should be easy!
Here's the 5-min perlscript inspired by rupello's answer:
#!/usr/bin/perl
# For a live histogram of rtt latencies, save this file as /tmp/hist.pl and chmod +x /tmp/hist.pl, then run:
# tshark -i wlp2s0 -Y "tcp.analysis.ack_rtt and ip.dst==192.168.0.0/16" -e tcp.analysis.ack_rtt -T fields -E separator=, -E quote=d | /tmp/hist.pl
# Don't forget to update the interface "wlp2s0" and "and ip.dst==..." bits as appropriate, type "ip addr" to get those.
#t[$m=0]=20;
#t[++$m]=10;
#t[++$m]=5;
#t[++$m]=2;
#t[++$m]=1;
#t[++$m]=0.9;
#t[++$m]=0.8;
#t[++$m]=0.7;
#t[++$m]=0.6;
#t[++$m]=0.5;
#t[++$m]=0.4;
#t[++$m]=0.3;
#t[++$m]=0.2;
#t[++$m]=0.1;
#t[++$m]=0.05;
#t[++$m]=0.04;
#t[++$m]=0.03;
#t[++$m]=0.02;
#t[++$m]=0.01;
#t[++$m]=0.005;
#t[++$m]=0.001;
#t[++$m]=0;
#h[0]=0;
while (<>) {
s/\"//g; $n=$_; chomp($n); $o++;
for ($i=$m;$i>=0;$i--) { if ($n<=$t[$i]) { $h[$i]++; $i=-1; }; };
if ($i==-1) { $h[0]++; };
print "\033c";
for (0..$m) { printf "%6s %6s %8s\n",$t[$_],sprintf("%3.2f",$h[$_]/$o*100),$h[$_]; };
}
The newer versions of tshark seem to work better with a "stdbuf -i0 -o0 -e0 " in front of the "tshark".
PS Does anyone know if wireshark has DNS and ICMP rtt stats built in or how to easily get those?
2018 Update: See https://github.com/dagelf/pping