I have a PCAP file I exported from wireshark that's rather large. The PCAP looks as follows:
No. Time Source Destination Protocol Length Info
1 0 192.168.100.180 8.8.8.8 DNS 95 Standard query 0xf948 A detectportal.firefox.com OPT
2 0.000159827 192.168.100.180 8.8.8.8 DNS 95 Standard query 0xaf8a AAAA detectportal.firefox.com OPT
3 0.002743676 8.8.8.8 192.168.100.180 DNS 206 Standard query response 0xf948 A detectportal.firefox.com CNAME detectportal.prod.mozaws.net CNAME prod.detectportal.prod.cloudops.mozgcp.net A 34.107.221.82 OPT
4 0.002774349 8.8.8.8 192.168.100.180 DNS 218 Standard query response 0xaf8a AAAA detectportal.firefox.com CNAME detectportal.prod.mozaws.net CNAME prod.detectportal.prod.cloudops.mozgcp.net AAAA 2600:1901:0:38d7:: OPT
12 0.168337688 192.168.100.180 8.8.8.8 DNS 82 Standard query 0x6b67 A mozilla.org OPT
13 0.170840019 8.8.8.8 192.168.100.180 DNS 98 Standard query response 0x6b67 A mozilla.org A 63.245.208.195 OPT
14 0.201381247 192.168.100.180 8.8.8.8 DNS 82 Standard query 0xce22 AAAA
...
I want to split the PCAP file into 10 smaller PCAPs based on the time column. Basically, 10 PCAP files, each file spanning about 50 seconds.
I think editcap may help me, but I'm not sure what the proper command would be. (Never even heard of editcap until ~2 days ago lol)
You can use editcap as below:
editcap -i <seconds per file> <input_file> <output_file_format>
More details here - https://www.wireshark.org/docs/man-pages/editcap.html
Related
I have hundreds of files in a directory that I would like to count the occurrence of a string in each file.
I would like the output to be a summary file that contains the original file name plus the count (ideally on the same line)
for example
file1 6
file2 3
file3 4
etc
Thanks for your consideration
CAUTION: I am pretty much an enthusiastic amateur, so take everything with a grain of salt.
Several questions for you - depending on your answers, the solution below may need some adjustments.
Are all your files in the same directory, or do you also need to look through subdirectories and sub-subdirectories, etc.? Below I make the simplest assumption - that all your files are in a single directory.
Are all your files text files? In the example below, the directory will contain text files, executable files, symbolic links, and directories; the count will only be given for text files. (What linux believe to be text files, anyway.)
There may be files that do not contain the searched-for string at all. Those are not included in the output below. Do you need to show them too, with a count of 0?
I assume by "count occurrences" you mean all of them - even if the string appears more than once on the same line. (Which is why a simple grep -c won't cut it, as that only counts lines that contain the substring, no matter how many times each.)
Do you need to include hidden files (whose name begins with a period)? In my code below I assumed you don't.
Do you care that the count appears first, and then the file name?
OK, so here goes.
[oracle#localhost test]$ ls -al
total 20
drwxr-xr-x. 3 oracle oinstall 81 Apr 3 18:42 .
drwx------. 39 oracle oinstall 4096 Apr 3 18:42 ..
-rw-r--r--. 1 oracle oinstall 40 Apr 3 17:44 aa
lrwxrwxrwx. 1 oracle oinstall 2 Apr 3 18:04 bb -> aa
drwxr-xr-x. 2 oracle oinstall 6 Apr 3 17:40 d1
-rw-r--r--. 1 oracle oinstall 38 Apr 3 17:56 f1
-rw-r--r--. 1 oracle oinstall 0 Apr 3 17:56 f2
-rwxr-xr-x. 1 oracle oinstall 123 Apr 3 18:15 zfgrep
-rw-r--r--. 1 oracle oinstall 15 Apr 3 18:42 .zz
Here's the command to count 'waca' in the text files in this directory (not recursive). I define a variable substr to hold the desired string. (Note that it could also be a regular expression, more generally - but I didn't test that so you will have to, if that's your use case.)
[oracle#localhost test]$ substr=waca
[oracle#localhost test]$ find . -maxdepth 1 -type f \
> -exec grep -osHI "$substr" {} \; | sed "s/^\.\/\(.*\):$substr$/\1/" | uniq -c
8 aa
2 f1
1 .zz
Explanation: I use find to find just the files in the current directory (excluding directories, links, and whatever other trash I may have in the directory). This will include the hidden files, and it will include binary files, not just text. In this example I find in the current directory, but you can use any path instead of . I limit the depth to 1, so the command only applies to files in the current directory - the search is not recursive. Then I pass the results to grep. -o means find all matches (even if multiple matches per line of text) and show each match on a separate line. -s is for silent mode (just in case grep thinks of printing messages), -H is to include file names (even when there is only one file matching the substring), and -I is to ignore binary files.
Then I pass this to sed so that from each row output by grep I keep just the file name, without the leading ./ and without the trailing :waca. This step may not be necessary - if you don't mind the output like this:
8 ./aa:waca
2 ./f1:waca
1 ./.zz:waca
Then I pass the output to uniq -c to get the counts.
You can then redirect the output to a file, if that's what you need. (Left as a trivial exercise - since I forgot that was part of the requirement, sorry.)
Thanks for the detailed answer it provides me with ideas for future projects.
In my case the files were all the same format (output from another script) and the only files in the directory.
I found the answer in another thread
grep -c -R 'xxx'
I have a log file that looks like this:
May 25 05:34:16 server sshd[1203]: Received disconnect from 192.0.2.2 port 39102:11
May 25 05:34:16 server sshd[1203]: Disconnected from 192.0.2.1 port 39102
Now i want to extract all of the ip addresses and the date/time strings at the beginning using grep. I already know how to get the ips:
grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /log.txt
and the dates/times:
grep -o '[A-Z][a-z][a-z] [0-3][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]' /log.txt
but i don't know how to get both at the same time in a format like:
May 25 05:34:16 192.0.2.1
I've read something like:
grep -oE 'match1|match2' /log.txt
but that doesn't seem to work.
Printing two matches in the single line is easier with awk, following will print date(by printing $1,$2,$3 and all the valid IP address.
gawk '{match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/,a);split(a[0],b,".")} b[1]<=255&& b[2]<=255 && b[3]<=255 && b[4]<=255 &&length(a[0]){print $1,$2,$3, a[0]}' log_file
May 25 05:34:16 192.0.2.2
May 25 05:34:16 192.0.2.1
Explanation: First use match function to capture all the strings having format of digit.digit.digit.digit and store them into an array called "a" , then split the captured array(a) by dot(.) characters and check if each is <= 255 to ensure the IP address is valid.
Note that: GNU awk is used here.
Also note that, the regex you mentioned will also print invalid IP addresses (Eg: 333.222.555.666).
You could use your 2 patterns in a capturing group and use those in the replacement using sed:
sed -i -E 's#^([A-Z][a-z][a-z] [0-3][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]).* ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).*$#\1 \2#g' log.txt
That will match:
^ Start of string
([A-Z][a-z][a-z] [0-3][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]) Your date/time like pattern
.* Match any char 0+ times
([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) Match space followed your ip like pattern
.* Match any char 0+ times
$ End of string
Result
May 25 05:34:16 192.0.2.2
May 25 05:34:16 192.0.2.1
With any awk in any shell on any UNIX box:
$ awk '{print $1, $2, $3, $(NF-2)}' file
May 25 05:34:16 192.0.2.2
May 25 05:34:16 192.0.2.1
I have several commands similar to:
ping -i 60 8.8.8.8 | while read pong; do echo "$(date): $pong" >> /security/latencytracking/pingcapturetest2.txt; done
output:
Tue Feb 4 15:13:39 EST 2014: 64 bytes from 8.8.8.8: icmp_seq=0 ttl=50
time=88.844 ms
I then search the results using:
cat /security/latencytracking/pingcapturetest* | egrep 'time=........ ms|time=......... ms'
I am looking for latency anomalies over X ms.
Is there a way to search better than I am doing and search over the past 1,2,3, etc. hours as opposed to from the start of the file? This could get tedious over time.
You could add unix timestamp to your log, and then search based on that:
ping -i 60 8.8.8.8 | while read pong; do
echo "$(date +"%s"): $pong" >> log.txt
done
Your log will have entries like:
1391548048: 64 bytes from 8.8.8.8: icmp_req=1 ttl=47 time=20.0 ms
Then search with a combination of date and awk:
Using GNU Date (Linux etc):
awk -F: "\$1 > $(date -d '1 hour ago' +'%s')" log.txt
or BSD Date (Mac OSX, BSD)
awk -F: "\$1 > $(date -j -v '-1H' +%s)" log.txt
The command uses date -d to translate english time-sentence (or date -v for the same task on BSD/OSX) to unix timestamp. awk then compares the logged timestamp (first field before the :) with the generated timestamp and prints all log-lines which have a higher value, ie newer.
If you are familiar with R:
1. I'd slurp the whole thing in with read.table(), drop the unnecessary columns
2. then do whatever calculations you like
Unless you have tens of millions of records, then R might be a bit slow.
Plan B:
1. use cut to nuke anything you dont need and then goto the plan above.
You can also do it with bash. You can compare dates, as follows:
Crop the date field. You can convert that date into the number of seconds since midnight of 1st Jan 1970
date -d "Tue Feb 4 15:13:39 EST 2014" '+%s'
you compare that number against the number of seconds you got one hour ago,
reference=$(date --date='-1 hour' '+%s')
This way you get all records from last hour. Then you can filter after the length of the delay
i have a large pcap with more than 1000 tcp flows. i want to filter major flows say with packets greater than 100. if i go to conversations and right click on those flows, i can filter those flows, but then i have to do it several times and since i have huge pcap, it may exceed 100. is there any other quick display filter i can use which will give me flows having number of packets > n (n being any +ve integer).
say some filter like :
flow.num_pkt > 100
which can give me all such flows.
thanks a lot,
any help will be greatly appreciated.
Bro is an apt tool for connection-oriented analysis. To find the number of packets per flow, you run simply run Bro on the trace and extract the value from the logs:
bro -r trace.pcap
bro-cut id.orig_h id.orig_p id.resp_h id.resp_p orig_pkts resp_pkts < conn.log \
| awk '$5+$6 > 100 {print $1,$2,$3,$4,$5,$6}' \
| sort -rn -k 5 \
| head
This gives the following output:
192.168.1.105 49325 137.226.34.227 80 73568 146244
192.168.1.105 49547 198.189.255.74 80 16764 57098
192.168.1.105 49531 198.189.255.74 80 5186 14843
192.168.1.105 49255 198.189.255.73 80 4749 32164
192.168.1.104 1422 69.147.86.184 80 2657 2656
192.168.1.105 49251 198.189.255.74 80 2254 13854
192.168.1.1 626 224.0.0.1 626 2175 0
192.168.1.105 49513 198.189.255.82 80 2010 3852
192.168.1.103 2026 151.207.243.129 80 1953 2570
192.168.1.105 49330 143.166.11.10 64334 1514 3101
The tool bro-cut ships with Bro and provides a convenient way to extract certain named columns from the logs. For this task, you want:
id.orig_h: IP of the connection originator (source)
id.orig_p: Transport-layer port of the connection originator (source)
id.resp_h: IP of the connection responder (destination)
id.resp_p: Transport-layer port of the connection responder (source)
orig_pkts: Number of packets sent by the originator
resp_pkts: Number of packets sent by the responder
Note the awk filter expression:
awk '$5+$6 > 100 {print ...}'
It restricts the output to those connections that have a total number of packets greater than 100.
Unless you have fixed-size packets, I encourage you to also investigate other metrics, such as packet size (IP or TCP payload). These are readily in the connection logs via the orig_bytes and resp_bytes columns.
I want to programme my own anti-distraction tool. I can not / do not want to use the hosts file or third-party apps. When using IPsec or Windows Firewall, it only accepts IP addresses. There is
youtube.[264 TLD]
www.youtube.[264 TLD]
subdomains.youtube.[264 TLD]
Apparently, there is no way any more to get a complete list of YouTube sub domains.
Is there a way to somehow obtain all YouTube IP addresses and block them on the IP address level other than using brute force subdomain pinging?
The way to find all the IP addresses associated with a URL is first to find the AS Number. You can get it in Networktools: asinfo Reverse IP Lookup, Whois, Ping, RBL Check, DNS Records, Traceroute, Host information.
For YouTube, get it on Networktools: asinfo youtube.com
There you get the AS number (ASN):
Primary ASN : 15169
Now, type this in the terminal:
whois -h whois.radb.net -- '-i origin AS15169' | grep ^route
And there you will get all the IP addresses. The list is long, but you can find similar addresses that can be grouped in a subnet.
The YouTube current address list is:
64.18.0.0/20
64.233.160.0/19
66.102.0.0/20
66.249.80.0/20
72.14.192.0/18
74.125.0.0/16
173.194.0.0/16
207.126.144.0/20
209.85.128.0/17
216.58.208.0/20
216.239.32.0/19
If you have your own DNS server you can block domains from resolving. Here is a guide to block domains in a BIND DNS server.
To get all the IP addresses there isn't any way. You can install dig. It's available on Linux, but not on Windows.
Then it is to time enter dig youtube.com. It'll show you all the resolved IP addresses from the DNS cache.
See my result of dig youtube.com A:
; <<>> DiG 9.7.0-P1 <<>> youtube.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13442
;; flags: qr rd ra; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;youtube.com. IN A
;; ANSWER SECTION:
youtube.com. 83 IN A 74.125.235.47
youtube.com. 83 IN A 74.125.235.46
youtube.com. 83 IN A 74.125.235.45
youtube.com. 83 IN A 74.125.235.44
youtube.com. 83 IN A 74.125.235.43
youtube.com. 83 IN A 74.125.235.42
youtube.com. 83 IN A 74.125.235.41
youtube.com. 83 IN A 74.125.235.40
youtube.com. 83 IN A 74.125.235.39
youtube.com. 83 IN A 74.125.235.38
youtube.com. 83 IN A 74.125.235.37
youtube.com. 83 IN A 74.125.235.36
youtube.com. 83 IN A 74.125.235.35
youtube.com. 83 IN A 74.125.235.34
youtube.com. 83 IN A 74.125.235.33
youtube.com. 83 IN A 74.125.235.32
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Feb 18 22:26:58 2012
;; MSG SIZE rcvd: 285
This may help: Obtain Google IP address ranges
Which points to:
https://www.gstatic.com/ipranges/goog.json