Ruby: How should I remove duplicate lines from an Array? - ruby-on-rails

Question

I may not be using the correct terminology but here goes..
I'm displaying IPS alerts on a dashboard app and there are many duplicate lines. For example, if one script kiddie is trying to brute force an RDP server, I could get 150 Alerts but could be slimmed down to about 5 because that's how many hosts they are going after. So I'm trying to remove the duplicate alerts, and I'm looking to use the sid, src_addr, and dst_addr as my metrics to determine if they are duplicates.
Currently I display #filtered_snort_detail_query using this code:
This is my view
<% if #filtered_snort_detail_query.count > 0 %>
<table>
<tr>
<th>Timestamp</th>
<th>Tag Info</th>
<th>Message</th>
</tr>
<% #filtered_snort_detail_query.each do |d|
text_msg = d['_source']['message']
if d['_source']['message'].nil?
end
%>
<tr>
<td class='timestamp'><%= d['_source']['#timestamp'].to_time %></td>
<td class='tags'><%= d['_source']['tags'] %></td>
<td class='message'><%= text_msg %></td>
</tr>
<% end %>
</table>
<% else %>
<div> No Results Returned. </div>
<% end %>
Here is my controller
if #es_snort_detail_query.count > 0
#filtered_snort_detail_query = Array.new
#es_snort_detail_query.each do |ips_detail|
next if ips_detail['_source']['type'] != 'snort-ips'
next if ips_detail['_source']['#timestamp'] < #ts
#filtered_snort_detail_query.push(ips_detail)
end
end
Here is what I think I need to do to get the metrics I need to compare lines in my controller.
I'm just not sure the best way to look at each line of #filtered_snort_detail_query and build a new array to display in my view using these parameters:
show me all lines, but not if sid_data, src_ip_data, and dst_ip_data happen two or more times.
if #es_snort_detail_query.count > 0
#filtered_snort_detail_query = Array.new
#es_snort_detail_query.each do |ips_detail|
next if ips_detail['_source']['type'] != 'snort-ips'
next if ips_detail['_source']['#timestamp'] < #ts
#filtered_snort_detail_query.push(ips_detail)
end
if #filtered_snort_detail_query.count > 0
ip_src = Array.new
ip_dst = Array.new
sid = Array.new
#filtered_snort_detail_query.each do |find_ip, find_sid|
unless find_ip.nil?
sid_data = find_sid.scan(/\[\d+\:\d+\:\d+\]/)
src_ip_data = find_ip.scan(/(?:[0-9]{1,3}\.){3}[0-9]{1,3}/)
dst_ip_data = find_ip.scan(/(?:[0-9]{1,3}\.){3}[0-9]{1,3}/)
sid.push(sid_data[0]) unless sid_data[0].nil?
ip_src.push(src_ip_data[0]) unless src_ip_data[0].nil?
ip_dst.push(dst_ip_data[1]) unless dst_ip_data[1].nil?
end
end
end
end

Answer 1

Sorry if I misunderstood the question.
If you have a bunch of objects in an array, and you want to remove duplicates based on a subset of their properties, you can use the uniq method with a block:
queries.uniq do |query|
[query.sid_data, query.src_ip_data, query.dst_ip_data]
end
It will compare the queries based on the array created in the block, and remove duplicates.
Go to http://www.compileonline.com/execute_ruby_online.php, copy and paste the code below, and click execute script in the top left.
queries = [
{ :a => "ab", :b => "ba" },
{ :a => "ab", :b => "xy" },
{ :a => "xy", :b => "xy" }
]
unique_queries = queries.uniq do |q|
q[:a]
end
puts unique_queries
See? The comparaison was done only based on the value of the :a key. That's the same principle.