Sidekiq job doesn't seem to be parsing - ruby-on-rails

Question

I'm trying to mimic what a previous developer did to parse an XML file in my Rails app and am stuck. From what I can tell, my job completes, but nothing is being posted as it should be, so I'm guessing my parsing file is incorrect (however, it works fine when testing with the raw file on my localhost). So, where am I going wrong here?
This is Sidekiq log output, just to confirm job is happening and not showing any errors in processing:
2016-05-25T13:51:04.499Z 8977 TID-oxs3s9lng ParseTestData JID-2a01971539c887cac3bf3374:1 INFO: start
2016-05-25T13:51:04.781Z 8977 TID-oxs3s9l3g GenerateNotifications JID-2a01971539c887cac3bf3374:2 INFO: start
2016-05-25T13:51:04.797Z 8977 TID-oxs3s9lng ParseTestData JID-2a01971539c887cac3bf3374:1 INFO: done: 0.297 sec
2016-05-25T13:51:04.824Z 8977 TID-oxs3s9l3g GenerateNotifications JID-2a01971539c887cac3bf3374:2 INFO: done: 0.043 sec
This is my Sidekiq job file, which iterates through the compressed files that get submitted through my API. The file in question that I'm working on is nmap_poodle_scan.xml:
class ParseTestData
include Sidekiq::Worker
# Order matters. Parse network hosts first to ensure we uniquely identify network hosts by their mac address.
PARSERS = {
"network_hosts.xml" => Parsers::NetworkHostParser,
"nmap_tcp_service_scan.xml" => Parsers::TcpServiceScanParser,
"nmap_shellshock_scan.xml" => Parsers::ShellshockScanParser,
"hydra.out" => Parsers::HydraParser,
"events.log" => Parsers::EventParser,
"nmap_poodle_scan.xml" => Parsers::PoodleScanParser
}
def perform(test_id)
test = Test.find(test_id)
gzip = if Rails.env.development?
Zlib::GzipReader.open(test.data.path)
else
file = Net::HTTP.get(URI.parse(test.data.url))
Zlib::GzipReader.new(StringIO.new(file))
end
# Collect entries from tarball
entries = {}
tar_extract = Gem::Package::TarReader.new(gzip)
tar_extract.rewind
tar_extract.each do |entry|
entries[File.basename(entry.full_name)] = entry.read
end
# Preserve parse order by using the parser hash to initiate parser executions.
PARSERS.each_pair do |filename, parser|
next unless entry = entries[filename]
parser.run!(test, entry)
end
end
end
Which grabs nmap_poodle_scan.xml:
<host starttime="1464180941" endtime="1464180941"><status state="up" reason="arp-response" reason_ttl="0"/>
<address addr="10.10.10.1" addrtype="ipv4"/>
<address addr="4C:E6:76:3F:2F:77" addrtype="mac" vendor="Buffalo.inc"/>
<hostnames>
<hostname name="DD-WRT" type="PTR"/>
</hostnames>
Nmap scan report for DD-WRT (10.10.10.1)
<ports><extraports state="closed" count="996">
<extrareasons reason="resets" count="996"/>
</extraports>
<table key="CVE-2014-3566">
<elem key="title">SSL POODLE information leak</elem>
<elem key="state">VULNERABLE</elem>
<table key="ids">
<elem>OSVDB:113251</elem>
<elem>CVE:CVE-2014-3566</elem>
</table>
<table key="description">
<elem> The SSL protocol 3.0, as used in OpenSSL through 1.0.1i and
other products, uses nondeterministic CBC padding, which makes it easier
for man-in-the-middle attackers to obtain cleartext data via a
padding-oracle attack, aka the "POODLE" issue.</elem>
</table>
<table key="dates">
<table key="disclosure">
<elem key="year">2014</elem>
<elem key="month">10</elem>
<elem key="day">14</elem>
</table>
</table>
<elem key="disclosure">2014-10-14</elem>
<table key="check_results">
<elem>TLS_RSA_WITH_3DES_EDE_CBC_SHA</elem>
</table>
<table key="refs">
<elem>https://www.imperialviolet.org/2014/10/14/poodle.html</elem>
<elem>http://osvdb.org/113251</elem>
<elem>https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-3566</elem>
<elem>https://www.openssl.org/~bodo/ssl-poodle.pdf</elem>
</table>
</table>
</script></port>
</ports>
<times srtt="4665" rttvar="556" to="100000"/>
</host>
Which should submit to PoodleScanParser:
module Parsers
class PoodleScanParser < NmapScanParser
def self.run!(test, content)
super(test, content, "//host//ports[.//elem[#key='state'][contains(text(), 'VULNERABLE')]]") do |host, network_host_test|
logger.info "Something cool"
IssueFinder.match(cve_id: "CVE-2014-3566").each do |issue|
Result.generate!(network_host_test.id, issue.id)
end
end
end
end
end
Which inherits from NmapScanParser. This file is parser is confirmed to work fine, so I know it's not the issue:
module Parsers
class NmapScanParser
def self.run!(test, content, xpath)
document = Nokogiri::XML(content)
document.remove_namespaces!
document.xpath(xpath).each do |host|
ip_address = host.at_xpath("address[#addrtype='ipv4']").at_xpath("#addr").value
vendor = host.at_xpath("address[#addrtype='mac']").at_xpath("#vendor").value rescue "Unknown"
hostname = host.at_xpath("hostnames/hostname").at_xpath("#name").value rescue "Hostname Unknown"
os = host.at_xpath("os/osmatch").at_xpath("#name").value rescue "Unknown"
os_vendor = host.at_xpath("os/osmatch/osclass").at_xpath("#vendor").value rescue "Unknown"
network_host_test = NetworkHostTest.generate!(test, ip_address: ip_address, hostname: hostname, vendor: vendor, os: os, os_vendor: os_vendor)
# If we didn't find a network host, that's because our network_hosts file didn't have this entry.
next unless network_host_test
yield(host, network_host_test)
end
end
end
end
I've confirmed the parser works on my localhost with the same raw output as above using a plain ruby file, and running ruby poodle_parser.rb:
require 'nokogiri'
document = Nokogiri::XML(File.open("poodle_results.xml"))
document.remove_namespaces!
document.xpath("//host[.//elem/#key='state']").each do |host|
ip_address = host.at_xpath("address[#addrtype='ipv4']").at_xpath("#addr").value
result = host.at_xpath("//ports//elem[#key='state']").content
puts "#{ip_address} #{result}"
end
Which outputs what I would expect in terminal:
10.10.10.1 VULNERABLE
So, in the end, I expect a Result to be generated, but it's not. I'm not seeing any errors in the Rails log on my localhost, nor am I seeing anything indicating an error in the Sidekiq logs either!
I decided to add a logger.info line to my PoodleScanParser to see if the Parser is even running as it should be. Assuming I did this correctly, the Parser doesn't look like it's running.

Answer 1

Well, the answer has nothing to do with Sidekiq, instead it was the output, which Nokogiri was dying on. Turns out Nmap was adding a non-XML line at the beginning of the XML file "Starting Nmap 7.12". So, Nokogiri was simply dying there.
I guess moral of the story is to make sure your XML output is what you Nokogiri intends it to be!