SAX parsing a bunch of dead presidents with Nokogiri HTML parser? - ruby-on-rails

Question

I would like to parse USA presidents on the "List of Presidents of the United States" wiki page.
I can do this with a bunch of XPath and loops. But SAx parsing is so fast and I would like to learn how to implement that.
The Nokogiri document gave me an HTML SAX parsing example:
class MyDoc < Nokogiri::XML::SAX::Document
def start_element name, attributes = []
puts "found a #{name}"
end
end
parser = Nokogiri::HTML::SAX::Parser.new(MyDoc.new)
parser.parse(File.read(ARGV[0], 'rb'))
But which methods do I use to define all the HTML elements and their content that I want to grab?

Answer 1

With SAX, you have to define callback methods in your parser for each 'event'. You have to keep track of state yourself. It is very crude. For example, to get president names from the page, you can do this:
class MyDoc < Nokogiri::XML::SAX::Document
def start_element name, attributes = []
if name == "li"
#inside_li = true
end
end
def characters(chars)
if #inside_li
puts "found an <li> containing the string '#{chars}'"
end
end
def end_element name
if name == "li"
puts "ending #{name}"
#inside_li = false
end
end
end
The above can be thought of as the rough equivalent of the statement:
doc.xpath('//li').map(&:text)
Which starts with the following output:
ending li
found an <li> containing the string 'Grover Cleveland'
ending li
found an <li> containing the string 'William McKinley'
ending li
found an <li> containing the string 'Theodore Roosevelt'
So far so good, However, it also outputs a lot of cruft, ending with:
found an <li> containing the string 'Disclaimers'
ending li
found an <li> containing the string 'Mobile view'
ending li
found an <li> containing the string '
'
found an <li> containing the string '
'
ending li
found an <li> containing the string '
'
found an <li> containing the string '
'
ending li
So to make this more precise and not get the li elements you don't care about, you'd have to keep track of which container elements you are in by adding more if clauses to start_element, characters, etc. And if you have nested elements of the same name, you'll have to keep track of counters yourself, or implement a stack to push and pop the elements you see. It gets VERY messy very fast.
SAX is best for filters where you don't care about the DOM, you're just doing some basic transformations.
Instead, consider using a single XPath statement, such as
doc.xpath("//table[contains(.//div, 'Presidents of the United States')]//ol/li").map(&:text)
This says "Find the table which contains a div with the words 'Presidents of the United States' and return the text from all the ordered list items within it". This can be done in SAX, but it would be a lot of messy code.
Output of the above XPath:
["George Washington", "John Adams", "Thomas Jefferson", "James Madison", "James Monroe", "John Quincy Adams", "Andrew Jackson", "Martin Van Buren", "William Henry Harrison", "John Tyler", "James K. Polk", "Zachary Taylor", "Millard Fillmore", "Franklin Pierce", "James Buchanan", "Abraham Lincoln", "Andrew Johnson", "Ulysses S. Grant", "Rutherford B. Hayes", "James A. Garfield", "Chester A. Arthur", "Grover Cleveland", "Benjamin Harrison", "Grover Cleveland", "William McKinley", "Theodore Roosevelt", "William Howard Taft", "Woodrow Wilson", "Warren G. Harding", "Calvin Coolidge", "Herbert Hoover", "Franklin D. Roosevelt", "Harry S. Truman", "Dwight D. Eisenhower", "John F. Kennedy", "Lyndon B. Johnson", "Richard Nixon", "Gerald Ford", "Jimmy Carter", "Ronald Reagan", "George H. W. Bush", "Bill Clinton", "George W. Bush", "Barack Obama"]