I need to put the text content from an html element to a csv file. With the ruby csv gem, it seems that the primary write method for wrapped Strings and IOs only converts a string even if an object is specified.
For example:
Searchresults = puts browser.divs(class: 'results_row').map(&:text)
csv << %w(Searchresults)
returns only "searchresults" in the csv file.
It seems like there should be a way to specify the text from the div element to be put and not just a literal string.
Edit:
Okay arieljuod and spickermann were right. Now I am getting text content from the div element output to the csv, but not all of it like when I output to the console. The div element "results_row" has two a elements with text content. It also has a child div "results_subrow" with a paragraph of text content that is not getting written to the csv.
HTML:
<div class="bodytag" style="padding-bottom:30px; overflow:visible">
<h2>Search Results for "serialnum3"</h2>
<div id="results_banner">
Products
<span>Showing 1 to 2 of 2 results</span>
</div>
<div class="pg_dir"></div>
<div class="results_row">
FUJI
50mm lens
<div class="results_subrow">
<p>more product info</p>
</div>
</div>
<div class="results_row">
FUJI
50mm lens
<div class="results_subrow">
<p>more product info 2</p>
</div>
</div>
<div class="pg_dir"></div>
My code:
search_results = browser.divs(class: 'results_row').map(&:text)
csv << search_results
I'm thinking that including the child div "results_subrow" in the locator will find what I am missing. Like:
search_results = browser.divs(class: 'results_row', 'results_subrow').map(&:text)
csv << search_results
%w[Searchresults] creates an array containing the word Searchresults. You probably want something like this:
# assign the array returned from `map` to the `search_results` variable
search_results = browser.divs(class: 'results_row').map(&:text)
# output the `search_results`. Note that the return value of `puts` is `nil`
# therefore something like `Searchresults = puts browser...` doesn't work
puts search_results
# append `search_results` to your csv
csv << search_results
Related
I have:
<ul>
<li>text1</li>
<li>text2 </li>
</ul>
Right now I get the text from <li> like this:
result = page.css(' ul li').text
The problem is, as a result I get a string with no spaces like
text1text2
I want it to be divided with <br>, like text1<br>text2<br>.
How do I do this?
From "Searching a XML/HTML Document"
:
methods xpath and css actually return a NodeSet, which acts very much
like an array, and contains matching nodes from the document.
So, if you want to concatenate all texts from all <li> tags, then you should work with the css method result as with a collection:
page.css('ul li') # selects all li tags and returns collection of Node objects
.map(&:text) # maps collection of li nodes into array of corresponding texts
.join('<br>') # concatenates all nodes texts into a single string with <br> separator
See: http://ruby.bastardsbook.com/chapters/html-parsing/
I have some HTML pages where the contents to be extracted are marked with HTML comments like below.
<html>
.....
<!-- begin content -->
<div>some text</div>
<div><p>Some more elements</p></div>
<!-- end content -->
...
</html>
I am using Nokogiri and trying to extract the HTML between the <!-- begin content --> and <!-- end content --> comments.
I want to extract the full elements between these two HTML comments:
<div>some text</div>
<div><p>Some more elements</p></div>
I can get the text-only version using this characters callback:
class TextExtractor < Nokogiri::XML::SAX::Document
def initialize
#interesting = false
#text = ""
#html = ""
end
def comment(string)
case string.strip # strip leading and trailing whitespaces
when /^begin content/ # match starting comment
#interesting = true
when /^end content/
#interesting = false # match closing comment
end
def characters(string)
#text << string if #interesting
end
end
I get the text-only version with #text but I need the full HTML stored in #html.
Extracting content between two nodes is not a normal thing we'd do; Normally we'd want content inside a particular node. Comments are nodes, they're just special types of nodes.
require 'nokogiri'
doc = Nokogiri::HTML(<<EOT)
<body>
<!-- begin content -->
<div>some text</div>
<div><p>Some more elements</p></div>
<!-- end content -->
</body>
EOT
By looking for a comment containing the specified text it's possible to find a starting node:
start_comment = doc.at("//comment()[contains(.,'begin content')]") # => #<Nokogiri::XML::Comment:0x3fe94994268c " begin content ">
Once that's found then a loop is needed that stores the current node, then looks for the next sibling until it finds another comment:
content = Nokogiri::XML::NodeSet.new(doc)
contained_node = start_comment.next_sibling
loop do
break if contained_node.comment?
content << contained_node
contained_node = contained_node.next_sibling
end
content.to_html # => "\n <div>some text</div>\n <div><p>Some more elements</p></div>\n"
I have a html string. In that string I want to parse all <p> tags and apply additional inline style.
Additional Style: style="margin:0px;padding:0px;" or it could be something else
Case1:
input string: <p>some string</p>
output string: <p style="margin:0px;padding:0px;">some string</p>
Case2:
input string: <p style="text-align:right;" >some string</p>
output string: <p style="text-align:right;margin:0px;padding:0px;">some string</p>
Case3:
input string: <p align="justify">some string</p>
output string: <p style="margin:0px;padding:0px;" align="justify">some string</p>
Right now I am using regex like this
myHtmlString.gsub("<p", "<p style = \"margin:0px;padding:0px\"")
Which works fine except it removes previous styling. I am using Ruby (ROR).
I need help to tweak this a bit.
You can do this using Nokogiri, by setting [:style] on the relevant Nodes.
require "nokogiri"
inputs = [
'<p>some string</p>',
'<p style="text-align:right;" >some string</p>',
'<p align="justify">some string</p>'
]
inputs.each do |input|
noko = Nokogiri::HTML::fragment(input)
noko.css("p").each do |tag|
tag[:style] = (tag[:style] || "") + "margin:0px;padding:0px;"
end
puts noko.to_html
end
This will loop through all elements matching the css selector p, and set the style attribute like you want.
Output:
<p style="margin:0px;padding:0px;">some string</p>
<p style="text-align:right;margin:0px;padding:0px;">some string</p>
<p align="justify" style="margin:0px;padding:0px;">some string</p>
I recommend against using regex for this, as in general HTML can't be properly parsed by regex. That said, as long as your input data is consistent, regex will still work. You want to match whatever content is already in a p element's style attribute using parentheses, then insert it in the substitution string:
myHtmlString.gsub(/<p( style="(.*)")?/,
"<p style=\"#{$2};margin:0px;padding:0px\"")
Here's how the match pattern works:
/ #regex delimiter
<p #match start of p tag
( #open paren used to group, everything in this group gets saved in $1
style=" #open style attribute
(.*) #group contents of style attribute, gets saved to $2
" #close style attribute
)? #question mark makes everything in the paren group optional
/ #regex delimiter
I ended up doing something like this, I had to do this just before sending the email. I know this is not the best way to do it but worth sharing here. Solutions given by #sgroves and #Dobert are really good and helpful.
But I din't want to included Nokogiri, though I have picked the idea from above 2 solutions only. Thanks.
Here is my code ( I am new to ROR so nothing much fancy here, I used it in HAML block)
myString.gsub!(/<p[^>]*>/) do |match|
match1 = match
style1_arr = match1.scan(/style=".*"/)
unless style1_arr.blank?
style1 = style1_arr.first.sub("style=", "").gsub(/\"/, "").to_s
style2 = style1 + "margin:0px;padding:0px;"
match2 = match1.sub(/style=".*"/, "style=\"#{style2.to_s}\"")
else
match2 = match1.sub(/<p/, "<p style = \"margin:0px;padding:0px;\"")
end
end
Now myString will be updated string.(notice the ! after gsub)
I'm trying to remove whitespace from an HTML fragment between <p> tags
<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>
as you can see, there always is a blank space between the <p> </p> tags.
The problem is that the blank spaces create <br> tags when saving the string into my database.
Methods like strip or gsub only remove the whitespace in the nodes, resulting in:
<p>FooBar</p> <p>barbarbar</p> <p>bla</p>
whereas I'd like to have:
<p>Foo Bar</p><p>bar bar bar</p><p>bla</p>
I'm using:
Nokogiri 1.5.6
Ruby 1.9.3
Rails
UPDATE:
Occasionally there are children nodes of the <p>Tags that generate the same problem: white space between
Sample Code
Note: the Code normally is in one Line, I reformatted it because it would be unbearable otherwise...
<p>
<p>
<strong>Selling an Appartment</strong>
</p>
<ul>
<li>
<p>beautiful apartment!</p>
</li>
<li>
<p>near the train station</p>
</li>
.
.
.
</ul>
<ul>
<li>
<p>10 minutes away from a shopping mall </p>
</li>
<li>
<p>nice view</p>
</li>
</ul>
.
.
.
</p>
How would I strip those white spaces aswell?
SOLUTION
It turns out that I messed up using the gsub method and didn't further investigate the possibility of using gsub with regex...
The simple solution was adding
data = data.gsub(/>\s+</, "><")
It deleted whitespace between all different kinds of nodes... Regex ftw!
This is how I'd write the code:
require 'nokogiri'
doc = Nokogiri::HTML::DocumentFragment.parse(<<EOT)
<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>
EOT
doc.search('p, ul, li').each { |node|
next_node = node.next_sibling
next_node.remove if next_node && next_node.text.strip == ''
}
puts doc.to_html
It results in:
<p>Foo Bar</p><p>bar bar bar</p><p>bla</p>
Breaking it down:
doc.search('p')
looks for only the <p> nodes in the document. Nokogiri returns a NodeSet from search, or a nil if nothing matched. The code loops over the NodeSet, looking at each node in turn.
next_node = node.next_sibling
gets the pointer to the next node following the current <p> node.
next_node.remove if next_node && next_node.text.strip == ''
next_node.remove removes the current next_node from the DOM if the next node isn't nil and its text isn't empty when stripped, in otherwords, if the node has only whitespace.
There are other techniques to locate only the TextNodes if all of them should be stripped from the document. That's risky, because it can end up deleting all blanks between tags, causing run-on sentences and joined words, which probably isn't what you want.
A first solution can be to remove empty text nodes, a quick way to do this for your exact case can be:
require 'nokogiri'
doc = Nokogiri::HTML("<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>")
doc.css('body').first.children.map{|node| node.to_s.strip}.compact.join
This won't work for nested elements as-is but should give you a good path for start.
UPDATE:
You can actually optimise a little with:
require 'nokogiri'
doc = Nokogiri::HTML::DocumentFragment.parse("<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>")
doc.children.map{|node| node.to_s.strip}.compact.join
Here is all the possible task you can be looking for which deals with unnecessary whitespaces(including unicode one) in parsing output.
html = "<p>A paragraph.<em> </em> <br><br><em>
</em></p><p><em> </em>
</p><p><em>
</em><strong><em>\" Quoted Text \" </em></strong></p>
<ul><li><p>List 1</p></li><li><p>List 2</p></li><li><p>List 3 </p>
<p><br></p><p><br><em> </em><br>
A text content.<br><em><br>
</em></p></li></ul>"
doc = Nokogiri::HTML.fragment(html)
doc.traverse { |node|
# removes any whitespace node
node.remove if node.text.gsub(/[[:space:]]/, '') == ''
# replace mutiple consecutive spaces with single space
node.content = node.text.gsub(/[[:space:]]{2,}/, ' ') if node.text?
}
# Gives you html without any text node including <br> or multiple spaces anywhere in the text of html
puts doc.to_html
# Gives text of html, concatenating li items with a space between them
# By default li items text are concatenated without the space
Nokogiri::HTML(doc.to_html).xpath('//text()').map(&:text).join(' ')
#Output
# "A paragraph. \" Quoted Text \" \n List 1 \n List 2 \n \n List 3 \n A text content. \n \n"
# To Remove newline character '\n'
Nokogiri::HTML(doc.to_html).xpath('//text()').map(&:text).join(' ').gsub(/\n+/,'')
#Output
# "A paragraph. \" Quoted Text \" List 1 List 2 List 3 A text content."
Note: If you are not using fragment in case of a complete html doc then you might have to replace traverse with other function like search.
data.squish does the same thing and is way more readable.
#item
creates a div with id="item"
.box#item
creates a div with class="box" and id="item"
.box#="item "+x
creates a div with class="box" and a comment '#="item"+x'
.box#
="item"+x
throws "Illegal element: classes and ids must have values."
How do I get set the id to a variable?
There are two ways:
The long form way (define the id as if it were a regular attribute):
.box{:id => "item_#{x}"}
produces this (x is what ever x.to_s evaluates to):
<div class="box" id="item_x">
The short form way:
.box[x]
produces the following assuming x is an instance of item:
<div class="box item" id="item_45">
See the HAML reference for more information.
You can set the id and class in HAML the following ways
The normal way
.box.item#item
<div id="item" class="box item"></div>
If you need to interpolation you can use this format
.box{id: "item_#{123}", class: "item_#{123}"}
<div id="item_123" class="box item_123"></div>
This format generates the class and id using the object reference
# app/controllers/items_controller.rb
#item = Item.find(123)
.box[#item]
<div id="item_123" class="box item"></div>
If you need to prefix something
.box[#item, :custom]
<div id="custom_item_123" class="box custom_item"></div>
If you need a custom class and id generation you need to add the following method to model.
class CrazyUser < ActiveRecord::Base
def haml_object_ref
"customized_item"
end
end
And then you will get the customized class
.box[#item]
<div id="customized_item_123" class="box customized_item"></div>
Refer:
http://haml.info/docs/yardoc/file.REFERENCE.html#object-reference-
http://haml.info/docs/yardoc/file.REFERENCE.html