How to parse an HTML webpage and remove <br> tags? - html-parsing

i need to parse a website that contains <p> tags (many of them) i want to get them and put them on a csv file (in same column).
After testing, i'm seeing the paragraphs are not on the same column, it's because of the <br> that are on <p> tags example :
HTML :
<div class="text">
<p> hello <br> friends </p>
<p> parsing is cool <br> using <br> simpleHTMLdom </p>
</div>
When i parse the html below i get the two <p> but not on same csv "column".
My code :
if($html_book_page->find('.text')){
foreach($html_book_page->find('div[class=text] p') as $bookPreview){
array_push($book, $bookPreview->plaintext);
}
}
$book is the array containing all text and i put $book on csv like :
fputcsv($open_csv, array_values($book), ',', ' ');
Any way to get :
(header of csv : TEXT ) and inside :
"Hello friends parsing is cool using simpleHTMLdom" ? Because for moment i have "Hello" and in another column i've "friends" .. "parsing is cool" ..."using".... "simpleHTMLdom"
Thank you all

Why don't you do a jQuery.remove() before your CSV insert? Something like this:
$('.text p').find('br').remove()
If you don't want to permanently remove <br> from the page, you could do something like this in your for-loop:
foreach($html_book_page - > find('div[class=text] p') as $bookPreview) {
$bookPreview.innerHTML.replace("<br>", "");
array_push($book, $bookPreview - > plaintext);
}

Related

How to write method to csv not just string? Ruby csv gem

I need to put the text content from an html element to a csv file. With the ruby csv gem, it seems that the primary write method for wrapped Strings and IOs only converts a string even if an object is specified.
For example:
Searchresults = puts browser.divs(class: 'results_row').map(&:text)
csv << %w(Searchresults)
returns only "searchresults" in the csv file.
It seems like there should be a way to specify the text from the div element to be put and not just a literal string.
Edit:
Okay arieljuod and spickermann were right. Now I am getting text content from the div element output to the csv, but not all of it like when I output to the console. The div element "results_row" has two a elements with text content. It also has a child div "results_subrow" with a paragraph of text content that is not getting written to the csv.
HTML:
<div class="bodytag" style="padding-bottom:30px; overflow:visible">
<h2>Search Results for "serialnum3"</h2>
<div id="results_banner">
Products
<span>Showing 1 to 2 of 2 results</span>
</div>
<div class="pg_dir"></div>
<div class="results_row">
FUJI
50mm lens
<div class="results_subrow">
<p>more product info</p>
</div>
</div>
<div class="results_row">
FUJI
50mm lens
<div class="results_subrow">
<p>more product info 2</p>
</div>
</div>
<div class="pg_dir"></div>
My code:
search_results = browser.divs(class: 'results_row').map(&:text)
csv << search_results
I'm thinking that including the child div "results_subrow" in the locator will find what I am missing. Like:
search_results = browser.divs(class: 'results_row', 'results_subrow').map(&:text)
csv << search_results
%w[Searchresults] creates an array containing the word Searchresults. You probably want something like this:
# assign the array returned from `map` to the `search_results` variable
search_results = browser.divs(class: 'results_row').map(&:text)
# output the `search_results`. Note that the return value of `puts` is `nil`
# therefore something like `Searchresults = puts browser...` doesn't work
puts search_results
# append `search_results` to your csv
csv << search_results

How to apply additional inline style to html tags in ruby?

I have a html string. In that string I want to parse all <p> tags and apply additional inline style.
Additional Style: style="margin:0px;padding:0px;" or it could be something else
Case1:
input string: <p>some string</p>
output string: <p style="margin:0px;padding:0px;">some string</p>
Case2:
input string: <p style="text-align:right;" >some string</p>
output string: <p style="text-align:right;margin:0px;padding:0px;">some string</p>
Case3:
input string: <p align="justify">some string</p>
output string: <p style="margin:0px;padding:0px;" align="justify">some string</p>
Right now I am using regex like this
myHtmlString.gsub("<p", "<p style = \"margin:0px;padding:0px\"")
Which works fine except it removes previous styling. I am using Ruby (ROR).
I need help to tweak this a bit.
You can do this using Nokogiri, by setting [:style] on the relevant Nodes.
require "nokogiri"
inputs = [
'<p>some string</p>',
'<p style="text-align:right;" >some string</p>',
'<p align="justify">some string</p>'
]
inputs.each do |input|
noko = Nokogiri::HTML::fragment(input)
noko.css("p").each do |tag|
tag[:style] = (tag[:style] || "") + "margin:0px;padding:0px;"
end
puts noko.to_html
end
This will loop through all elements matching the css selector p, and set the style attribute like you want.
Output:
<p style="margin:0px;padding:0px;">some string</p>
<p style="text-align:right;margin:0px;padding:0px;">some string</p>
<p align="justify" style="margin:0px;padding:0px;">some string</p>
I recommend against using regex for this, as in general HTML can't be properly parsed by regex. That said, as long as your input data is consistent, regex will still work. You want to match whatever content is already in a p element's style attribute using parentheses, then insert it in the substitution string:
myHtmlString.gsub(/<p( style="(.*)")?/,
"<p style=\"#{$2};margin:0px;padding:0px\"")
Here's how the match pattern works:
/ #regex delimiter
<p #match start of p tag
( #open paren used to group, everything in this group gets saved in $1
style=" #open style attribute
(.*) #group contents of style attribute, gets saved to $2
" #close style attribute
)? #question mark makes everything in the paren group optional
/ #regex delimiter
I ended up doing something like this, I had to do this just before sending the email. I know this is not the best way to do it but worth sharing here. Solutions given by #sgroves and #Dobert are really good and helpful.
But I din't want to included Nokogiri, though I have picked the idea from above 2 solutions only. Thanks.
Here is my code ( I am new to ROR so nothing much fancy here, I used it in HAML block)
myString.gsub!(/<p[^>]*>/) do |match|
match1 = match
style1_arr = match1.scan(/style=".*"/)
unless style1_arr.blank?
style1 = style1_arr.first.sub("style=", "").gsub(/\"/, "").to_s
style2 = style1 + "margin:0px;padding:0px;"
match2 = match1.sub(/style=".*"/, "style=\"#{style2.to_s}\"")
else
match2 = match1.sub(/<p/, "<p style = \"margin:0px;padding:0px;\"")
end
end
Now myString will be updated string.(notice the ! after gsub)

How do I remove white space between HTML nodes?

I'm trying to remove whitespace from an HTML fragment between <p> tags
<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>
as you can see, there always is a blank space between the <p> </p> tags.
The problem is that the blank spaces create <br> tags when saving the string into my database.
Methods like strip or gsub only remove the whitespace in the nodes, resulting in:
<p>FooBar</p> <p>barbarbar</p> <p>bla</p>
whereas I'd like to have:
<p>Foo Bar</p><p>bar bar bar</p><p>bla</p>
I'm using:
Nokogiri 1.5.6
Ruby 1.9.3
Rails
UPDATE:
Occasionally there are children nodes of the <p>Tags that generate the same problem: white space between
Sample Code
Note: the Code normally is in one Line, I reformatted it because it would be unbearable otherwise...
<p>
<p>
<strong>Selling an Appartment</strong>
</p>
<ul>
<li>
<p>beautiful apartment!</p>
</li>
<li>
<p>near the train station</p>
</li>
.
.
.
</ul>
<ul>
<li>
<p>10 minutes away from a shopping mall </p>
</li>
<li>
<p>nice view</p>
</li>
</ul>
.
.
.
</p>
How would I strip those white spaces aswell?
SOLUTION
It turns out that I messed up using the gsub method and didn't further investigate the possibility of using gsub with regex...
The simple solution was adding
data = data.gsub(/>\s+</, "><")
It deleted whitespace between all different kinds of nodes... Regex ftw!
This is how I'd write the code:
require 'nokogiri'
doc = Nokogiri::HTML::DocumentFragment.parse(<<EOT)
<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>
EOT
doc.search('p, ul, li').each { |node|
next_node = node.next_sibling
next_node.remove if next_node && next_node.text.strip == ''
}
puts doc.to_html
It results in:
<p>Foo Bar</p><p>bar bar bar</p><p>bla</p>
Breaking it down:
doc.search('p')
looks for only the <p> nodes in the document. Nokogiri returns a NodeSet from search, or a nil if nothing matched. The code loops over the NodeSet, looking at each node in turn.
next_node = node.next_sibling
gets the pointer to the next node following the current <p> node.
next_node.remove if next_node && next_node.text.strip == ''
next_node.remove removes the current next_node from the DOM if the next node isn't nil and its text isn't empty when stripped, in otherwords, if the node has only whitespace.
There are other techniques to locate only the TextNodes if all of them should be stripped from the document. That's risky, because it can end up deleting all blanks between tags, causing run-on sentences and joined words, which probably isn't what you want.
A first solution can be to remove empty text nodes, a quick way to do this for your exact case can be:
require 'nokogiri'
doc = Nokogiri::HTML("<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>")
doc.css('body').first.children.map{|node| node.to_s.strip}.compact.join
This won't work for nested elements as-is but should give you a good path for start.
UPDATE:
You can actually optimise a little with:
require 'nokogiri'
doc = Nokogiri::HTML::DocumentFragment.parse("<p>Foo Bar</p> <p>bar bar bar</p> <p>bla</p>")
doc.children.map{|node| node.to_s.strip}.compact.join
Here is all the possible task you can be looking for which deals with unnecessary whitespaces(including unicode one) in parsing output.
html = "<p>A paragraph.<em> </em> <br><br><em>
</em></p><p><em> </em>
</p><p><em>
</em><strong><em>\" Quoted Text \" </em></strong></p>
<ul><li><p>List 1</p></li><li><p>List 2</p></li><li><p>List 3 </p>
<p><br></p><p><br><em> </em><br>
A text content.<br><em><br>
</em></p></li></ul>"
doc = Nokogiri::HTML.fragment(html)
doc.traverse { |node|
# removes any whitespace node
node.remove if node.text.gsub(/[[:space:]]/, '') == ''
# replace mutiple consecutive spaces with single space
node.content = node.text.gsub(/[[:space:]]{2,}/, ' ') if node.text?
}
# Gives you html without any text node including <br> or multiple spaces anywhere in the text of html
puts doc.to_html
# Gives text of html, concatenating li items with a space between them
# By default li items text are concatenated without the space
Nokogiri::HTML(doc.to_html).xpath('//text()').map(&:text).join(' ')
#Output
# "A paragraph. \" Quoted Text \" \n List 1 \n List 2 \n \n List 3 \n A text content. \n \n"
# To Remove newline character '\n'
Nokogiri::HTML(doc.to_html).xpath('//text()').map(&:text).join(' ').gsub(/\n+/,'')
#Output
# "A paragraph. \" Quoted Text \" List 1 List 2 List 3 A text content."
Note: If you are not using fragment in case of a complete html doc then you might have to replace traverse with other function like search.
data.squish does the same thing and is way more readable.

very simple ruby combobox - text parse prolem

tried to create simple dropdown in ruby - like that :
<%= select_tag(:origin_id, '<option value="1">Lisbon</option>') %>
and I received an empy one.
this is the generated html
<select id="origin_id" name="origin_id"><option alue="1">Lisbon</option>/select>
Mark your HTML as safe to avoid Rails to escape it.
<%= select_tag(:origin_id, '<option value="1">Lisbon</option>'.html_safe) %>

how to BOLD a fragment of the linkText in an Html.ActionLink?

I have this:
<li><%:Html.ActionLink(user.Email.Replace(Model.SearchString, "<b>" + Model.SearchString + "</b>"), "LoginEdit", "Admin", new { area = "Staff", webUserKey = user.WebUserKey }, null)%>, last login: <%:loginString%></li>
As you can see, I want the portion of the Email string that matches the Model.SearchString to be bolded. I can't figure out the syntax to make this happen, given the context of my code.
Any ideas?
The goal is something like this (assuming user searched for "john"):
<b>john</b>#gmail.com
Whenever I encounter a situation likes this, I try my best not to embed HTML inside HTML helpers. In addition, I think breaking up your code will help in future maintenance - you're doing a lot in a single function call.
I would prefer doing it this way:
<li>
<a href="<%: Url.Action("LoginEdit", "Admin", new { area = "Staff", webUserKey =user.WebUserKey }) %>">
<%: user.Email.Replace(Model.SearchString, "") %>
<b><%: Model.SearchString %></b>
</a>
last login: <%: loginString %>
</li>
It's a few more lines of code, but it makes it much easier to decipher what's going on.
I think the issue is that the output of <%: %> is HTML encoded. So your <b> tag is probably encoded and you see the actual tag in the rendered HTML instead of the bold text.
If user.Email is a trusted value you could skip HTML encoding the output.
<li><%= Html.ActionLink(user.Email.Replace(Model.SearchString, "<b>" + Model.SearchString + "</b>"), "LoginEdit", "Admin", new { area = "Staff", webUserKey = user.WebUserKey }, null)%>, last login: <%:loginString%></li>
For more information see: http://haacked.com/archive/2009/09/25/html-encoding-code-nuggets.aspx

Resources