I am trying to find the best way to find a word on a page using Nokogiri.
I have a page which has the following text.
<p>Modelo: ABC123-A</p>
I would like to find the "Modelo:" text, and then get the model number after it.
I have had a look around but can't seem to find. So, I thought I would post on here and see if anyone with experience of Nokogiri could shed some light on this for me.
Use p:contains selector and get the matching p nodes.
doc = Nokogiri::HTML('<html><body><p>Modelo: ABC123-A</p><br/><p>Nothing here</p><p>Modelo: 4321</p></body></html>')
doc.css('p:contains("Modelo")').map { |x| x.text.split(': ').last }
#=> ["ABC123-A", "4321"]
A simple example:
doc = Nokogiri::HTML('<html><body><p>Modelo: ABC123-A</p></body></html>')
doc.css('p').first.content # => Modelo: ABC123-A
str.split( ': ' )[-1] # => ABC123-A
You could also try Oga, it's lighter than Nokogiri.
Related
I am trying to detect the urls from a text and replace them by wrapping in quotes like below:
original text: Hey, it is a url here www.example.com
required text: Hey, it is a url here "www.example.com"
original text show my input value and required text represents the required output. I searched a lot on web but could not find any possible solution. I already have tried URL.extract feature but that doesn't seem to detect URLs without http or https. Below are the examples of some of urls I want to deal with. Kindly let me know if you know the solution.
ANQUETIL-DUPERRON Abraham-Hyacinthe, KIEFFER Jean-Luc, www.hominides.net/html/actualites/outils-preuve-presence-hominides-asie-0422.php,Les Belles lettres, 2001.
https://www.ancient-code.com/indian-archeologists-stumbleacross-ruins-great-forgotten-civilization-mizoram/
www.jstor.org/stable/24084454
www.biorespire.com/2016/03/22/une-nouvelle-villeantique-d%C3%A9couverte-en-inde/
insu.cnrs.fr/terre-solide/terre-et-vie/de-nouvellesdatations-repoussent-l-age-de-l-apparition-d-outils-surle-so
www.cerege.fr/spip.php?page=pageperso&id_user=94
Find words who look like urls:
str = "ANQUETIL-DUPERRON Abraham-Hyacinthe, KIEFFER Jean-Luc, www.hominides.net/html/actualites/outils-preuve-presence-hominides-asie-0422.php,Les Belles lettres, 2001.\n\nhttps://www.ancient-code.com/indian-archeologists-stumbleacross-ruins-great-forgotten-civilization-mizoram/\n\nwww.jstor.org/stable/24084454\n\nwww.biorespire.com/2016/03/22/une-nouvelle-villeantique-d%C3%A9couverte-en-inde/\n\ninsu.cnrs.fr/terre-solide/terre-et-vie/de-nouvellesdatations-repoussent-l-age-de-l-apparition-d-outils-surle-so\n\nwww.cerege.fr/spip.php?page=pageperso&id_user=94"
str.split.select{|w| w[/(\b+\.\w+)/]}
This will give you an array of words which have no spaces and include a one or more . characters which MIGHT work for your use case.
puts str.split.select{|w| w[/(\b+\.\w+)/]}
www.hominides.net/html/actualites/outils-preuve-presence-hominides-asie-0422.php,
https://www.ancient-code.com/indian-archeologists-stumbleacross-ruins-great-forgotten-civilization-mizoram/
www.jstor.org/stable/24084454
www.biorespire.com/2016/03/22/une-nouvelle-villeantique-d%C3%A9couverte-en-inde/
insu.cnrs.fr/terre-solide/terre-et-vie/de-nouvellesdatations-repoussent-l-age-de-l-apparition-d-outils-surle-so
www.cerege.fr/spip.php?page=pageperso&id_user=94
Updated
Complete solution to modify your string:
str_with_quote = str.clone # make a clone for the `gsub!`
str.split.select{|w| w[/(\b+\.\w+)/]}
.each{|url| str_with_quote.gsub!(url, '"' + url + '"')}
Now your cloned object wraps urls inside double quotes
puts str_with_quote
Will give you this output
ANQUETIL-DUPERRON Abraham-Hyacinthe, KIEFFER Jean-Luc, "www.hominides.net/html/actualites/outils-preuve-presence-hominides-asie-0422.php,Les" Belles lettres, 2001.
"https://www.ancient-code.com/indian-archeologists-stumbleacross-ruins-great-forgotten-civilization-mizoram/"
"www.jstor.org/stable/24084454"
"www.biorespire.com/2016/03/22/une-nouvelle-villeantique-d%C3%A9couverte-en-inde/"
"insu.cnrs.fr/terre-solide/terre-et-vie/de-nouvellesdatations-repoussent-l-age-de-l-apparition-d-outils-surle-so"
"www.cerege.fr/spip.php?page=pageperso&id_user=94"
I'm looking to parse a RSS feed such as http://feeds.soundcloud.com/users/soundcloud:users:70685608/sounds.rss using Nokogiri. However despite how many different ways I try to parse the document, I'm unable to get the data elements I want.
For example, if I want to get the itunes:name block, I've tried the following below and none of these is returning what I want.
page = HTTParty.get('http://feeds.soundcloud.com/users/soundcloud:users:70685608/sounds.rss')
parse_feed = Nokogiri::HTML(page)
#tried 1
parse_feed.css('channel|itunes:owner')
#tried 2
parse_feed.css('itunes:name')
#tried 3
parse_feed.xpath('//itunes:email')
Am I just way off here on my xpath or css searches to get the block I'm looking for?
It worked for me by switching Nokogiri::HTML(page) with Nokogiri::XML(page). Then, I could use xpath to extract the itunes:name block:
page = HTTParty.get('http://feeds.soundcloud.com/users/soundcloud:users:70685608/sounds.rss')
parse_feed = Nokogiri::XML(page)
itunes_name = parse_feed.xpath("//itunes:name")
# do stuff
So I have recently switched from Dreamweaver to Aptana Studio 3 and I have been playing around with the whole custom snippet feature. For the life of me though I cannot figure out how to take a selection/highlighted text and wrap it with my own custom code and/or text. I have looked around the internet for three days now and cannot find anything regarding snippets. I have found some things using commands and key combinations, but I am wanting to create and use a snippet and trying to modify what I have found is not producing good fruit.
I have been able to create my own category and some basic snippets that insert straight text, but nothing that uses a selection.
I have absolutely NO experience with Ruby so forgive me if what follows is completely atrocious. I have more experience with PHP, HTML, Javascript, Java, etc. Here is what I have so far.
snippet "Selection Test" do |snip|
snip.trigger = "my_code"
snip.input = :selection
selection = ENV['TM_SELECTED_TEXT'] || ''
snip.expansion = "<test>$selection</test>\n"
snip.category = "My Snippets"
end
I haven't done much with custom Snippets, but if it helps, there is an example in the HTML bundle of a snippet that surrounds the selected text with <p></p> tags when you do Ctrl + Shift + W. You can see the code for it in snippets.rb in the HTML bundle:
with_defaults :scope => 'text.html - source', :input => :none, :output => :insert_as_snippet do |bundle|
command t(:wrap_selection_in_tag_pair) do |cmd|
cmd.key_binding = "CONTROL+SHIFT+W"
cmd.input = :selection
cmd.invoke do |context|
selection = ENV['TM_SELECTED_TEXT'] || ''
if selection.length > 0
"<${1:p}>${2:#{selection.gsub('/', '\/')}}</${1:p}>"
else
"<${1:p}>$0</${1:p}>"
end
end
end
end
I fiddled around with putting it into the PHP bundle for a few minutes under CTRL + Shift + P and got it working in HTML files, which was not my goal... but was progress. I may play around with it some more later, but in the meantime, maybe you know enough after all of your research to get something put together. I would be interested to see your results if you get this figured out.
I am trying to parse og meta tags using the HTTParty gem using this code:
link = http://www.usatoday.com/story/gameon/2013/01/08/nfl-jets-tony-sparano-fired/1817037/
# link = http://news.yahoo.com/chicago-lottery-winners-death-ruled-homicide-181627271.html
resp = HTTParty.get(link)
ret_body = resp.body
# title
og_title = ret_body.match(/\<[Mm][Ee][Tt][Aa] property\=\"og:title\"\ content\=\"(.*?)\"\/\>/)
og_title = og_title[1].to_s
The problem is that it worked on some sites (yahoo!) but not others (usa today)
Don't parse HTML with regular expressions, because they're too fragile for anything but the simplest problems. A tiny change to the HTML can break the pattern, causing you to begin a slow battle of maintaining an ever expanding pattern. It's a war you won't win.
Instead, use a HTML parser. Ruby has Nokogiri, which is excellent. Here's how I'd do what you want:
require 'nokogiri'
require 'httparty'
%w[
http://www.usatoday.com/story/gameon/2013/01/08/nfl-jets-tony-sparano-fired/1817037/
http://news.yahoo.com/chicago-lottery-winners-death-ruled-homicide-181627271.html
].each do |link|
resp = HTTParty.get(link)
doc = Nokogiri::HTML(resp.body)
puts doc.at('meta[property="og:title"]')['content']
end
Which outputs:
Jets fire offensive coordinator Tony Sparano
Chicago lottery winner's death ruled a homicide
Perhaps I can offer an easier solution? Check out the OpenGraph gem.
It's a simple library for parsing Open Graph protocol information from web sites and should solve your problem.
Solution:
og_title = ret_body.match(/\<[Mm][Ee][Tt][Aa] property\=\"og:title\"\ content\=\"(.*?)\"[\s\/\>|\/\>]/)
og_title = og_title[1].to_s
Trailing whitespace messed up the parsing so make sure to check for that. I added an OR clause to the regex to allow for both trailing and non trailing whitespace.
Suppose I want to turn this :
http://en.wikipedia.org/wiki/Anarchy
into this :
en.wikipedia.org
or even better, this :
wikipedia.org
Is this even possible in regex?
Why use a regex when Ruby has a library for it? The URI library:
ruby-1.9.1-p378 > require 'uri'
=> true
ruby-1.9.1-p378 > uri = URI.parse("http://en.wikipedia.org/wiki/Anarchy")
=> #<URI::HTTP:0x000001010a2270 URL:http://en.wikipedia.org/wiki/Anarchy>
ruby-1.9.1-p378 > uri.host
=> "en.wikipedia.org"
ruby-1.9.1-p378 > uri.host.split('.')
=> ["en", "wikipedia", "org"]
Splitting the host is one way to separate the domains, but I'm not aware of a reliable way to get the base domain -- you can't just count, in the event of a URL like "http://somedomain.otherdomain.school.ac.uk" vs "www.google.com".
/http:\/\/([^\/]*).*/ will produce en.wikipedia.org from the string you provided.
/http:\/\/.{0,3}\.([^\/]*).*/ will produce wikipedia.org.
yes
Now I know you haven't asked for how, and you haven't specified a language, but I'll answer anyway... (note, this works for all language subsites, not just en.wikipedia...)
perl:
$url =~ s,http://[a-z]{2}\.(wikipedia\.org)/.*,$1,;
ruby:
url = url.sub(/http:\/\/[a-z]{2}\.(wikipedia\.org)\/.*/, '\1')
php:
$url = preg_replace('|http://[a-z]{2}.(wikipedia.org)/.*|, '$1', $url);
Of course, for this particular example, you don't even need a regex, just this will do:
url = 'wikipedia.org'
but I jest...
you probably want to handle any URL and pull out the domain part, and it should also work for domains in different countries, eg: foo.co.uk.
In which case, I'd use Mark Rushakoff's solution to get the hostname and then a regex to pull out the domain:
domain = host.sub(/^.*\.([^.]+\.[^.]+(\.[a-z]{2})?)$/, '\1')
Hope this helps
Also, if you want to learn more, I have a regex tute online: http://tech.bluesmoon.info/2006/04/beginning-regular-expressions.html
Sure all you would have to do is search on http://(.*)/wiki/Anarchy
In Perl (Sorry I don't know Ruby, but I expect it's similar)
$string_to_search =~ s/http:////(.)//. should give you wikipedia.org
to get rid of the en, you can simply search on http:////en(.)//......
That should do it.
Update: In case you're not familiar with Regex, I would recommend picking up a Regex book, this one really rocks and I like it: REGEX BOOK,Mastering Regular Expressions, I saw it on half.com the other day for 14.99 used, but to clarify what i suggested above, is to look for the string http://en, then for anything until you find a / this is all captured in $1 (in perl, not sure if it's the same in ruby), a simple print $1 will print the string.
Update: #2 sorry the star in the regex is not showing up for some reason, so where you see the . in the () and after the // just imagine a *, oh and I forgot for the en part add a /. at the end that way you don't end up with .wikipedia.org