in `escape': undefined method `gsub' for #<URI::HTTP:0x007fa07cb01e08> (NoMethodError) - ruby-on-rails

Hi I am trying to scrape a web page "take the links" go to that links and "to scrape it" too.
require 'rubygems'
require 'scrapi'
require 'uri'
Scraper::Base.parser :html_parser
web = "http://......"
def sub_web(linksubweb)
uri = URI.parse(URI.encode(linksubweb))
end
scraper = Scraper.define do
array :items
process "div.mozaique>div", :items => Scraper.define {
process "p>a", :title => :text
process "div.thumb>a", :link => "#href"
result :title, :link,
}
result :items
end
uri = URI.parse(URI.encode(web))
scraper.scrape(uri).each do |pag|
link_full = uri + pag.link.to_str
puts pag.title
sub_web(link_full)
puts
end
And I have the following error
e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /Users/sss/web/app/views/admin/topics/webconector.rb
Title 1
http://mydomain/user34/top5
/Users/sss/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/common.rb:304:in `escape': undefined method `gsub' for #<URI::HTTP:0x007fa07cb01e08> (NoMethodError)
from /Users/sss/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/common.rb:623:in `escape'
from ../app/views/admin/topics/conectaweb.rb:11:in `sub_web'
from ../app/views/admin/topics/conectaweb.rb:34:in `block in <top (required)>'
from ../views/admin/topics/conectaweb.rb:29:in `each'
from ../app/views/admin/topics/conectaweb.rb:29:in `<top (required)>'
from -e:1:in `load'
from -e:1:in `<main>'
Process finished with exit code 1

try using uri = URI.parse(URI.encode(linksubweb.to_s)) this should work. The problem is that method requires a string argument so you have to first convert the URI::HTTP object into string.

Related

rails mechanize .click undefined method `click' for #

I have created a mechanize task in rails. it is very simple
task :estimateone => :environment do
require 'mechanize'
mechanize = Mechanize.new
page = mechanize.get('https://www.theurbanlist.com/brisbane/a-list/50-brisbane-cafes-you-should-have-eaten-breakfast-at')
page.css('ol li a').each do |link|
link.click
end
end
but I get this error, what am I doing wrong?
rake aborted!
NoMethodError: undefined method `click' for #<Nokogiri::XML::Element:0x00007ffb4b281830>
/Users/jeremybray/RubymineProjects/OpportunityFinder/lib/tasks/getlead.rake:8:in `block (2 levels) in <top (required)>'
/Users/jeremybray/.rvm/gems/ruby-2.4.2/gems/nokogiri-1.8.4/lib/nokogiri/xml/node_set.rb:204:in `block in each'
/Users/jeremybray/.rvm/gems/ruby-2.4.2/gems/nokogiri-1.8.4/lib/nokogiri/xml/node_set.rb:203:in `upto'
/Users/jeremybray/.rvm/gems/ruby-2.4.2/gems/nokogiri-1.8.4/lib/nokogiri/xml/node_set.rb:203:in `each'
/Users/jeremybray/RubymineProjects/OpportunityFinder/lib/tasks/getlead.rake:7:in `block in <top (required)>'
/Users/jeremybray/.rvm/gems/ruby-2.4.2#global/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/Users/jeremybray/.rvm/gems/ruby-2.4.2/bin/ruby_executable_hooks:15:in `eval'
/Users/jeremybray/.rvm/gems/ruby-2.4.2/bin/ruby_executable_hooks:15:in `<main>'
As click is a method of Mechanize, you should use your instance of it as the receiver of the click method, not the link, that's a Nokogiri one.
As the doc states, it receives the element, clicks it and returns the fetched page.
Try with:
-- snip --
page.css('ol li a').each do |link|
mechanize.click(link)
end
Page#links_with will return links, css will return html elements:
page.links_with(css: 'ol li a').each do |link|
link.click
end

NoMethodError: undefined method in routes_with_grape

I am trying to set up a rails grape api with the below structure.
app
api
api
V1
user.rb
app.rb
I am getting this error when i run routes_with_grape
rake aborted!
NoMethodError: undefined method `ast' for "/api/ping(.json)":String
/usr/local/rvm/gems/ruby-1.9.3-p551/gems/actionpack-4.2.1/lib/action_dispatch/journey/path/pattern.rb:14:in `initialize'
/usr/local/rvm/gems/ruby-1.9.3-p551/gems/grape-rails-routes-1.0/lib/rails/tasks/routes_with_grape.rake:9:in `new'
/usr/local/rvm/gems/ruby-1.9.3-p551/gems/grape-rails-routes-1.0/lib/rails/tasks/routes_with_grape.rake:9:in `block (3 levels) in <top (required)>'
/usr/local/rvm/gems/ruby-1.9.3-p551/gems/grape-rails-routes-1.0/lib/rails/tasks/routes_with_grape.rake:8:in `each'
/usr/local/rvm/gems/ruby-1.9.3-p551/gems/grape-rails-routes-1.0/lib/rails/tasks/routes_with_grape.rake:8:in `block (2 levels) in <top (required)>'
/usr/local/rvm/gems/ruby-1.9.3-p551/gems/grape-rails-routes-1.0/lib/rails/tasks/routes_with_grape.rake:6:in `each'
/usr/local/rvm/gems/ruby-1.9.3-p551/gems/grape-rails-routes-1.0/lib/rails/tasks/routes_with_grape.rake:6:in `block in <top (required)>'
/usr/local/rvm/gems/ruby-1.9.3-p551/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-1.9.3-p551/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => routes_with_grape
(See full trace by running task with --trace)
here is my code
user.rb
module V1
class User < Grape::API
desc 'Returns pong.'
get :ping do
{ ping: params[:pong] || 'pong' }
end
end
end
app.rb
class API < Grape::API
prefix 'api'
format :json
mount ::V1::User
end
Thanks!
Ditch that gem and use this code (copied from here):
namespace :grape do
desc "Grape API Routes"
task :routes => :environment do
mapped_prefix = '/api' # where mounted API in routes.rb
params_str = ' params:'
desc_limit = 45
route_info = API.routes.map {|r| [r, r.instance_variable_get(:#options)] }
max_desc_size = route_info.map{|_,info| (info[:description] || '')[0..desc_limit].size }.max
max_method_size = route_info.map{|_,info| info[:method].size }.max
max_version_size = route_info.map{|_,info| info[:version].size }.max
max_path_size = route_info.map{|_,info| info[:path].sub(':version', info[:version]).size }.max
max_params_digits = route_info.map{|_,info| info[:params].size.to_s.size }.max
format_str = format(
'%%%ds %%%ds %%%ds %%%ds%%-%ds | %%%ds%%%ds %%s',
max_desc_size + 1,
max_version_size,
max_method_size,
mapped_prefix.size,
max_path_size,
max_params_digits,
params_str.size)
route_info.each do |_,info|
fields = [
info[:description] ? info[:description][0..desc_limit] : '',
info[:version],
info[:method],
mapped_prefix,
info[:path].sub(':version', info[:version]),
info[:params].size.to_s,
params_str,
info[:params].first.inspect,
]
puts format(format_str, *fields)
info[:params].drop(1).each do |param|
puts format(format_str, *([''] * (fields.size-1)) + [param.inspect])
end
end
end
end
paste it into Rakefile and modify:
route_info = API.routes.map
with your API mounting point in my case:
under app/api/web/api.rb I have
module Web
class API < Grape::API
prefix 'api'
mount Web::V1::Root
# mount API::V2::Root (next version)
end
end
So I replaced API.routes by Web::API.routes

String with URL-ENCODING

Hello I have an string with URL-ENCODING , %3A , %2F ...
http%3A%2F%2Fmydomain.com%2Fimage%2Fflv%2F1%2F8%2Fa%2Fimage_18060.jpg%3Fe%3D13777194
I would like to replace the "special characters" http://www.w3schools.com/tags/ref_urlencode.asp to ASCII characters , I am programing in Rails but i dont have idea from where i can start. Do you have any ideas?
Thank you very much for your help!
hummm
in this way Can I use URI.unescape?
scraper = Scraper.define do
array :items
process "div.mozaique>div", :items => Scraper.define {
process "div.thumb>a", URI.unescape(:link) => "#href"
result :link
}
result :items
end
/Users/jcr/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/common.rb:331:in `unescape': undefined method `gsub' for :link:Symbol (NoMethodError)
from /Users/jcr/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/uri/common.rb:649:in `unescape'
from /Users/jcr/web/sss/app/controllers/cweb.rb:17:in `block (2 levels) in main_web'
from /Users/jcr/.rvm/gems/ruby-1.9.3-p448/gems/scrapi-2.0.0/lib/scraper/base.rb:986:in `module_eval'
from /Users/jcr/.rvm/gems/ruby-1.9.3-p448/gems/scrapi-2.0.0/lib/scraper/base.rb:986:in `define'
from /Users/jcr/web/sss/app/controllers/cweb.rb:15:in `block in main_web'
from /Users/jcr/.rvm/gems/ruby-1.9.3-p448/gems/scrapi-2.0.0/lib/scraper/base.rb:986:in `module_eval'
from /Users/jcr/.rvm/gems/ruby-1.9.3-p448/gems/scrapi-2.0.0/lib/scraper/base.rb:986:in `define'
from /Users/jcr/web/sss/app/controllers/cweb.rb:13:in `main_web'
from /Users/jcr/web/sss/app/controllers/cweb.rb:57:in `<top (required)>'
from -e:1:in `load'
from -e:1:in `<main>'
You can use URI.unescape:
irb(main):003:0> require 'uri'
=> true
irb(main):006:0> URI.unescape("http%3A%2F%2Fmydomain.com%2Fimage%2Fflv%2F1%2F8%2Fa%2Fimage_18060.jpg%3Fe%3D13777194")
=> "http://mydomain.com/image/flv/1/8/a/image_18060.jpg?e=13777194"

How to load extra fields into a model from thinking-sphinx?

I use thinking-sphinx like this :
class Project < ActiveRecord::Base
define_index do
indexes title
indexes comments.message, as: :comment_message
indexes category_projects.description, as: :category_description
end
class << self
def text_search(word)
ThinkingSphinx.search(word, include: :comments)
end
end
end
When I do this Project.text_search('test') it works but I want to do this :
#projects = Project.text_search('test')
#projects.each do |project|
puts project.comment_message
end
For the moment I have this message :
undefined method `comment_message' for #<ThinkingSphinx::Search:0x000000057e11c0>
from /home/dougui/.rvm/gems/ruby-1.9.3-p194-perf#comment_my_projects/bundler/gems/thinking-sphinx-b293abdbdf0c/lib/thinking_sphinx/search.rb:185:in `method_missing'
from (irb):1
from /home/dougui/.rvm/gems/ruby-1.9.3-p194-perf#comment_my_projects/gems/railties-3.2.6/lib/rails/commands/console.rb:47:in `start'
from /home/dougui/.rvm/gems/ruby-1.9.3-p194-perf#comment_my_projects/gems/railties-3.2.6/lib/rails/commands/console.rb:8:in `start'
from /home/dougui/.rvm/gems/ruby-1.9.3-p194-perf#comment_my_projects/gems/railties-3.2.6/lib/rails/commands.rb:41:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
So, how to load fields into a model from thinking-sphinx?
I didn't find anything one the doc.
Thanks!
Answered on the TS mailing list as well, but essentially, the fields aren't returned by Sphinx, so you just need to refer to the associations/columns that the fields are built from:
projects.each do |project|
puts project.comments.message.join("\n")
end

rails not allowing me to call a method from inside another method - rails keeps saying the method is undefined

In my model for tutoring sessions, I have code that triggers reminder texts at different times. Everything worked fine, until I tried some refactoring, and now I am having issues.
def send_reminder_text(texts_batch)
texts_batch.each do |text|
page_number = Refugee.find(text.refugee_id)[:last_page]
body_of_text = text[:begin_time].in_time_zone.strftime("Burma Reminder: upcoming session at %I:%M%p beginning on page #{page_number}.
Please email jek2141#columbia.edu to reschedule or cancel the session.")
text.begin_text(body_of_text)
end
end
def self.deliver_pm_reminder_text
texts_batch = TutoringSession.batch_for_pm_reminder_text
send_reminder_text(texts_batch)
end
def self.deliver_just_before_reminder_text
texts_batch = TutoringSession.batch_for_just_before_reminder_text
send_reminder_text(texts_batch)
end
When I call the deliver_just_before_reminder_text function, I get the following error message:
irb(main):006:0> TutoringSession.send_reminder_text
NoMethodError: undefined method `send_reminder_text' for #<Class:0x00000003cfe5d8>
from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record /base.rb:1088:in `method_missing'
from (irb):6
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.1.1/lib/rails/commands/console.rb:45:in `start'
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.1.1/lib/rails/commands/console.rb:8:in `start'
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.1.1/lib/rails/commands.rb:40:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
This is in spite of the fact that the send_reminder_text is clearly defined above.
Your method declaration specifies that the method is on objects of the TutoringSession class:
def send_reminder_text(texts_batch)
But you're trying to call it as if it were a method on the class itself:
irb(main):006:0> TutoringSession.send_reminder_text
Try changing your definition to:
def self.send_reminder_text(texts_batch)

Resources