I'm using the following code to scrape an eBay listing using the scrAPI gem:
I installed this by executing:
gem install scrapi
I'm also overriding its default text parser by declaring:
Scraper::Base.parser :html_parser
The problem is that I keep receiving the following error on the auctions array size. Not sure what I'm doing wrong? Both size and length don't work.
Scraper.rb:31:in `<class:ScraperDemo>': undefined method `size' for nil:NilClass (No
MethodError)Scraper.rb:10:in `<main>'
I just run via the commandline:
ruby Scraper.rb
Code:
#!/usr/bin/env ruby
require 'open-uri'
require 'httparty'
require 'json'
require 'scrapi'
Scraper::Base.parser :html_parser
class ScraperDemo
ebay_auction = Scraper.define do
process "h3.ens>a", :description=>:text, :url=>"#href"
process "td.ebcPr>span", :price=>:text
process "div.ebPicture >a>img", :image=>"#src"
result :description, :url, :price, :image
end
ebay = Scraper.define do
array :auctions
process "table.ebItemlist tr.single", :auctions=>ebay_auction
result :auctions
end
auctions = ebay.scrape(URI.parse('http://search.ebay.com/ipod-nano_W0QQcatrefZC6QQfromZR3QQfsooZ1QQfsopZ1QQkeywordZonQQsacatZQ2d1QQstrkwZipod'))
# No. of channels found
puts auctions.size # error occurs on this line number
# First auction:
auction = auctions[0]
puts auction.description
puts auction.url
end
I ended up using Nokogiri as my scraper of choice.
Related
I have plugin that takes attribute from post's front matter and uses it in permalink. Problem is I need to clean up any accents and diacritics from the string before putting it in to the permalink. Ruby on rails has method called parametrize which does exactly what I need but I have no idea how to use it in plugin.
This is plugins code I have:
module JekyllCustomPermalink
class CustomPermalink < Jekyll::Generator
safe true
priority :low
def generate(site)
# nothing to do, wait for hook
end
Jekyll::Hooks.register :documents, :pre_render do |doc|
begin
# check if jekyll can resolve the url template
doc.url
rescue NoMethodError => error
begin
if !doc.collection.metadata.fetch("custom_permalink_placeholders").is_a?(Array)
raise CustomPermalinkSetupError, "The custom placeholders need to be an array! Check the settings of your '#{doc.collection.label}' collection."
end
def doc.url_template
#custom_url_template ||= collection.metadata.fetch("custom_permalink_placeholders").inject(collection.url_template){|o,m| o.sub ":" + m, data[m].to_s.parameterize}
end
rescue KeyError
# "custom_permalink_placeholders"
raise CustomPermalinkSetupError, "No custom placeholders defined for the '#{doc.collection.label}' collection. Define an array of placeholders under the key 'custom_permalink_placeholders'. \nCaused by: " + error.to_s
end
end
end
end
end
but I get this error:
john#arch-thinkpad ~/P/blog (master)> bundle exec jekyll serve --trace
Configuration file: /home/john/Projects/lyricall/_config.yml
Source: /home/john/Projects/lyricall
Destination: /home/john/Projects/lyricall/_site
Incremental build: disabled. Enable with --incremental
Generating...
Jekyll Feed: Generating feed for posts
Liquid Exception: undefined method `parameterize' for "Žďořšťáčik":String in feed.xml
bundler: failed to load command: jekyll (/home/john/.gem/ruby/3.0.0/bin/jekyll)
/usr/lib/ruby/gems/3.0.0/gems/jekyll_custom_permalink-0.0.1/lib/jekyll_custom_permalink/custom_permalink.rb:20:in `block in url_template': undefined method `parameterize' for "Žďořšťáčik":String (NoMethodError)
What am I doing wrong ? How can I use this method which should be part of a string class but apparently it is not ? How can I achieve same result without ruby on rails framework ?
INFO:
jekyll 4.1.1
ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [x86_64-linux]
Thank you for help
Rails additions to base Ruby classes, like String#parameterize, are part of the Active Support Core Extensions. The activesupport gem can be installed and used independent of Rails.
To keep the default footprint low, ActiveSupport allows you to require only the individual extensions you want to use. In your case, you will need to require the string inflection extensions:
require 'active_support/core_ext/string/inflections'
"Kurt Gödel".parameterize
=> "kurt-godel"
I am using gem differ https://github.com/pvande/differ
I have a helper
require 'differ'
module AnswersHelper
def self.getDiff (text1, text2)
Differ.format = :html
diff = Differ.diff_by_word(#current, #original)
end
end
But I get an error No such file to load -- differ
If I remove require line
I get an error at that line
Differ.format = :html
uninitialized constant QuestionsController::Differ
When I tried following commands in rails console it worked
require 'differ'
diff = Differ.diff_by_word("text1","text2)
I have gem differ in my gemfile
and also I tried
require_relative 'differ'
and
require './differ'
UPD: seems restarting server helps, I'll check it right now
Restarting server helped........
Trying to write a custom irb for a gem to ease debugging. At the point where the shell loads and you can use it like ruby console but running into this wall
MyClass.get_last_instance
=>
_.attributes
=> {'attribute'=> 'test'}
The instance was found but a blank string is echo'ed. Here are the requires involved in starting the shell
require 'irb'
require 'irb/completion'
require 'debugger'
I tried reading through the rails source code, didn't get very far, mostly because I didn't really know what I was looking for. I think I'm just missing a require of a part of rails that echos objects.
create a .irbrc in your home path for ubuntu/osx and use the below code, it will work. Also you can add additional gems also debugger or irb
# print SQL to STDOUT
if ENV.include?('RAILS_ENV') && !Object.const_defined?('RAILS_DEFAULT_LOGGER')
require 'logger'
end
# Autocomplete
require 'irb/completion'
# Prompt behavior
ARGV.concat [ "--readline", "--prompt-mode", "simple" ]
# History
require 'irb/ext/save-history'
IRB.conf[:SAVE_HISTORY] = 100
IRB.conf[:HISTORY_FILE] = "#{ENV['HOME']}/.irb-save-history"
# Easily print methods local to an object's class
class Object
def local_methods
(methods - Object.instance_methods).sort
end
end
# copy a string to the clipboard
def pbcopy(string)
`echo "#{string}" | pbcopy`
string
end
require "rubygems"
I'm trying to move a current working task (in production and in the console) to use delayed_job in a Rails 2 app but keep getting the error:
ThermalImageJob failed with NameError: uninitialized constant Barby::Code128B
I've pored through others' code searching for an answer to no avail. Here's my code:
/lib/thermal_image_job.rb
class ThermalImageJob < Struct.new(:order_id)
def perform
order = Order.find(order_id)
order.tickets.each do |ticket|
ticket.barcodes.each do |barcode|
barcode.generate_thermal_image
end
end
end
end
/app/controllers/orders_controller.rb
Delayed::Job.enqueue(ThermalImageJob.new(#order.id))
/app/models/barcode.rb
def generate_thermal_image(format=:gif)
filename = "#{barcode}_thermal.#{format}"
temp_file_path = File.join("#{RAILS_ROOT}", 'tmp', filename)
unless FileTest.exists?(temp_file_path)
barcode_file = File.new(temp_file_path, 'w')
code = Barby::Code128B.new(barcode)
....
end
Gemfile
gem "delayed_job", "2.0.7"
gem "daemons", "1.0.10"
Well, after much head banging, I figured it out, so I'm posting this to help the next person. The problem was that it couldn't find the barby libs, so I added a require at the beginning of my class:
require "barby/outputter/rmagick_outputter"
require "barby/barcode/code_128"
Trying to go through the tekpub rack tutorial but run into this error.
Boot Error
Something went wrong while loading app.ru
LoadError: cannot load such file -- haiku
There is a file named haiku.rb in the same directory as the app I am trying to run but I get the above error while trying to run the program. Here is the code:
class EnvironmentOutput
def initialize(app=nil)
#app = app
end
def call(env)
out = ""
unless(#app.nil?)
response = #app.call(env)[2]
out+=response
end
env.keys.each {|key| out+="<li>#{key}=#{env[key]}</li>"}
["200",{"Content-Type" => "text/html"},[out]]
end
end
require 'haml'
require 'haiku'
class MyApp
def call(env)
poem = Haiku.new.random
template = File.open("views/index.haml").read
engine = Haml::Engine.new(template)
out = engine.render(Object.new, :poem => poem)
["200",{"Content-Type" => "text/html"}, out]
end
end
use EnvironmentOutput
run MyApp.new
I'm sure its a small error as the code is the same as in the tutorial and it works for him...
Thanks
You'll want to read up on ruby load path (either $LOAD_PATH or $:). By default, ruby has a load path which includes wherever your gems are installed, which is why you can do require 'haml' without providing the full path to where your haml gem is located.
When you type require 'haiku', you're basically telling ruby to look for some file called haiku.rb somewhere in it's load path, and the LoadError comes from ruby not finding your haiku.rb file in any of the directories listed in $LOAD_PATH (or $:, which is just shorthand for $LOAD_PATH).
You can solve this in one of (at least) two ways:
change require 'haiku' to require File.dirname(__FILE__) + '/haiku.rb' to explicitly tell ruby what file to load
add the current working directory to your load path: $:.push(File.dirname(__FILE__)). This way you can keep the require 'haiku' part.