I have spent all day on this and I cannot figure out how to use multithreading in rails, without causing the following error:
Uncaught exception: Circular dependency detected while autoloading constant Docket::System::ExecutableFactory
/Users/myuser/.rvm/gems/ruby-2.1.2#core/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:478:in `load_missing_constant'
/Users/myuser/.rvm/gems/ruby-2.1.2#core/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:180:in `const_missing'
The code works without threads, but as soon as I introduce threads, that is when I get the "circular dependency" error.
Below is the code in question:
def run
records.in_groups_of(4) do |records|
workers = records.map do |record|
Thread.new do
begin
options[:record] = record
county = create_county options
county.prepare
if county.ready
county.crawl
end
rescue ThreadError => e
puts "THREAD ERROR: #{e}"
end
end
end
workers.map(&:join)
end
end
...
def crawl
cmd = "#{executable_factory.path} #{crawler_path}"
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
stdout_is = stdout.read
stderr_is = stderr.read
end
end
Docket::System::ExecutableFactory is one of my classes, but I have it properly nested. Directory structure:
\docket
\system
executable_factory.rb
And I have it properly namedspace:
module Docket
module System
class ExecutableFactory
...
end
end
end
So why does this error occur with threads and how can I fix it?
Related
I have a rake task that loops through rows in CSV file, and inside that loop, there's a begin/rescue block to catch any possible raised exception. But when I run it, it keeps on saying 'rake aborted!' and it is not entering the rescue block
CSV.foreach(path, :headers => true) do |row|
id = row.to_hash['id'].to_i
if id.present?
begin
# call to mymethod
rescue => ex
puts "#{ex} error executing task"
end
end
end
...
def mymethod(...)
...
begin
response = RestClient.post(...)
rescue => ex
raise Exception.new('...')
end
end
Expected: It should finish looping all the rows of the CSV
Actual result: It stops after reaching the 'raise' exception saying that:
rake aborted!
Exception: error message here
...
Caused by:
RestClient::InternalServerError: 500 Internal Server Error
You can use next to skip the faulty step of loop:
CSV.foreach(path, :headers => true) do |row|
id = row.to_hash['id'].to_i
if id.present?
begin
method_which_doing_the_staff
rescue SomethingException
next
end
end
end
And raise the exception inside your method:
def method_which_doing_the_staff
stuff
...
raise SomethingException.new('hasd')
end
I solved this issue by just commenting out the line that is raising an exception because it seems like it the quickest fix for now.
# raise Exception.new('...')
I'm still open to other suggestions if there are any better ways to do it.
I use the capybara-webkit gem to scrape data from certain pages in my Rails application. I've noticed, what seems to be "random" / "sporadic", that the application will crash with the following error:
Capybara::Webkit::ConnectionError: /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/bin/webkit_server failed to start.
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/server.rb:56:in `parse_port'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/server.rb:42:in `discover_port'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/server.rb:26:in `start'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/connection.rb:67:in `start_server'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/connection.rb:17:in `initialize'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/driver.rb:16:in `new'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/driver.rb:16:in `initialize'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit.rb:15:in `new'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit.rb:15:in `block in <top (required)>'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-2.7.1/lib/capybara/session.rb:85:in `driver'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-2.7.1/lib/capybara/session.rb:233:in `visit'
It happens even after it's already connected and accessed a website multiple times before. Here's a code snippet of what I'm using currently...
if site.url.present?
begin
# Visit the URL
session = Capybara::Session.new(:webkit)
session.visit(site.url) # here is where the error occurs...
document = Nokogiri::HTML.parse(session.body)
# Load configuration options for Development Group
roster_table_selector = site.development_group.table_selector
header_row_selector = site.development_group.table_header_selector
row_selector = site.development_group.table_row_selector
row_offset = site.development_group.table_row_selector_offset
header_format_type = site.config_header_format_type
# Get the Table and Header Row for processing
roster_table = document.css(roster_table_selector)
header_row = roster_table.css(header_row_selector)
header_hash = retrieve_headers(header_row, header_format_type)
my_object = process_rows(roster_table, header_hash, site, row_selector, row_offset)
rescue ::Capybara::Webkit::ConnectionError => e
raise e
rescue OpenURI::HTTPError => e
if e.message == '404 Not Found'
raise "404 Page not found..."
else
raise e
end
end
end
I've even thought perhaps I don't find out why it's happening necessarily - but just recover when it does. So I was going to do a "retry" in the rescue block for the error but it appears the server is just down - so I get the same result when retrying. Perhaps someone knows of a way I can check if the server is down and restart it then perform a retry? Thanks for the help!
So after further investigating it appears that I was generating a new Capybara::Session for each iteration of my loop. I moved it outside of the loop and also added Capybara.reset_sessions! at the end of my loop. Not sure if that helps with anything -- but the issue seems to have been resolved. I'll monitor it for the next hour or so. Below is an example of my ActiveJob code now...
class ScrapeJob < ActiveJob::Base
queue_as :default
include Capybara::DSL
def perform(*args)
session = Capybara::Session.new(:webkit)
Site.where(config_enabled: 1).order(:code).each do |site|
process_roster(site, session)
Capybara.reset_sessions!
end
end
def process_roster(site, session)
if site.roster_url.present?
begin
# Visit the Roster URL
session.visit(site.roster_url)
document = Nokogiri::HTML.parse(session.body)
# processing code...
# pass the session that was created as the final parameter..
my_object = process_rows( ..., session)
rescue ::Capybara::Webkit::ConnectionError => e
raise e
rescue OpenURI::HTTPError => e
if e.message == '404 Not Found'
raise "404 Page not found..."
else
raise e
end
end
end
end
end
We use airbrake throughout our codebase which works great except for custom classes that do not inherit from a Rails hierarchy (i.e. ActiveRecord).
Some code:
class Offer
def counter(amount, qty=1)
begin
offers_response = viyet_mage_rest_client.counter_offer(#id.to_i, amount, qty.to_i)
rescue => e
notify_airbrake(e)
offers_response = []
end
end
end
In airbrake we see:
NoMethodError: undefined method `notify_airbrake' for Offer:Class
As the error. I want the actual error and not an error for reporting an error!
Our airbrake.rb file:
if Rails.env.production? || Rails.env.staging?
Airbrake.configure do |config|
config.api_key = Settings.airbrake.api_key
config.secure = true
end
end
Any help would be appreciated!
The method you need is Airbrake.notify_or_ignore(). For your given example:
class Offer
def counter(amount, qty=1)
begin
offers_response = viyet_mage_rest_client.counter_offer(#id.to_i, amount, qty.to_i)
rescue => e
Airbrake.notify_or_ignore(e)
offers_response = []
end
end
end
More detail can be found here: https://github.com/airbrake/airbrake/wiki/Using-Airbrake-with-plain-Ruby
Whats the best way to test concurrency locally? i.e. i want to test 10 concurrent hits. I am aware of services like Blitz. However, I am trying to find a simpler way of doing it locally to test against race conditions.
Any ideas? Via Curl maybe?
Check out Apache Bench (ab). Basic usage is dead simple:
ab -n 100 -c 10 http://your.application
For locally testing race conditions in the tests you can use helpers like this
# call block in a forked process
def fork_with_new_connection(config, object = nil, options={})
raise ArgumentError, "Missing block" unless block_given?
options = {
:stop => true, :index => 0
}.merge(options)
fork do
# stop the process after fork
Signal.trap('STOP') if options[:stop]
begin
ActiveRecord::Base.establish_connection(config)
yield(object)
ensure
ActiveRecord::Base.remove_connection
end
end
end
# call multiply times blocks
def multi_times_call_in_fork(count=3, &block)
raise ArgumentError, "Missing block" unless block_given?
config = ActiveRecord::Base.remove_connection
pids = []
count.times do |index|
pids << fork_with_new_connection(config, nil, :index=>index, &block)
end
# continue forked processes
Process.kill("CONT", *pids)
Process.waitall
ActiveRecord::Base.establish_connection(config)
end
# example
multi_times_call_in_fork(5) do
# do something with race conditions
# add asserts
end
I am using rescue to handle all the heavy lifting background tasks,
In my library/parsers/file.rb I have
Resque.enqueue(Hello)
This will redirect app/workers/file.rb where I have
class Hello
def self.perform(page)
.......
.......
end
rescue Exception => e
log "error: #{e}"
end
end
my lib/tasks/resque.rake file is
require "resque/tasks"
task "resque:setup" => :environment
I am able to queue the jobs buts when i try to execute the job using
rake resque:work QUEUE=*
it is throwing an error by saying
argument error
wrong number of arguments (0 for 1)
what am I doing wrong in this?
pjumble is exactly right, you're not passing the page.
Resque.enqueue(Hello, page_id)
enqueue takes the Job followed by the args which go into the perform action. If you had:
class Hello
def self.perform(page_number, page_foo, page_bar)
...
end
end
Then you would do this:
Resque.enqueue(Hello, page_number, page_foo, page_bar)