I am using lock! in my code and want to catch the exception thrown if lock! fails for some reason (e.g. cannot get the lock). What kind of exceptions can lock! throw? I checked the ruby docs but couldn't find the specific Exception classes.
Thanks.
When in doubt, probe.
Consider the following pair of functions:
def long_hold
ActiveRecord::Base.transaction do
u = User.find(220)
u.lock!
sleep 100.seconds
u.email="foo#bar.com"
u.save!
end
end
def short_hold
ActiveRecord::Base.transaction do
u = User.find(220)
u.lock!
u.email="foo#bar.com"
u.save!
end
end
In my setup (OSX 10.11, ruby 2.2.4, rails 4.2, postgres 9.5), running long_hold in one rails console and then running short_hold in a second console, I observe short_hold blocks until long_hold completes; moreover, instrumenting the code with puts, we see that while long_hold is sleeping, short_hold is waiting to acquire the lock.
Assuming no caveats about the independence of rails consoles, this suggests that no exceptions are thrown if a second process tries to lock a row that is already locked, but that process blocks until the first completes.
Here is the source for that locking call. It calls reload and its source looks like this:
# File lib/active_record/base.rb, line 2333
2333: def reload(options = nil)
2334: clear_aggregation_cache
2335: clear_association_cache
2336: #attributes.update(self.class.find(self.id, options).instance_variable_get('#attributes'))
2337: #attributes_cache = {}
2338: self
2339: end
so when you call reload(:lock => lock) as the call to lock does it it really updating the attributes of that record.
There are a lot of different situations here. You could try to lock a record that dosnt exist, or lock one that has been locked elsewhere. What error are you interested in catching?
Related
I have a Rails/SQLite3 setup which is getting exceptions
ActiveRecord::StatementInvalid (SQLite3::BusyException: database is locked):
Now I understand that SQLite3 cannot do parallel writes, I'd be happy if they were blocked until the other write was finished, so done serially, rather than raising. This is something to do with default SQLite3 handler in C falling foul of Ruby's GIL, apparently. The fix, according to this post, is to install one's own "busy handler" for the SQLite3 connection, and I have this code in a Rails initializer
# config/initializers/sqlite3.rb
if ActiveRecord::Base.connection.adapter_name == 'SQLite' then
if raw_connection = ActiveRecord::Base.connection.raw_connection then
puts 'installing busy handler'
raw_connection.busy_handler do |count|
puts 'QUACK'
end
puts 'done'
else
raise RuntimeError 'no DB raw connection!'
end
end
On starting Rails (Puma in the console) I get the expected
installing busy handler
done
but on running parallel requests I get the exception with no QUACK, i.e., it seems that my handler has not been called. Is this a misunderstanding of the scope of initializers? Is there a "correct" way to get this handler installed?
(Obviously the real handler will not just say QUACK, but I find that my "real" handler has no effect so replace it by a debugging version.)
I applied "mongoid-locker" gem on my app but during concurrent request it got failed and got error "LockError: could not get lock".So can anyone help me out.
By default, with_lock does not wait for other locks to complete, so if you actually have a concurrent access, you will get the LockError raised if you don't tell it to wait.
Try it like so:
object = Object.first
object.with_lock wait:true do
object.foo = "bar"
object.save!
end
I use a Sidekiq queue to process communications with an unreliable, 3rd party API. Since this API is often down for a couple minutes at a time and then back up again, Sidekiq has been handy. When a connection issue happens, an error is raised and Sidekiq throws the job back in the queue to be retried again later, after some time has passed.
I use NewRelic to not only help debug crashes, but also for monitoring. My problem is that this current methodology above creates errors in NewRelic. If the 3rd party API is down for more than a couple of minutes, the error count accumulates enough to cause notifications to send out through NewRelic.
What I'd like to do is only raise an error from my worker when a certain number of retries have occurred for a job. I'm using sidekiq_retries_exhausted to do this. My problem is that I'm not quite sure how to put jobs back in the queue after they have an error without raising an error.
Does Sidekiq provide any facilities to return a job to a queue, increment the number of retries for the job, and have it sit there until it's due to run again, as if an exception was raised in the worker class?
You raise a specific error and tell the error service to ignore errors of that type. For NewRelic:
https://docs.newrelic.com/docs/agents/ruby-agent/installation-configuration/ruby-agent-configuration#error_collector.ignore_errors
Here is what I did to keep intentional retry errors out of AirBrake:
class TaskWorker
include Sidekiq::Worker
class RetryNotAnError < RuntimeError
end
def perform task_id
task = Task.find(task_id)
task.do_cool_stuff
if task.finished?
#log.debug "Task #{task_id} was successful."
return false
else
#log.debug "Task #{task_id} will try again later."
raise RetryNotAnError, task_id
end
end
end
Tell Airbrake to ignore it:
Airbrake.configure do |config|
config.ignore << 'RetryNotAnError'
end
It's good to make your exception name OBVIOUSLY not an error (e.g. RetryLaterNotAnError), as it will still show up in logs and such, and you don't want to freak people out when they see a bunch of them.
ps. That said, I would really like to see Sidekiq to provide an explicit, errorless retry mechanism.
If using Sidekiq Enterprise, one other option might be to utilize the optional set of additional error types that will then get treated as Sidekiq::Limiter::OverLimit violations.
For my purposes, I've used a new error class and then added it to the list in the config. Here are the notes from the sidekiq-ent code (not in the public sidekiq repo) on how to modify your config file:
# An optional set of additional error types which would be
# treated as a rate limit violation, so the job would automatically
# be rescheduled as with Sidekiq::Limiter::OverLimit.
#
# Sidekiq::Limiter.errors << MyApp::TooMuch
# Sidekiq::Limiter.errors = [Foo::Error, MyApp::Limited]
Inside the specific job you can specify the max_retries, or it will default to 20:
sidekiq_options max_limiter_retries: 10
Inside the job, I'll rescue the "expected" intermittent error that I'd rather not ignore completely and then raise the error I've added to the list, something like this:
rescue RestClient::RequestTimeout => e
raise SidekiqSoftRetry.new(e.inspect)
end
Here's what that looks like in my initialization file-- and Mike Perham was kind enough to respond with the option to update the global retry limit.
class SidekiqSoftRetry < RuntimeError
end
Sidekiq::Limiter::DEFAULT_OPTIONS[:reschedule] = 10
Sidekiq::Limiter.configure do |config|
config.errors.concat(
[
SidekiqSoftRetry,
]
)
end
I've a rails app that I'm crashing on purpose.. it's local and I'm just hitting ctrl + c and killing it mid way through processing records..
To my mind the records in the block shouldn't have been committed.. Is this a postgres "error" or a rails "error", or a dave ERROR?
ActiveRecord::Base.transaction do
UploadStage.where("id in (#{ids.join(',')})").update_all(:status => 2);
records.each do |record|
record.success = process_line(record.id, klas, record.hash_value).to_s[0..250]
record.status = 1000
record.save();
end
end
I generate my ids by reading out all the records where the status is 1.
Nothing but this function sets the status to 1000..
If the action crashes for what ever reason, I'd expect there to be no records in the database with status = 2...
This is not what I'm seeing though. Half the records have status 1000, the other half have status 2.. .
Am I missing something?
How can I make sure there are no 2's if the app crashes?
EDIT:
I found this link http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/
As I suspected and as confirmed by dave's update, it looks like ActiveRecord will commit a half-finished transaction under some circumstances when you kill a thread. Woo, safe! See dave's link for detailed explanation and mitigation options.
If you're simulating hard crash (host OS crash or plug-pull), control-C is absolutely not the right approach. Use Control-\ to send a SIGQUIT, which is generally not handled, or use kill -KILL to hard-kill the process with no opportunity to do cleanup. Control-C sends SIGINT which is a gentle signal that's usually attached to a clean shutdown handler.
In general, if you're debugging issues like this, you should enable detailed query logging and see what Rails is doing. Use log_statement = 'all' in postgresql.conf then examine the PostgreSQL logs.
I'm having an issue trying to get a timeout when connecting via TCPSocket to a remote resource that isn't available. It just hangs indefinitely without timing out. Ideally I'd want it to try reconnect every 2 minutes or so, but the TCPSocket.new call seems to block. I've tried using timeout() but that doesn't do anything either. Trying the same call in an IRB instance works perfectly fine, but when it's in Rails, it fails. Anyone have a work around for this?
My code looks something as follows:
def self.connect!
##connection = TCPSocket.new IP, 4449
end
def self.send(cmd)
puts "send "
unless ##connection
self.connect!
end
loop do
begin
##connection.puts(cmd)
return
rescue IOError
sleep(self.get_reconnect_delay)
self.connect!
end
end
end
Unfortunately, there is currently no way to set timeouts on TCPSocket directly.
See http://bugs.ruby-lang.org/issues/5101 for the feature request. You will have use the basic Socket class and set socket options.