My Rails application is using ActiveJob + DelayedJob to execute some background jobs.
I am trying to figure out what is the way to define what happens on failure (not on error) - meaning, if DelayedJob has marked the job as failed, after the allowed 3 attempts, I want to perform some operation.
This is what I know so far:
DelayedJob has the aptly named failure hook.
This hook is not supported in ActiveJob
ActiveJob has a rescue_from method
The rescue_from method is probably not the right solution, since I do not want to do something on each exception, but rather only after 3 attempts (read: only after DelayedJob has deemed the job as failed).
ActiveJob has an after_perform hook, which I cannot utilize since (as far as I can see) it is not called when perform fails.
Any help is appreciated.
You may already find the solution to this, but for people who still struggle on this issue, you can use ActiveJob rety_on method with a block to run custom logic when maximum attempts have reached but still failed.
class RemoteServiceJob < ApplicationJob
retry_on(CustomAppException) do |job, error|
ExceptionNotifier.caught(error)
end
def perform(*args)
# Might raise CustomAppException
end
end
You can find more info about Exception handling in ActiveJob in https://api.rubyonrails.org/v6.0.3.2/classes/ActiveJob/Exceptions/ClassMethods.html
Related
Rails ActiveJob has a retry_on hook that allows you to customize retry behavior. For example:
retry_on AnotherCustomAppException, wait: ->(executions) { executions * 2 }
Rails also passes the current retry number as executions with the job_data and there's a retry_job method for further customization.
However, if you use the delayed_job_active_record gem as your backend, it looks like there's a separate config called max_attempts that controls the retry behavior.
My question is, if you use the delayed_job_active_record backend, can you still use retry_on without issues?
If you can't use retry_on, then what would be an appropriate strategy for imitating that customization of rescues?
When you're using delayed_job as a backend to ActiveJob, you end up with two retry mechanisms: first from ActiveJob, configurable using the retry_on method, and second from delayed_job, which is controlled by the max_attempts variable.
You can turn off the retry behaviour from delayed_job with the following:
# in config/initializers/delayed_job.rb
Delayed::Worker.max_attempts = 1
Now your retries are controlled entirely by the ActiveJob retry_on call, which should result in predictable behaviour.
I need for testing purpose to reproduce an ActiveJob::DeserializationError on rails console, I need to test the behaviour manually on testing environment.
class WrapperJob < ApplicationJob
discard_on ActiveJob::DeserializationError
def perform(event)
# doing something
end
end
The WrapperJob aims to perform something on the event, but during the enqueue time it might happen to remove event from database. I need a way please to reproduce ActiveJob::DeserializationError. right now I tried to pass a missing event, or a missing GlobalID but actually fail on the # doing something part instead of raising DeserializationError.
Thanks.
I am confused about how ActiveJob handles retries when there is an exception raised during the execution of a job. The Rails Guide about ActiveJob has this example:
10.1 Retrying or Discarding failed jobs
It's also possible to retry or discard a job if an exception is raised during execution. For example:
class RemoteServiceJob < ApplicationJob
retry_on CustomAppException # defaults to 3s wait, 5 attempts
discard_on ActiveJob::DeserializationError
def perform(*args)
# Might raise CustomAppException or ActiveJob::DeserializationError
end
end
To get more details see the API Documentation for ActiveJob::Exceptions.
That means there is a method to explicitly tell ActiveJob to retry a job on certain exceptions and at the same time, there is a method to explicitly tell ActiveJob to discard a job on certain exceptions.
But how does ActiveJob handle exceptions when the developer didn't define retry_on or discard_on? What is the default behavior? Would it discard the job? Would it retry the job? And if, how often and in what interval?
I'm using sidekiq to process thousands of jobs per hour - all of which ping an external API (Google). One out of X thousand requests will return an unexpected (or empty) result. As far as I can tell, this is unavoidable when dealing with an external API.
Currently, when I encounter such response, I raise an Exception so that the retry logic will automatically take care of it on the next try. Something is only really wrong with the same job fails over and over many times. Exceptions are handled by Airbrake.
However my airbrake gets clogged up with these mini-outages that aren't really 'issues'. I'd like Airbrake to only be notified of these issues if the same job has failed X times already.
Is it possible to either
disable the automated airbrake integration so that I can use the sidekiq_retries_exhausted to report the error manually via Airbrake.notify
Rescue the error somehow so it doesn't notify Airbrake but keep retrying it?
Do this in a different way that I'm not thinking of?
Here's my code outline
class GoogleApiWorker
include Sidekiq::Worker
sidekiq_options queue: :critical, backtrace: 5
def perform
# Do stuff interacting with the google API
rescue Exception => e
if is_a_mini_google_outage? e
# How do i make it so this harmless error DOES NOT get reported to Airbrake but still gets retried?
raise e
end
end
def is_a_mini_google_outage? e
# check to see if this is a harmless outage
end
end
As far as I know Sidekiq has a class for retries and jobs, you can get your current job through arguments (comparing - cannot he effective) or jid (in this case you'd need to record the jid somewhere), check the number of retries and then notify or not Airbrake.
https://github.com/mperham/sidekiq/wiki/API
https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/api.rb
(I just don't give more info because I'm not able to)
if you look for Sidekiq solution https://blog.eq8.eu/til/retry-active-job-sidekiq-when-exception.html
if you are more interested in configuring Airbrake so you don't get these errors untill certain retry check Airbrake::Sidekiq::RetryableJobsFilter
https://github.com/airbrake/airbrake#airbrakesidekiqretryablejobsfilter
I have an app with both sidekiq and delayed job gems installed. When I trigger handle_asynchronously in active record models it appear to be handled by sidekiq while I would like to trigger delayed_job.
Is there a way to desactivate sidekiq for a specific model?
UPDATE:
Sidekiq now provides ways to either disable its delay module completely or alias it as sidekiq_delay. Please check this to see how to do it. https://github.com/mperham/sidekiq/wiki/Delayed-Extensions#disabling-extensions
For older version of sidekiq:
I use this monkey patch to make it so that calling .sidekiq_delay() goes to sidekiq and .delay() is goes to DelayedJob. According the answer by Viren, I think this may also solve your problem.
The patch is less complex (just a bunch of aliases), and gives you the power to consciously decide which delay you are actually calling.
As I mention in the comment In order to get it working you have to redefine/basically monkey patch the handle_asynchronously method something like this
Anywhere you like (but make sure it loaded )
in your config/initializers/patch.rb the code look like this
module Patch
def handle_asynchronously(method, opts = {})
aliased_method, punctuation = method.to_s.sub(/([?!=])$/, ''), $1
with_method, without_method = "#{aliased_method}_with_delay#{punctuation}", "#{aliased_method}_without_delay#{punctuation}"
define_method(with_method) do |*args|
curr_opts = opts.clone
curr_opts.each_key do |key|
if (val = curr_opts[key]).is_a?(Proc)
curr_opts[key] = if val.arity == 1
val.call(self)
else
val.call
end
end
end
## Replace this with other syntax
# delay(curr_opts).__send__(without_method, *args)
__delay__(curr_opts).__send__(without_method, *args)
end
alias_method_chain method, :delay
end
end
Module.send(:include,Patch)
And I believe rest all will follow then they way it should :)
Reason:
Delayed::Job include delay method on Object and Sidekiq include it delay method over ActiveRecord
Hence when the class try to invoke delay it look up it ancestors class (including the Eigen Class)
and it find the method define or included in ActiveRecord::Base class (which is sidekiq delay)
why does __delay__ work because alias define the copy of the existing method which is delay method of DelayedJob , hence when you invoke the __delay__ method it invoke delay method define DelayedJob
include to Object
Note:
Although the solution is bit patch but the it works . Keeping in mind that every direct .delay methid invocation is invoking delay method of the SideKiq and not DelayedJob to invoke the DelayedJob delay method you always has call it this way __delay__
Suggestion :
Monkey Patching is just a bad practice on my personal note I would rather not use 2 entirely different background processing library for a single application to achieve the same task. If the task is process thing in background why cant it be done with a single library either delayed_job or sidekiq (why it is that you required both of them )
So the point and to simply thing make your background processing an ease with respect to future I sincerely advice you take any one of the two library for background processing and I feel that would the valid answer for your question instead of monkey patching an doing other crazy stuff
Hope this help