Rails ActiveJob has a retry_on hook that allows you to customize retry behavior. For example:
retry_on AnotherCustomAppException, wait: ->(executions) { executions * 2 }
Rails also passes the current retry number as executions with the job_data and there's a retry_job method for further customization.
However, if you use the delayed_job_active_record gem as your backend, it looks like there's a separate config called max_attempts that controls the retry behavior.
My question is, if you use the delayed_job_active_record backend, can you still use retry_on without issues?
If you can't use retry_on, then what would be an appropriate strategy for imitating that customization of rescues?
When you're using delayed_job as a backend to ActiveJob, you end up with two retry mechanisms: first from ActiveJob, configurable using the retry_on method, and second from delayed_job, which is controlled by the max_attempts variable.
You can turn off the retry behaviour from delayed_job with the following:
# in config/initializers/delayed_job.rb
Delayed::Worker.max_attempts = 1
Now your retries are controlled entirely by the ActiveJob retry_on call, which should result in predictable behaviour.
Related
I need to consume SQS events with my rails application. I've written a Sidekiq job which does a long polling like this:
class SqsConsumerWorker
include Sidekiq::Worker
def perform
...
poller = Aws::SQS::QueuePoller.new(<queue_url>, client: <sqs_instance>)
poller.poll(wait_time_seconds: 20, max_number_of_messages: 10, visibility_timeout: 180) do |messages|
messages.each do |message|
puts message.inspect
end
end
end
end
First problem was when to initiate this job. Currently I've moved the invocation to rails initializer where I've overriden the Sidekiq config.on(:startup) block to call this job. This will help me to start the job on every deployment. (I have also written some logic in this initializer to check the number of workers are not above some limit etc.)
I wanted to understand is there a better way to solve this problem? I've seen the gem shoryuken which abstracts out these things. But I need more control over the consumer and thought of having my own implementation. I also need to understand how to scale up and scale down the number of consumers with this approach.
I am confused about how ActiveJob handles retries when there is an exception raised during the execution of a job. The Rails Guide about ActiveJob has this example:
10.1 Retrying or Discarding failed jobs
It's also possible to retry or discard a job if an exception is raised during execution. For example:
class RemoteServiceJob < ApplicationJob
retry_on CustomAppException # defaults to 3s wait, 5 attempts
discard_on ActiveJob::DeserializationError
def perform(*args)
# Might raise CustomAppException or ActiveJob::DeserializationError
end
end
To get more details see the API Documentation for ActiveJob::Exceptions.
That means there is a method to explicitly tell ActiveJob to retry a job on certain exceptions and at the same time, there is a method to explicitly tell ActiveJob to discard a job on certain exceptions.
But how does ActiveJob handle exceptions when the developer didn't define retry_on or discard_on? What is the default behavior? Would it discard the job? Would it retry the job? And if, how often and in what interval?
My Rails application is using ActiveJob + DelayedJob to execute some background jobs.
I am trying to figure out what is the way to define what happens on failure (not on error) - meaning, if DelayedJob has marked the job as failed, after the allowed 3 attempts, I want to perform some operation.
This is what I know so far:
DelayedJob has the aptly named failure hook.
This hook is not supported in ActiveJob
ActiveJob has a rescue_from method
The rescue_from method is probably not the right solution, since I do not want to do something on each exception, but rather only after 3 attempts (read: only after DelayedJob has deemed the job as failed).
ActiveJob has an after_perform hook, which I cannot utilize since (as far as I can see) it is not called when perform fails.
Any help is appreciated.
You may already find the solution to this, but for people who still struggle on this issue, you can use ActiveJob rety_on method with a block to run custom logic when maximum attempts have reached but still failed.
class RemoteServiceJob < ApplicationJob
retry_on(CustomAppException) do |job, error|
ExceptionNotifier.caught(error)
end
def perform(*args)
# Might raise CustomAppException
end
end
You can find more info about Exception handling in ActiveJob in https://api.rubyonrails.org/v6.0.3.2/classes/ActiveJob/Exceptions/ClassMethods.html
I want to run an infinite loop on a separate thread that starts as soon as the app initializes (in an initializer). Here's what it might look like:
# in config/initializers/item_loop.rb
Thread.new
loop do
Item.find_each do |item|
# Get price from third-party api and update record.
item.update_price!
# Need to wait a little between requests to avoid getting throttled.
sleep 5
end
end
end
I tend to accomplish this by running batch updates in recurring background jobs. But this doesn't make sense since I don't really need parallelization, downtime, or queueing, I just want to update one item at a time in a single thread, forever.
Yet there are multiple things that concern me:
Leaked Connections: Should I open up a new connection_pool for the thread? Should I use a gem like safely to avoid crashing the thread?
Thread Safety: Should I be worried about race conditions? Should I make use of Mutex and synchronize? Does using ActiveRecord::Base.transaction impact thread safety?
Deadlock: Should I use Rails.application.executor.wrap?
Concurrent Ruby/Sleep Intervals: Should I use TimerTask from concurrent-ruby gem instead of sleep or something other than Thread.new?
Information on any of these subjects is appreciated.
Usually to perform a job in a background process(non web-server process) a background workers manager is used. Rails has a specific interface for that manager called ActiveJob There are few implementation of a background workers manager - Sidekiq, DelayedJob, Resque, etc. Sidekiq is preferred. Returning back to actual problem - you may create a schedule to run UpdatePriceJob every interval using gem sidekiq-scheduler Another nice extension for throttling Sidekiq workers is sidekiq-throttler
Some code snippets:
# app/workers/update_price_worker.rb
# Actual Worker class
class UpdatePriceWorker
include Sidekiq::Worker
sidekiq_options throttle: { threshold: 720, period: 1.hour }
def perform(item_id)
Item.find(item_id).update_price!
end
end
# app/workers/update_price_master_worker.rb
# Master worker that loops over items
class UpdatePriceMasterWorker
include Sidekiq::Worker
def perform
Item.find_each { |item| UpdatePriceWorker.perform_async item.id }
end
end
# config/sidekiq.yml
:schedule:
update_price:
cron: '0 */4 * * *' # Runs once per 4 hours - depends on how many Items are there
class: UpdatePriceMasterWorker
Idea of this setup - we run MasterWorker every 4 hours(this depends on how much time it takes to update all items). Master worker creates jobs to update price of an every particular item. UpdatePriceWorker is throttled to max 720 RPH.
I use rails runner x (god gem or k8s) in our similar case.
Rails runner runs in another process so that we do not have to worry about connection-leak and thread-safety.
God-gem or k8s supports concurrency and monitoring the job failure. Running 1 process with some specific sleep-time would promise third-party API throttles (running N process with N API-key could support speed up).
I think deadlock would happen in any concurrency situation.
I do not think this loop + sleep approach is a design flaw, because:
cron always starts based on schedule so that long running jobs could run simultaneously. We need to add a logic to avoid job overlapping. Rather, just loop + sleep keeps maximum throughput without any job overlap.
ActiveJob is good for one-shot long-running task, but it does not fit for daemon.
TLDR; How can I test that a PORO argument for an asynchronous ActionMailer action (using Sidekiq) serializes and deserializes correctly?
Sidekiq provides RSpec matchers for testing that a job is enqueued and performing a job (with given arguments).
--
To give you some context, I have a Ruby on Rails 4 application with an ActionMailer. Within the ActionMailer is a method that takes in a PORO as an argument - with references to data I need in the email. I use Sidekiq to handle the background jobs. It turns out that there was an issue in deserializing the argument that it would fail when Sidekiq decided to perform the job. I haven't been able to find a way to test the correctness of the un/marshaling such that the PORO I called the action with is being used when performed.
For example:
Given an ActionMailer with an action
class ApplicationMailer < ActionMailer::Base
def send_alert profile
#profile = profile
mail(to: profile.email)
end
end
...
I would use it like this
profile = ProfileDetailsService.new(user)
ApplicationMailer.send_alert(profile).deliver_later
...
And have a test like this (but this fails)
let(:profile) { ProfileDetailsService.new(user) }
it 'request an email to be sent' do
expect {
ApplicationMailer.send_alert(profile).deliver_later
}.to have_enqueued_job.on_queue('mailers')
end
Any assistance would be much appreciated.
You can test it in a synchronous way (using only IRB and without the need of start the Sidekiq workers).
Let's say your worker class has the following implementation:
class Worker
include Sidekiq::Worker
def perform(poro_obj)
puts poro_obj.inspect
# ...
end
end
You can open IRB (bundle exec irb) and type the following commands:
require 'sidekiq'
require 'worker'
Worker.new.perform
Thus, you will execute the code in a synchronous way AND using Sidekiq (note that we're invoking the perform method instead of the perform_async).
I think this is the best way to debug your code.