I have a Rufus "every job" that runs periodically, but sometimes it may fail to perform it's task.
On failure, I would like to reschedule a retry soon rather than wait until the next cycle.
class PollProducts
def initialize()
end
def call(job)
puts "Updating products"
begin
# Do something that might fail
raise if rand(3) == 1
rescue Exception => e
puts "Request failed - recheduling: #{e}"
# job.in("5s") <-- What can I do?
end
end
end
scheduler.every '6h', PollProducts.new, :first_in => '5s', :blocking => true
Is this possible?
Ok this worked for me:
job.scheduler.in '5s', self
Related
I have written retry logic for sending mails and trying to cover test cases for it.
def send_mail(mail,log_obj)
begin
attempts ||= 1
mail.deliver_now
rescue Errno::ECONNREFUSED, Errno::ECONNRESET, Net::OpenTimeout, Net::ReadTimeout => e
CommonLogger.input_log(log_obj,"Timeout: #{e} (attempt ##{ attempts })")
if (attempts += 1) <= 4
sleep(1)
Rails.logger.info ("Retrying...")
retry
else
Rails.logger.error("Retry attempts exceeded.")
raise e
end
end
end
You can use and_raise to raise errors
https://relishapp.com/rspec/rspec-mocks/docs/configuring-responses/raising-an-error
allow(mail).to receive(:deliver_now).and_raise(Net::OpenTimeout)
allow_any_instance_of(MailClass).to receive(:deliver_now).and_raise(Net::OpenTimeout)
I'm using Sidekiq for delayed jobs with sidekiq-status and sidekiq-ent gems. I've created a worker which is reponsible to update minor status to false when user is adult and has minor: true. This worker should be fired every day at midnight ET. Like below:
#initializers/sidekiq.rb
config.periodic do |mgr|
# every day between midnight 0 5 * * *
mgr.register("0 5 * * *", MinorWorker)
end
#app/workers/minor_worker.rb
class MinorWorker
include Sidekiq::Worker
def perform
User.adults.where(minor: true).remove_minor_status
rescue => e
Rails.logger.error("Unable to update minor field. Exception: #{e.message} : #{e.backtrace.join('\n')}")
end
end
#models/user.rb
class User < ApplicationRecord
scope :adults, -> { where('date_of_birth <= ?', 18.years.ago) }
def self.remove_minor_status
update(minor: false)
end
end
No I want to check this on my local machine - to do so I'm using gem 'timecop' to timetravel:
#application.rb
config.time_zone = 'Eastern Time (US & Canada)'
#config/environments/development.rb
config.after_initialize do
t = Time.local(2021, 12, 21, 23, 59, 0)
Timecop.travel(t)
end
After firing up sidekiq by bundle exec sidekiq and bundle exec rails s I'm waiting a minute and I see that worker shows up:
2021-12-21T22:59:00.130Z 25711 TID-ovvzr9828 INFO: Managing 3 periodic jobs
2021-12-21T23:00:00.009Z 25711 TID-ovw69k4ao INFO: Enqueued periodic job SettlementWorker with JID ddab15264f81e0b417e7dd83 for 2021-12-22 00:00:00 +0100
2021-12-21T23:00:00.011Z 25711 TID-ovw69k4ao INFO: Enqueued periodic job MinorWorker with JID 0bcd6b76d6ee4ff9e7850b35 for 2021-12-22 00:00:00 +0100
But it didn't do anything, the user's minor status is still set to minor: true:
2.4.5 :002 > User.last.date_of_birth
=> Mon, 22 Dec 2003
2.4.5 :001 > User.last.minor
=> true
Did I miss something?
EDIT
I have to add that when I'm trying to call this worker on rails c everything works well. I've got even a RSpec test which also passes:
RSpec.describe MinorWorker, type: :worker do
subject(:perform) { described_class.new.perform }
context 'when User has minor status' do
let(:user1) { create(:user, minor: true) }
it 'removes minor status' do
expect { perform }.to change { user1.reload.minor }.from(true).to(false)
end
context 'when user is adult' do
let(:registrant2) { create(:registrant) }
it 'not change minor status' do
expect(registrant2.reload.minor).to eq(false)
end
end
end
end
Since this is the class method update won't work
def self.remove_minor_status
update(minor: false)
end
Make use of #update_all
def self.remove_minor_status
update_all(minor: false)
end
Also, I think it's best practice to have some test cases to ensure the working of the methods.
As of now you can try this method from rails console and verify if they actually work
test "update minor status" do
user = User.create(date_of_birth: 19.years.ago, minor: true)
User.adults.where(minor: true).remove_minor_status
assert_equal user.reload.minor, false
end
I think you need to either do update_all or update each record by itself, like this:
User.adults.where(minor: true).update_all(minor: false)
or
class MinorWorker
include Sidekiq::Worker
def perform
users = User.adults.where(minor: true)
users.each { |user| user.remove_minor_status }
rescue => e
Rails.logger.error("Unable to update minor field. Exception: #{e.message} : #{e.backtrace.join('\n')}")
end
end
You may also want to consider changing update to update! so it throws an error if failing to be caught by your rescue in the job:
def self.remove_minor_status
update!(minor: false)
end
I want to get exception error message out of the sidekiq job. when I set back_trace option to true it retries my job but I want to exit from job when error raises and get error message.
if I find that process ended successful or fail is enough.
def perform(text)
begin
fail StandardError, 'Error!'
rescue
fail 'EEE' # I want to get this error when call job
end
end
# call
NormalJob.perform_async('test')
# I want to get error here after call
If I were you I would try gem sidekiq-status. It has several options, which can be helpful in such situations:
You can retrieve status of your worker:
job_id = MyJob.perform_async(*args)
# :queued, :working, :complete or :failed , nil after expiry (30 minutes)
status = Sidekiq::Status::status(job_id)
Sidekiq::Status::queued? job_id
Sidekiq::Status::working? job_id
Sidekiq::Status::complete? job_id
Sidekiq::Status::failed? job_id
Also you have options for Tracking progress, saving and retrieveing data associated with job
class MyJob
include Sidekiq::Worker
include Sidekiq::Status::Worker # Important!
def perform(*args)
# your code goes here
# the common idiom to track progress of your task
total 100 # by default
at 5, "Almost done"
# a way to associate data with your job
store vino: 'veritas'
# a way of retrieving said data
# remember that retrieved data is always is String|nil
vino = retrieve :vino
end
end
job_id = MyJob.perform_async(*args)
data = Sidekiq::Status::get_all job_id
data # => {status: 'complete', update_time: 1360006573, vino: 'veritas'}
Sidekiq::Status::get job_id, :vino #=> 'veritas'
Sidekiq::Status::at job_id #=> 5
Sidekiq::Status::total job_id #=> 100
Sidekiq::Status::message job_id #=> "Almost done"
Sidekiq::Status::pct_complete job_id #=> 5
Another option is to use sidekiq batches status
This is what batches allow you to do!
batch = Sidekiq::Batch.new
batch.description = "Batch description (this is optional)"
batch.notify(:email, :to => 'me#example.org')
batch.jobs do
rows.each { |row| RowWorker.perform_async(row) }
end
puts "Just started Batch #{batch.bid}"
b = Sidekiq::Batch.new(bid) # bid is a method on Sidekiq::Worker that gives access to the Batch ID associated to the job.
b.jobs do
SomeWorker.perform_async(1)
sleep 1
# Uh oh, Sidekiq has finished all outstanding batch jobs
# and fires the complete message!
SomeWorker.perform_async(2)
end
status = Sidekiq::Batch::Status.new(bid)
status.total # jobs in the batch => 98
status.failures # failed jobs so far => 5
status.pending # jobs which have not succeeded yet => 17
status.created_at # => 2012-09-04 21:15:05 -0700
status.complete? # if all jobs have executed at least once => false
status.join # blocks until the batch is considered complete, note that some jobs might have failed
status.failure_info # an array of failed jobs
status.data # a hash of data about the batch which can easily be converted to JSON for javascript usage
It can be used out of the box
We have an application that makes hundreds of API calls to external services. Sometimes some calls take too much time to respond.
I am using the rake_timeout gem to find time consuming process, so, Timeout::Error will be thrown whenever some request is taking too long to respond. I am rescuing this error and doing a retry on that method:
def new
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
#Below code will be repeated in every method
tries = 0
rescue Timeout::Error => e
tries += 1
retry if tries <= 3
logger.error e.message
end
This lets the method fully re-run it. This is very verbose and I am repeating it every time.
Is there any way to do this so that, if the Timeout:Error occurrs, it will retry that method automatically three times?
I have a little module for that:
# in lib/retryable.rb
module Retryable
# Options:
# * :tries - Number of tries to perform. Defaults to 1. If you want to retry once you must set tries to 2.
# * :on - The Exception on which a retry will be performed. Defaults to Exception, which retries on any Exception.
# * :log - The log level to log the exception. Defaults to nil.
#
# If you work with something like ActiveRecord#find_or_create_by_foo, remember to put that call in a uncached { } block. That
# forces subsequent finds to hit the database again.
#
# Example
# =======
# retryable(:tries => 2, :on => OpenURI::HTTPError) do
# # your code here
# end
#
def retryable(options = {}, &block)
opts = { :tries => 1, :on => Exception }.merge(options)
retry_exception, retries = opts[:on], opts[:tries]
begin
return yield
rescue retry_exception => e
logger.send(opts[:log], e.message) if opts[:log]
retry if (retries -= 1) > 0
end
yield
end
end
and than in your model:
extend Retryable
def new
retryable(:tries => 3, :on => Timeout::Error, :log =>:error) do
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
end
...
end
You could do something like this:
module Foo
def self.retryable(options = {})
retry_times = options[:times] || 10
try_exception = options[:on] || Exception
yield if block_given?
rescue *try_exception => e
retry if (retry_times -= 1) > 0
raise e
end
end
Foo.retryable(on: Timeout::Error, times: 5) do
# your code here
end
You can even pass multiple exceptions to "catch":
Foo.retryable(on: [Timeout::Error, StandardError]) do
# your code here
end
I think what you need is the retryable gem.
With the gem, you can write your method like below
def new
retryable :on => Timeout::Error, :times => 3 do
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
end
end
Please read the documentation for more information on how to use the gem and the other options it provides
you could just write a helper-method for that:
class TimeoutHelper
def call_and_retry(tries=3)
yield
rescue Timeout::Error => e
tries -= 1
retry if tries > 0
Rails.logger.error e.message
end
end
(completely untested) and call it via
TimeoutHelper.call_and_retry { [your code] }
I'm using Timeout::timeout(1) for a process that takes longer than 1 second, though it only occasionally triggers a timeout. When it does, rescue captures it in different ways each time. Here's a sample of my code:
require 'timeout'
...
begin
status = Timeout::timeout(1) {
open(file_url) do |foo|
feed = RSS::Parser.parse(foo)
some_method_call(arg1, arg2)
#other stuff
end
}
rescue Timeout::Error
Rails.logger.debug "Timeout"
return nil
rescue Exception => ex
Rails.logger.debug "EXCEPTION - #{ex.message}"
return nil
end
Here are the three scenarios I encounter with the same input:
Process runs to completion and takes longer than 60 seconds
Process times out and hangs, printing only execution expired in development.log
Process times out, is rescued properly, prints "Timeout" in development.log, and returns nil
Why is this so inconsistent?
UPDATE
After reducing the timeout to 0.0001s, the process is timing out consistently and as expected. It seems that the open(file_url) block was opening faster than 1 second, and despite everything within the block taking more than 1 second, the Timeout was only triggered if the opening itself took longer than 1 second.
This however did not explain the execution expired exception. To test this, I moved the Timeout::timeout(0.0001) to within the open block. The code looks like the following:
require 'timeout'
...
begin
open(file_url) do |foo|
status = Timeout::timeout(0.0001) do
begin
feed = RSS::Parser.parse(foo)
some_method_call(arg1, arg2)
#other stuff
rescue Timeout::Error
Rails.logger.debug "Timeout 2"
rescue Exception => ex
Rails.logger.debug "EXCEPTION 2 - #{ex.message}"
end
end
end
rescue Timeout::Error
Rails.logger.debug "Timeout"
return nil
rescue Exception => ex
Rails.logger.debug "EXCEPTION - #{ex.message}"
return nil
end
Now, I'm consistently receiving the output EXCEPTION 2 - execution expired. Why is it that the Timeout::Error is not being triggered here?
Your inner
rescue Exception => ex
Rails.logger.debug "EXCEPTION 2 - #{ex.message}"
end
keeps the outer timeoutblock from raising Timeout::Error.
Removing that rescuestatement should do the trick.
If you really need to catch any exception, replace it with:
rescue StandardError => ex
Rails.logger.debug "EXCEPTION 2 - #{ex.message}"
end
Internally (within the Timeout block) Timeout does not use Timeout::Error. If it did, then every garden-variety rescue would catch it, and you don't want that. So it creates a new Exception and uses that, so that it hopefully blows through all normal error handling and actually makes the code stop running.
Check out the timeout.rb code in ruby200/lib/ruby/2.0.0. It's quite short, and pretty informative.
In particular, you can pass your own Exception in as the second parameter of Timeout::timeout, and Timeout will use that. So you can catch it inside your code, if you want.
Note that Logger currently traps all exceptions that happen while writing, and doesn't re-raise, so it breaks Timeout. I've filed a bug report.