I have jobs of a particular type that I'd like to have retry more frequently than set by the default Sidekiq interval. Is this currently possible? Ideally the job would retry every 5 seconds for up to a minute. Not entirely sure this is currently something that's trivial to plugin to a Sidekiq job.
According to: https://github.com/mperham/sidekiq/wiki/Error-Handling you can do this:
class Worker
include Sidekiq::Worker
sidekiq_retry_in do |count|
5
end
end
I'm answering if calling 10.minutes in the block works, because I can't comment on an answer.
According to the Sidekiq code, you need to pass an Integer, or either :kill or :discard symbols.
10.minutes returns an instance of ActiveSupport::Duration
This means that the following would work:
class Worker
include Sidekiq::Worker
sidekiq_retry_in { 10.minutes.to_i }
end
Source
If you need other range, can use minute syntax
sidekiq_retry_in do |_count|
10.minutes
end
Related
I have been trying to see if this is possible and so far have found nothing so I will try and ask specifically
Is it possible to have a sidekiq worker which can recive a method as for example a lambda method and pass on arguments to it?
Example case:
I need to make some heavy computation on my server and my options are to either make a specific sidekiq worker for the job which will only be done 1 time ever and will end up cloddering my code base, or make a worker which could lets say accept something like:
lot_of_work.each do |args|
Workers::Tmp::LetsGo.perform_async(args) { |a| a.lets_go }
end
I've tried looking through old stackoverflow posts and documentation for sidekiq.
I've tried the above method which I hoped worked as a normal method but it does not.
I would have liked it to execute the method which was pass to the worker such that I do not need to make workers for 1 time cases and dont have to use single thread computation.
I found a solution to this problem, there are probably better ones but this worked for me.
Make a worker like this:
module Workers
module Default
class TesterWorker
include Sidekiq::Worker
sidekiq_options queue: :default, retry: false
def perform(method_name, method)
eval(method)
send(method_name)
end
end
end
end
After this you simply just have to write your code as a string like this:
methode_name = 'tester'
spec = "def test; puts 1; end"
Workers::Default::TesterWorker.perform_async(methode_name, spec)
And this will execute the for example the puts 1 action on the sidekiq ^^
I encountered an issue with sidekiq: I want to set timeout for jobs, meaning when a job has process time greater than timeout then that job will stop.
I have searched how to set global timeout config in file sidekiq.yml. But I want to set separate timeout for difference separate jobs meaning one of classes to define worker will have particular timeout config.
Can you help me. Thanks so much.
There's no approved way to do this. You cannot stop a thread safely while it is executing. You need to change your job to check periodically if it should stop.
You can set network timeouts on any 3rd party calls you are making so that they time out.
You can wrap your job code inside a timeout block like the below:
Timeout::timeout(2.hours) do
***. do possibly long-running task *****
end
The job will fail automatically after 2 hours.
This is the same method as yassen suggested, but more concrete.
class MyCustomWorker
include Sidekiq::Worker
def perform
begin
Timeout::timeout(30.minutes) do # set timeout to 30 minutes
perform_job()
end
rescue Timeout::Error
Rails.logger.error "timeout reached for worker"
end
end
def perform_job
# worker logic here
end
end
I have some methods that works with API of third party app. To do it on button click is no problem, but it should be permanent process.
How to run them background? And how to pause the cycle for make some other works with same API and resume the cycle after the job is done.
Now I read about ActiveJob, but its has time dependences only...
UPDATE
I've tried to make it with whenever and sidekiq, task runs, but it do nothing. Where to look for logs I can't understand.
**schedule.rb**
every 1.minute do
runner "UpdateWorker.perform_async"
end
**update_worker.rb**
class UpdateWorker
include Sidekiq::Worker
include CommonMods
def perform
logger.info "Things are happening."
logger.debug "Here's some info: #{hash.inspect}"
myMethod
end
def myMethod
....
....
....
end
end
It's not exactly what I need, but better then nothing. Can somebody explain me with examples?
UPDATE 2 After manipulating with code it's absolutely necessary to restart sidekiq . With this problem is solved, but I'm not sure that this is the best way.
You can define a job which enqueues itself:
class MyJob < ActiveJob::Base
def perform(*args)
# Do something unless some flag is raised
ensure
self.class.set(wait: 1.hour).perform_later(*args)
end
end
There are several libraries to schedule jobs on a regular basis. For example you could use to sidekiq-cron to run a job every minute.
If you want to pause it for some time, you could set a flag somewhere (Redis/database/file) and skip execution as long it is detected.
On a somewhat related note: don't use sidetiq. It was really great but it's not maintained anymore and has incompatibilities to current Sidekiq versions.
Just enqueue next execution in ensure section after job completes after checking some flag that indicates that it should.
Also i recommend adding some delay there so that you don't end up with dead loop on some error inside job
I dont know ActiveJobs, but I can recommend the whenever gem to create cron (periodic background) jobs. Basically you end up writing a rake tasks. Like this:
desc 'send digest email'
task send_digest_email: :environment do
# ... set options if any
UserMailer.digest_email_update(options).deliver!
end
I never added a rake task to itself but for repeated processing you could do somehow like this (from answers to this specific question)
Rake::Task["send_digest_email"].execute
I have a rails app in which I would like to use the delayed_jobs gem to send texts/emails in background processes at certain times of the day. This is how I have the relevant parts of my app set up right now:
class SomeClass
after_create :send_reminder
def when_to_run
self.date_time - 1.hour
end
def send_reminder
MessageHandler.new().send_message
end
handle_asynchronously :send_reminder, run_at: Proc.new { |i| i.when_to_run }, queue: "Messages"
end
The MessageHandler class is a separate class I've defined which actually houses the methods for sending texts (with Twilio) and emails (with Mailgun).
After starting delayed_job (bin/delayed_job start) and creating an instance of SomeClass, the delayed_job log reads as follows:
Job SomeClass#send_reminder_without_delay (id=686) RUNNING
I'm not sure why it is running send_reminder_without_delay, and it's doing it every time. I've tried using MessageHandler.new().delay.send_message in the send_reminder method instead of MessageHandler.new().send_message, but that hasn't gotten me anywhere either.
I've searched high and low for answers and keep coming up short - any help would be much appreciated!
I think your problem is in run_at: Proc.new { |i| i.when_to_run }.
I'm not entirely sure what the date_time method does but have you checked whether it actually returns time that is more than 1 hour in the future?
Otherwise the run_At will receive a time in the past and run the method.
if the return value of date_time is similar to this:
def when_to_run
Time.now - 1.hour
end
Then the queue will run as soon as you start delayed job.
Check if it your run_at time is in the future.
We were using Rails 4.2 and set the queue adapter. We needed to use this line in our config:
config.active_job.queue_adapter = :delayed_job
I have a sidekiq worker that shouldn't take more than 30 seconds, but after a few days I'll find that the entire worker queue stops executing because all of the workers are locked up.
Here is my worker:
class MyWorker
include Sidekiq::Worker
include Sidekiq::Status::Worker
sidekiq_options queue: :my_queue, retry: 5, timeout: 4.minutes
sidekiq_retry_in do |count|
5
end
sidekiq_retries_exhausted do |msg|
store({message: "Gave up."})
end
def perform(id)
begin
Timeout::timeout(3.minutes) do
got_lock = with_semaphore("lock_#{id}") do
# DO WORK
end
end
rescue ActiveRecord::RecordNotFound => e
# Handle
rescue Timeout::Error => e
# Handle
raise e
end
end
def with_semaphore(name, &block)
Semaphore.get(name, {stale_client_timeout: 1.minute}).lock(1, &block)
end
end
And the semaphore class we use. (redis-semaphore gem)
class Semaphore
def self.get(name, options = {})
Redis::Semaphore.new(name.to_sym,
:redis => Application.redis,
stale_client_timeout: options[:stale_client_timeout] || 1.hour,
)
end
end
Basically I'll stop the worker and it will state done: 10000 seconds, which the worker should NEVER be running for.
Anyone have any ideas on how to fix this or what is causing it? The workers are running on EngineYard.
Edit: One additional comment. The # DO WORK has a chance to fire off a PostgresSQL function. I have noticed in logs some mention of PG::TRDeadlockDetected: ERROR: deadlock detected. Would this cause the worker to never complete even with a timeout set?
Given you want to ensure unique job execution, i would attempt removing all locks and delegate job uniqueness control to a plugin like Sidekiq Unique Jobs
In this case, even if sidetiq enqueue the same job id twice, this plugin ensures it will be enqueued/processed a single time.
You might also try the ActiveRecord with_lock mechanism: http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html
I have had a similar problem before. To solve this problem, you should stop using Timeout.
As explained in this article, you should never use Timeout in a Sidekiq job. If you use Timeout, Sidekiq processes and threads can easily break.
Not only Ruby, but also Java has a similar problem. Stopping a thread from the outside is inherently dangerous, regardless of the language.
If you continue to have the same problem after deleting Timeout, check that if you are using threads carelessly in your code.
As Sidekiq's architecture is so sophisticated, in almost all cases, the source of the bug is outside of Sidekiq.