I have a SchedulerJob:
class SchedulerJob < ActiveJob::Base
queue_as :scheduler
def perform
logger.debug("Start")
DelayedJob.set(wait: 10.seconds).perform_later
logger.debug("After Job 1")
DelayedJob.set(wait: 20.seconds).perform_later
logger.debug("After Job 2")
DelayedJob.set(wait: 30.seconds).perform_later
logger.debug("End")
end
end
and a DelayedJob:
class DelayedJob < ActiveJob::Base
queue_as :delayed_jobs
def perform
puts "I'm done"
end
end
If I call SchedulerJob.new.perform the job runs in just a few milliseconds. If I call SchedulerJob.perform_later to run the job in Sidekiq it takes about 90 seconds to finish, and by looking at the logs I can tell that each of those .perform_later calls takes about 30 seconds each.
Why would this happen?
By definition,
perform_later will be performed as soon as queuing system is free
perform will be performed regardless of your queue status
My guess is that when you call perform_later, there are some jobs running in the queue.
The problem was the initializer. This was the solution:
My guess is you’ve destroyed all concurrency by hacking in one global $REDIS connection into the pool. Don’t do that. Let Sidekiq create and manage the pool of connections.
Related
I wish I could decide on which queue to go to work.
This is because if the job is scheduled by the server (cronjob) it must be run on a slow queue, if instead it is run by the user it will go on a fast queue.
How can I run this in Resque?
Controller
MyJob.perform_later(id, :fast)
Rake task
MyJob.perform_later(id, :slow)
Job
class MyJob < ApplicationJob
queue_as :default #<-- This has to be dynamic
def perform(item_id, queue_name)
....
end
I see you are using ActiveJob, you can set the queue by using set method:
Controller
MyJob.set(queue: :fast).perform_later(id)
Rake task
MyJob.set(queue: :slow).perform_later(id)
set method allows you to set more thing than just queue, you can also set eg priority or when should job be performed. See the documentation https://api.rubyonrails.org/v5.2.3/classes/ActiveJob/Core/ClassMethods.html#method-i-set
Note: I presume, you already have slow and fast Resque queues in place and running and only want to use them
I encountered an issue with sidekiq: I want to set timeout for jobs, meaning when a job has process time greater than timeout then that job will stop.
I have searched how to set global timeout config in file sidekiq.yml. But I want to set separate timeout for difference separate jobs meaning one of classes to define worker will have particular timeout config.
Can you help me. Thanks so much.
There's no approved way to do this. You cannot stop a thread safely while it is executing. You need to change your job to check periodically if it should stop.
You can set network timeouts on any 3rd party calls you are making so that they time out.
You can wrap your job code inside a timeout block like the below:
Timeout::timeout(2.hours) do
***. do possibly long-running task *****
end
The job will fail automatically after 2 hours.
This is the same method as yassen suggested, but more concrete.
class MyCustomWorker
include Sidekiq::Worker
def perform
begin
Timeout::timeout(30.minutes) do # set timeout to 30 minutes
perform_job()
end
rescue Timeout::Error
Rails.logger.error "timeout reached for worker"
end
end
def perform_job
# worker logic here
end
end
I found this Schedule one-time jobs in Rails
but this only shows how schedule one-time. I am interested in scheduling a recurring job.
Delayed_job has this
self.delay(:run_at => 1.minute.from_now)
How do I do something like that in Rails 4.2/Active Job?
Similar to rab3's answer, since ActiveJob has support for callbacks, I was thinking of doing something like
class MyJob < ActiveJob::Base
after_perform do |job|
# invoke another job at your time of choice
self.class.set(:wait => 10.minutes).perform_later(job.arguments.first)
end
def perform(the_argument)
# do your thing
end
end
activejob callbacks
If you want to delay the job execution to 10 minutes later, two options:
SomeJob.set(wait: 10.minutes).perform_later(record)
SomeJob.new(record).enqueue(wait: 10.minutes)
Delay to a specific moment from now use wait_until.
SomeJob.set(wait_until: Date.tomorrow.noon).perform_later(record)
SomeJob.new(record).enqueue(wait_until: Date.tomorrow.noon)
Details please refer to http://api.rubyonrails.org/classes/ActiveJob/Base.html.
For recurring jobs, you just put SomeJob.perform_now(record) in a cronjob (whenever).
If you use Heroku, just put SomeJob.perform_now(record) in a scheduled rake task. Please read more about scheduled rake task here: Heroku scheduler.
You can just re-enqueue the job at the end of the execution
class MyJob < ActiveJob::Base
RUN_EVERY = 1.hour
def perform
# do your thing
self.class.perform_later(wait: RUN_EVERY)
end
end
If you're using resque as your ActiveJob backend, you can use a combination of resque-scheduler's Scheduled Jobs and active_scheduler (https://github.com/JustinAiken/active_scheduler, which wraps the scheduled jobs to work properly with ActiveJob).
Say I have a worker class that looks like this:
class BuilderWorker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(order_id)
if(order_id == 5)
# How can I finish the job here? Say I want to finish it with a status of FAIL or COMPLETE.
end
end
end
I am looking for a way to finish a job from the Worker class, and when finished give it the status of FAILED. The finish should be quiet, (not raising an exception)
With Sidekiq there are only two job results:
success - a job returns from the perform method without error
failure - a job raises an error, going onto the retry queue to be retried in the future
Your complicated scenario is called application logic and Sidekiq cannot provide it. It is up to you to write that logic.
I have a sidekiq worker that shouldn't take more than 30 seconds, but after a few days I'll find that the entire worker queue stops executing because all of the workers are locked up.
Here is my worker:
class MyWorker
include Sidekiq::Worker
include Sidekiq::Status::Worker
sidekiq_options queue: :my_queue, retry: 5, timeout: 4.minutes
sidekiq_retry_in do |count|
5
end
sidekiq_retries_exhausted do |msg|
store({message: "Gave up."})
end
def perform(id)
begin
Timeout::timeout(3.minutes) do
got_lock = with_semaphore("lock_#{id}") do
# DO WORK
end
end
rescue ActiveRecord::RecordNotFound => e
# Handle
rescue Timeout::Error => e
# Handle
raise e
end
end
def with_semaphore(name, &block)
Semaphore.get(name, {stale_client_timeout: 1.minute}).lock(1, &block)
end
end
And the semaphore class we use. (redis-semaphore gem)
class Semaphore
def self.get(name, options = {})
Redis::Semaphore.new(name.to_sym,
:redis => Application.redis,
stale_client_timeout: options[:stale_client_timeout] || 1.hour,
)
end
end
Basically I'll stop the worker and it will state done: 10000 seconds, which the worker should NEVER be running for.
Anyone have any ideas on how to fix this or what is causing it? The workers are running on EngineYard.
Edit: One additional comment. The # DO WORK has a chance to fire off a PostgresSQL function. I have noticed in logs some mention of PG::TRDeadlockDetected: ERROR: deadlock detected. Would this cause the worker to never complete even with a timeout set?
Given you want to ensure unique job execution, i would attempt removing all locks and delegate job uniqueness control to a plugin like Sidekiq Unique Jobs
In this case, even if sidetiq enqueue the same job id twice, this plugin ensures it will be enqueued/processed a single time.
You might also try the ActiveRecord with_lock mechanism: http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html
I have had a similar problem before. To solve this problem, you should stop using Timeout.
As explained in this article, you should never use Timeout in a Sidekiq job. If you use Timeout, Sidekiq processes and threads can easily break.
Not only Ruby, but also Java has a similar problem. Stopping a thread from the outside is inherently dangerous, regardless of the language.
If you continue to have the same problem after deleting Timeout, check that if you are using threads carelessly in your code.
As Sidekiq's architecture is so sophisticated, in almost all cases, the source of the bug is outside of Sidekiq.