polling with delayed_job - ruby-on-rails

I have a process which takes generally a few seconds to complete so I'm trying to use delayed_job to handle it asynchronously. The job itself works fine, my question is how to go about polling the job to find out if it's done.
I can get an id from delayed_job by simply assigning it to a variable:
job = Available.delay.dosomething(:var => 1234)
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| id | priority | attempts | handler | last_error | run_at | locked_at | failed_at | locked_by | created_at | updated_at |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| 4037 | 0 | 0 | --- !ru... | | 2011-04-... | | | | 2011-04... | 2011-04-... |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
But as soon as it completes the job it deletes it and searching for the completed record returns an error:
#job=Delayed::Job.find(4037)
ActiveRecord::RecordNotFound: Couldn't find Delayed::Backend::ActiveRecord::Job with ID=4037
#job= Delayed::Job.exists?(params[:id])
Should I bother to change this, and maybe postpone the deletion of complete records? I'm not sure how else I can get a notification of it's status. Or is polling a dead record as proof of completion ok? Anyone else face something similar?

Let's start with the API. I'd like to have something like the following.
#available.working? # => true or false, so we know it's running
#available.finished? # => true or false, so we know it's finished (already ran)
Now let's write the job.
class AwesomeJob < Struct.new(:options)
def perform
do_something_with(options[:var])
end
end
So far so good. We have a job. Now let's write logic that enqueues it. Since Available is the model responsible for this job, let's teach it how to start this job.
class Available < ActiveRecord::Base
def start_working!
Delayed::Job.enqueue(AwesomeJob.new(options))
end
def working?
# not sure what to put here yet
end
def finished?
# not sure what to put here yet
end
end
So how do we know if the job is working or not? There are a few ways, but in rails it just feels right that when my model creates something, it's usually associated with that something. How do we associate? Using ids in database. Let's add a job_id on Available model.
While we're at it, how do we know that the job is not working because it already finished, or because it didn't start yet? One way is to actually check for what the job actually did. If it created a file, check if file exists. If it computed a value, check that result is written. Some jobs are not as easy to check though, since there may be no clear verifiable result of their work. For such case, you can use a flag or a timestamp in your model. Assuming this is our case, let's add a job_finished_at timestamp to distinguish a not yet ran job from an already finished one.
class AddJobIdToAvailable < ActiveRecord::Migration
def self.up
add_column :available, :job_id, :integer
add_column :available, :job_finished_at, :datetime
end
def self.down
remove_column :available, :job_id
remove_column :available, :job_finished_at
end
end
Alright. So now let's actually associate Available with its job as soon as we enqueue the job, by modifying the start_working! method.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options))
update_attribute(:job_id, job.id)
end
Great. At this point I could've written belongs_to :job, but we don't really need that.
So now we know how to write the working? method, so easy.
def working?
job_id.present?
end
But how do we mark the job finished? Nobody knows a job has finished better than the job itself. So let's pass available_id into the job (as one of the options) and use it in the job. For that we need to modify the start_working! method to pass the id.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options.merge(:available_id => id))
update_attribute(:job_id, job.id)
end
And we should add the logic into the job to update our job_finished_at timestamp when it's done.
class AwesomeJob < Struct.new(:options)
def perform
available = Available.find(options[:available_id])
do_something_with(options[:var])
# Depending on whether you consider an error'ed job to be finished
# you may want to put this under an ensure. This way the job
# will be deemed finished even if it error'ed out.
available.update_attribute(:job_finished_at, Time.current)
end
end
With this code in place we know how to write our finished? method.
def finished?
job_finished_at.present?
end
And we're done. Now we can simply poll against #available.working? and #available.finished? Also, you gain the convenience of knowing which exact job was created for your Available by checking #available.job_id. You can easily turn it into a real association by saying belongs_to :job.

I ended up using a combination of Delayed_Job with an after(job) callback which populates a memcached object with the same ID as the job created. This way I minimize the number of times I hit the database asking for the status of the job, instead polling the memcached object. And it contains the entire object I need from the completed job, so I don't even have a roundtrip request. I got the idea from an article by the github guys who did pretty much the same thing.
https://github.com/blog/467-smart-js-polling
and used a jquery plugin for the polling, which polls less frequently, and gives up after a certain number of retries
https://github.com/jeremyw/jquery-smart-poll
Seems to work great.
def after(job)
prices = Room.prices.where("space_id = ? AND bookdate BETWEEN ? AND ?", space_id.to_i, date_from, date_to).to_a
Rails.cache.fetch(job.id) do
bed = Bed.new(:space_id => space_id, :date_from => date_from, :date_to => date_to, :prices => prices)
end
end

I think that the best way would be to use the callbacks available in the delayed_job.
These are:
:success, :error and :after.
so you can put some code in your model with the after:
class ToBeDelayed
def perform
# do something
end
def after(job)
# do something
end
end
Because if you insist of using the obj.delayed.method, then you'll have to monkey patch Delayed::PerformableMethod and add the after method there.
IMHO it's far better than polling for some value which might be even backend specific (ActiveRecord vs. Mongoid, for instance).

The simplest method of accomplishing this is to change your polling action to be something similar to the following:
def poll
#job = Delayed::Job.find_by_id(params[:job_id])
if #job.nil?
# The job has completed and is no longer in the database.
else
if #job.last_error.nil?
# The job is still in the queue and has not been run.
else
# The job has encountered an error.
end
end
end
Why does this work? When Delayed::Job runs a job from the queue, it deletes it from the database if successful. If the job fails, the record stays in the queue to be ran again later, and the last_error attribute is set to the encountered error. Using the two pieces of functionality above, you can check for deleted records to see if they were successful.
The benefits to the method above are:
You get the polling effect that you were looking for in your original post
Using a simple logic branch, you can provide feedback to the user if there is an error in processing the job
You can encapsulate this functionality in a model method by doing something like the following:
# Include this in your initializers somewhere
class Queue < Delayed::Job
def self.status(id)
self.find_by_id(id).nil? ? "success" : (job.last_error.nil? ? "queued" : "failure")
end
end
# Use this method in your poll method like so:
def poll
status = Queue.status(params[:id])
if status == "success"
# Success, notify the user!
elsif status == "failure"
# Failure, notify the user!
end
end

I'd suggest that if it's important to get notification that the job has completed, then write a custom job object and queue that rather than relying upon the default job that gets queued when you call Available.delay.dosomething. Create an object something like:
class DoSomethingAvailableJob
attr_accessor options
def initialize(options = {})
#options = options
end
def perform
Available.dosomething(#options)
# Do some sort of notification here
# ...
end
end
and enqueue it with:
Delayed::Job.enqueue DoSomethingAvailableJob.new(:var => 1234)

The delayed_jobs table in your application is intended to provide the status of running and queued jobs only. It isn't a persistent table, and really should be as small as possible for performance reasons. Thats why the jobs are deleted immediately after completion.
Instead you should add field to your Available model that signifies that the job is done. Since I'm usually interested in how long the job takes to process, I add start_time and end_time fields. Then my dosomething method would look something like this:
def self.dosomething(model_id)
model = Model.find(model_id)
begin
model.start!
# do some long work ...
rescue Exception => e
# ...
ensure
model.finish!
end
end
The start! and finish! methods just record the current time and save the model. Then I would have a completed? method that your AJAX can poll to see if the job is finished.
def completed?
return true if start_time and end_time
return false
end
There are many ways to do this but I find this method simple and works well for me.

Related

cancelling a sheduled Sidekiq job in Rails

Some Sidekiq jobs in my app are scheduled to change the state of a resource to cancelled unless a user responds within a certain timeframe. There is a lot of information about how to best accomplish this task, but none of it actually cancels the job.
To cancel a job, the code in the wiki says:
class MyWorker
include Sidekiq::Worker
def perform(thing_id)
return if cancelled?
thing = Thing.find thing_id
thing.renege!
end
def cancelled?
Sidekiq.redis {|c| c.exists("cancelled-#{jid}") }
end
def self.cancel!(jid)
Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
end
end
Yet here it's suggested that I do something like
def perform(thing_id)
thing = Thing.find thing_id
while !cancel?(thing)
thing.ignore!
end
end
def cancel?(thing_id)
thing = Thing.find thing_id
thing.matched? || thing.passed?
end
What's confusing about this and similar code on the wiki is none of it actually cancels the job. The above example just performs an update on thing if cancelled? returns false (as it should), but doesn't cancel if and when it returns true in the future. It just fails with an aasm transition error message and gets sent to the RetrySet. Calling MyWorker.cancel! jid in model code throws an undefined variable error. How can I access that jid in the model? How can actually cancel or delete that specific job? Thanks!
# The wiki code
class MyWorker
include Sidekiq::Worker
def perform(thing_id)
return if cancelled?
# do actual work
end
def cancelled?
Sidekiq.redis {|c| c.exists("cancelled-#{jid}") }
end
def self.cancel!(jid)
Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
end
end
# create job
jid = MyWorker.perform_async("foo")
# cancel job
MyWorker.cancel!(jid)
You can do this but it won't be efficient. It's a linear scan for find a scheduled job by JID.
require 'sidekiq/api'
Sidekiq::ScheduledSet.new.find_job(jid).try(:delete)
Alternatively your job can look to see if it's still relevant when it runs.
Ok, so turns out I had one question already answered. One of the code sets I included was a functionally similar version of the code from the wiki. The solution to the other question ("how can I access that jid in the model?") seems really obvious if you're not still new to programming, but basically: store the jid in a database column and then retrieve/update it whenever it's needed! Duh!

How to find associated Resque jobs for an ActiveRecord model object?

I need to be able to find queued and/or working jobs and/or failed jobs for a model object, say for example when the model object is destroyed we want to find all and either decide not to delete or destroy the jobs (conditionally).
Is there a recommended way to do this before I reinvent the wheel?
Example:
If you want to create a before_destroy callback that destroys all jobs when the object is destroyed (queued and failed jobs) and only destroy if there are no working jobs
Some pseudo code of what I am thinking to do for this example use case:
Report model
class Report < ActiveRecord::Base
before_destroy :check_if_working_jobs, :destroy_queued_and_failed_jobs
def check_if_working_jobs
# find all working jobs related to this report object
working_jobs = ProcessReportWorker.find_working_jobs_by_report_id(self.id)
return false unless working_jobs.empty?
end
def destroy_queued_and_failed_jobs
# find all jobs related to this report object
queued_jobs = ProcessReportWorker.find_queued_jobs_by_report_id(self.id)
failed_jobs = ProcessReportWorker.find_failed_jobs_by_report_id(self.id)
# destroy/remove all jobs found
(queued_jobs + failed_jobs).each do |job|
# destroy the job here ... commands?
end
end
end
Report processing worker class for resque / redis backed jobs
class ProcessReportWorker
# find the jobs by report id which is one of the arguments for the job?
# envisioned as separate methods so they can be used independently as needed
def self.find_queued_jobs_by_report_id(id)
# parse all jobs in all queues to find based on the report id argument?
end
def self.find_working_jobs_by_report_id(id)
# parse all jobs in working queues to find based on the report id argument?
end
def self.find_failed_jobs_by_report_id(id)
# parse all jobs in failed queue to find based on the report id argument?
end
end
Is this approach on track with what needs to happen?
What are the missing pieces above to find the queued or working jobs by model object id and then destroy it?
Are there already methods in place to find and/or destroy by associated model object id that I have missed in the documentation or my searching?
Update: Revised the usage example to only use working_jobs as a way to check to see if we should delete or not vs suggesting we will try to delete working_jobs also. (because deleting working jobs is more involved than simply removing the redis key entries)
Its been quiet here with no responses so I managed to spend the day tackling this myself following the path I was indicating in my question. There may be a better solution or other methods available but this seems to get the job done so far. Feel free to comment if there are better options here for the methods used or if it can be improved further.
The overall approach here is you need to search through all jobs (queued, working, failed) and filtering out only jobs for the class and queue that are relevant and that match the object record id you are looking for in the correct index position of the args array. For example (after confirming the class and queue match) if the argument position 0 is where the object id is, then you can test to see if args[0] matches the object id.
Essentially, a job is associated to the object id if: job_class == class.name && job_queue == #queue && job_args[OBJECT_ID_ARGS_INDEX].to_i == object_id
Queued Jobs: To find all queued jobs you need to collect all redis
entries with the keys named queue:#{#queue} where #queue is the
name of the queue your worker class is using. Modify accordingly by
looping through multiple queues if you are using multiple queues for
a particular worker class. Resque.redis.lrange("queue:#{#queue}",0,-1)
Failed Jobs: To find all queued jobs you need to collect all redis
entries with the keys named failed (unless you are using multiple
failure queues or some other than default setup). Resque.redis.lrange("failed",0,-1)
Working Jobs: To find all working jobs you can use Resque.workers
which contains an array of all workers and the jobs that are running. Resque.workers.map(&:job)
Job: Each job in each of the above lists will be an encoded hash. You
can decode the job into a ruby hash using Resque.decode(job).
Class and args: For queued jobs, the class and args keys are job["class"]
and job["args"]. For failed and working jobs these are job["payload"]["class"] and job["payload"]["args"].
Queue: For each of the failed and working jobs found, the queue will be job["queue"]. Before testing the args list for the object id, you only want jobs that match the class and queue. Your queued jobs list will already be limited to the queue you collected.
Below are the example worker class and model methods to find (and to remove) jobs that are associated to the example model object (report).
Report processing worker class for resque / redis backed jobs
class ProcessReportWorker
# queue name
#queue = :report_processing
# tell the worker class where the report id is in the arguments list
REPORT_ID_ARGS_INDEX = 0
# <snip> rest of class, not needed here for this answer
# find jobs methods - find by report id (report is the 'associated' object)
def self.find_queued_jobs_by_report_id report_id
queued_jobs(#queue).select do |job|
is_job_for_report? :queued, job, report_id
end
end
def self.find_failed_jobs_by_report_id report_id
failed_jobs.select do |job|
is_job_for_report? :failed, job, report_id
end
end
def self.find_working_jobs_by_report_id report_id
working_jobs.select do |worker,job|
is_job_for_report? :working, job, report_id
end
end
# association test method - determine if this job is associated
def self.is_job_for_report? state, job, report_id
attributes = job_attributes(state, job)
attributes[:klass] == self.name &&
attributes[:queue] == #queue &&
attributes[:args][REPORT_ID_ARGS_INDEX].to_i == report_id
end
# remove jobs methods
def self.remove_failed_jobs_by_report_id report_id
find_failed_jobs_by_report_id(report_id).each do |job|
Resque::Failure.remove(job["index"])
end
end
def self.remove_queued_jobs_by_report_id report_id
find_queued_jobs_by_report_id(report_id).each do |job|
Resque::Job.destroy(#queue,job["class"],*job["args"])
end
end
# reusable methods - these methods could go elsewhere and be reusable across worker classes
# job attributes method
def self.job_attributes(state, job)
if state == :queued && job["args"].present?
args = job["args"]
klass = job["class"]
elsif job["payload"] && job["payload"]["args"].present?
args = job["payload"]["args"]
klass = job["payload"]["class"]
else
return {args: nil, klass: nil, queue: nil}
end
{args: args, klass: klass, queue: job["queue"]}
end
# jobs list methods
def self.queued_jobs queue
Resque.redis.lrange("queue:#{queue}", 0, -1)
.collect do |job|
job = Resque.decode(job)
job["queue"] = queue # for consistency only
job
end
end
def self.failed_jobs
Resque.redis.lrange("failed", 0, -1)
.each_with_index.collect do |job,index|
job = Resque.decode(job)
job["index"] = index # required if removing
job
end
end
def self.working_jobs
Resque.workers.zip(Resque.workers.map(&:job))
.reject { |w, j| w.idle? || j['queue'].nil? }
end
end
So then the usage example for Report model becomes
class Report < ActiveRecord::Base
before_destroy :check_if_working_jobs, :remove_queued_and_failed_jobs
def check_if_working_jobs
# find all working jobs related to this report object
working_jobs = ProcessReportWorker.find_working_jobs_by_report_id(self.id)
return false unless working_jobs.empty?
end
def remove_queued_and_failed_jobs
# find all jobs related to this report object
queued_jobs = ProcessReportWorker.find_queued_jobs_by_report_id(self.id)
failed_jobs = ProcessReportWorker.find_failed_jobs_by_report_id(self.id)
# extra code and conditionals here for example only as all that is really
# needed is to call the remove methods without first finding or checking
unless queued_jobs.empty?
ProcessReportWorker.remove_queued_jobs_by_report_id(self.id)
end
unless failed_jobs.empty?
ProcessReportWorker.remove_failed_jobs_by_report_id(self.id)
end
end
end
The solution needs to be modified if you use multiple queues for the worker class or if you have multiple failure queues. Also, redis failure backend was used. If a different failure backend is used, changes may be required.

Can I call delayed_job with max attempts of 1?

I have a method that I run asynchronously
User.delay(queue: 'users').grab_third_party_info(user.id)
In case this fails, I want it to not retry. My default retries are 3, which I cannot change. I just want to have this only try once. The following doesn't seem to work:
User.delay(queue: 'users', attempts: 3).grab_third_party_info(user.id)
Any ideas?
This isn't my favorite solution, but if you need to use the delay method that you can set the attempts: to one less your max attempts. So in your case the following should work
User.delay(queue: 'users', attempts: 2).grab_third_party_info(user.id)
Better yet you could make it safer by using Delayed::Worker.max_attempts
User.delay(queue: 'users', attempts: Delayed::Worker.max_attempts-1).grab_third_party_info(user.id)
This would enter it into your delayed_jobs table as if it already ran twice so when it runs again it will be at the max attempts.
From https://github.com/collectiveidea/delayed_job#custom-jobs
To set a per-job max attempts that overrides the Delayed::Worker.max_attempts you can define a max_attempts method on the job
NewsletterJob = Struct.new(:text, :emails) do
def perform
emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
end
def max_attempts
3
end
end
Does this help you?
You have to use a Custom Job.
Just like #lazzi showed, you have to create a custom job in order to override the max_attempts.
As you can see in the README here, the only params that the .delay method take are:
priority
run_at
queue
And if you think about it, a value for max_attempts is not stored in the delayed_jobs table, only the attempts are stored, so there's no way for it to be persisted.
The only way to do it is to create a custom job that gets re-instantiated when the delayed job worker processes the job. It then reads the value from the max_attempts method and uses that to determine if the current attempts in the table record equals or exceeds the max_attempts value.
In your case, the simplest way to do it would be something like this:
# Inside your user.rb
class User < ApplicationRecord
FetchThirdPartyInfoJob = Struct.new( :user ) do
def perform
User.grab_third_party_info(user.id) # REFACTOR: Make this an instance method so you don't need to pass the User's id to it.
end
def queue_name
"users"
end
def max_attempts
3
end
end
end
Then run it wherever you need to by using enqueue, like this:
Delayed::Job.enqueue( User::FetchThirdPartyInfoJob.new( user ) )
I also added a little REFACTOR comment on your code because User.grab_third_party_info(user.id) looks to be incorrectly setup as a class method that you then pass the instance id to instead of just calling it directly on the user instance. I can't think of a reason why you would want this, but if there is, please leave it in the comments so we can all learn.

Change status of an object after a set of async jobs is complete using sidekiq

I'm using sidekiq to deal with async jobs, and after some complexity added, I'm having difficulties to be aware of the state of the jobs.
Here's the deal:
I have a model Batch that calls an async method after it's commited:
# app/models/batch.rb
class Batch < ActiveRecord::Base
after_commit :calculate, on: :create
def calculate
job_id = BatchWorker.perform_async(self.id)
# update_column skips callbacks and validations!
self.update_column(:job_id, job_id)
end
end
The worker reads data from the model and calls an async job for each data, as follows:
# app/workers/batch_worker.rb
class BatchWorker
def perform(batch_id)
batch = Batch.find(batch_id)
## read data to 'tab'
tab.each do |ts|
obj = batch.item.create(name: ts[0], data: ts[1])
job_id = ItemWorker.perform_async(obj.id)
obj.update_attribute(:job_id, job_id)
end
end
end
The problem is: Those async jobs perform calculations, and I can't allow the download results link be available before it's complete, so I need to know when all "children-jobs" are done, so I can change a status attribute from the Batch model. In other words, I don't need to know if all jobs have been queued, but instead, if all async jobs generated by ItemWorker have been performed, and are now complete.
What would be the best way to attain this? Does it make sense in the "parallel computation world"?
Obs.: I'm not sure about storing the job_id in db, since it seems to be volatile.
Perhaps using Redis for this could be a good fit, seeing as you already have it in your infrastructure and configured in your Rails app (due to Sidekiq)
Redis has an inbuilt publish/subscribe engine, as well as atomic operations on keys - making it suitable for managing the type of concurrency you are looking for.
Maybe something roughly like this:
class BatchWorker
def perform(batch_id)
batch = Batch.find(batch_id)
redis = Redis.new
redis.set "jobs_remaining_#{batch_id}", tab.count
redis.subscribe("batch_task_complete.#{batch_id}") do |on|
on.message do |event, data|
if redis.decr("jobs_remaining_#{batch_id}") < 1
#UPDATE STATUS HERE
redis.del "jobs_remaining_#{batch_id}"
end
end
end
tab.each do |ts|
obj = batch.item.create(name: ts[0], data: ts[1])
job_id = ItemWorker.perform_async(obj.id, batch_id)
end
end
end
class ItemWorker
def perform item_id, batch_id=nil
#DO STUFF
if batch_id
Redis.new.publish "batch_task_complete.#{batch_id}"
end
end
end

delayed_job. Delete job from queue

class Radar
include Mongoid::Document
after_save :post_on_facebook
private
def post_on_facebook
if self.user.settings.post_facebook
Delayed::Job.enqueue(::FacebookJob.new(self.user,self.body,url,self.title),0,self.active_from)
end
end
end
class FacebookJob < Struct.new(:user,:body,:url,:title)
include SocialPluginsHelper
def perform
facebook_client(user).publish_feed('', :message => body, :link => url, :name => title)
end
end
I want execute post_on_facebook method at specific date. I store this date at "active_from" field.
Code above is working and job is executed at correct date.
But in some cases I first create Radar object and send some job to Delayed Job queue. After that I update this object and send another job to Delayed Job.
This is wrong behavior because I wan't execute job only once at correct time. In this implementation I will have 2 jobs which will be executed. How I can delete previous job so only updated one will be executed ?
Rails 3.0.7
Delayed Job => 2.1.4 https://github.com/collectiveidea/delayed_job
ps: sorry for my english I try do my best
Sounds like you want to de-queue any jobs if a radar object gets updated and re-queue.
Delayed::Job.enqueue should return a Delayed::Job record, so you can grab the ID off of that and save it back onto the Radar record (create a field for it on radar document) so you can find it again later easily.
You should change it to a before_save so you don't enter an infinite loop of saving.
before_save :post_on_facebook
def post_on_facebook
if self.user.settings.post_facebook && self.valid?
# delete existing delayed_job if present
Delayed::Job.find(self.delayed_job_id).destroy if self.delayed_job_id
# enqueue job
dj = Delayed::Job.enqueue(
::FacebookJob.new(self.user,self.body,url,self.title),0,self.active_from
)
# save id of delayed job on radar record
self.delayed_job_id = dj.id
end
end
did you try storing the id from the delayed job and then store it for possible deletion:
e.g
job_id = Delayed::Job.enqueue(::FacebookJob.new(self.user,self.body,url,self.title),0,self.active_from)
job = Delayed::Job.find(job_id)
job.delete

Resources