I have a method that I run asynchronously
User.delay(queue: 'users').grab_third_party_info(user.id)
In case this fails, I want it to not retry. My default retries are 3, which I cannot change. I just want to have this only try once. The following doesn't seem to work:
User.delay(queue: 'users', attempts: 3).grab_third_party_info(user.id)
Any ideas?
This isn't my favorite solution, but if you need to use the delay method that you can set the attempts: to one less your max attempts. So in your case the following should work
User.delay(queue: 'users', attempts: 2).grab_third_party_info(user.id)
Better yet you could make it safer by using Delayed::Worker.max_attempts
User.delay(queue: 'users', attempts: Delayed::Worker.max_attempts-1).grab_third_party_info(user.id)
This would enter it into your delayed_jobs table as if it already ran twice so when it runs again it will be at the max attempts.
From https://github.com/collectiveidea/delayed_job#custom-jobs
To set a per-job max attempts that overrides the Delayed::Worker.max_attempts you can define a max_attempts method on the job
NewsletterJob = Struct.new(:text, :emails) do
def perform
emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
end
def max_attempts
3
end
end
Does this help you?
You have to use a Custom Job.
Just like #lazzi showed, you have to create a custom job in order to override the max_attempts.
As you can see in the README here, the only params that the .delay method take are:
priority
run_at
queue
And if you think about it, a value for max_attempts is not stored in the delayed_jobs table, only the attempts are stored, so there's no way for it to be persisted.
The only way to do it is to create a custom job that gets re-instantiated when the delayed job worker processes the job. It then reads the value from the max_attempts method and uses that to determine if the current attempts in the table record equals or exceeds the max_attempts value.
In your case, the simplest way to do it would be something like this:
# Inside your user.rb
class User < ApplicationRecord
FetchThirdPartyInfoJob = Struct.new( :user ) do
def perform
User.grab_third_party_info(user.id) # REFACTOR: Make this an instance method so you don't need to pass the User's id to it.
end
def queue_name
"users"
end
def max_attempts
3
end
end
end
Then run it wherever you need to by using enqueue, like this:
Delayed::Job.enqueue( User::FetchThirdPartyInfoJob.new( user ) )
I also added a little REFACTOR comment on your code because User.grab_third_party_info(user.id) looks to be incorrectly setup as a class method that you then pass the instance id to instead of just calling it directly on the user instance. I can't think of a reason why you would want this, but if there is, please leave it in the comments so we can all learn.
Related
I building a rewards system for a coffee shop. Basically a customer can sign up for a year subscription. Right now when they sign up the active attribute is toggled to true. I'm trying to write a method that will toggle the attribute to false after a year passes. I have a method right now that I want to use but I don't know where to use it at? I also have a failing test. I'll show my current code for clarity.
Controller:
def create
#subscriber = Subscriber.new(subscriber_params)
if #subscriber.save
#subscriber.touch(:subscription_date)
#subscriber.update(active: true)
SubscriberMailer.welcome_subscriber(#subscriber).deliver_now
flash[:notice] = "Subscriber Has Been Successfully Created"
redirect_to new_subscriber_path(:subscriber)
else
render "new"
end
end
Model method I want to use:
def not_active(subscriber)
if subscription_date < 1.year.ago
self.update(active: false)
end
end
Failing Test:
it "sets active to false after a year" do
subscriber = create(:subscriber)
subscriber.update(active: true)
Time.now + 366.days
expect(subscriber.active).to eq(false)
end
So hopefully this idea is clear. I just want to update to active: false if the user was created over a year ago.
You must run the not_active method in order for the method to have an effect. The method has no way of knowing what the date is today and updating a subscriber unless it is actually run. I agree with matt that you would likely run this method in a sidekiq job daily on on all of your subscribers who subscribed a year or longer ago and are active (You can write a scope for this). This way you can call the not_active method and set each subscriber's active appropriately, or write it as a Subscriber class method and apply it to the results of your scope. In the case of testing the not_active method itself all you need to do is call it and test the result. Its also not clear to me why the not_active method takes a subscriber as an arg, it seems like it would make more sense to just call it from a subscriber instance. Is this not whats already happening? I would personally call this method something like deactivate!, as its making changes. not_active kind of sounds like it would return a boolean or an inactive subscriber. I would also recommend using update! instead of update in not_active. update! will raise an error if the update fails. Adding to time.now does actually change the time. You can use rspec mocks to fake the current time if you need to. In any case here is what your not_active test might look like:
it "sets active to false after a year" do
subscriber = Subscriber.create(subscription_date: (1.year.ago - 1.day), active: true)
#changed not_active to deactivate, called from instance instead of passing in subscriber
subscriber.deactivate!
expect(subscriber.active?).to eq(false)
end
You can also write a test for the other case
it "does not deactivate a recent subscriber" do
subscriber = Subscriber.create(subscription_date: Date.today, active: true)
subscriber.deactivate!
expect(subscriber.active?).to eq(true)
end
A simple solution to this would be to use cron. There is a rubygem to interface with cron, called whenever. The setup is simple and well documented.
With cron setup on your server, you would create some kind of class method that would iterate through Subscribers, calling the not_active method.
Btw, if the not_active method is defined within your Subscriber model, you won't need to pass subscriber as an argument, as self will be implicitly set to the subscriber.
The code would end up looking something like:
in subscriber.rb
def self.set_subscribers_to_inactive
find_each(active: false) do |subscriber|
subscriber.inactive!
end
end
def inactive!
update(active: false) if subscription_date < 1.year.ago
end
in schedule.rb
every 1.day do
runner "Subscriber.set_subscribers_to_inactive"
end
As mentioned, your test is not actually calling the not_active method.
it "sets active to false after a year" do
last_year = DateTime.current - 366.days
subscriber = create(:subscriber, active: true, subscription_date: last_year)
subscriber.inactive!
expect(subscriber.active).to eq false
end
Take a look at cron and whenever gem which works on top of cron. You just need to write a super simple script which will extract data from DB and update it.
Another way to solve your problem is not to update anything. You only need *_expires_at column and check if its value less than current date.
It is pretty agile method, because by using activation_expires_at column you are able to implement #active? method and .active scope to select only users with active subscriptions.
I'm trying to get the job which starts an action in this particular action.
Let me explain.
class MyClass
def go_for_it(delay = true)
if delay
delay(run_at: 2.minutes.from_now).go_for_it(false)
else
# How can I know if I was called by a DelayedJob AND if yes, which one ?
puts "I'll do it"
end
end
end
my_class = MyClass.new
my_class.delay(run_at: 2.minutes.from_now).go_for_it
My aim here is to make restrictions on jobs creation. I don't want go_for_it method called twice but this method can delay again itself according to some reasons. If I add those lines to go_for_it:
calling_method = caller_locations[0].label
job = Delayed::Job.where(queue: "my_queue").first
puts job.payload_object.id
# => id of MyClass if recorded
puts job.payload_object.method_name
# => :go_for_it
In the case of go_for_it delaying itself, these data are not enough because job variable can be itself and then it's not a second different call of got_for_it. It's just itself delayed again.
What I need to know here is which job call run or invoke_job on go_for_it method.
If I'm understanding well, you need to know which job is actually running.
You can use a custom job with a before hook to do an action before running the job, also you'll have totally access to job object at this moment.
Example :
class MyClassJob
def initialize(my_object: MyClass.new)
#my_object = my_object
end
def before(job)
binding.pry
another_job = Delayed::Job.where(queue: "my_queue").where('id <> ?', job.id)
end
def perform
#my_object.go_for_it
end
end
MyClassJob.new().delay.perform
I've a method named update inside my DailyOrdersController:
def update
if #daily_order.update( daily_order_params.merge({default_order:false}) )
respond_or_redirect(#daily_order)
else
render :edit
end
end
My DailyOrder model:
before_save :refresh_total
def refresh_total
# i do something here
end
What I'm trying to do now is, I want the refresh_total callback to be skipped if the update request is coming from current_admin.
I have 2 user model generated using Devise gem:
User (has current_user)
Admin (has current_admin)
I try to make it like this:
def update
if current_admin
DailyOrder.skip_callback :update, :before, :refresh_total
end
if #daily_order.update( daily_order_params.merge({default_order:false}) )
respond_or_redirect(#daily_order)
else
render :edit
end
end
But it's not working and still keep calling the refresh_total callback if the update request is coming from current_admin (when the logged-in user is admin user).
What should I do now?
I think this is all what you need:
http://guides.rubyonrails.org/active_record_callbacks.html#conditional-callbacks
If you skip callback, you should enable it later. Anyway, this does not look as the best solution. Perhaps you could avoid the callbacks otherwise.
One way would be to use update_all:
DailyOrder.where(id: #daily_order.id).update_all( daily_order_params.merge({default_order:false}) )
Or you could do something like this:
#in the model:
before_validation :refresh_total
#in the controller
#daily_order.assign_attributes( daily_order_params.merge({default_order:false}) )
#daily_order.save(validate: current_admin.nil?)
or maybe it would be the best to add a new column to the model: refresh_needed and then you would conditionally update that column on before_validation, and on before_save you would still call the same callback, but conditionally to the state of refresh_needed. In this callback you should reset that column. Please let me know if you would like me to illustrate this with some code.
This may come in handy:
http://www.davidverhasselt.com/set-attributes-in-activerecord/
UPDATE
Even better, you can call update_columns.
Here is what it says in the documentation of the method:
Updates the attributes directly in the database issuing an UPDATE SQL
statement and sets them in the receiver:
user.update_columns(last_request_at: Time.current)
This is the fastest way to update attributes because it goes straight to
the database, but take into account that in consequence the regular update
procedures are totally bypassed. In particular:
\Validations are skipped.
\Callbacks are skipped.
+updated_at+/+updated_on+ are not updated.
This method raises an ActiveRecord::ActiveRecordError when called on new
objects, or when at least one of the attributes is marked as readonly.
I have a method like this that goes through an array to find different APIs and launch a delayed_job instance for every API found like this.
def refresh_users_list
apis_array.each do |api|
api.myclass.new.delay.get_and_create_or_update_users
end
end
I have an after_filter on users#index controller to trigger this method. This is creating many jobs to be triggered that will eventually cause too many connections problems on Heroku.
I'm wondering if there's a way I can check for the presence of a Job in the database by each of the API that the array iterates. This would be very helpful so I can only trigger a particular refresh if that api wasn't updated on a given time.
Any idea how to do this?
In config/application.rb, add the following
config.autoload_paths += Dir["#{config.root}/app/jobs/**/"]
Create a new directory at app/jobs/.
Create a file at app/jobs/api_job.rb that looks like
class ApiJob < Struct.new(:attr1, :attr2, :attr3)
attr_accessor :token
def initialize(*attrs)
self.token = self.class.token(attr1, attr2, attr3)
end
def display_name
self.class.token(attr1, attr2, attr3)
end
#
# Class methods
#
def self.token(attr1, attr2, attr3)
[name.parameterize, attr1.id, attr2.id, attr3.id].join("/")
end
def self.find_by_token(token)
Delayed::Job.where("handler like ?", "%token: #{token}%")
end
end
Note: You will replace attr1, attr2, and attr3 with whatever number of attributes you need (if any) to pass to the ApiJob to perform the queued task. More on how to call this in a moment
For each of your API's that you queue some get_and_create_or_update_users method for you'll create another Job. For example, if I have some Facebook api model, I might have a class at app/jobs/facebook_api_job.rb that looks like
class FacebookApiJob < ApiJob
def perform
FacebookApi.new.get_and_create_or_update_users(attr1, attr2, attr3)
end
end
Note: In your Question you did not pass any attributes to get_and_create_or_update_users. I am just showing you where you would do this if you need the job to have attributes passed to it.
Finally, wherever your refresh_users_list is defined, define something like this job_exists? method
def job_exists?(tokens)
tokens = [tokens] if !tokens.is_a?(Array) # allows a String or Array of tokens to be passed
tokens.each do |token|
return true unless ApiJob.find_by_token(token).empty?
end
false
end
Now, within your refresh_users_list and loop, you can build new tokens and call job_exists? to check if you have queued jobs for the API. For example
# Build a token
def refresh_users_list
apis_array.each do |api|
token = ApiJob.token(attr1, attr2, attr3)
next if job_exists?(token)
api.myclass.new.delay.get_and_create_or_update_users
end
end
Note: Again I want to point out, you won't be able to just drop in the code above and have it work. You must tailor it to your application and the job's you're running.
Why is this so complicated?
From my research, there's no way to "tag" or uniquely identify a queued job through what delayed_job provides. Sure, each job has a unique :id attribute. You could store the ID values for each created job in some hash somewhere
{
"FacebookApi": [1, 4, 12],
"TwitterApi": [3, 193, 44],
# ...
}
and then check corresponding hash key for an ID, but I find this limiting, and not always sufficient for the problem When you need to identify a specific job by multiple attributes like above, we must create a way to find these jobs (without loading every job into memory and looping over them to see if one matches our criteria).
How is this working?
The Struct that the ApiJob extends has a :token attribute. This token is based on the attributes passed (attr1, attr2, attr3) and is built when a new class extending ApiJob is instantiated.
The find_by_token class method simply searches the string representation of the job in the delayed_job queue for a match based on a token built using the same token class method.
I have a process which takes generally a few seconds to complete so I'm trying to use delayed_job to handle it asynchronously. The job itself works fine, my question is how to go about polling the job to find out if it's done.
I can get an id from delayed_job by simply assigning it to a variable:
job = Available.delay.dosomething(:var => 1234)
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| id | priority | attempts | handler | last_error | run_at | locked_at | failed_at | locked_by | created_at | updated_at |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| 4037 | 0 | 0 | --- !ru... | | 2011-04-... | | | | 2011-04... | 2011-04-... |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
But as soon as it completes the job it deletes it and searching for the completed record returns an error:
#job=Delayed::Job.find(4037)
ActiveRecord::RecordNotFound: Couldn't find Delayed::Backend::ActiveRecord::Job with ID=4037
#job= Delayed::Job.exists?(params[:id])
Should I bother to change this, and maybe postpone the deletion of complete records? I'm not sure how else I can get a notification of it's status. Or is polling a dead record as proof of completion ok? Anyone else face something similar?
Let's start with the API. I'd like to have something like the following.
#available.working? # => true or false, so we know it's running
#available.finished? # => true or false, so we know it's finished (already ran)
Now let's write the job.
class AwesomeJob < Struct.new(:options)
def perform
do_something_with(options[:var])
end
end
So far so good. We have a job. Now let's write logic that enqueues it. Since Available is the model responsible for this job, let's teach it how to start this job.
class Available < ActiveRecord::Base
def start_working!
Delayed::Job.enqueue(AwesomeJob.new(options))
end
def working?
# not sure what to put here yet
end
def finished?
# not sure what to put here yet
end
end
So how do we know if the job is working or not? There are a few ways, but in rails it just feels right that when my model creates something, it's usually associated with that something. How do we associate? Using ids in database. Let's add a job_id on Available model.
While we're at it, how do we know that the job is not working because it already finished, or because it didn't start yet? One way is to actually check for what the job actually did. If it created a file, check if file exists. If it computed a value, check that result is written. Some jobs are not as easy to check though, since there may be no clear verifiable result of their work. For such case, you can use a flag or a timestamp in your model. Assuming this is our case, let's add a job_finished_at timestamp to distinguish a not yet ran job from an already finished one.
class AddJobIdToAvailable < ActiveRecord::Migration
def self.up
add_column :available, :job_id, :integer
add_column :available, :job_finished_at, :datetime
end
def self.down
remove_column :available, :job_id
remove_column :available, :job_finished_at
end
end
Alright. So now let's actually associate Available with its job as soon as we enqueue the job, by modifying the start_working! method.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options))
update_attribute(:job_id, job.id)
end
Great. At this point I could've written belongs_to :job, but we don't really need that.
So now we know how to write the working? method, so easy.
def working?
job_id.present?
end
But how do we mark the job finished? Nobody knows a job has finished better than the job itself. So let's pass available_id into the job (as one of the options) and use it in the job. For that we need to modify the start_working! method to pass the id.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options.merge(:available_id => id))
update_attribute(:job_id, job.id)
end
And we should add the logic into the job to update our job_finished_at timestamp when it's done.
class AwesomeJob < Struct.new(:options)
def perform
available = Available.find(options[:available_id])
do_something_with(options[:var])
# Depending on whether you consider an error'ed job to be finished
# you may want to put this under an ensure. This way the job
# will be deemed finished even if it error'ed out.
available.update_attribute(:job_finished_at, Time.current)
end
end
With this code in place we know how to write our finished? method.
def finished?
job_finished_at.present?
end
And we're done. Now we can simply poll against #available.working? and #available.finished? Also, you gain the convenience of knowing which exact job was created for your Available by checking #available.job_id. You can easily turn it into a real association by saying belongs_to :job.
I ended up using a combination of Delayed_Job with an after(job) callback which populates a memcached object with the same ID as the job created. This way I minimize the number of times I hit the database asking for the status of the job, instead polling the memcached object. And it contains the entire object I need from the completed job, so I don't even have a roundtrip request. I got the idea from an article by the github guys who did pretty much the same thing.
https://github.com/blog/467-smart-js-polling
and used a jquery plugin for the polling, which polls less frequently, and gives up after a certain number of retries
https://github.com/jeremyw/jquery-smart-poll
Seems to work great.
def after(job)
prices = Room.prices.where("space_id = ? AND bookdate BETWEEN ? AND ?", space_id.to_i, date_from, date_to).to_a
Rails.cache.fetch(job.id) do
bed = Bed.new(:space_id => space_id, :date_from => date_from, :date_to => date_to, :prices => prices)
end
end
I think that the best way would be to use the callbacks available in the delayed_job.
These are:
:success, :error and :after.
so you can put some code in your model with the after:
class ToBeDelayed
def perform
# do something
end
def after(job)
# do something
end
end
Because if you insist of using the obj.delayed.method, then you'll have to monkey patch Delayed::PerformableMethod and add the after method there.
IMHO it's far better than polling for some value which might be even backend specific (ActiveRecord vs. Mongoid, for instance).
The simplest method of accomplishing this is to change your polling action to be something similar to the following:
def poll
#job = Delayed::Job.find_by_id(params[:job_id])
if #job.nil?
# The job has completed and is no longer in the database.
else
if #job.last_error.nil?
# The job is still in the queue and has not been run.
else
# The job has encountered an error.
end
end
end
Why does this work? When Delayed::Job runs a job from the queue, it deletes it from the database if successful. If the job fails, the record stays in the queue to be ran again later, and the last_error attribute is set to the encountered error. Using the two pieces of functionality above, you can check for deleted records to see if they were successful.
The benefits to the method above are:
You get the polling effect that you were looking for in your original post
Using a simple logic branch, you can provide feedback to the user if there is an error in processing the job
You can encapsulate this functionality in a model method by doing something like the following:
# Include this in your initializers somewhere
class Queue < Delayed::Job
def self.status(id)
self.find_by_id(id).nil? ? "success" : (job.last_error.nil? ? "queued" : "failure")
end
end
# Use this method in your poll method like so:
def poll
status = Queue.status(params[:id])
if status == "success"
# Success, notify the user!
elsif status == "failure"
# Failure, notify the user!
end
end
I'd suggest that if it's important to get notification that the job has completed, then write a custom job object and queue that rather than relying upon the default job that gets queued when you call Available.delay.dosomething. Create an object something like:
class DoSomethingAvailableJob
attr_accessor options
def initialize(options = {})
#options = options
end
def perform
Available.dosomething(#options)
# Do some sort of notification here
# ...
end
end
and enqueue it with:
Delayed::Job.enqueue DoSomethingAvailableJob.new(:var => 1234)
The delayed_jobs table in your application is intended to provide the status of running and queued jobs only. It isn't a persistent table, and really should be as small as possible for performance reasons. Thats why the jobs are deleted immediately after completion.
Instead you should add field to your Available model that signifies that the job is done. Since I'm usually interested in how long the job takes to process, I add start_time and end_time fields. Then my dosomething method would look something like this:
def self.dosomething(model_id)
model = Model.find(model_id)
begin
model.start!
# do some long work ...
rescue Exception => e
# ...
ensure
model.finish!
end
end
The start! and finish! methods just record the current time and save the model. Then I would have a completed? method that your AJAX can poll to see if the job is finished.
def completed?
return true if start_time and end_time
return false
end
There are many ways to do this but I find this method simple and works well for me.