so some context, I got some advice here:
Scheduling events in Ruby on Rails
aand have been tying to implement it today. I cant seem to make it work though. this is my scheduler job that is used to move my questions around between a delayed queue and a ready to send out queue (i've since decided to use email instead of SMS)
require 'Assignment'
require 'QuestionMailer'
module SchedulerJob
#delayed_queue = :delayed_queue
#ready_queue
def self.perform()
#delayed_queue.each do |a|
if(Time.now >= a.question.schedule)
#ready_queue << a
#delayed_queue.delete(a)
end
end
push_questions
end
def self.gather()
assignments = Assignment.find :all
assignments.each do |a|
#delayed_queue << a unless #delayed_queue.include? a
end
end
private
def self.push_questions
#ready_queue.each do |a|
QuestionMailer.question(a)
end
end
end
I use a callback on_create to call the gather method every time an assignment is created, and then the perform action actually does the sending of emails when resque runs.
I'm getting a strange error from the callback though.
undefined method `include?' for :delayed_queue:Symbol
here is the code from the assignment model
class Assignment < ActiveRecord::Base
belongs_to :user
belongs_to :question
attr_accessible :title, :body, :user_id, :question_id , :response , :correct
after_create :queue_assignments
def grade
self.correct = (response == self.question.solution) unless response == nil
end
def queue_assignments
SchedulerJob.gather
end
Any ideas what's going on? I think this is a problem with my understanding of how these queue's work with resque-scheduler. I assumed that if the queues were list-like objects then I could operate on them , but it appears that it a symbol instead of something with methode like include? I assume the << notation for adding something to it is also invalid.
Also please advise if this isn't the way to go about handling this kind of job scheduling
It appears you may have not restarted your Rails app after adding the new method gather to the SchedulerJob module. Try restarting your app to resolve this.
You may also be able to add the directory containing your Resque worker to Rails' watchable_dirs array so that changes you make to Resque worker modules in development don't require restarting your app. See this blog post for details:
http://wondible.com/2012/01/13/rails-3-2-autoloading-in-theory/
Related
I have a rake task that loops through bookings and sends an email for each one using .deliver method (which I got from here (which I'm conscious is now 7 years old).
The problem is, sometimes some of the emails don't get sent. Here is my code
# Select bookings starting soon
bookings = Booking.where('start_time < ?', 24.hours.since)
# Email a reminder
bookings.each do |booking|
customer = booking.customer
CustomerMailer.reminder_24h(customer, booking).deliver
end
Since the loop is in a rake task, I don't think there's any value in calling .deliver_later, so I just use .deliver like in the rails cast
I am curious to know if there are best practices that can help using Action Mailer, for example should there be a sleep 2 between each email? Or should I always use .deliver_later to relieve the load on the server? Are there any other rails-related reasons that my code may not work (or, worse, I am using any anti patterns that I should refactor?)
TL;DR why would emails sent in a loop like in the code above occasionally fail to send
No an answer, but some advice from another forum.
Sending emails is a process that is filled with potential failures. It is always a good idea to do it in a background job that can be re-tried in case of intermittent errors like networks etc. and also skipped due to faulty addresses.
Here is a sketch of what may work:
# Reminder process rake task
namespace :bookings do
desc "Deliver reminders to upcoming bookings"
task remind_upcoming: :environment do
EnqueueUpcomingBookingReminders.call(UpcomingBookingRemindersQuery.call)
end
end
class EnqueueUpcomingBookingReminders
def self.call(bookings_scope)
booking_communication_attrs =
bookings_scope
.pluck(:id)
.map { |id| {booking_id: id, type: "reminder"} }
communications_result =
BookingCommunication.insert_all(booking_communication_attrs, unique_by: %i[booking_id type])
# Email a reminder
communications_result.rows.flatten.each do |communication_id|
DeliverBookingCommunicationJob.perform_later(communication_id)
end
end
end
class UpcomingBookingRemindersQuery
def self.call(scope: Booking)
Booking
.upcoming_this_day
.left_outer_joins(:communications)
.merge(BookingCommunication.reminder)
.where(communications: {id: nil})
end
end
class Booking
has_many :communications, class_name: "BookingCommunication"
def self.upcoming_this_day
where(starts_at:, (Time.current..24.hours.from_now))
end
end
class BookingCommunication
belongs_to :booking
enum step: {confirmation: "confirmation", reminder: "reminder"} # combination of this and the booking id should be unique
enum status: {pending: "pending", delivered: "delivered", canceled: "canceled", failed: "failed"} # should default to pending at database layer
end
class DeliverBookingCommunicationJob < ApplicationJob
def perform(communication_id)
communication = BookingCommunication.find_by(communication_id)
# Guard against state that invalidates us running this job
return unless communication
return unless communication.pending?
return communication.canceled! if communication.booking.canceled? # This should probably live in the cancel booking process
booking = communication.booking
mailer = CustomerMailer.with(customer: booking.customer, booking: booking)
case communication.step
when "reminder"
mailer.reminder_24h.deliver_now
else
# log unknown communication step, send to error tracking but dont raise since we do not want job to run again
end
communication.delivered!
rescue SomeEmailRelatedError => err
communication.failed!
# deliver err to error tracking service
end
end
Some Sidekiq jobs in my app are scheduled to change the state of a resource to cancelled unless a user responds within a certain timeframe. There is a lot of information about how to best accomplish this task, but none of it actually cancels the job.
To cancel a job, the code in the wiki says:
class MyWorker
include Sidekiq::Worker
def perform(thing_id)
return if cancelled?
thing = Thing.find thing_id
thing.renege!
end
def cancelled?
Sidekiq.redis {|c| c.exists("cancelled-#{jid}") }
end
def self.cancel!(jid)
Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
end
end
Yet here it's suggested that I do something like
def perform(thing_id)
thing = Thing.find thing_id
while !cancel?(thing)
thing.ignore!
end
end
def cancel?(thing_id)
thing = Thing.find thing_id
thing.matched? || thing.passed?
end
What's confusing about this and similar code on the wiki is none of it actually cancels the job. The above example just performs an update on thing if cancelled? returns false (as it should), but doesn't cancel if and when it returns true in the future. It just fails with an aasm transition error message and gets sent to the RetrySet. Calling MyWorker.cancel! jid in model code throws an undefined variable error. How can I access that jid in the model? How can actually cancel or delete that specific job? Thanks!
# The wiki code
class MyWorker
include Sidekiq::Worker
def perform(thing_id)
return if cancelled?
# do actual work
end
def cancelled?
Sidekiq.redis {|c| c.exists("cancelled-#{jid}") }
end
def self.cancel!(jid)
Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
end
end
# create job
jid = MyWorker.perform_async("foo")
# cancel job
MyWorker.cancel!(jid)
You can do this but it won't be efficient. It's a linear scan for find a scheduled job by JID.
require 'sidekiq/api'
Sidekiq::ScheduledSet.new.find_job(jid).try(:delete)
Alternatively your job can look to see if it's still relevant when it runs.
Ok, so turns out I had one question already answered. One of the code sets I included was a functionally similar version of the code from the wiki. The solution to the other question ("how can I access that jid in the model?") seems really obvious if you're not still new to programming, but basically: store the jid in a database column and then retrieve/update it whenever it's needed! Duh!
Earlier, I had posted this question – and thought it was resolved:
Rails background worker always fails first time, works second
However, after continuing with tests and development, the error is back again, but in a slightly different way.
I'm using Sidekiq (with Rails 3.2.8, Ruby 1.9.3) to run background processes, after_save. Below is the code for my model, worker, and controller.
Model:
class Post < ActiveRecord::Base
attr_accessible :description,
:name,
:key
after_save :process
def process
ProcessWorker.perform_async(id, key) if key.present?
true
end
def secure_url
key.match(/(.*\/)+(.*$)/)[1]
end
def nonsecure_url
key.gsub('https', 'http')
end
end
Worker:
class ProcessWorker
include Sidekiq::Worker
def perform(id, key)
post = Post.find(id)
puts post.nonsecure_url
end
end
(Updated) Controller:
def create
#user = current_user
#post = #user.posts.create(params[:post])
render nothing: true
end
Whenever jobs are first dispatched, no matter the method, they fail initially:
undefined method `gsub' for nil:NilClass
Then, they always succeed on the first retry.
I've come across the following github issue, that appears to be resolved – relating to this same issue:
https://github.com/mperham/sidekiq/issues/331
Here, people are saying that if they create initializers to initialize the ActiveRecord methods on the model, that it resolves their issue.
To accomplish this, I've tried creating an initializer in lib/initializers called sidekiq.rb, with the following, simply to initialize the methods on the Post model:
Post.first
Now, the first job created completes successfully the first time. This is good. However, a second job created fails the first time – and completes upon retry... putting me right back to where I started.
This is really blowing my mind – has anyone had the same issue? Any help is appreciated.
Change your model callback from after_save to after_commit for the create action. Sometimes, sidekiq can initialize your worker before the model actually finishes saving to the database.
after_commit :process, :on => :create
I have a process which takes generally a few seconds to complete so I'm trying to use delayed_job to handle it asynchronously. The job itself works fine, my question is how to go about polling the job to find out if it's done.
I can get an id from delayed_job by simply assigning it to a variable:
job = Available.delay.dosomething(:var => 1234)
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| id | priority | attempts | handler | last_error | run_at | locked_at | failed_at | locked_by | created_at | updated_at |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| 4037 | 0 | 0 | --- !ru... | | 2011-04-... | | | | 2011-04... | 2011-04-... |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
But as soon as it completes the job it deletes it and searching for the completed record returns an error:
#job=Delayed::Job.find(4037)
ActiveRecord::RecordNotFound: Couldn't find Delayed::Backend::ActiveRecord::Job with ID=4037
#job= Delayed::Job.exists?(params[:id])
Should I bother to change this, and maybe postpone the deletion of complete records? I'm not sure how else I can get a notification of it's status. Or is polling a dead record as proof of completion ok? Anyone else face something similar?
Let's start with the API. I'd like to have something like the following.
#available.working? # => true or false, so we know it's running
#available.finished? # => true or false, so we know it's finished (already ran)
Now let's write the job.
class AwesomeJob < Struct.new(:options)
def perform
do_something_with(options[:var])
end
end
So far so good. We have a job. Now let's write logic that enqueues it. Since Available is the model responsible for this job, let's teach it how to start this job.
class Available < ActiveRecord::Base
def start_working!
Delayed::Job.enqueue(AwesomeJob.new(options))
end
def working?
# not sure what to put here yet
end
def finished?
# not sure what to put here yet
end
end
So how do we know if the job is working or not? There are a few ways, but in rails it just feels right that when my model creates something, it's usually associated with that something. How do we associate? Using ids in database. Let's add a job_id on Available model.
While we're at it, how do we know that the job is not working because it already finished, or because it didn't start yet? One way is to actually check for what the job actually did. If it created a file, check if file exists. If it computed a value, check that result is written. Some jobs are not as easy to check though, since there may be no clear verifiable result of their work. For such case, you can use a flag or a timestamp in your model. Assuming this is our case, let's add a job_finished_at timestamp to distinguish a not yet ran job from an already finished one.
class AddJobIdToAvailable < ActiveRecord::Migration
def self.up
add_column :available, :job_id, :integer
add_column :available, :job_finished_at, :datetime
end
def self.down
remove_column :available, :job_id
remove_column :available, :job_finished_at
end
end
Alright. So now let's actually associate Available with its job as soon as we enqueue the job, by modifying the start_working! method.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options))
update_attribute(:job_id, job.id)
end
Great. At this point I could've written belongs_to :job, but we don't really need that.
So now we know how to write the working? method, so easy.
def working?
job_id.present?
end
But how do we mark the job finished? Nobody knows a job has finished better than the job itself. So let's pass available_id into the job (as one of the options) and use it in the job. For that we need to modify the start_working! method to pass the id.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options.merge(:available_id => id))
update_attribute(:job_id, job.id)
end
And we should add the logic into the job to update our job_finished_at timestamp when it's done.
class AwesomeJob < Struct.new(:options)
def perform
available = Available.find(options[:available_id])
do_something_with(options[:var])
# Depending on whether you consider an error'ed job to be finished
# you may want to put this under an ensure. This way the job
# will be deemed finished even if it error'ed out.
available.update_attribute(:job_finished_at, Time.current)
end
end
With this code in place we know how to write our finished? method.
def finished?
job_finished_at.present?
end
And we're done. Now we can simply poll against #available.working? and #available.finished? Also, you gain the convenience of knowing which exact job was created for your Available by checking #available.job_id. You can easily turn it into a real association by saying belongs_to :job.
I ended up using a combination of Delayed_Job with an after(job) callback which populates a memcached object with the same ID as the job created. This way I minimize the number of times I hit the database asking for the status of the job, instead polling the memcached object. And it contains the entire object I need from the completed job, so I don't even have a roundtrip request. I got the idea from an article by the github guys who did pretty much the same thing.
https://github.com/blog/467-smart-js-polling
and used a jquery plugin for the polling, which polls less frequently, and gives up after a certain number of retries
https://github.com/jeremyw/jquery-smart-poll
Seems to work great.
def after(job)
prices = Room.prices.where("space_id = ? AND bookdate BETWEEN ? AND ?", space_id.to_i, date_from, date_to).to_a
Rails.cache.fetch(job.id) do
bed = Bed.new(:space_id => space_id, :date_from => date_from, :date_to => date_to, :prices => prices)
end
end
I think that the best way would be to use the callbacks available in the delayed_job.
These are:
:success, :error and :after.
so you can put some code in your model with the after:
class ToBeDelayed
def perform
# do something
end
def after(job)
# do something
end
end
Because if you insist of using the obj.delayed.method, then you'll have to monkey patch Delayed::PerformableMethod and add the after method there.
IMHO it's far better than polling for some value which might be even backend specific (ActiveRecord vs. Mongoid, for instance).
The simplest method of accomplishing this is to change your polling action to be something similar to the following:
def poll
#job = Delayed::Job.find_by_id(params[:job_id])
if #job.nil?
# The job has completed and is no longer in the database.
else
if #job.last_error.nil?
# The job is still in the queue and has not been run.
else
# The job has encountered an error.
end
end
end
Why does this work? When Delayed::Job runs a job from the queue, it deletes it from the database if successful. If the job fails, the record stays in the queue to be ran again later, and the last_error attribute is set to the encountered error. Using the two pieces of functionality above, you can check for deleted records to see if they were successful.
The benefits to the method above are:
You get the polling effect that you were looking for in your original post
Using a simple logic branch, you can provide feedback to the user if there is an error in processing the job
You can encapsulate this functionality in a model method by doing something like the following:
# Include this in your initializers somewhere
class Queue < Delayed::Job
def self.status(id)
self.find_by_id(id).nil? ? "success" : (job.last_error.nil? ? "queued" : "failure")
end
end
# Use this method in your poll method like so:
def poll
status = Queue.status(params[:id])
if status == "success"
# Success, notify the user!
elsif status == "failure"
# Failure, notify the user!
end
end
I'd suggest that if it's important to get notification that the job has completed, then write a custom job object and queue that rather than relying upon the default job that gets queued when you call Available.delay.dosomething. Create an object something like:
class DoSomethingAvailableJob
attr_accessor options
def initialize(options = {})
#options = options
end
def perform
Available.dosomething(#options)
# Do some sort of notification here
# ...
end
end
and enqueue it with:
Delayed::Job.enqueue DoSomethingAvailableJob.new(:var => 1234)
The delayed_jobs table in your application is intended to provide the status of running and queued jobs only. It isn't a persistent table, and really should be as small as possible for performance reasons. Thats why the jobs are deleted immediately after completion.
Instead you should add field to your Available model that signifies that the job is done. Since I'm usually interested in how long the job takes to process, I add start_time and end_time fields. Then my dosomething method would look something like this:
def self.dosomething(model_id)
model = Model.find(model_id)
begin
model.start!
# do some long work ...
rescue Exception => e
# ...
ensure
model.finish!
end
end
The start! and finish! methods just record the current time and save the model. Then I would have a completed? method that your AJAX can poll to see if the job is finished.
def completed?
return true if start_time and end_time
return false
end
There are many ways to do this but I find this method simple and works well for me.
Basically what I want to do is to log an action on MyModel in the table of MyModelLog. Here's some pseudo code:
class MyModel < ActiveRecord::Base
validate :something
def something
# test
errors.add(:data, "bug!!")
end
end
I also have a model looking like this:
class MyModelLog < ActiveRecord::Base
def self.log_something
self.create(:log => "something happened")
end
end
In order to log I tried to :
Add MyModelLog.log_something in the something method of MyModel
Call MyModelLog.log_something on the after_validation callback of MyModel
In both cases the creation is rolled back when the validation fails because it's in the validation transaction. Of course I also want to log when validations fail. I don't really want to log in a file or somewhere else than the database because I need the relationships of log entries with other models and ability to do requests.
What are my options?
Nested transactions do seem to work in MySQL.
Here is what I tried on a freshly generated rails (with MySQL) project:
./script/generate model Event title:string --skip-timestamps --skip-fixture
./script/generate model EventLog error_message:text --skip-fixture
class Event < ActiveRecord::Base
validates_presence_of :title
after_validation_on_create :log_errors
def log_errors
EventLog.log_error(self) if errors.on(:title).present?
end
end
class EventLog < ActiveRecord::Base
def self.log_error(event)
connection.execute('BEGIN') # If I do transaction do then it doesn't work.
create :error_message => event.errors.on(:title)
connection.execute('COMMIT')
end
end
# And then in script/console:
>> Event.new.save
=> false
>> EventLog.all
=> [#<EventLog id: 1, error_message: "can't be blank", created_at: "2010-10-22 13:17:41", updated_at: "2010-10-22 13:17:41">]
>> Event.all
=> []
Maybe I have over simplified it, or missing some point.
Would this be a good fit for an Observer? I'm not sure, but I'm hoping that exists outside of the transaction... I have a similar need where I might want to delete a record on update...
I've solved a problem like this by taking advantage of Ruby's variable scoping. Basically I declared an error variable outside of a transaction block then catch, store log message, and raise the error again.
It looks something like this:
def something
error = nil
ActiveRecord::Base.transaction do
begin
# place codez here
rescue ActiveRecord::Rollback => e
error = e.message
raise ActiveRecord::Rollback
end
end
MyModelLog.log_something(error) unless error.nil?
end
By declaring the error variable outside of the transaction scope the contents of the variable persist even after the transaction has exited.
I am not sure if it applies to you, but i assume you are trying to save/create a model from your controller. In the controller it is easy to check the outcome of that action, and you most likely already do to provide the user with a useful flash; so you could easily log an appropriate message there.
I am also assuming you do not use any explicit transactions, so if you handle it in the controller, it is outside of the transaction (every save and destroy work in their own transaction).
What do you think?
MyModelLog.log_something should be done using a different connection.
You can make MyModelLog model always use a different connection by using establish_connection.
class MyModelLog < ActiveRecord::Base
establish_connection Rails.env # Use different connection
def self.log_something
self.create(:log => "something happened")
end
end
Not sure if this is the right way to do logging!!
You could use a nested transaction. This way the code in your callback executes in a different transaction than the failing validation. The Rails documentations for ActiveRecord::Transactions::ClassMethods discusses how this is done.