I have problem with sending recurring mails with Sidekiq and Sidetiq. I'v tried almost everything and I didn't find the solution.
I have Sidekiq worker which looks like this:
class InvoiceEmailSender
include Sidekiq::Worker
include Sidetiq::Schedulable
recurrence {minutely(2)}
def perform(invoice_id, action)
#invoice = Invoice.find(invoice_id.to_i)
if action == "invoice"
send_invoice
else
send_reminder
end
end
private
def send_invoice
if #invoice.delivery_date == Date.today
InvoiceMailer.delay.send_invoice(#invoice)
else
InvoiceMailer.delay_for(#invoice.delivery_date.to_time).send_invoice(#invoice)
end
end
def send_reminder
InvoiceMailer.delay.send_invoice_reminder(#invoice) unless #invoice.paid?
end
end
End in controller I use it in this way:
InvoiceEmailSender.perform_async(#invoice.id, "invoice")
And when I try to send this emails I have the following error in sidekiq console:
2014-08-26T05:36:01.107Z 4664 TID-otcc5idts WARN: {"retry"=>true, "queue"=>"default", "class"=>"InvoiceEmailSender", "args"=>[1409031120.0, 1409031240.0], "jid"=>"06dc732831c24e1a6f78d929", "enqueued_at"=>1409031120.7438812, "error_message"=>"Couldn't find Invoice with 'id'=1409031120", "error_class"=>"ActiveRecord::RecordNotFound", "failed_at"=>1409031249.1003482, "retry_count"=>2, "retried_at"=>1409031361.1066737}
2014-08-26T05:36:01.107Z 4664 TID-otcc5idts WARN: Couldn't find Invoice with 'id'=1409031120
2014-08-26T05:36:01.107Z 4664 TID-otcc5idts WARN: /home/mateusz/.rvm/gems/ruby-2.0.0-p0#rails4/gems/activerecord-4.1.2/lib/active_record/relation/finder_methods.rb:320:in `raise_record_not_found_exception!'
In sideiq web monitor in scheduled tab it looks like this:
Please help because I have not idea what is going on...
The data passed in looks like epoch timestamps, turns out Sidetiq passes the last and current times as the 2 parameters to your worker, according to the documentation.
I'm not sure how you go about having custom parameters with a scheduled worker, you'll probably need a different strategy like instead of trying to create more scheduled workers, just have 1 (or two, since it looks like you made this class do 2 jobs) scheduled worker(s) that processes a list of work to do every so often.
Related
Something simple, I'm sure but I've been searching and can't find an answer.
in brief: I want to set up a daily mailer to email a lists of tasks daily.
I have a worker (scheduled every minute, and only puts'ing for dev):
class DailyReminderWorker
include Sidekiq::Worker
include Sidetiq::Schedulable
recurrence do
hourly.minute_of_hour(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59)
end
def perform
User.find_each do |user|
#user = user.name
puts "user name is #{#user}"
#reminder = Remindarrr.where(user_id: user.id)
#reminder.each do |r|
puts r.title
end
end
end
end
I know I can call this with the following in a controller:
DailyReminderWorker.perform_async
This works and outputs every minute but every time the page refreshes it also fires. (less useful for a daily mailer.)
How do you call the worker to queue the job without it firing immediately?
Where would you put the perform.async?
Thanks a lot!
If you are on a Linux platform, just use cron. very easy to setup.
Some Sidekiq jobs in my app are scheduled to change the state of a resource to cancelled unless a user responds within a certain timeframe. There is a lot of information about how to best accomplish this task, but none of it actually cancels the job.
To cancel a job, the code in the wiki says:
class MyWorker
include Sidekiq::Worker
def perform(thing_id)
return if cancelled?
thing = Thing.find thing_id
thing.renege!
end
def cancelled?
Sidekiq.redis {|c| c.exists("cancelled-#{jid}") }
end
def self.cancel!(jid)
Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
end
end
Yet here it's suggested that I do something like
def perform(thing_id)
thing = Thing.find thing_id
while !cancel?(thing)
thing.ignore!
end
end
def cancel?(thing_id)
thing = Thing.find thing_id
thing.matched? || thing.passed?
end
What's confusing about this and similar code on the wiki is none of it actually cancels the job. The above example just performs an update on thing if cancelled? returns false (as it should), but doesn't cancel if and when it returns true in the future. It just fails with an aasm transition error message and gets sent to the RetrySet. Calling MyWorker.cancel! jid in model code throws an undefined variable error. How can I access that jid in the model? How can actually cancel or delete that specific job? Thanks!
# The wiki code
class MyWorker
include Sidekiq::Worker
def perform(thing_id)
return if cancelled?
# do actual work
end
def cancelled?
Sidekiq.redis {|c| c.exists("cancelled-#{jid}") }
end
def self.cancel!(jid)
Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
end
end
# create job
jid = MyWorker.perform_async("foo")
# cancel job
MyWorker.cancel!(jid)
You can do this but it won't be efficient. It's a linear scan for find a scheduled job by JID.
require 'sidekiq/api'
Sidekiq::ScheduledSet.new.find_job(jid).try(:delete)
Alternatively your job can look to see if it's still relevant when it runs.
Ok, so turns out I had one question already answered. One of the code sets I included was a functionally similar version of the code from the wiki. The solution to the other question ("how can I access that jid in the model?") seems really obvious if you're not still new to programming, but basically: store the jid in a database column and then retrieve/update it whenever it's needed! Duh!
given the delayed job worker,
class UserCommentsListWorker
attr_accessor :opts
def initialize opts = {}
#opts = opts
end
def perform
UserCommentsList.new(#opts)
end
def before job
p 'before hook', job
end
def after job
p 'after hook', job
end
def success job
p 'success hook', job
end
def error job, exception
p '4', exception
end
def failure job
p '5', job
end
def enqueue job
p '-1', job
end
end
When I run Delayed::Job.enqueue UserCommentsListWorker.new(client: client) from a rails console, I can get repeated sequences of print statements and a proper delayed job lifecyle even hooks to print including the feedback from the worker that the job was a success.
Including the same call to run the worker via a standard rails controller endpoint like;
include OctoHelper
include QueryHelper
include ObjHelper
include StructuralHelper
class CommentsController < ApplicationController
before_filter :authenticate_user!
def index
if params['updateCache'] == 'true'
client = build_octoclient current_user.octo_token
Delayed::Job.enqueue UserCommentsListWorker.new(client: client)
end
end
end
I'm noticing that the worker will run and created the delayed job, but none of the hooks get called and the worker nevers logs the job as completed.
Notice the screenshot,
Jobs 73,75,76 were all triggered via a roundtrip to the above referenced endpoint while job 74 was triggered via the rails console, what is wrong with my setup and/or what am I failing to notice in this process? I will stress that the first time the webserver hits the above controller endpoint, the job queues and runs properly but all subsequent instances where the job should run properly appear to be doing nothing and giving me no feedback in the process.
i would also highlight that i'm never seeing the failure, error or enqueue hooks run.
thanks :)
The long and the short of the answer to this problem was that if you notice, i was attempting to store a client object in the delayed job notification which was causing problems, so therefore, don't store complex objects in the job, just work with basic data ids 1 or strings foo or booleans true etc. capisce?
I want to trigger mail to be sent one hour before an appointment comes up. I am using the at field from the #appointment instance variable.
class AppointmentController < ApplicationController
def create
if DateTime.now + 1.hour > #appointment.at
AppointmentReminder.send_appointment_email(#appointment).deliver_now
end
end
end
This works if the appointment was created within an hour, but if the appointment was created in the future... then our poor customer won't be notified. Is there a mechanism where Rails can automatically deliver the email at the right time? I don't want to use a cronjob or rake task.
I'd recommend looking at background processing systems like Sidekiq or Sucker Punch which can be configured to perform jobs "later".
This way when the appointment is created you can schedule the job to execute at the correct time. You'll need to add checks to make sure when the job finally runs that it's still legitimate, etc.
http://sidekiq.org
https://github.com/brandonhilkert/sucker_punch
As you tagged your question as related to rails 4.2 then Active Job exactly what you need.
You could use whenever to run a block of code on a schedule. Say, ever 5 minutes, looks for appointments that are starting within the next hour and send an email.
To prevent multiple servers from sending an email, you could have a status on the appointment to keep track of if the email has been sent.
Then, using postgres, you can use this SQL to grab records to send and use the database to decide which server is going to send out a specific email:
Email.find_by_sql("WITH updated AS (
UPDATE emails SET status = 'processing' where lower(status) = 'new' RETURNING id
)
SELECT *
FROM emails
WHERE id IN (SELECT id FROM updated)
order by id asc
")
I will share how I have done it. It works just fine.
First, install whenever gem.
You should have your mailer. Here is mine:
class WeeklyDigestMailer < ApplicationMailer
default :from => "bla#bla.org"
# Subject can be set in your I18n file at config/locales/en.yml
# with the following lookup:
#
# en.weekly_digest_mailer.weekly_promos.subject
#
helper ApplicationHelper
def weekly_promos(suscriptor, promos)
#promos = promos
mail(:to => "<#{suscriptor.email}>", :subject => "Mercadillo digital semanal")
end
end
Of course, you need to style your view.
Then, you create a rake task (in lib/tasks). Just like this:
desc 'send digest email'
task send_weekly_email: :environment do
#promociones = Promo.where("validez > ?", Time.zone.now).order("created_at DESC")
if (#promociones.count > 0)
#suscriptors = Suscriptor.where(email_confirmation: true)
#suscriptors.each do |suscriptor|
WeeklyDigestMailer.weekly_promos(suscriptor, #promociones).deliver_now
end
end
end
Finally, you configure your schedule with whenever gem. As I want to send the mails all thrusdays at 9 am, I just put it:
every :thursday, at: '9:00 am' do # Use any day of the week or :weekend, :weekday
rake "send_weekly_email"
end
One important point: since you are using a rake task, use deliver_now instead of deliver_later because if the task finish before all emails have been sent, the rest will be undelivered.
That's all.
I have a process which takes generally a few seconds to complete so I'm trying to use delayed_job to handle it asynchronously. The job itself works fine, my question is how to go about polling the job to find out if it's done.
I can get an id from delayed_job by simply assigning it to a variable:
job = Available.delay.dosomething(:var => 1234)
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| id | priority | attempts | handler | last_error | run_at | locked_at | failed_at | locked_by | created_at | updated_at |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
| 4037 | 0 | 0 | --- !ru... | | 2011-04-... | | | | 2011-04... | 2011-04-... |
+------+----------+----------+------------+------------+-------------+-----------+-----------+-----------+------------+-------------+
But as soon as it completes the job it deletes it and searching for the completed record returns an error:
#job=Delayed::Job.find(4037)
ActiveRecord::RecordNotFound: Couldn't find Delayed::Backend::ActiveRecord::Job with ID=4037
#job= Delayed::Job.exists?(params[:id])
Should I bother to change this, and maybe postpone the deletion of complete records? I'm not sure how else I can get a notification of it's status. Or is polling a dead record as proof of completion ok? Anyone else face something similar?
Let's start with the API. I'd like to have something like the following.
#available.working? # => true or false, so we know it's running
#available.finished? # => true or false, so we know it's finished (already ran)
Now let's write the job.
class AwesomeJob < Struct.new(:options)
def perform
do_something_with(options[:var])
end
end
So far so good. We have a job. Now let's write logic that enqueues it. Since Available is the model responsible for this job, let's teach it how to start this job.
class Available < ActiveRecord::Base
def start_working!
Delayed::Job.enqueue(AwesomeJob.new(options))
end
def working?
# not sure what to put here yet
end
def finished?
# not sure what to put here yet
end
end
So how do we know if the job is working or not? There are a few ways, but in rails it just feels right that when my model creates something, it's usually associated with that something. How do we associate? Using ids in database. Let's add a job_id on Available model.
While we're at it, how do we know that the job is not working because it already finished, or because it didn't start yet? One way is to actually check for what the job actually did. If it created a file, check if file exists. If it computed a value, check that result is written. Some jobs are not as easy to check though, since there may be no clear verifiable result of their work. For such case, you can use a flag or a timestamp in your model. Assuming this is our case, let's add a job_finished_at timestamp to distinguish a not yet ran job from an already finished one.
class AddJobIdToAvailable < ActiveRecord::Migration
def self.up
add_column :available, :job_id, :integer
add_column :available, :job_finished_at, :datetime
end
def self.down
remove_column :available, :job_id
remove_column :available, :job_finished_at
end
end
Alright. So now let's actually associate Available with its job as soon as we enqueue the job, by modifying the start_working! method.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options))
update_attribute(:job_id, job.id)
end
Great. At this point I could've written belongs_to :job, but we don't really need that.
So now we know how to write the working? method, so easy.
def working?
job_id.present?
end
But how do we mark the job finished? Nobody knows a job has finished better than the job itself. So let's pass available_id into the job (as one of the options) and use it in the job. For that we need to modify the start_working! method to pass the id.
def start_working!
job = Delayed::Job.enqueue(AwesomeJob.new(options.merge(:available_id => id))
update_attribute(:job_id, job.id)
end
And we should add the logic into the job to update our job_finished_at timestamp when it's done.
class AwesomeJob < Struct.new(:options)
def perform
available = Available.find(options[:available_id])
do_something_with(options[:var])
# Depending on whether you consider an error'ed job to be finished
# you may want to put this under an ensure. This way the job
# will be deemed finished even if it error'ed out.
available.update_attribute(:job_finished_at, Time.current)
end
end
With this code in place we know how to write our finished? method.
def finished?
job_finished_at.present?
end
And we're done. Now we can simply poll against #available.working? and #available.finished? Also, you gain the convenience of knowing which exact job was created for your Available by checking #available.job_id. You can easily turn it into a real association by saying belongs_to :job.
I ended up using a combination of Delayed_Job with an after(job) callback which populates a memcached object with the same ID as the job created. This way I minimize the number of times I hit the database asking for the status of the job, instead polling the memcached object. And it contains the entire object I need from the completed job, so I don't even have a roundtrip request. I got the idea from an article by the github guys who did pretty much the same thing.
https://github.com/blog/467-smart-js-polling
and used a jquery plugin for the polling, which polls less frequently, and gives up after a certain number of retries
https://github.com/jeremyw/jquery-smart-poll
Seems to work great.
def after(job)
prices = Room.prices.where("space_id = ? AND bookdate BETWEEN ? AND ?", space_id.to_i, date_from, date_to).to_a
Rails.cache.fetch(job.id) do
bed = Bed.new(:space_id => space_id, :date_from => date_from, :date_to => date_to, :prices => prices)
end
end
I think that the best way would be to use the callbacks available in the delayed_job.
These are:
:success, :error and :after.
so you can put some code in your model with the after:
class ToBeDelayed
def perform
# do something
end
def after(job)
# do something
end
end
Because if you insist of using the obj.delayed.method, then you'll have to monkey patch Delayed::PerformableMethod and add the after method there.
IMHO it's far better than polling for some value which might be even backend specific (ActiveRecord vs. Mongoid, for instance).
The simplest method of accomplishing this is to change your polling action to be something similar to the following:
def poll
#job = Delayed::Job.find_by_id(params[:job_id])
if #job.nil?
# The job has completed and is no longer in the database.
else
if #job.last_error.nil?
# The job is still in the queue and has not been run.
else
# The job has encountered an error.
end
end
end
Why does this work? When Delayed::Job runs a job from the queue, it deletes it from the database if successful. If the job fails, the record stays in the queue to be ran again later, and the last_error attribute is set to the encountered error. Using the two pieces of functionality above, you can check for deleted records to see if they were successful.
The benefits to the method above are:
You get the polling effect that you were looking for in your original post
Using a simple logic branch, you can provide feedback to the user if there is an error in processing the job
You can encapsulate this functionality in a model method by doing something like the following:
# Include this in your initializers somewhere
class Queue < Delayed::Job
def self.status(id)
self.find_by_id(id).nil? ? "success" : (job.last_error.nil? ? "queued" : "failure")
end
end
# Use this method in your poll method like so:
def poll
status = Queue.status(params[:id])
if status == "success"
# Success, notify the user!
elsif status == "failure"
# Failure, notify the user!
end
end
I'd suggest that if it's important to get notification that the job has completed, then write a custom job object and queue that rather than relying upon the default job that gets queued when you call Available.delay.dosomething. Create an object something like:
class DoSomethingAvailableJob
attr_accessor options
def initialize(options = {})
#options = options
end
def perform
Available.dosomething(#options)
# Do some sort of notification here
# ...
end
end
and enqueue it with:
Delayed::Job.enqueue DoSomethingAvailableJob.new(:var => 1234)
The delayed_jobs table in your application is intended to provide the status of running and queued jobs only. It isn't a persistent table, and really should be as small as possible for performance reasons. Thats why the jobs are deleted immediately after completion.
Instead you should add field to your Available model that signifies that the job is done. Since I'm usually interested in how long the job takes to process, I add start_time and end_time fields. Then my dosomething method would look something like this:
def self.dosomething(model_id)
model = Model.find(model_id)
begin
model.start!
# do some long work ...
rescue Exception => e
# ...
ensure
model.finish!
end
end
The start! and finish! methods just record the current time and save the model. Then I would have a completed? method that your AJAX can poll to see if the job is finished.
def completed?
return true if start_time and end_time
return false
end
There are many ways to do this but I find this method simple and works well for me.