I am using the most basic version of delayed_job in a Rails app. I have the max time allowed for a delayed_job set at 10 minutes. I would like to get the hooks/callbacks working so I can do something after a job stop executing at the 10 minute mark.
I have this set in my rails app:
config.active_job.queue_adapter = :delayed_job
This is how I normally queue a job:
object.delay.object_action
The hook/callback example is for a named job but the basic, getting started steps are not for a named job. So I don't think I have a named job. Here is the example given to get the callbacks working:
class ParanoidNewsletterJob < NewsletterJob
def enqueue(job)
record_stat 'newsletter_job/enqueue'
end
def perform
emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
end
def before(job)
record_stat 'newsletter_job/start'
end
def after(job)
record_stat 'newsletter_job/after'
end
def success(job)
record_stat 'newsletter_job/success'
end
def error(job, exception)
Airbrake.notify(exception)
end
def failure(job)
page_sysadmin_in_the_middle_of_the_night
end
end
I would love to get the after or error hooks/callbacks to fire.
Where do I put these callbacks in my Rails app to have them fire for the basic delayed_job setup? If I should be using ActiveJob callbacks where do you put those callbacks given delayed_job is being used?
You cannot use object.delay.object_action convenience syntax if you want more advanced features like callbacks. The #delay convenience method will generate a job object that works similar to this:
# something like this is already defined in delayed_job
class MethodCallerJob
def initialize(object, method, *args)
#object = object
#method = method
#args = args
end
def perform
#object.send(#method, *#args)
end
end
# `object.delay.object_action` does the below automatically for you
# instantiates a job with your object and method call
job = MethodCallerJob.new(object, :object_action, [])
Delayed::Job.enqueue(job) # enqueues it for running later
then later, in the job worker, something like the below happens:
job = Delayed::Job.find(job_id) # whatever the id turned out to be
job.invoke_job # does all the things, including calling #perform and run any hooks
job.delete # if it was successful
You have to create what the delayed_job README calls "Custom Jobs", which are just plain POROs that have #perform defined at a minimum. Then you can customize it and add all the extra methods that delayed_job uses for extra features like max_run_time, queue_name, and the ones you want to use: callbacks & hooks.
Sidenote: The above info is for using delayed_job directly. All of the above is possible using ActiveJob as well. You just have to do it the ActiveJob way by reading the documentation & guides on how, just as I've linked you to the delayed_job README, above.
You can create delayed_job hooks/callback by something like this
module Delayed
module Plugins
class TestHooks < Delayed::Plugin
callbacks do |lifecycle|
lifecycle.before(:perform) do |_worker, job|
.....
end
end
end
end
end
And need this plugin to initializer
config/initializers/delayed_job.rb
require_relative 'path_to_test_plugin'
Delayed::Worker.plugins << Delayed::Plugins::TestHooks
Similar to perform there are also hooks for success failure and error.
And similar to 'before' you can also capture the 'after' hooks.
Related
Can I use the hooks of a delayed job to stop it before running?
If so how?
class Jobs::SomeJob < Struct.new
def before(job)
if should_not_run_job
# Do I raise an exception?
# Is there an official way to stop a Job from running?
end
end
def perform
# Working...
end
def queue_name
return "SomeJob";
end
end
Do I raise an error?
Which hook is recommended to use?
Raising an exception wouldn't work because then the retry mechanism would kick in and would retry running the job a couple of times.
Instead, I would just add a guard clause that in the first line of the perform method that returns without doing anything when the conditions is not true anymore. It depends on how your condition looks like but something like this might work for you:
def perform
return if job_should_not_run_anymore?
# Working
end
private
def job_should_not_run_anymore?
# Condition
end
When your condition is based on the job itself that is only available in a hook method like before but not in the perform method then I would store the result of the condition in an instance variable and check that variable in the perform method like this:
def before(job)
#outdated = job.run_at > 15.minutes.ago
end
def perform
return if #outdated
# Working
end
I have a job created with rails g job cleanUp.
Is where any option to check if Job is running? Something like this CleanUpJob.isRunning?
If where is no way to make it without additional gems, which will be the simplest? delayed_job?
Second thing to control Job is progress, any thoughts how to implement CleanUpJob.progress or progress_job should be my choice?
briefly:
I need to create a job with two methods (isRunning?, progress).
I don't really want additional tables if possible.
You can use Class: Sidekiq::ScheduledSet for this purpose.
Documentation here
This Class is used in Sidekiq web interface.
Example:
Save job id (jid) when set job. Then you can call it for queried instance
def is_running?
require 'sidekiq/api'
ss = Sidekiq::ScheduledSet.new
jobs = ss.select {|ret| ret.jid == self.jid}
jobs.any?
end
Or, you can set DB flag inside Job with around_perform hook.
class SomeJob < ActiveJob::Base
queue_as :some_job
around_perform do |job, block|
start_import_process_log job.arguments[0], job.arguments[1] || {}
block.call
finish_import_process_log
end
# ...
private
def start_import_process_log import_process, options={}
#some actions
end
def finish_import_process_log
end
end
In this example associated log record is created.
Or you can use before_perform/ after_perform.
In my practice I'm using creting log records on long tasks.
When I need to find and kill job as example - I'm using Sidekiq::ScheduledSet.
I'm using delayed_job and I'm very happy with it (especially with the workless extension).
But I'd like to set that ALL mails from my app are sent asynchronously.
Indeed, the solution offered for mailers
# without delayed_job
Notifier.signup(#user).deliver
# with delayed_job
Notifier.delay.signup(#user)
doesn't suit me because:
it is not easily maintainable
mails sent from gems are not sent asynchronously (devise, mailboxer)
I could use this kind of extension https://github.com/mhfs/devise-async but I'd rather figure out a solution for the whole app at once.
Can't I extend ActionMailer to override the .deliver method (like here https://stackoverflow.com/a/4316543/1620081 but it is 4 years old, like pretty much all the doc I found on the topic)?
I'm using Ruby 1.9 and Rails 3.2 with activerecord.
Thanks for support
A simple solution would be to write a utility method on the Notifier object as follows:
class Notifier
def self.deliver(message_type, *args)
self.delay.send(message_type, *args)
end
end
Send the sign up email as follows:
Notifier.deliver(:signup, #user)
The utility method provides a single point where if needed you could replace delayed job with resque or sidekiq solutions.
If you have your ActiveJob and concurrency library set-up is done documentation here.The most simple solution is to override you device send_devise_notification instance methods involved with the transactions mails like shown here
class User < ApplicationRecord
# whatever association you have here
devise :database_authenticatable, :confirmable
after_commit :send_pending_devise_notifications
# whatever methods you have here
protected
def send_devise_notification(notification, *args)
if new_record? || changed?
pending_devise_notifications << [notification, args]
else
render_and_send_devise_message(notification, *args)
end
end
private
def send_pending_devise_notifications
pending_devise_notifications.each do |notification, args|
render_and_send_devise_message(notification, *args)
end
pending_devise_notifications.clear
end
def pending_devise_notifications
#pending_devise_notifications ||= []
end
def render_and_send_devise_message(notification, *args)
message = devise_mailer.send(notification, self, *args)
# Deliver later with Active Job's `deliver_later`
if message.respond_to?(:deliver_later)
message.deliver_later
# Remove once we move to Rails 4.2+ only, as `deliver` is deprecated.
elsif message.respond_to?(:deliver_now)
message.deliver_now
else
message.deliver
end
end
end
I used resque-scheduler for delay jobs in previous code:
Resque.enqueue_in(options[:delay].seconds, self, context)
Now I want to include resque-status to do the job but have no idea how they can work together. The latest resque-status source code supports scheduler, as in the source code:
https://github.com/quirkey/resque-status/blob/master/lib/resque/plugins/status.rb
# Wrapper API to forward a Resque::Job creation API call into a Resque::Plugins::Status call.
# This is needed to be used with resque scheduler
# http://github.com/bvandenbos/resque-scheduler
def scheduled(queue, klass, *args)
self.enqueue_to(queue, self, *args)
end
end
But I'm not sure how to use it. Shall I just call SampleJob.scheduled(queue, myclass, :delay => delay) instead of SampleJob.create(options)?
======================================================================
Also, there is Support for resque-status (and other custom jobs):
https://github.com/bvandenbos/resque-scheduler
Some Resque extensions like resque-status use custom job classes with a slightly different API signature. Resque-scheduler isn't trying to support all existing and future custom job classes, instead it supports a schedule flag so you can extend your custom class and make it support scheduled job.
Let's pretend we have a JobWithStatus class called FakeLeaderboard
class FakeLeaderboard < Resque::JobWithStatus
def perform
# do something and keep track of the status
end
end
And then a schedule:
create_fake_leaderboards:
cron: "30 6 * * 1"
queue: scoring
custom_job_class: FakeLeaderboard
args:
rails_env: demo
description: "This job will auto-create leaderboards for our online demo and the status will update as the worker makes progress"
But it seems only for recurring jobs. I can find params of cron, but not delay. So how can I handle delayed jobs with it?
Thanks!
I had the same issue and I solved by myself by implementing a module which provide a runner for "statused" jobs.
module Resque # :nodoc:
# Module to include in your worker class to get resque-status
# and resque-scheduler integration
module ScheduledJobWithStatus
extend ActiveSupport::Concern
included do
# Include status functionalities
include Resque::Plugins::Status
end
# :nodoc:
module ClassMethods
# This method will use a custom worker class to enqueue jobs
# with resque-scheduler plugin with status support
def enqueue_at(timestamp, *args)
class_name = self.to_s # store class name since plain class object are not "serializable"
Resque.enqueue_at(timestamp, JobWithStatusRunner, class_name, *args)
end
# Identical to enqueue_at but takes number_of_seconds_from_now
# instead of a timestamp.
def enqueue_in(number_of_seconds_from_now, *args)
enqueue_at(Time.now + number_of_seconds_from_now, *args)
end
end
end
# Wrapper worker for enqueuing
class JobWithStatusRunner
# default queue for scheduling jobs with status
#queue = :delayed
# Receive jobs from {Resque::ScheduledJobWithStatus} queue them in Resque
# with support for status informations
def self.perform(status_klass, *args)
# Retrieve original worker class
klass = status_klass.to_s.constantize
# Check if supports status jobs
unless klass.included_modules.include? Resque::Plugins::Status
Rails.logger.warn("Class #{klass} doesn't support jobs with status")
return false
end
Rails.logger.debug("Enqueing jobs #{klass} with arguments #{args}")
klass.create(*args)
rescue NameError
Rails.logger.error("Unable to enqueue jobs for class #{status_klass} with args #{args}")
false
end
end
end
In this way you can enqueue your jobs with this simple syntax:
# simple worker class
class SleepJob
# This provides integrations with both resque-status and resque-scheduler
include Resque::ScheduledJobWithStatus
# Method triggered by resque
def perform
total = (options['length'] || 60).to_i
1.upto(total) { |i| at(i, total, "At #{i} of #{total}"); sleep(1) }
end
end
# run the job delayed
SleepJob.enqueue_in(5.minutes)
# or
SleepJob.enqueue_at(5.minutes.from_now)
Just drop the module in resque initializer or in lib folder. In the latter case remember to require it somewhere.
resque-scheduler will call the scheduled method on the supplied class whenever it creates the job for your workers to consume.
It also nicely passes both the queue name and class name as a string so you can create a class level method to handle scheduled job creation.
While Fabio's answer will solve the problem, it is a bit over-engineered to answer your specific question.
Assuming you have a JobWithStatus class that all of your resque-status workers inherit from, you need only add the scheduled method to it for it to work with the resque-scheduler as so:
class JobWithStatus
include Resque::Plugins::Status
def self.scheduled(queue, klass, *args)
Resque.constantize(klass).create(*args)
end
end
I cannot log messages from my delayed_job process. Here is the job that is being run.
class MyJob
def initialize(blahblah)
#blahblah = blahblah
#logger = Logger.new(File.join(Rails.root, 'log', 'delayed_job.log'))
end
def perform
#logger.add Logger::INFO, "logging from delayed_job"
#do stuff
end
end
I've tried various logging levels, and I have config.log_level = :debug in my environment configuration. I run delayed_job from monit. I'm using delayed_job 3.0.1 with ruby 1.9.3 and rails 3.0.10.
An explation could be that the job gets initialized only once on producer side. Then it gets serialized, delivered through the queue (database for example) and unserialized in the worker. But the initialize method is not being called in the worker process again. Only the perform method is called via send.
However you can reuse the workers logger to write to the log file:
class MyJob
def perform
say "performing like hell"
end
def say(text)
Delayed::Worker.logger.add(Logger::INFO, text)
end
end
Don't forget to restart the workers.
In RAILS_ROOT/config/initializers have a file called delayed_job_config.rb with these lines:
Delayed::Worker.logger = Rails.logger
Delayed::Worker.logger.auto_flushing = true
Remember to re-start your workers after doing this.
Let me know if this helps
I don't see why you would set the logger in the job. When I've done this I set the worker to to use a specific file on start e.g. Logger.new("log/worker_#{worker_number}") which ensures that each worker outputs to it's own file and you don't have to worry about multiple workers writing to the same file at the same time (messy).
Also, in plain ol' ruby you can call #logger.info "logging from delayed_job".
Finally, i'm pretty sure that 'perform' is called directly by your worker and instantiated, so you can refactor to:
class MyJob
def perform(blahblah)
#logger.add Logger::INFO, "logging from delayed_job"
#blahblah = blahblah
#do stuff
end
end
This is working just fine for me in Rails 3.2:
class FiveMinuteAggregateJob < Struct.new(:link, :timestamp)
def perform
Rails.logger.info 'yup.'
end
end