In my Rails app, I am using sidekiq for job scheduling to lift heavy tasks. So, I have this code in a lot of places:
if Rails.env.development? or Rails.env.test?
#Call the method directly
else #Rails.env.production?
#Call the job via sidekiq that calls the said method
end
Is there any way to clean this out? I am reading software design patterns these days. I am not able to apply that knowledge here. Can you suggest how can this be cleaned up or written in a way that is more manageable?
You can put
require 'sidekiq/testing'
Sidekiq::Testing.inline!
in your development.rb and test.rb config files to get the behaviour you are after. In your applications business logic you would remove the environment conditional and just call the worker (which will now run synchronously in test and development).
How about trying refactoring like following?
module Util
extend self
def execute_method_or_delay_its_execution(obj:, method_name:)
if Rails.env.development? or Rails.env.test?
obj.send(method_name)
else #Rails.env.production?
#Call the job via sidekiq that calls the said method
end
end
end
MyClass1
def my_method
Util.execute_method_or_delay_its_execution(obj: self, method_name: :my_method)
end
end
MyClass2
def my_method
Util.execute_method_or_delay_its_execution(obj: self, method_name: :my_method)
end
end
and then just invoke the methods on object as it should be and the internal delegation should take care of your desired direct execution or delayed execution
mc_1 = MyClass1.new
mc_1.my_method
mc_2 = MyClass2.new
mc_2.my_method
Hope that helps. Thanks.
Related
I am using the most basic version of delayed_job in a Rails app. I have the max time allowed for a delayed_job set at 10 minutes. I would like to get the hooks/callbacks working so I can do something after a job stop executing at the 10 minute mark.
I have this set in my rails app:
config.active_job.queue_adapter = :delayed_job
This is how I normally queue a job:
object.delay.object_action
The hook/callback example is for a named job but the basic, getting started steps are not for a named job. So I don't think I have a named job. Here is the example given to get the callbacks working:
class ParanoidNewsletterJob < NewsletterJob
def enqueue(job)
record_stat 'newsletter_job/enqueue'
end
def perform
emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
end
def before(job)
record_stat 'newsletter_job/start'
end
def after(job)
record_stat 'newsletter_job/after'
end
def success(job)
record_stat 'newsletter_job/success'
end
def error(job, exception)
Airbrake.notify(exception)
end
def failure(job)
page_sysadmin_in_the_middle_of_the_night
end
end
I would love to get the after or error hooks/callbacks to fire.
Where do I put these callbacks in my Rails app to have them fire for the basic delayed_job setup? If I should be using ActiveJob callbacks where do you put those callbacks given delayed_job is being used?
You cannot use object.delay.object_action convenience syntax if you want more advanced features like callbacks. The #delay convenience method will generate a job object that works similar to this:
# something like this is already defined in delayed_job
class MethodCallerJob
def initialize(object, method, *args)
#object = object
#method = method
#args = args
end
def perform
#object.send(#method, *#args)
end
end
# `object.delay.object_action` does the below automatically for you
# instantiates a job with your object and method call
job = MethodCallerJob.new(object, :object_action, [])
Delayed::Job.enqueue(job) # enqueues it for running later
then later, in the job worker, something like the below happens:
job = Delayed::Job.find(job_id) # whatever the id turned out to be
job.invoke_job # does all the things, including calling #perform and run any hooks
job.delete # if it was successful
You have to create what the delayed_job README calls "Custom Jobs", which are just plain POROs that have #perform defined at a minimum. Then you can customize it and add all the extra methods that delayed_job uses for extra features like max_run_time, queue_name, and the ones you want to use: callbacks & hooks.
Sidenote: The above info is for using delayed_job directly. All of the above is possible using ActiveJob as well. You just have to do it the ActiveJob way by reading the documentation & guides on how, just as I've linked you to the delayed_job README, above.
You can create delayed_job hooks/callback by something like this
module Delayed
module Plugins
class TestHooks < Delayed::Plugin
callbacks do |lifecycle|
lifecycle.before(:perform) do |_worker, job|
.....
end
end
end
end
end
And need this plugin to initializer
config/initializers/delayed_job.rb
require_relative 'path_to_test_plugin'
Delayed::Worker.plugins << Delayed::Plugins::TestHooks
Similar to perform there are also hooks for success failure and error.
And similar to 'before' you can also capture the 'after' hooks.
I'm Just wondering, Is there a chaining concept in ruby.
I wanted to execute series of async tasks or methods one after the other. Is it possible?
Thanks,
Ravi
You might want to create a process class, something like:
class MyProcess
PROCESS_STEPS = %w(
step_one
step_two
step_three
)
class << self
def next_step
new.next_step
end
end # Class Methods
#======================================================================
# Instance Methods
#======================================================================
def next_step
PROCESS_STEPS.each do |process_step|
send(process_step) if send("do_#{process_step}?")
end
end
def step_one
# execute step one task
end
def do_step_one?
# some logic
end
def step_two
# execute step two task
end
def do_step_two?
# some logic
end
def step_three
# execute step three task
end
def do_step_three?
# some logic
end
end
You would probably put that in:
app
|- processes
| |- my_process.rb
Then, at the end of each task, do something like:
MyProcess.next_step
Javascript, where Promises were first introduced, is also synchronous, promises being an abstraction over callbacks in the strictest sense
There are concurrency libraries for Ruby, some of which capture the spirit of Promises to a certain extent, a google search for promise.rb yields some promising results:
https://github.com/lgierth/promise.rb
https://github.com/ruby-concurrency/concurrent-ruby
Perhaps these are not idiomatic ruby, but they do offer some useful paradigms
As far as i can tell, promise.rb is the most commonly used gem for an async mechanism adhering to the js Promise/A+ standard.
This article does a decent job of introducing it: https://medium.com/#gauravbasti2006/lets-keep-our-promise-in-ruby-e45925182fdc
concurrent-ruby is most widely used to implement concurrency related features like promises, similar to other widely used languages. The documentation is pretty straightforward as well:
https://github.com/ruby-concurrency/concurrent-ruby/blob/master/docs-source/promises.in.md
For chaining asynchronous tasks you can use the following:
https://github.com/ruby-concurrency/concurrent-ruby/blob/master/docs-source/promises.in.md#chaining
I have a job created with rails g job cleanUp.
Is where any option to check if Job is running? Something like this CleanUpJob.isRunning?
If where is no way to make it without additional gems, which will be the simplest? delayed_job?
Second thing to control Job is progress, any thoughts how to implement CleanUpJob.progress or progress_job should be my choice?
briefly:
I need to create a job with two methods (isRunning?, progress).
I don't really want additional tables if possible.
You can use Class: Sidekiq::ScheduledSet for this purpose.
Documentation here
This Class is used in Sidekiq web interface.
Example:
Save job id (jid) when set job. Then you can call it for queried instance
def is_running?
require 'sidekiq/api'
ss = Sidekiq::ScheduledSet.new
jobs = ss.select {|ret| ret.jid == self.jid}
jobs.any?
end
Or, you can set DB flag inside Job with around_perform hook.
class SomeJob < ActiveJob::Base
queue_as :some_job
around_perform do |job, block|
start_import_process_log job.arguments[0], job.arguments[1] || {}
block.call
finish_import_process_log
end
# ...
private
def start_import_process_log import_process, options={}
#some actions
end
def finish_import_process_log
end
end
In this example associated log record is created.
Or you can use before_perform/ after_perform.
In my practice I'm using creting log records on long tasks.
When I need to find and kill job as example - I'm using Sidekiq::ScheduledSet.
I cannot log messages from my delayed_job process. Here is the job that is being run.
class MyJob
def initialize(blahblah)
#blahblah = blahblah
#logger = Logger.new(File.join(Rails.root, 'log', 'delayed_job.log'))
end
def perform
#logger.add Logger::INFO, "logging from delayed_job"
#do stuff
end
end
I've tried various logging levels, and I have config.log_level = :debug in my environment configuration. I run delayed_job from monit. I'm using delayed_job 3.0.1 with ruby 1.9.3 and rails 3.0.10.
An explation could be that the job gets initialized only once on producer side. Then it gets serialized, delivered through the queue (database for example) and unserialized in the worker. But the initialize method is not being called in the worker process again. Only the perform method is called via send.
However you can reuse the workers logger to write to the log file:
class MyJob
def perform
say "performing like hell"
end
def say(text)
Delayed::Worker.logger.add(Logger::INFO, text)
end
end
Don't forget to restart the workers.
In RAILS_ROOT/config/initializers have a file called delayed_job_config.rb with these lines:
Delayed::Worker.logger = Rails.logger
Delayed::Worker.logger.auto_flushing = true
Remember to re-start your workers after doing this.
Let me know if this helps
I don't see why you would set the logger in the job. When I've done this I set the worker to to use a specific file on start e.g. Logger.new("log/worker_#{worker_number}") which ensures that each worker outputs to it's own file and you don't have to worry about multiple workers writing to the same file at the same time (messy).
Also, in plain ol' ruby you can call #logger.info "logging from delayed_job".
Finally, i'm pretty sure that 'perform' is called directly by your worker and instantiated, so you can refactor to:
class MyJob
def perform(blahblah)
#logger.add Logger::INFO, "logging from delayed_job"
#blahblah = blahblah
#do stuff
end
end
This is working just fine for me in Rails 3.2:
class FiveMinuteAggregateJob < Struct.new(:link, :timestamp)
def perform
Rails.logger.info 'yup.'
end
end
I am trying to write a rspec for delayed job.
Currently I use delayed job as follows:
class IncomingMailsController < ApplicationController
...
MailingJob.new(#incoming_mail.id).perform
...
end
Then in /lib/mailing_job.rb:
class MailingJob < Struct.new(:mailing_id)
def perform
.......
How can I test that with rspec? Right now I have:
/spec/lib/mailing_job/mailingjob_spec.rb
require 'spec_helper'
describe MailingJob do
include DelayedJobSpecHelper
it "should have been worked on if I do something that queues jobs" do
#incoming_mail = IncomingMail.create(.........)
MailingJob.new(#incoming_mail.id).perform
#IncomingMail.method_that_queues_jobs
work_off
MailingJob.should be_worked_on
end
end
/spec/lib/delayed_job_spec_helper.rb
module DelayedJobSpecHelper
# http://erikonrails.snowedin.net/?p=230
def work_off
Delayed::Job.all.each do |job|
job.payload_object.perform
job.destroy
end
end
end
But this errors with:
1) MailingJob should have been worked on if I do something that queues jobs
Failure/Error: MailingJob.should be_worked_on
NoMethodError:
undefined method `worked_on?' for MailingJob:Class
# ./spec/lib/mailing_job/mailingjob_spec.rb:19
Ideas? Thanks
You're queueing your job incorrectly in the controller. perform is called by the worker, not by you.
Delayed::Job.enqueue MailingJob.new(#incoming_mail.id)
Similarly, don't call perform in your spec, that's what work_off does.
The structure of the spec should be this:
Access controller method that queues the job
Tell DelayedJob to work off the queue
Check to see that it did what it was supposed to do
I'm not sure where be_worked_on comes in, as worked_on? is not defined anywhere in DelayedJob (and indeed, is the cause of your error). I would instead check something was done, like a mail was sent, or whatever your job is supposed to do.