I want to put a heavy lifting method into a background job. Rails 5 ActiveJob or the use of Redis. Not sure which one I should use.
Basically there will be an API that uses a gem and to stuff things from that API call to my local database.
Controller:
...
before_action :get_api
def do_later
GetApiJob.perform_later(foo)
# Call foo later
end
def foo
#apis.map do |api|
puts api.title
end
end
private
def get_api
#apis = ShopifyAPI::Product.find(:all)
end
...
GetApiJob:
...
queue_as :default
def perform(a)
a
# Expect to see a list, if any, of api's name
end
...
When I call do_later it will put foo into a background job. Doing that sample code, I get:
ActiveJob::SerializationError
Should I be using Sidekiq for this?
ActiveJob is just a common interface between Rails application and different background job runners. You cannot use ActiveJob alone, you still need to add sidekiq (and Redis) or delayed_job or something else.
ActiveJob does the serialization of passed arguments in your Rails application and then deseriales this on the background job side. But you cannot serialize anything, you can only serialize basic types like Fixnum, String, Float, arrays of those basic values, hashes or ActiveRecord objects. ActiveRecord objects are serialized using GlobalId.
In your case you are passing a collection returned from shopify api client, which is not an ActiveRecord collection and ActiveJob doesn't know how to serialize it.
It will be best if you move api call to the background job itself.
Controller
# No before_action
def do_later
# No arguments, because we are fetching all products
GetApiJob.perform_later
end
GetApiJob
queue_as :default
def perform
# Fetch list of products
products = ShopifyAPI::Product.find(:all)
# Process list of products
end
Related
I am using the most basic version of delayed_job in a Rails app. I have the max time allowed for a delayed_job set at 10 minutes. I would like to get the hooks/callbacks working so I can do something after a job stop executing at the 10 minute mark.
I have this set in my rails app:
config.active_job.queue_adapter = :delayed_job
This is how I normally queue a job:
object.delay.object_action
The hook/callback example is for a named job but the basic, getting started steps are not for a named job. So I don't think I have a named job. Here is the example given to get the callbacks working:
class ParanoidNewsletterJob < NewsletterJob
def enqueue(job)
record_stat 'newsletter_job/enqueue'
end
def perform
emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
end
def before(job)
record_stat 'newsletter_job/start'
end
def after(job)
record_stat 'newsletter_job/after'
end
def success(job)
record_stat 'newsletter_job/success'
end
def error(job, exception)
Airbrake.notify(exception)
end
def failure(job)
page_sysadmin_in_the_middle_of_the_night
end
end
I would love to get the after or error hooks/callbacks to fire.
Where do I put these callbacks in my Rails app to have them fire for the basic delayed_job setup? If I should be using ActiveJob callbacks where do you put those callbacks given delayed_job is being used?
You cannot use object.delay.object_action convenience syntax if you want more advanced features like callbacks. The #delay convenience method will generate a job object that works similar to this:
# something like this is already defined in delayed_job
class MethodCallerJob
def initialize(object, method, *args)
#object = object
#method = method
#args = args
end
def perform
#object.send(#method, *#args)
end
end
# `object.delay.object_action` does the below automatically for you
# instantiates a job with your object and method call
job = MethodCallerJob.new(object, :object_action, [])
Delayed::Job.enqueue(job) # enqueues it for running later
then later, in the job worker, something like the below happens:
job = Delayed::Job.find(job_id) # whatever the id turned out to be
job.invoke_job # does all the things, including calling #perform and run any hooks
job.delete # if it was successful
You have to create what the delayed_job README calls "Custom Jobs", which are just plain POROs that have #perform defined at a minimum. Then you can customize it and add all the extra methods that delayed_job uses for extra features like max_run_time, queue_name, and the ones you want to use: callbacks & hooks.
Sidenote: The above info is for using delayed_job directly. All of the above is possible using ActiveJob as well. You just have to do it the ActiveJob way by reading the documentation & guides on how, just as I've linked you to the delayed_job README, above.
You can create delayed_job hooks/callback by something like this
module Delayed
module Plugins
class TestHooks < Delayed::Plugin
callbacks do |lifecycle|
lifecycle.before(:perform) do |_worker, job|
.....
end
end
end
end
end
And need this plugin to initializer
config/initializers/delayed_job.rb
require_relative 'path_to_test_plugin'
Delayed::Worker.plugins << Delayed::Plugins::TestHooks
Similar to perform there are also hooks for success failure and error.
And similar to 'before' you can also capture the 'after' hooks.
I would like to create a module that when included, makes every active record query execute as a block that I will wrap.
Specifically I'm using https://github.com/zendesk/active_record_shards to shard to multiple databases. In this scenario we have 10 duplicate databases (same schema but different client instances) and 1 unique database. By default, any calls to our models that belong to the 10 duplicate databases will be wrapped in an application controller around_action that does
ActiveRecord::Base.on_shard(database_name) do
yield
end
However we have a set of models that should only execute queries against this 1 unique database. Rather than execute every ActiveRecord query on those models using something like
ActiveRecord::Base.on_shard('unique_db') { Model.find(id) }
I would rather make a module that when included executes the active record query as a block inside the on_shard method so they always execute agains the unique_db
Let me know if I'm going down the wrong path and should just stick with my helper method which basically just shortens the length
module UniqeDB
def self.exec!
ActiveRecord::Base.on_shard('unique_db') do
yield
end
end
end
Thanks!
Update:
I was able to do something like this, however it overrides every method with (*args, &block) however some methods (like default_scope) only take a block and then it throws exceptions
included do
ActiveRecord::Base.methods.each do |name|
define_singleton_method name do |*args, &block|
ActiveRecord::Base.on_shard('unique_db') do
super(*args, &block)
end
end
end
end
Update: I opened an issue with the gem to see if there is something i'm missing
https://github.com/zendesk/active_record_shards/issues/87
Include the below module in any controller like 'include ExecuteInUniqueDB'
All action inside the controller will only in unique_db
module ExecuteInUniqueDB
def self.included(base)
base.class_eval do
around_action :run_on_unique_db
end
end
def run_on_unique_db(&block)
ActiveRecord::Base.on_shard('unique_db', &block)
end
end
I am using celluloid to perform some jobs in the background, some of the jobs are using the DB and for that I use the below code in a cell(actor):
ActiveRecord::Base.connection_pool.with_connection do
User.find(123)
end
In some cases I have some services, which are queuing the DB but with some other logic in them. Can I send them to ...with_connection or I need to wrap the call inside? Is there a better way of doing all this?
ActiveRecord::Base.connection_pool.with_connection do
SomeService.new(params).get_records
end
class SomeService
def initialise(param)
#params = params
end
def get_records
some_logic
ServicePersistenceModel.where(id: prams.id)
end
end
I used resque-scheduler for delay jobs in previous code:
Resque.enqueue_in(options[:delay].seconds, self, context)
Now I want to include resque-status to do the job but have no idea how they can work together. The latest resque-status source code supports scheduler, as in the source code:
https://github.com/quirkey/resque-status/blob/master/lib/resque/plugins/status.rb
# Wrapper API to forward a Resque::Job creation API call into a Resque::Plugins::Status call.
# This is needed to be used with resque scheduler
# http://github.com/bvandenbos/resque-scheduler
def scheduled(queue, klass, *args)
self.enqueue_to(queue, self, *args)
end
end
But I'm not sure how to use it. Shall I just call SampleJob.scheduled(queue, myclass, :delay => delay) instead of SampleJob.create(options)?
======================================================================
Also, there is Support for resque-status (and other custom jobs):
https://github.com/bvandenbos/resque-scheduler
Some Resque extensions like resque-status use custom job classes with a slightly different API signature. Resque-scheduler isn't trying to support all existing and future custom job classes, instead it supports a schedule flag so you can extend your custom class and make it support scheduled job.
Let's pretend we have a JobWithStatus class called FakeLeaderboard
class FakeLeaderboard < Resque::JobWithStatus
def perform
# do something and keep track of the status
end
end
And then a schedule:
create_fake_leaderboards:
cron: "30 6 * * 1"
queue: scoring
custom_job_class: FakeLeaderboard
args:
rails_env: demo
description: "This job will auto-create leaderboards for our online demo and the status will update as the worker makes progress"
But it seems only for recurring jobs. I can find params of cron, but not delay. So how can I handle delayed jobs with it?
Thanks!
I had the same issue and I solved by myself by implementing a module which provide a runner for "statused" jobs.
module Resque # :nodoc:
# Module to include in your worker class to get resque-status
# and resque-scheduler integration
module ScheduledJobWithStatus
extend ActiveSupport::Concern
included do
# Include status functionalities
include Resque::Plugins::Status
end
# :nodoc:
module ClassMethods
# This method will use a custom worker class to enqueue jobs
# with resque-scheduler plugin with status support
def enqueue_at(timestamp, *args)
class_name = self.to_s # store class name since plain class object are not "serializable"
Resque.enqueue_at(timestamp, JobWithStatusRunner, class_name, *args)
end
# Identical to enqueue_at but takes number_of_seconds_from_now
# instead of a timestamp.
def enqueue_in(number_of_seconds_from_now, *args)
enqueue_at(Time.now + number_of_seconds_from_now, *args)
end
end
end
# Wrapper worker for enqueuing
class JobWithStatusRunner
# default queue for scheduling jobs with status
#queue = :delayed
# Receive jobs from {Resque::ScheduledJobWithStatus} queue them in Resque
# with support for status informations
def self.perform(status_klass, *args)
# Retrieve original worker class
klass = status_klass.to_s.constantize
# Check if supports status jobs
unless klass.included_modules.include? Resque::Plugins::Status
Rails.logger.warn("Class #{klass} doesn't support jobs with status")
return false
end
Rails.logger.debug("Enqueing jobs #{klass} with arguments #{args}")
klass.create(*args)
rescue NameError
Rails.logger.error("Unable to enqueue jobs for class #{status_klass} with args #{args}")
false
end
end
end
In this way you can enqueue your jobs with this simple syntax:
# simple worker class
class SleepJob
# This provides integrations with both resque-status and resque-scheduler
include Resque::ScheduledJobWithStatus
# Method triggered by resque
def perform
total = (options['length'] || 60).to_i
1.upto(total) { |i| at(i, total, "At #{i} of #{total}"); sleep(1) }
end
end
# run the job delayed
SleepJob.enqueue_in(5.minutes)
# or
SleepJob.enqueue_at(5.minutes.from_now)
Just drop the module in resque initializer or in lib folder. In the latter case remember to require it somewhere.
resque-scheduler will call the scheduled method on the supplied class whenever it creates the job for your workers to consume.
It also nicely passes both the queue name and class name as a string so you can create a class level method to handle scheduled job creation.
While Fabio's answer will solve the problem, it is a bit over-engineered to answer your specific question.
Assuming you have a JobWithStatus class that all of your resque-status workers inherit from, you need only add the scheduled method to it for it to work with the resque-scheduler as so:
class JobWithStatus
include Resque::Plugins::Status
def self.scheduled(queue, klass, *args)
Resque.constantize(klass).create(*args)
end
end
I want to use delayed_job to execute a function from controller. The function is stored in module lib/site_request.rb:
module SiteRequest
def get_data(query)
...
end
handle_asynchronously :get_data
end
query_controller.rb:
class QueryController < ApplicationController
include SiteRequest
def index
#query = Query.find_or_initialize_by_word(params[:query])
if #query.new_record?
#query.save
get_data(#query)
flash[:notice] = "Request for data is sent to server."
end
end
end
I also tried to remove handle_asynchronously clause from module and use delay.get_data(#query), both do not executed silently (without delayed_job code works)
I had trouble trying to use the built-in delay methods myself, too. The pattern I settled on in my own coding was to enqueue DelayedJobs myself, giving them a payload object from which to work. This should work for you too and would seem to make sense even. (This way, you may not even need your SiteRequest module, for example.)
class MyModuleName < Struct.new(:query)
def perform
# TODO
end
end
Then, instead of calling get_data(query) after saving, enqueue with:
Delayed::Job.enqueue(MyModuleName.new(query))
I found the same issue. My environment is:
Ruby 2.1.7
Rails 4.2.6
activejob (4.2.6)
delayed_job (4.1.2)
delayed_job_active_record (4.1.1)
MY solutions:
Turn the module into a class.
Instantiate a object from the class and apply the method to the instance.
It seems that ActiveJob can enqueue only instances.
In your case:
Class SiteRequest
def initialize
end
def get_data(query)
...
end
handle_asynchronously :get_data
end
def index
...
q= SiteRequest.new
q.get_data(#query)
flash[:notice] = "Request for data is sent to server."
end
end