I've implemented Ahoy for tracking events inside my application and I want to move the processing into a background job.
My setup works fine when I call perform_now (the job gets processed), but changing to perform_later doesn't work. From the logs, it seems the job doesn't even get enqueued.
config/initializers/ahoy.rb
class Ahoy::Store < Ahoy::BaseStore
def track_visit(data)
AhoyTrackVisitJob.perform_later(#options, data)
end
def track_event(data)
AhoyTrackEventJob.perform_later(#options, data)
end
end
app/jobs/ahoy_track_event_job.rb
class AhoyTrackEventJob < ApplicationJob
queue_as :default
def perform(options, data)
Ahoy::DatabaseStore.new(options).track_event(data)
end
end
I've tried with both Sidekiq and SuckerPunch:
development.rb
config.active_job.queue_adapter = :sidekiq
Related
I am using Sidekiq
on my gemfile I have:
#gemfile
gem 'sidekiq', '~> 6.0.7'
Now my model looks like
class Special < ActiveRecord::Base
def set_queue
if send_at.present? # send at is a data time
SpecialQueueClientsJob.perform_at(self.send_at, self)
end
end
end
This is the code to my job and this works perfect when I just call it normally SpecialQueueClientsJob.perform_later(special: self)
class SpecialQueueClientsJob < ApplicationJob
queue_as :default
def perform(special)
...
end
end
https://github.com/mperham/sidekiq
I am getting an error undefined method perform_at' for SpecialQueueClientsJob `
I am looking at https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
Sidekiq allows you to schedule the time when a job will be executed.
You use perform_in(interval_in_seconds, *args) or
perform_at(timestamp, *args) rather than the standard
perform_async(*args):
MyWorker.perform_in(3.hours, 'mike', 1)
MyWorker.perform_at(3.hours.from_now, 'mike', 1) This is useful for
example if you want to send the user an email 3 hours after they sign
up. Jobs which are scheduled in the past are enqueued for immediate
execution.
I have a Rails app with 2 jobs (ImportCsvJob and ProcessCsvJob). So that I can visibly hint in the app that there are still jobs in the queue I have this helper method (inside application_helper.rb):
module ApplicationHelper
def queued_job_count
Sidekiq::Queue.new.size
end
end
Then I use it on my index controller which is then passed to the view for processing and giving the visual hint in the app.
def index
#still_have_jobs = !queued_job_count.zero?
end
However, this works when I still had 1 background Job (ImportCsvJob), but when I added the (ProcessCsvJob) it does not work anymore.
import_csv_job.rb
require 'open-uri'
class ImportCsvJob < ActiveJob::Base
queue_as :default
def perform(csv_record)
csv_record[:object_changes] = ApplicationController.helpers.generate_hash(csv_record[:object_changes])
ObjectRecord.create(csv_record)
end
end
process_csv_job.rb
class ProcessCsvJob < ActiveJob::Base
queue_as :default
def perform(csv_path)
csv_file = open(csv_path,'rb:UTF-8')
options = {
row_sep: :auto, col_sep: ",",
user_provided_headers: [:object_id, :object_type, :timestamp, :object_changes],
remove_empty_values: true,
headers_in_file: true
}
SmarterCSV.process(csv_file, options) do |array|
ImportCsvJob.perform_later(array.first)
end
end
end
and lastly, in the model where this is called:
ProcessCsvJob.perform_later(gdrive.uploaded_file_link)
When I try to debug in Rails console using Sidekiq::Queue.new.size, it still gives out 0.
Running:
redis-server
bundle exec sidekiq
A job that is executing is not enqueued anymore. The Sidekiq process has already popped it off the queue and is executing it. The queue is empty but the job is not finished yet.
So, basically I added a monitoring for sidekiq using the web interface to see what was happening:
And as I inspected, there were no enqueued tasks nor scheduled since most of the job is set to perform almost immediately (on parallel).
Thus here's my solution to know if the count of busy jobs:
module ApplicationHelper
def queued_job_count
Sidekiq::ProcessSet.new.first['busy']
end
end
and then on the index:
def index
#still_have_jobs = !queued_job_count.zero?
end
it works! :)
A Rails app has a background job that calls a service object as below
class ObjectUpdateJob < ActiveJob::Base
queue_as :default
def perform(object_id)
ObjectUpdater.call(object_id)
end
end
class ObjectUpdater
def self.call(*args)
if #object = Object.find( args[:object_id] )
#... update some stuff from external services
else
#... raise an error and prevent the background job from attempting to retry
end
end
end
This may be a simple question, but my experience with error handling is limited and I'd appreciate a second thought on this.
What is the 'Rails way' of handling this 'record does not exist' case, and ensuring the background job is informed so that no retries are attempted?
You could just put a guard clause into worker's perform action:
class ObjectUpdateJob < ActiveJob::Base
queue_as :default
def perform(object_id)
return unless Object.find(object_id)
ObjectUpdater.call(object_id)
end
end
But for more protection you can put some check to the job call:
ObjectUpdateJob.perform_now(object_id) if Object.find(object_id)
This way you just won't let the inexistent object to be processed.
If you use Sidekiq you can set a flag to :retry => false so that jobs do not retry automatically.
Additionally Sidekiq has built in error handling mechanism. Here is more information regarding the matter: https://github.com/mperham/sidekiq/wiki/Error-Handling
Hope that helps.
I'm using ActiveJob with delayed_job (4.0.6) in the background and I want to find a scheduled job to deleted it.
For instance, if I have
class MyClass
def my_method
perform_stuff
MyJob.set(wait: 1.month.from_now).perform_later(current_user)
end
end
Then, if I edit MyClass instance and call my_method again, I want to cancel that job and schedule a new one.
As suggested in this post http://www.sitepoint.com/delayed-jobs-best-practices, I added two columns to the Delayed Job Table:
table.integer :delayed_reference_id
table.string :delayed_reference_type
add_index :delayed_jobs, [:delayed_reference_id], :name => 'delayed_jobs_delayed_reference_id'
add_index :delayed_jobs, [:delayed_reference_type], :name => 'delayed_jobs_delayed_reference_type'
So this way I may find a delayed Job and destroy it. But I wanted to do that inside a ActiveJob class, to maintain the pattern of jobs in my project.
I wanted to do something like:
class MyJob < ActiveJob::Base
after_enqueue do |job|
user = self.arguments.first
job.delayed_reference_id = user.id,
job.delayed_reference_type = "User"
end
def perform(user)
delete_previous_job_if_exists(user_id)
end
def delete_previous_job_if_exists(user_id)
Delayed::Job.find_by(delayed_reference_id: 1, delayed_reference_type: 'User').delete
end
end
But that doesn't work.
Anyone had this kind of issue?
Two changes:
1. updated the after_enqueue callback so that you can update the
delayed_jobs table directly
2. fixed a typo where delayed_reference_id was hard coded as 1
This should work:
class MyJob < ActiveJob::Base
after_enqueue do |job|
user = self.arguments.first
delayed_job = Delayed::Job.find(job.provider_job_id)
delayed_job.update(delayed_reference_id:user.id,delayed_reference_type:'User')
end
def perform(user)
delete_previous_job_if_exists(user.id)
end
def delete_previous_job_if_exists(user_id)
Delayed::Job.find_by(delayed_reference_id: user_id, delayed_reference_type: 'User').delete
end
end
If you want to access the Delayed::Job from your worker, in an initializer, you can monkey patch the JobWrapper class.
module ActiveJob
module QueueAdapters
class DelayedJobAdapter
class JobWrapper
def perform(job)
end
end
end
end
end
Any idea how to get the Delayed::Job id from the ActiveJob enqueuing? When I enqueue a job I get back an instance of ActiveJob::Base with a #job_id, but that job id seems to be internal to ActiveJob. My best guess so far is just to walk down the most recently created jobs:
active_job_id = GenerateReportJob.perform_later(self.id).job_id
delayed_job = Delayed::Job.order(id: :desc).limit(5).detect do |job|
YAML.load(job.handler).job_data['job_id'] == active_job_id
end
but that seems all kinds of hacky. Kind of surprised ActiveJob isn't returning the ID from Delayed::Job, especially since that is what is explicitly returned when the job gets enqueued.
== EDIT
Looks like I'm not the only one (https://github.com/rails/rails/issues/18821)
In case anyone finds this in the future: Rails just accepted a patch to allow you to get this id from provider_job_id in Rails 5. You can get it to work with a patch like
ActiveJob::QueueAdapters::DelayedJobAdapter.singleton_class.prepend(Module.new do
def enqueue(job)
provider_job = super
job.provider_job_id = provider_job.id
provider_job
end
def enqueue_at(job, timestamp)
provider_job = super
job.provider_job_id = provider_job.id
provider_job
end
end)
Inspired by the answer of Beguene and some reverse engineering of the Rails 5 ActiveJob code, I have made it work with Rails 4.2 by
1) adding following code in lib/active_job/queue_adapters/delayed_job_adapter.rb or config/initializers/delayed_job.rb (both locations have worked):
# file: lib/active_job/queue_adapters/delayed_job_adapter.rb
module ActiveJob
module Core
# ID optionally provided by adapter
attr_accessor :provider_job_id
end
module QueueAdapters
class DelayedJobAdapter
class << self
def enqueue(job) #:nodoc:
delayed_job = Delayed::Job.enqueue(JobWrapper.new(job.serialize), queue: job.queue_name)
job.provider_job_id = delayed_job.id
delayed_job
end
def enqueue_at(job, timestamp) #:nodoc:
delayed_job = Delayed::Job.enqueue(JobWrapper.new(job.serialize), queue: job.queue_name, run_at: Time.at(timestamp))
job.provider_job_id = delayed_job.id
delayed_job
end
end
class JobWrapper #:nodoc:
attr_accessor :job_data
def initialize(job_data)
#job_data = job_data
end
def perform
Base.execute(job_data)
end
end
end
end
end
The attr_accessor :provider_job_id statement is needed in Rails 4.2, since it is used in the enqueue method and is not yet defined in 4.2.
Then we can make use of it like follows:
2) define our own ActiveJob class:
# file: app/jobs/my_job.rb
class MyJob < ActiveJob::Base
queue_as :default
def perform(object, performmethod = method(:method))
# Do something later
returnvalue = object.send(performmethod)
returnvalue
end
end
end
3) Now we can create a new job anywhere in the code:
job = MyJob.perform_later(Myobject, "mymethod")
This will put the method Myobject.mymethod into the queue.
4) The code in 1) helps us to find the Delayed Job that is associated with our job:
delayed_job = Delayed::Job.find(job.provider_job_id)
5) finally, we can do, whatever we need to do with the delayed_job, e.g. delete it:
delayed_job.delete
Note: in Rails 5, step 1) will not be needed anymore, since the exact same code is integral part of Rails 5.
I made it work in Rails 4.2 using the new patch from Rails 5 like this:
create the file lib/active_job/queue_adapters/delayed_job_adapter.rb
module ActiveJob
module QueueAdapters
class DelayedJobAdapter
class << self
def enqueue(job) #:nodoc:
delayed_job = Delayed::Job.enqueue(JobWrapper.new(job.serialize), queue: job.queue_name)
job.provider_job_id = delayed_job.id
delayed_job
end
def enqueue_at(job, timestamp) #:nodoc:
delayed_job = Delayed::Job.enqueue(JobWrapper.new(job.serialize), queue: job.queue_name, run_at: Time.at(timestamp))
job.provider_job_id = delayed_job.id
delayed_job
end
end
class JobWrapper #:nodoc:
attr_accessor :job_data
def initialize(job_data)
#job_data = job_data
end
def perform
Base.execute(job_data)
end
end
end
end
end
Instead of removing the job from the queue if it is cancelled you could model a cancellation of the job itself.
Then, when you come to run the GenerateReportJob you can first check for a cancellation of the report. If there is one then you can destroy the cancellation record and drop out of the report generation. If there is no cancellation then you can carry on as normal.