I have a Rails app with 2 jobs (ImportCsvJob and ProcessCsvJob). So that I can visibly hint in the app that there are still jobs in the queue I have this helper method (inside application_helper.rb):
module ApplicationHelper
def queued_job_count
Sidekiq::Queue.new.size
end
end
Then I use it on my index controller which is then passed to the view for processing and giving the visual hint in the app.
def index
#still_have_jobs = !queued_job_count.zero?
end
However, this works when I still had 1 background Job (ImportCsvJob), but when I added the (ProcessCsvJob) it does not work anymore.
import_csv_job.rb
require 'open-uri'
class ImportCsvJob < ActiveJob::Base
queue_as :default
def perform(csv_record)
csv_record[:object_changes] = ApplicationController.helpers.generate_hash(csv_record[:object_changes])
ObjectRecord.create(csv_record)
end
end
process_csv_job.rb
class ProcessCsvJob < ActiveJob::Base
queue_as :default
def perform(csv_path)
csv_file = open(csv_path,'rb:UTF-8')
options = {
row_sep: :auto, col_sep: ",",
user_provided_headers: [:object_id, :object_type, :timestamp, :object_changes],
remove_empty_values: true,
headers_in_file: true
}
SmarterCSV.process(csv_file, options) do |array|
ImportCsvJob.perform_later(array.first)
end
end
end
and lastly, in the model where this is called:
ProcessCsvJob.perform_later(gdrive.uploaded_file_link)
When I try to debug in Rails console using Sidekiq::Queue.new.size, it still gives out 0.
Running:
redis-server
bundle exec sidekiq
A job that is executing is not enqueued anymore. The Sidekiq process has already popped it off the queue and is executing it. The queue is empty but the job is not finished yet.
So, basically I added a monitoring for sidekiq using the web interface to see what was happening:
And as I inspected, there were no enqueued tasks nor scheduled since most of the job is set to perform almost immediately (on parallel).
Thus here's my solution to know if the count of busy jobs:
module ApplicationHelper
def queued_job_count
Sidekiq::ProcessSet.new.first['busy']
end
end
and then on the index:
def index
#still_have_jobs = !queued_job_count.zero?
end
it works! :)
Related
I've implemented Ahoy for tracking events inside my application and I want to move the processing into a background job.
My setup works fine when I call perform_now (the job gets processed), but changing to perform_later doesn't work. From the logs, it seems the job doesn't even get enqueued.
config/initializers/ahoy.rb
class Ahoy::Store < Ahoy::BaseStore
def track_visit(data)
AhoyTrackVisitJob.perform_later(#options, data)
end
def track_event(data)
AhoyTrackEventJob.perform_later(#options, data)
end
end
app/jobs/ahoy_track_event_job.rb
class AhoyTrackEventJob < ApplicationJob
queue_as :default
def perform(options, data)
Ahoy::DatabaseStore.new(options).track_event(data)
end
end
I've tried with both Sidekiq and SuckerPunch:
development.rb
config.active_job.queue_adapter = :sidekiq
I'm using ActiveJob with SideKiq and with the code below I'm getting differing results when using perform_later than perform_now.
When using perform_now my code creates the specified folder and file.
When using perform_later I can see the code executes and doesn't get thrown into the Retry queue of Sidekiq, yet doesn't create the folder or file.
If there is anything else I could produce to help troubleshoot the issue please let me know, as I've likely just overlooked it.
app/controllers/concerns/pdf_player_reports.rb
module PdfPlayerReports
extend ActiveSupport::Concern
included do
# before_action :set_player
# before_action :set_user_report
end
def director_report_pdf
#players = Player.where(id: params["id"])
html = render_to_string template: "players/director_summary_report.pdf.erb"
CreatePdfJob.perform_later(html)
#CreatePdfJob.perform_now(html)
end
end
app/jobs/create_pdf_job.rb
class CreatePdfJob < ActiveJob::Base
queue_as :high_priority
def perform(*args)
generate_pdf_from_string(args.first)
end
def generate_pdf_from_string(html)
# create pdf from passed in HTML string
pdf = WickedPdf.new.pdf_from_string(html)
# Create the directory for the pdfs that will be gernereated incases where it doesn't already exist
pdf_directory_path = Rails.root.join('public','uploads','pdfs')
unless File.directory? pdf_directory_path
FileUtils.mkdir_p(pdf_directory_path)
end
# Open the specified file and then write the contents to disk
File.open(pdf_directory_path.join('output.pdf'),'wb') do |f|
f << pdf
end
end
end
The issue (to my understanding) seems to be that I need to cancel my sidekiq process and restart it (sidekiq -C config/sidekiq.yml) anytime I change code on the Job file.
I have installed the Ruby gem Delayed_Job to run tasks in a queue, but it shows some behavior that I don't understand. Delayed_Job is using my local active_record so a very standard installation.
I have the code for a job in a file called test_job.rb in my /lib folder
class TestJob
# Create a entry in the database to track the execution of jobs
DatabaseJob = Struct.new(:text, :emails) do
def perform
# Perform Test Code
end
end
def enqueue
#enqueue the job
Delayed::Job.enqueue DatabaseJob.new('lorem ipsum...', 'test email')
end
end
When I try to call the code from a controller like this, the first time the job seems to get submitted (is listed in rake jobs:work) but it does not run:
require 'test_job'
class ExampleController < ApplicationController
def index
end
def job
# Create a new job instance
job = TestJob.new
# Enqueue the job into Delay_Job
job.enqueue
end
end
Then when I change the controller code to do what my lib class does, it works perfectly. The job does not only get submitted to the queue, but also runs and completes without failures.
require 'test_job'
class ExampleController < ApplicationController
# Create a entry in the database to track the execution of jobs
DatabaseJob = Struct.new(:text, :emails) do
def perform
# Perform Test Code
end
end
def index
end
def job
#enqueue the job
Delayed::Job.enqueue DatabaseJob.new('lorem ipsum...', 'test email')
end
end
The strange thing is that when I switch back to calling the lib job class it works without a problem. Then it does not matter whether the struct is directly defined in the controller or in the class in the lib folder.
Defining the struct inside the controller and submitting the job to the queue this way always seems to work, but afterwards also the lib class starts working and sometimes the lib class works even after a restart of the rails server.
Any ideas? Thank you very much for the help.
Best,
Bastian
A Rails app has a background job that calls a service object as below
class ObjectUpdateJob < ActiveJob::Base
queue_as :default
def perform(object_id)
ObjectUpdater.call(object_id)
end
end
class ObjectUpdater
def self.call(*args)
if #object = Object.find( args[:object_id] )
#... update some stuff from external services
else
#... raise an error and prevent the background job from attempting to retry
end
end
end
This may be a simple question, but my experience with error handling is limited and I'd appreciate a second thought on this.
What is the 'Rails way' of handling this 'record does not exist' case, and ensuring the background job is informed so that no retries are attempted?
You could just put a guard clause into worker's perform action:
class ObjectUpdateJob < ActiveJob::Base
queue_as :default
def perform(object_id)
return unless Object.find(object_id)
ObjectUpdater.call(object_id)
end
end
But for more protection you can put some check to the job call:
ObjectUpdateJob.perform_now(object_id) if Object.find(object_id)
This way you just won't let the inexistent object to be processed.
If you use Sidekiq you can set a flag to :retry => false so that jobs do not retry automatically.
Additionally Sidekiq has built in error handling mechanism. Here is more information regarding the matter: https://github.com/mperham/sidekiq/wiki/Error-Handling
Hope that helps.
I'm using resque to do some (long time) job. And I have a few classes with the same mixed-in module for queuing. Class Service substitutes in tests, that's why it standalone and (maybe too much) complicated. So the story is when I call
Campaign.perform(user_id)
directly, everything works fine, but when I try to use queue:
Resque.enqueue(Campaign, user_id)
Job created, but seems like do nothing. At least, nothing saves into the database. Which is main task of Campaign class. I can see in resque-web-interface that jobs creates and finished, and finished (to fast, almost just after create) but no result.
I'm new in Resque and not really sure it calls it all (confused how to debug it).
Does anybody have similar problem? thanks for any help.
Module:
module Synchronisable
def self.included(base)
base.extend ClassMethods
end
module ClassMethods
def perform(user_id)
save_objects("#{self.name}::Service".constantize.get_objects(user_id))
end
protected
def save_objects(objects)
raise ArgumentError "should be implemented"
end
end
class Service
def self.get_objects(user)
raise ArgumentError "should be implemented"
end
end
end
One of the classes:
class Campaign < ActiveRecord::Base
include Synchronisable
#queue = :app
class << self
protected
def save_objects(objects)
#some stuff to save objects
end
end
class Service
def self.get_objects(user_id)
#some stuff to get objects
end
end
end
This is a very old question so not sure how rails folder structure was back then but I had the same problem and issue was with inheritance. Seems if you are using Resque your job classes shouldn't inherit from ApplicationJob.
so if your code was like this in (app/jobs/campaign_job.rb):
class Campaign < ApplicationJob
#queue = :a_job_queue
def self.perform
#some background job
end
end
then remove the inheritance i.e "< ApplicationJob"
These jobs are almost certainly failing, due to an Exception. What is resque-web showing you on the Failures tab? You can also get this from the Rails console with:
Resque.info
or
Resque::Failure.all(0)
You should run your worker like this:
nohup QUEUE=* rake resque:work & &> log/resque_worker_QUEUE.log
This will output everything you debug to "log/resque_worker_QUEUE.log" and you will be able to find out what's wrong with your Campaign class.
Try this:
env TERM_CHILD=1 COUNT=2 "QUEUE=*" bundle exec rake resque:workers