Resque-Scheduler not working with ActiveJob in Rails 4.2 - ruby-on-rails
Has anyone been able to get scheduled jobs to work in Rails 4.2?
I am using resque, and I am attempting to use resque-scheduler to schedule jobs. I have a schedule that get loaded and the scheduler runs, and even looks like it is running the jobs but it doesn't do anything.
resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Starting
resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Loading Schedule
resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Scheduling friends
resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Schedules Loaded
resque-scheduler: [INFO] 2014-09-16T01:54:55-07:00: queueing FriendsJob (friends)
I can enqueue jobs like this and they get processed.
TestJob.enqueue(params[:id])
and I get this output in the worker log
got: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])
** [01:24:01 2014-09-16] 54841: resque-1.25.2: Processing default since 1410855841 [ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper]
** [01:24:01 2014-09-16] 54841: Running before_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])]
** [01:24:01 2014-09-16] 54841: resque-1.25.2: Forked 54882 at 1410855841
** [01:24:01 2014-09-16] 54882: Running after_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])]
Hello World!!
** [01:24:01 2014-09-16] 54882: done: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])
But when I try to schedule a job, it looks like the are getting enqueue but they well not doing anything.
Here is the schedule.yml
friends:
every: "30s"
queue: "friends"
class: "FriendsJob"
args: 8
description: "Friends jobs scheduler"
Here is the output from the scheduled job.
** [01:23:36 2014-09-16] 54841: got: (Job{friends} | FriendsJob | [8])
** [01:23:36 2014-09-16] 54841: resque-1.25.2: Processing friends since 1410855816 [FriendsJob]
** [01:23:36 2014-09-16] 54841: Running before_fork hooks with [(Job{friends} | FriendsJob | [8])]
** [01:23:36 2014-09-16] 54841: resque-1.25.2: Forked 54880 at 1410855816
** [01:23:36 2014-09-16] 54880: Running after_fork hooks with [(Job{friends} | FriendsJob | [8])]
** [01:23:36 2014-09-16] 54880: done: (Job{friends} | FriendsJob | [8])
After reading this http://dev.mikamai.com/post/96343027199/rails-4-2-new-gems-active-job-and-global-id
I am suspecting it has something to do with ActiveJob and GlobalId.
Take a look at the difference enqueued
** [01:24:01 2014-09-16] 54841: Running before_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])]
vs scheduled
** [01:23:36 2014-09-16] 54841: Running before_fork hooks with [(Job{friends} | FriendsJob | [8])]
The jobs themselves were generated via
rails g job <JobName>
They look like this:
class TestJob < ActiveJob::Base
queue_as :default
def perform(friend_id)
friend = Friend.find(friend_id)
name = friend.name.swapcase
puts "Hello World!!"
end
end
class FriendsJob < ActiveJob::Base
queue_as :friends
def perform(friend_id)
friend = Friend.find(friend_id)
name = friend.name.swapcase
puts "Hello World!!"
end
end
Any help with this will be greatly appreciated and thanks in advance.
********* UPDATE *********
I've removed the action_job railtie and I am using Resque and Resque-Scheduler only, and scheduled jobs are working now. So this does seems to be related to ActionJob/GlobalId. I've open an issue in the rails project. Hopefully, they will fix it soon.
****** SECOND UPDATE *********
I got an update on this. Cristianbica who works in the ActiveJob codebase said this.
"We haven't thought at recurring jobs so far and we don't support this as none of the adapters support this without an external gem. However this is a very nice feature but I don't think we can make it in time for 4.2. Also I'm not sure it will suitable to be included in rails"
https://github.com/JustinAiken/active_scheduler is a gem that wraps the one into the other
It seems that ActiveJob in Rails 4.2 is not supported by resque-scheduler. Therefore, jobs are not scheduled correctly, that explains the difference in the log when a job is enqueued using ActiveJob API and resque-scheduler.
To fix this, we should find a way to schedule job within ActiveJob wrapper:
ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper
Resque-scheduler provides the way to support extensions that are not supported by default. To do this, we should extend custom job class to support the #scheduled method. This way we can enqueue job manually using ActiveJob API.
The easiest way is to write general code method in the base job, and then extend all jobs from it:
# custom base job
class Job < ActiveJob::Base
# method called by resque-scheduler to enqueue job in a queue
def self.scheduled(queue, klass, *args)
# create the job instance and pass the arguments
job = self.job_or_instantiate(*args)
# set correct queue
job.queue_name = queue
# enqueue job using ActiveJob API
job.enqueue
end
end
Reque-scheduler will call this method to schedule every single job extended from Job class. This way jobs are going to be enqueued within the ActiveJob wrapper. Result will be same as calling MyJob.perform_later(*args).
UPDATE: 4/4/16 - Whilst the below answer is still correct with the current version of Rails, I now use the active_scheduler gem created by Justin as mentioned in the answer above: https://stackoverflow.com/a/29155372/1299792
ORIGINAL ANSWER:
Avoid using ActiveJob if you need to schedule recurring jobs.
From the issue that Luis raised
As we are not currently supporting recurring jobs with ActiveJob we're going to close this. If it's possible to support recurring jobs I'm seeing this as a separate gem or in rails 5. Feature requests and talks around them are usually talked in the mailing list (https://groups.google.com/forum/#!forum/rubyonrails-core).
As a solution to your problem #luismadrigal I suggest you use the resque-scheduler way to do recurring job.
https://github.com/rails/rails/issues/16933#issuecomment-58945932
There was talk about creating a gem in the mailing list but I can't find any further information on it.
Related
Resque not failing an obviously buggy job
I have the following problem with Resque and Rails 6 (ruby 2.7.2). I am new to this gem and wanted to do some simple testing to see how things are handled, so I created a simple failing job:
class TestJob < ApplicationJob
#queue = :default
def self.perform(*args)
0/0
end
end
Obviously, when I go to the console and run it, I get the error:
irb(main):001:0> TestJob.perform
Traceback (most recent call last):
3: from (irb):1
2: from app/jobs/test_job.rb:5:in `perform'
1: from app/jobs/test_job.rb:5:in `/'
ZeroDivisionError (divided by 0)
But when I run my worker with the command:
QUEUE=* VVERBOSE=true rake resque:work
And then enqueue my job in rails console, the output I get is:
irb(main):002:0> Resque.enqueue(TestJob)
=> true
And in the resque logs:
** [21:32:57 2020-12-09] 15198: resque-2.0.0: Waiting for default
** [21:33:02 2020-12-09] 15198: Checking default
** [21:33:02 2020-12-09] 15198: Found job on default
** [21:33:02 2020-12-09] 15198: resque-2.0.0: Processing default since 1607545982 [TestJob]
** [21:33:02 2020-12-09] 15198: got: (Job{default} | TestJob | [])
** [21:33:02 2020-12-09] 15198: resque-2.0.0: Forked 15765 at 1607545982
** [21:33:02 2020-12-09] 15765: done: (Job{default} | TestJob | [])
Additionally, in the resque web UI it counts this process as successful. The number of failed jobs stays 0. Furthermore, any puts and raise I put into the perform method are not showing in resque/rails console logs.
I have done a basic resque setup on a fresh rails app following the README on the resque's github page. I did not do anything custom. Obviously I restarted my console and worker after I made changes to the test_job.rb file, so this is not a problem of loading (maybe?). I also tried changing self.perform to perform.
I don't know why, but this is the solution. Rather than running: Resque.enqueue(TestJob) one should run: TestJob.perform_later The thing is, the first way is from the Resque README, and the second one is from the Rails docs. I guess I was fixated on doing this the first way, but it looks like going with the rails conventions is always best.
Rails 5 - Resque not processing enqueued job
I'm trying to do enqueue a simple job using Resque 1.26.0 (and Redis-rb 3.3.1). The job doesn't seem to be processing the perform function because resque-web is processing each job and shows 0 failures. The jobs also are being processed instantly.
The jobs are enqueued from a controller action with
Resque.enqueue(TestJob, url)
The job itself looks like
class TestJob < ApplicationJob
#queue = :tags_queue
Logger.new("log/resque_worker_QUEUE.log").fatal("thing")
def self.perform(url)
Logger.new("log/resque_worker_QUEUE.log").fatal("other thing")
logger.fatal("more errors please")
myDivideByZeroVar= 1/0
raise "error"
Logger.new("log/resque_worker_QUEUE.log").fatal("other thing")
logger.fatal("more errors please")
end
end
A rake task is also set up:
require 'resque/tasks'
task "resque:setup" => :environment
The redis-server is running.
The worker is started with rake resque:work QUEUE=*. Using verbose logging doesn't show anything useful.
The log file only shows the first fatal error string "thing". None of the other errors are logged that are inside perform.
What am I doing wrong here?
Solved this. The job needed to be called using ActiveJob instead of using Resque.enqueue. TestJob.perform_later(url) worked just fine.
Resque could not find job class ZF2 namespaces
Attempting to use the Redis backed PHP-Resque project https://github.com/chrisboulton/php-resque from within a ZF2 project. (Apigility to be specific)
What I'm having trouble with, is combining the ZF2 namespaces and the classes
for instance :
Controller
//Enqueue a worker
$args = array(
'name' => 'EMCP'
);
Resque::enqueue('default', 'phpresque\\V1\\Model\\MyResqueJob', $args);
MyResqueJob Class
namespace phpresque\V1\Model;
class MyResqueJob
{
public function perform()
{
// Work work work
echo "helloWorld";
}
}
Error message given by resque :
ubuntu#/zf2projectname/vendor/chrisboulton/php-resque$ QUEUE=* APP_INCLUDE=/zf2projectname/vendor/autoload.php VVERBOSE=1 php resque.php
** [03:01:37 2014-10-26] Sleeping for 5
** [03:01:42 2014-10-26] Checking default
** [03:01:42 2014-10-26] Found job on default
** [03:01:42 2014-10-26] got (Job{default} | ID: 38fa104b11de81731c15ba9c2f1853ab | phpresque\V1\Model\MyResqueJob | [{"name":"EMCP"}])
** [03:01:42 2014-10-26] Forked 30642 at 2014-10-26 03:01:42
** [03:01:42 2014-10-26] Processing default since 2014-10-26 03:01:42
** [03:01:42 2014-10-26] (Job{default} | ID: 38fa104b11de81731c15ba9c2f1853ab | phpresque\V1\Model\MyResqueJob | [{"name":"EMCP"}]) failed: Could not find job class phpresque\V1\Model\MyResqueJob.
** [03:01:42 2014-10-26] Checking default
Edit : I dropped using Resque but here's the suggestion below on how it could work.. I ended up using SlmQueue instead https://github.com/juriansluiman/SlmQueue
According to the ZF2 github community, the following should be done :
Shouldn't your APP_INCLUDE environment var contain the path to composer's init_autoloader.php? Just including the autoloader class will not suffice, as you not configuring and registering the autoloader anywhere.
and
YMMV, but what I've done to integrate my ZF2 app into Resque is create a separate entry point script that initializes the ZF2 app within the context of the Resque worker:
<?php
// resque_context.php
require_once 'vendor/autoload.php';
$application = \Zend\Mvc\Application::init(include 'config/application.config.php');
$config = $application->getServiceManager()->get('config');
And provide it's path to the Resque run script via the APP_INCLUDE environment variable.
Are there console commands to look at whats in the queue and to clear the queue in Sidekiq?
I'm used to using delayed_jobs method of going into the console to see whats in the queue, and the ease of clearing the queue when needed. Are there similar commands in Sidekiq for this? Thanks!
There is an ergonomic API for viewing and managing queues.
It is not required by default.
require 'sidekiq/api'
Here's the excerpt:
# get a handle to the default queue
default_queue = Sidekiq::Queue.new
# get a handle to the mailer queue
mailer_queue = Sidekiq::Queue.new("mailer")
# How many jobs are in the default queue?
default_queue.size # => 1001
# How many jobs are in the mailer queue?
mailer_queue.size # => 50
#Deletes all Jobs in a Queue, by removing the queue.
default_queue.clear
You can also get some summary statistics.
stats = Sidekiq::Stats.new
# Get the number of jobs that have been processed.
stats.processed # => 100
# Get the number of jobs that have failed.
stats.failed # => 3
# Get the queues with name and number enqueued.
stats.queues # => { "default" => 1001, "email" => 50 }
#Gets the number of jobs enqueued in all queues (does NOT include retries and scheduled jobs).
stats.enqueued # => 1051
I haven't ever used Sidekiq, so it's possible that there are methods just for viewing the queued jobs, but they would really just be wrappers around Redis commands, since that's basically all Sidekiq (and Resque) is:
# See workers
Sidekiq::Client.registered_workers
# See queues
Sidekiq::Client.registered_queues
# See all jobs for one queue
Sidekiq.redis { |r| r.lrange "queue:app_queue", 0, -1 }
# See all jobs in all queues
Sidekiq::Client.registered_queues.each do |q|
Sidekiq.redis { |r| r.lrange "queue:#{q}", 0, -1 }
end
# Remove a queue and all of its jobs
Sidekiq.redis do |r|
r.srem "queues", "app_queue"
r.del "queue:app_queue"
end
Unfortunately, removing a specific job is a little more difficult as you'd have to copy its exact value:
# Remove a specific job from a queue
Sidekiq.redis { |r| r.lrem "queue:app_queue", -1, "the payload string stored in Redis" }
You could do all of this even more easily via redis-cli :
$ redis-cli
> select 0 # (or whichever namespace Sidekiq is using)
> keys * # (just to get an idea of what you're working with)
> smembers queues
> lrange queues:app_queue 0 -1
> lrem queues:app_queue -1 "payload"
if there is any scheduled job. You may delete all the jobs using the following command: Sidekiq::ScheduledSet.new.clear if there any queues you wanted to delete all jobs you may use the following command: Sidekiq::Queue.new.clear Retries Jobs can be removed by the following command also: Sidekiq::RetrySet.new.clear There are more information here at the following link, you may checkout: https://github.com/mperham/sidekiq/wiki/API
There is a API for accessing real-time information about workers, queues and jobs. Visit here https://github.com/mperham/sidekiq/wiki/API
A workaround is to use the testing module (require 'sidekiq/testing') and to drain the worker (MyWorker.drain).
There were hanged 'workers' in default queue and I was able to see them though web interface. But they weren't available from console if I used Sidekiq::Queue.new.size
irb(main):002:0> Sidekiq::Queue.new.size
2014-03-04T14:37:43Z 17256 TID-oujb9c974 INFO: Sidekiq client with redis options {:namespace=>"sidekiq_staging"}
=> 0
Using redis-cli I was able to find them
redis 127.0.0.1:6379> keys *
1) "sidekiq_staging:worker:ip-xxx-xxx-xxx-xxx:7635c39a29d7b255b564970bea51c026-69853672483440:default"
2) "sidekiq_staging:worker:ip-xxx-xxx-xxx-xxx:0cf585f5e93e1850eee1ae4613a08e45-70328697677500:default:started"
3) "sidekiq_staging:worker:ip-xxx-xxx-xxx-xxx:7635c39a29d7b255b564970bea51c026-69853672320140:default:started"
...
The solution was:
irb(main):003:0> Sidekiq.redis { |r| r.del "workers", 0, -1 }
=> 1
Also in the Sidekiq v3 there is a command
Sidekiq::Workers.new.prune
But for some reason it didn't work for me that day
And if you want to clear the sidekiq retry queue, it's this: Sidekiq::RetrySet.new.clear
$ redis-cli > select 0 # (or whichever namespace Sidekiq is using) > keys * # (just to get an idea of what you're working with) > smembers queues > lrange queue:queue_name 0 -1 # (queue_name must be your relevant queue) > lrem queue:queue_name -1 "payload"
Rake task for clear all sidekiq queues: namespace :sidekiq do desc 'Clear sidekiq queue' task clear: :environment do require 'sidekiq/api' Sidekiq::Queue.all.each(&:clear) end end Usage: rake sidekiq:clear
This is not a direct solution for the Rails console, but for a quick monitoring of the Sidekiq task count and queue size, you can use sidekiqmon binary that ships with Sidekiq 6+: $ sidekiqmon Sidekiq 6.4.2 2022-07-25 11:05:56 UTC ---- Overview ---- Processed: 20,313,347 Failed: 57,120 Busy: 9 Enqueued: 17 Retries: 0 Scheduled: 37 Dead: 2,382 ---- Processes (1) ---- 36f993209f93:15:a498f85c6a12 [server] Started: 2022-07-25 10:49:43 +0000 (16 minutes ago) Threads: 10 (9 busy) Queues: default, elasticsearch, statistics ---- Queues (3) ---- NAME SIZE LATENCY default 0 0.00 elasticsearch 17 0.74 statistics 0 0.00
rake jobs:work working fine. problem with script/delayed_job start
I am calling function with LoadData.send_later(:test). LoadData is my class and test is my method. It's working fine while i am running rake jobs:work. But when i am running script/delayed_job start or run that time delayed_job.log shows error like TEastern Daylight Time: *** Starting job worker delayed_job host:KShah pid:5968 TEastern Daylight Time: * [Worker(delayed_job host:KShah pid:5968)] acquired lock on LoadData.load_test_data_with_delayed_job Could not load object for job: uninitialized constant LoadData TEastern Daylight Time: * [JOB] delayed_job host:KShah pid:5968 completed after 0.0310 TEastern Daylight Time: 1 jobs processed at 10.6383 j/s, 0 failed ... Any solution??
Try putting include LoadData in an initializer. I seem to remember DelayedJob including activerecord classes, notifiers etc, but not custom classes. Personally I'd put the class in your models directory. It's still dealing with data, even if it's not activerecord.
Try doing this: Delayed::Job.enqueue LoadData.test Also, a big gotcha that took me while to realize... if you make changes to the code restart rake jobs:work!