I am interested in setting up a monitoring service that will page me whenever there are too many jobs in the Resque queue (I have about 6 queues, I'll have different numbers for each queue). I also want to setup a very similar monitoring service that will alert me when I exceed a certain amount of failed jobs in my queue.
My question is, there is a lot of keys and confusion that I see affiliated with Resque on my redis server. I don't necessarily see a straight forward way to get a count of jobs per queue or the number of failed jobs. Is there currently a trivial way to grab this data from redis?
yes it's quite easy, given you're using the Resque gem:
require 'resque'
Resque.info
will return a hash
e.g/ =>
{
:pending => 54338,
:processed => 12772,
:queues => 2,
:workers => 0,
:working => 0,
:failed => 8761,
:servers => [
[0] "redis://192.168.1.10:6379/0"
],
:environment => "development"
}
So to get the failed job count, simply use:
Resque.info[:failed]
which would give
=> 8761 #in my example
To get the queues use:
Resque.queues
this returns a array
e.g./ =>
[
[0] "superQ",
[1] "anotherQ"
]
You may then find the number of jobs per queue:
Resque.size(queue_name)
e.g/ Resque.size("superQ") or Resque.size(Resque.queues[0]) .....
Here is a bash script which will monitor the total number of jobs queued and the number of failed jobs.
while :
do
let sum=0
let errors=$(redis-cli llen resque:failed)
for s in $(redis-cli keys resque:queue:*)
do
let sum=$sum+$(redis-cli llen $s)
done
echo $sum jobs queued, with $errors errors
sleep 1 # sleep 1 second, probably want to increase this
done
This is for Resque 1.X, 2.0 might have different key names.
There is also a method Resque.queue_sizes That returns a hash of the queue name and size
Resque.queue_sizes
=> {"default"=>0, "slow"=>0}
Related
I have a Ruby on Rails project in which there are millions of products with different urls. I have a function "test_response" that checks the url and returns either a true or false for the Product attribute marked_as_broken, either way the Product is saved and has its "updated_at"-attribute updated to the current Timestamp.
Since this is a very tedious process I have created a task which in turn starts off 15 tasks, each with a N/15 number of products to check. The first one should check from, for example, the first to the 10.000th, the second one from the 10.000nd to the 20.000nd and so on, using limit and offset.
This script works fine, it starts off 15 process but rather quickly completes one script after another far too early. It does not terminate, it finishes with a "Process exited with status 0".
My guess here is that using find_each together with a search for updated_at as well as in fact updating the "updated_at" while running the script changes everything and does not make the script go through the 10.000 items as supposed but I can't verify this.
Is there something inherently wrong by doing what I do here. For example, does "find_each" run a new sql query once in a while providing completely different results each time, than anticipated? I do expect it to provide the same 10.000 -> 20.000 but just split it up in pieces.
task :big_response_launcher => :environment do
nbr_of_fps = Product.where(:marked_as_broken => false).where("updated_at < '" + 1.year.ago.to_date.to_s + "'").size.to_i
nbr_of_processes = 15
batch_size = ((nbr_of_fps / nbr_of_processes))-2
heroku = PlatformAPI.connect_oauth(auth_code_provided_elsewhere)
(0..nbr_of_processes-1).each do |i|
puts "Launching #{i.to_s}"
current_offset = batch_size * i
puts "rake big_response_tester[#{current_offset},#{batch_size}]"
heroku.dyno.create('kopa', {
:command => "rake big_response_tester[#{current_offset},#{batch_size}]",
:attach => false
})
end
end
task :big_response_tester, [:current_offset, :batch_size] => :environment do |task,args|
current_limit = args[:batch_size].to_i
current_offset = args[:current_offset].to_i
puts "Launching with offset #{current_offset.to_s} and limit #{current_limit.to_s}"
Product.where(:marked_as_broken => false).where("updated_at < '" + 1.year.ago.to_date.to_s + "'").limit(current_limit).offset(current_offset).find_each do |fp|
fp.test_response
end
end
As many have noted in the comments, it seems like using find_each will ignore the order and limit. I found this answer (ActiveRecord find_each combined with limit and order) that seems to be working for me. It's not working 100% but it is a definite improvement. The rest seems to be a memory issue, i.e. I cannot have too many processes running at the same time on Heroku.
So I have
MyJob.perform_in(60, #user),
Which will perform my job in 60 seconds.
I want to cancel this Job if this line of code is ran again replacing it in the queue.
I have had no luck researching.
To get the stats related to SuckerPunch jobs
[14] pry(main)> SuckerPunch::Queue.stats
{
"CreateVVLinkJob" => {
"workers" => {
"total" => 1,
"busy" => 0,
"idle" => 1
},
"jobs" => {
"processed" => 1,
"failed" => 0,
"enqueued" => 0
}
}
}
And To Clear the previous jobs
[24] pry(main)> SuckerPunch::Queue.clear
[
[0] #<SuckerPunch::Queue:0x0000000b150da0 #__lock__=#<Mutex:0x0000000b150d50>, #__condition__=#<Thread::ConditionVariable:0x0000000b150d28>, #running=false, #name="CreateVVLinkJob", #pool=#<Concurrent::ThreadPoolExecutor:0x0000000b146ad0 #__lock__=#<Mutex:0x0000000b1469e0>, #__condition__=#<Thread::ConditionVariable:0x0000000b1469b8>, #min_length=2, #max_length=2, #idletime=60, #max_queue=0, #fallback_policy=:abort, #auto_terminate=false, #pool=[], #ready=[], #queue=[], #scheduled_task_count=1, #completed_task_count=1, #largest_length=1, #ruby_pid=22314, #gc_interval=30, #next_gc_time=25973.834404648, #StopEvent=#<Concurrent::Event:0x0000000b1468c8 #__lock__=#<Mutex:0x0000000b146878>, #__condition__=#<Thread::ConditionVariable:0x0000000b146850>, #set=true, #iteration=0>, #StoppedEvent=#<Concurrent::Event:0x0000000b1467d8 #__lock__=#<Mutex:0x0000000b146788>, #__condition__=#<Thread::ConditionVariable:0x0000000b146760>, #set=true, #iteration=0>>>
]
Hope this is useful !
There is no built-in method inside of the SuckerPunch framework, that I can see from the source code, of canceling a single queued up job or a job that is currently executing. It appears that clearing the job queue is an all or nothing function.
That said, it should be a trivial matter to add an extension method that queries the underlying ConcurrentTask framework and matches up your new job with an already queued job based on the value of the #user parameter.
I'm trying to use Sidekiq batches to group a set of related jobs together. However, the batch prematurely fires on complete callback's since the jobs method can't push all the jobs to Redis fast enough. The sidekiq documentation at https://github.com/mperham/sidekiq/wiki/Batches says this can be resolved by using the Sidekiq::Client.push_bulk method but the documentation is unclear on how to push to a batch in bulk. Can someone share an example of how to use push_bulk in the context of a batch?
Assume you want to process 100 users in a batch, one user ID per job.
b = Sidekiq::Batch.new
b.jobs do
users = User.select(:id).limit(100).map(&:id) # users is [1, 2, 3, etc...]
args = users.map {|uid| [uid] } # args is [[1], [2], [3], etc...]
Sidekiq::Client.push_bulk('class' => YourWorker, 'args' => args)
end
I realize there is a push_bulk option for sidekiq but I'm currently being limited by latency to redis, so passing multiple items via push_bulk still isn't going quickly enough (only about 50/s).
I've tried to increase the number of redis connections like so:
redis_conn = proc {
Redis.new({ :url => Rails.configuration.redis.url })
}
Sidekiq.configure_client do |config|
Sidekiq.configure_client do |config|
config.redis = ConnectionPool.new(size: 50, &redis_conn)
end
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
And then fire off separate threads (Thread.new) to actually perform_async on the various objects. What is interesting is any thread that isn't the first thread NEVER gets thrown into the sidekiq queue, it's like they're ignored entirely.
Does anyone know of a better way to do this?
Edit: Here is the push_bulk method I was trying which is actually slower:
user_ids = User.need_scraping.pluck(:id)
bar = ProgressBar.new(user_ids.count)
user_ids.in_groups_of(10000, false).each do |user_id_group|
Sidekiq::Client.push_bulk(
'args' => user_id_group.map{ |user_id| [user_id] },
'class' => ScrapeUser,
'queue' => 'scrape_user',
'retry' => true
)
end
Thanks!
You DO want to use push_bulk. You're limited by the latency/round-trip time to write elements to the redis queue backing sidekiq.
You're using multiple threads/connections to overcome a slow network, when you should really be removing extra network roundtrips.
Assuming you're trying to enqueuue 20k UserWorker jobs that take a user_id:
You would enqueue a single job via:
UserWorker.perform_async(user_id)
... which maps to:
Sidekiq::Client.push('class' => UserWorker, 'args' => [user_id] )
So the push_bulk version for 20k user_ids is:
# This example takes 20k user_ids in an array, chunks them into groups of 1000 ids,
# and batch sends them to redis as a group.
User.need_scraping.select('id').find_in_batches do |user_group|
sidekiq_items = user_group.map {|user| { 'class' => UserWorker, 'args' => [user.id] } }
Sidekiq::Client.push_bulk(sidekiq_items)
end
This turns 20k redis calls into 20 redis calls, with an average round trip time of 5ms (optimistic), that's 1sec vs. 100 seconds. Your mileage may vary.
EDIT:
Commenters seem confused about the behavior of the Sidekiq/Redis client for bulk enqueuing data.
The Sidekiq::Client.push_bulk() method takes an array of jobs to be enqueud. It translates these into Sidekiq job payload hashes, and then calls SideKiq::Client.raw_push() to deliver these payloads to redis. See source: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/client.rb#L158
SideKiq::Client.raw_push() takes a list of Sidekiq hash payloads, converts them to JSON, and then executes a redis MULTI command combining two redis commands. First, it adds to targeted queue to the list of active queues (redis SADD), then it pushes all of the job payloads to the targeted queue redis list object (redis LPUSH). This is a single redis command, executed together in a single redis atomic group.
If this is still slow, you likely have other problems (slow network, overloaded redis server, etc).
#Winfield's answer is correct, and he's absolutely right about latency. However, the correct syntax is actually as follows:
User.need_scraping.select('id').find_in_batches do |user_group|
Sidekiq::Client.push_bulk({ 'class' => UserWorker, 'args' => user_group.map {|user| [user.id] } })
end
Maybe it changed in the most recent Sidekiq (I was too lazy to check), but this is the correct syntax now.
In my sidekiq dashboard, I see on the left a box with the counters
Processed 168
Failed 111
Busy 0
Scheduled 0
Retries 0
Enqueued 0
How do I reset them all to 0?
To reset statistics:
Sidekiq::Stats.new.reset
ref: Add reset stats to Web UI summary box and method to API
Also, you can now clear specific stats:
single stat by Sidekiq::Stats.new.reset('failed')
or multiple stats by Sidekiq::Stats.new.reset('failed', 'processed')
(Thanks https://stackoverflow.com/users/2475008/tmr08c for update)
To reset processed jobs:
Sidekiq.redis {|c| c.del('stat:processed') }
and to reset failed jobs:
Sidekiq.redis {|c| c.del('stat:failed') }
Also, to reset specific days in the history panel, you can do:
Sidekiq.redis {|c| c.del('stat:processed:2015-07-02') }
Sidekiq.redis {|c| c.del('stat:failed:2015-07-02') }
And repeat for each day you want to clear.
This is useful if you had a wild job spawning and failing many times more than your usual and you get a history graph with a massive spike in it that makes all your usual history values effectively a flat line.
1. Clear retry set
Sidekiq::RetrySet.new.clear
2. Clear scheduled jobs
Sidekiq::ScheduledSet.new.clear
3. Clear 'Processed' and 'Failed' jobs
Sidekiq::Stats.new.reset
3. Clear 'Dead' jobs statistics
Sidekiq::DeadSet.new.clear
Font: https://gist.github.com/wbotelhos/fb865fba2b4f3518c8e533c7487d5354
Just to complement all good answers, reset counters using ruby interactive mode, doing this into console:
irb
irb(main):001:0> require 'sidekiq/api'
=> true
irb(main):002:0> Sidekiq.redis {|c| c.del('stat:processed') }
=> 1
irb(main):003:0> Sidekiq.redis {|c| c.del('stat:failed') }
=> 1
In case you want to delete the whole thing along with the history panel for specific dates, here is the helpful snippet:
from_date = Date.new(2016, 1, 1)
to_date = Date.today
Sidekiq.redis do |redis|
redis.del("stat:processed")
redis.del("stat:failed")
(from_date..to_date).each do |date|
redis.del("stat:processed:#{date}")
redis.del("stat:failed:#{date}")
end
end
This will also reset the history and delete everything from the Redis queue completely
Sidekiq.redis {|c| c.flushdb }