In my sidekiq dashboard, I see on the left a box with the counters
Processed 168
Failed 111
Busy 0
Scheduled 0
Retries 0
Enqueued 0
How do I reset them all to 0?
To reset statistics:
Sidekiq::Stats.new.reset
ref: Add reset stats to Web UI summary box and method to API
Also, you can now clear specific stats:
single stat by Sidekiq::Stats.new.reset('failed')
or multiple stats by Sidekiq::Stats.new.reset('failed', 'processed')
(Thanks https://stackoverflow.com/users/2475008/tmr08c for update)
To reset processed jobs:
Sidekiq.redis {|c| c.del('stat:processed') }
and to reset failed jobs:
Sidekiq.redis {|c| c.del('stat:failed') }
Also, to reset specific days in the history panel, you can do:
Sidekiq.redis {|c| c.del('stat:processed:2015-07-02') }
Sidekiq.redis {|c| c.del('stat:failed:2015-07-02') }
And repeat for each day you want to clear.
This is useful if you had a wild job spawning and failing many times more than your usual and you get a history graph with a massive spike in it that makes all your usual history values effectively a flat line.
1. Clear retry set
Sidekiq::RetrySet.new.clear
2. Clear scheduled jobs
Sidekiq::ScheduledSet.new.clear
3. Clear 'Processed' and 'Failed' jobs
Sidekiq::Stats.new.reset
3. Clear 'Dead' jobs statistics
Sidekiq::DeadSet.new.clear
Font: https://gist.github.com/wbotelhos/fb865fba2b4f3518c8e533c7487d5354
Just to complement all good answers, reset counters using ruby interactive mode, doing this into console:
irb
irb(main):001:0> require 'sidekiq/api'
=> true
irb(main):002:0> Sidekiq.redis {|c| c.del('stat:processed') }
=> 1
irb(main):003:0> Sidekiq.redis {|c| c.del('stat:failed') }
=> 1
In case you want to delete the whole thing along with the history panel for specific dates, here is the helpful snippet:
from_date = Date.new(2016, 1, 1)
to_date = Date.today
Sidekiq.redis do |redis|
redis.del("stat:processed")
redis.del("stat:failed")
(from_date..to_date).each do |date|
redis.del("stat:processed:#{date}")
redis.del("stat:failed:#{date}")
end
end
This will also reset the history and delete everything from the Redis queue completely
Sidekiq.redis {|c| c.flushdb }
Related
I have a Ruby on Rails project in which there are millions of products with different urls. I have a function "test_response" that checks the url and returns either a true or false for the Product attribute marked_as_broken, either way the Product is saved and has its "updated_at"-attribute updated to the current Timestamp.
Since this is a very tedious process I have created a task which in turn starts off 15 tasks, each with a N/15 number of products to check. The first one should check from, for example, the first to the 10.000th, the second one from the 10.000nd to the 20.000nd and so on, using limit and offset.
This script works fine, it starts off 15 process but rather quickly completes one script after another far too early. It does not terminate, it finishes with a "Process exited with status 0".
My guess here is that using find_each together with a search for updated_at as well as in fact updating the "updated_at" while running the script changes everything and does not make the script go through the 10.000 items as supposed but I can't verify this.
Is there something inherently wrong by doing what I do here. For example, does "find_each" run a new sql query once in a while providing completely different results each time, than anticipated? I do expect it to provide the same 10.000 -> 20.000 but just split it up in pieces.
task :big_response_launcher => :environment do
nbr_of_fps = Product.where(:marked_as_broken => false).where("updated_at < '" + 1.year.ago.to_date.to_s + "'").size.to_i
nbr_of_processes = 15
batch_size = ((nbr_of_fps / nbr_of_processes))-2
heroku = PlatformAPI.connect_oauth(auth_code_provided_elsewhere)
(0..nbr_of_processes-1).each do |i|
puts "Launching #{i.to_s}"
current_offset = batch_size * i
puts "rake big_response_tester[#{current_offset},#{batch_size}]"
heroku.dyno.create('kopa', {
:command => "rake big_response_tester[#{current_offset},#{batch_size}]",
:attach => false
})
end
end
task :big_response_tester, [:current_offset, :batch_size] => :environment do |task,args|
current_limit = args[:batch_size].to_i
current_offset = args[:current_offset].to_i
puts "Launching with offset #{current_offset.to_s} and limit #{current_limit.to_s}"
Product.where(:marked_as_broken => false).where("updated_at < '" + 1.year.ago.to_date.to_s + "'").limit(current_limit).offset(current_offset).find_each do |fp|
fp.test_response
end
end
As many have noted in the comments, it seems like using find_each will ignore the order and limit. I found this answer (ActiveRecord find_each combined with limit and order) that seems to be working for me. It's not working 100% but it is a definite improvement. The rest seems to be a memory issue, i.e. I cannot have too many processes running at the same time on Heroku.
Per docs I thought it would be (for everyday at 3pm)
daily.hour_of_day(15)
What I'm getting is a random mess. First, it's executing whenever I push to Heroku regardless of time, and then beyond that, seemingly randomly. So the latest push to Heroku was 1:30pm. It executed: twice at 1:30pm, once at 2pm, once at 4pm, once at 5pm.
Thoughts on what's wrong?
Full code (note this is for the Fist of Fury gem, but FoF is heavily influenced by Sidetiq so help from Sidetiq users would be great as well).
class Outstanding
include SuckerPunch::Job
include FistOfFury::Recurrent
recurs { daily.hour_of_day(15) }
def perform
ActiveRecord::Base.connection_pool.with_connection do
# Auto email lenders every other day if they have outstanding requests
lender_array = Array.new
Inventory.where(id: (Borrow.where(status1:1).all.pluck("inventory_id"))).each { |i| lender_array << i.signup.id }
lender_array.uniq!
lender_array.each { |l| InventoryMailer.outstanding_request(l).deliver }
end
end
end
Maybe you should use:
recurrence { daily.hour_of_day(15) }
instead of recurs?
I'm getting "Mysql2::Error: Lock wait timeout exceeded" errors in my production Rails app, and I'm looking for help debugging which transaction is locking the tables for an excessively long time. MySQL has a "slow query" log, but not a log of slow transactions, as far as I can tell.
Is there a way to log information about how long transactions take directly from ActiveRecord?
My team logs slow transactions like this:
In lib/slow_transaction_logger.rb:
module SlowTransactionLogger
LOG_IF_SLOWER_THAN_SECONDS = ENV.fetch("LOG_TRANSACTIONS_SLOWER_THAN_SECONDS", "5").to_f
def transaction(*args)
start = Time.now
value = super
finish = Time.now
duration = finish - start
return value if duration <= LOG_IF_SLOWER_THAN_SECONDS
backtrace = caller.
select { |path| path.start_with?(Rails.root.to_s) }.
reject { |row|
row.include?("/gems/") ||
row.include?("/app/middleware/") ||
row.include?("/config/initializers/")
}
puts "slow_transaction_logger: duration=#{duration.round(2)} start=#{start.to_s.inspect} finish=#{finish.to_s.inspect} class=#{name} caller=#{backtrace.to_json}"
value
end
end
In config/initializers/active_record.rb:
class ActiveRecord::Base
class << self
prepend SlowTransactionLogger
end
end
So by default, it logs transactions slower than 5 seconds, in every environment.
If you set a LOG_TRANSACTIONS_SLOWER_THAN_SECONDS environment variable (e.g. on Heroku), you can tweak that limit.
We log with puts intentionally because it ends up in the logs in Heroku, and we also we see it more easily in dev and tests. But feel free to use Rails.logger if you prefer.
I realize there is a push_bulk option for sidekiq but I'm currently being limited by latency to redis, so passing multiple items via push_bulk still isn't going quickly enough (only about 50/s).
I've tried to increase the number of redis connections like so:
redis_conn = proc {
Redis.new({ :url => Rails.configuration.redis.url })
}
Sidekiq.configure_client do |config|
Sidekiq.configure_client do |config|
config.redis = ConnectionPool.new(size: 50, &redis_conn)
end
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
And then fire off separate threads (Thread.new) to actually perform_async on the various objects. What is interesting is any thread that isn't the first thread NEVER gets thrown into the sidekiq queue, it's like they're ignored entirely.
Does anyone know of a better way to do this?
Edit: Here is the push_bulk method I was trying which is actually slower:
user_ids = User.need_scraping.pluck(:id)
bar = ProgressBar.new(user_ids.count)
user_ids.in_groups_of(10000, false).each do |user_id_group|
Sidekiq::Client.push_bulk(
'args' => user_id_group.map{ |user_id| [user_id] },
'class' => ScrapeUser,
'queue' => 'scrape_user',
'retry' => true
)
end
Thanks!
You DO want to use push_bulk. You're limited by the latency/round-trip time to write elements to the redis queue backing sidekiq.
You're using multiple threads/connections to overcome a slow network, when you should really be removing extra network roundtrips.
Assuming you're trying to enqueuue 20k UserWorker jobs that take a user_id:
You would enqueue a single job via:
UserWorker.perform_async(user_id)
... which maps to:
Sidekiq::Client.push('class' => UserWorker, 'args' => [user_id] )
So the push_bulk version for 20k user_ids is:
# This example takes 20k user_ids in an array, chunks them into groups of 1000 ids,
# and batch sends them to redis as a group.
User.need_scraping.select('id').find_in_batches do |user_group|
sidekiq_items = user_group.map {|user| { 'class' => UserWorker, 'args' => [user.id] } }
Sidekiq::Client.push_bulk(sidekiq_items)
end
This turns 20k redis calls into 20 redis calls, with an average round trip time of 5ms (optimistic), that's 1sec vs. 100 seconds. Your mileage may vary.
EDIT:
Commenters seem confused about the behavior of the Sidekiq/Redis client for bulk enqueuing data.
The Sidekiq::Client.push_bulk() method takes an array of jobs to be enqueud. It translates these into Sidekiq job payload hashes, and then calls SideKiq::Client.raw_push() to deliver these payloads to redis. See source: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/client.rb#L158
SideKiq::Client.raw_push() takes a list of Sidekiq hash payloads, converts them to JSON, and then executes a redis MULTI command combining two redis commands. First, it adds to targeted queue to the list of active queues (redis SADD), then it pushes all of the job payloads to the targeted queue redis list object (redis LPUSH). This is a single redis command, executed together in a single redis atomic group.
If this is still slow, you likely have other problems (slow network, overloaded redis server, etc).
#Winfield's answer is correct, and he's absolutely right about latency. However, the correct syntax is actually as follows:
User.need_scraping.select('id').find_in_batches do |user_group|
Sidekiq::Client.push_bulk({ 'class' => UserWorker, 'args' => user_group.map {|user| [user.id] } })
end
Maybe it changed in the most recent Sidekiq (I was too lazy to check), but this is the correct syntax now.
I am interested in setting up a monitoring service that will page me whenever there are too many jobs in the Resque queue (I have about 6 queues, I'll have different numbers for each queue). I also want to setup a very similar monitoring service that will alert me when I exceed a certain amount of failed jobs in my queue.
My question is, there is a lot of keys and confusion that I see affiliated with Resque on my redis server. I don't necessarily see a straight forward way to get a count of jobs per queue or the number of failed jobs. Is there currently a trivial way to grab this data from redis?
yes it's quite easy, given you're using the Resque gem:
require 'resque'
Resque.info
will return a hash
e.g/ =>
{
:pending => 54338,
:processed => 12772,
:queues => 2,
:workers => 0,
:working => 0,
:failed => 8761,
:servers => [
[0] "redis://192.168.1.10:6379/0"
],
:environment => "development"
}
So to get the failed job count, simply use:
Resque.info[:failed]
which would give
=> 8761 #in my example
To get the queues use:
Resque.queues
this returns a array
e.g./ =>
[
[0] "superQ",
[1] "anotherQ"
]
You may then find the number of jobs per queue:
Resque.size(queue_name)
e.g/ Resque.size("superQ") or Resque.size(Resque.queues[0]) .....
Here is a bash script which will monitor the total number of jobs queued and the number of failed jobs.
while :
do
let sum=0
let errors=$(redis-cli llen resque:failed)
for s in $(redis-cli keys resque:queue:*)
do
let sum=$sum+$(redis-cli llen $s)
done
echo $sum jobs queued, with $errors errors
sleep 1 # sleep 1 second, probably want to increase this
done
This is for Resque 1.X, 2.0 might have different key names.
There is also a method Resque.queue_sizes That returns a hash of the queue name and size
Resque.queue_sizes
=> {"default"=>0, "slow"=>0}