How to get the limit of my sidekiq worker? - ruby-on-rails

Question

I have 200000 users in my database
I need to iterate through each record to process something.
So I have a rake task to iterate each user, main logic will be in a worker. Now I want to get the limit of the workers those can be run simultaneously. If the limit is 50000, then I can divide my users into 4 sets, for each set I will call the worker seperately.
task:
namespace :users do
task data: :environment do
confirmed_users = User.where('confirmed_at IS NOT NULL').where('id <= 50000')
confirmed_users.each do |user|
MyWorker.perform_async(user.id)
end#confirmed_users.each do |user|
confirmed_users = User.where('confirmed_at IS NOT NULL').where('id > 50000 and id <= 100000')
confirmed_users.each do |user|
MyWorker.perform_async(user.id)
end#confirmed_users.each do |user|
confirmed_users = User.where('confirmed_at IS NOT NULL').where('id > 100000 and id <= 150000')
confirmed_users.each do |user|
MyWorker.perform_async(user.id)
end#confirmed_users.each do |user|
confirmed_users = User.where('confirmed_at IS NOT NULL').where('id > 200000')
confirmed_users.each do |user|
MyWorker.perform_async(user.id)
end#confirmed_users.each do |user|
end
end
If I can know the limit of sidekiq, I can make the user sets dynamically. And I wanted to know is this correct way to complete the process in less time. Or is there any way that I can process all my records in less time ?

Answer 1

Sidekiq only processes as many jobs concurrently as you have workers/threads. The rest will be placed in the queue and the queue is practically unlimited. No issues with 200k jobs.
Your issue probably comes from the slowness of querying for 200k jobs using 1 sql query and having to keep the result in memory while creating jobs from them.
Use find_each to tell Rails to find the records in batches and yield them one-by-one.
namespace :users do
task data: :environment do
User.where('confirmed_at IS NOT NULL').find_each do |user|
MyWorker.perform_async(user.id)
end
end
end
However, since you only need the id, not the entire user object, we can also remove the object initiation to speed it up more.
User.where('confirmed_at IS NOT NULL').in_batches.each do |batch|
batch.pluck(:id).each do |id|
MyWorker.perform_async(user.id)
end
end
And if that still is not fast enough, there is Sidekiq::Client.push_bulk. It will only make one request to redis for each batch. Might need to adjust batch size here.
User.where('confirmed_at IS NOT NULL').in_batches.each do |batch|
args = batch.pluck(:id).map { |id| [id] } # args is [[1], [2], [3], etc...]
Sidekiq::Client.push_bulk('class' => MyWorker, 'args' => args)
end