Tell Sidekiq to use all available Heroku workers - ruby-on-rails

Question

I need to batch process a large set of files (millions of database records) as quickly as possible. For this purpose, I split the files into 3 directories and set up Sidekiq with the standard configuration (no config file).
I then started 3 Heroku workers and called 3 methods, which started 3 Sidekiq workers, all with the "default" queue. Initially, Sidekiq used 2 Heroku workers and after a while it decided to use only 1 worker.
How can I force Sidekiq to use all 3 workers to get the job done asap?
Thanks

Answer 1

I found the solution at the bottom of this page: http://manuelvanrijn.nl/blog/2012/11/13/sidekiq-on-heroku-with-redistogo-nano/
# app/config/sidekiq.yml
:concurrency: 1
# Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq -e production -C config/sidekiq.yml
Also, if you have many workers and a free / cheap Redis instance, make sure you limit the number of connections from each worker to the Redis server:
# app/config/initializers/sidekiq.rb
require 'sidekiq'
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
Sidekiq.configure_server do |config|
config.redis = { :size => 2 }
end
You can calculate the maximum of connections here: http://manuelvanrijn.nl/sidekiq-heroku-redis-calc/

Answer 2

I wanted to clarify a few things about your question. Your question reads "Tell Sidekiq to use all available Heroku workers". In fact, for each Dyno, a sidekiq process will execute using a command like bundle exec sidekiq -e production -C config/sidekiq.yml. Each one of these Sidekiq processes can handle multiple threads as specified in the config/sidekiq.yml file with a line like: :concurrency: 3 which is what the Sidekiq docs recommend for a Heroku standard-2x dyno (read here for more details https://github.com/mperham/sidekiq/wiki/Heroku), which only has 1gb of memory.
But technically you don't need to tell Sidekiq to use all available Heroku processes. There is another key element of this, which is the Redis server. Our main app will publish messages to a Redis server. Each Sidekiq process running on a given Dyno can be configured with the same Queue and thus all are subscribed to that Redis queue and pull the messages from the Queue. This is clearly stated by the creator of Sidekiq from the Sidekiq github page: https://github.com/mperham/sidekiq/issues/3603.
There are a couple of key points to share the load. First restrict the concurrency of a given Sidekiq process to a number like I mentioned above. Second, you can also limit the connections to the Redis server from within Sidekiq.configure_client. Finally, think of the Heroku load balancing somewhat different from how ALB in AWS works. ALB is a round robin that distributes traffic to instances in Target Groups based on certain metrics defined in Launch Templates and Auto Scaling Groups, such as vCPU utilization, memory utilization and read/write io. Rather the load balancing here is more like a publish-subscribe system where Sidekiq instances take work when they are able to and based on their restrictions both on concurrency and connection limits to the Redis server.
Finally, Heroku discourages long running jobs. The longer your job runs the higher the amount of memory it will eat up. Heroku dynos are expensive. The standard-2x is 4x the cost of a t3.micro in AWS for the same vCPU and Memory (1gb). Furthermore, in AWS you can create a spot fleet, where you purchase compute for 10 percent of the cost of its on-demand price and then execute these spot instances as batch jobs. In fact, AWS also has a service called AWS Batch. The spot fleet option doesn't exist in Heroku. Therefore, it's important to keep price in mind and thus how long the job runs. Read this article here where Heroku delineates why it is bad running long running jobs in Heroku environment: https://devcenter.heroku.com/articles/scaling#understanding-concurrency. Try to keep a job under 2 minutes.