I was able to scale my Heroku workers by using heroku ps:scale worker=3 from CLI.
Do I also need to edit my Procfile like worker: env TERM_CHILD=1 QUEUE='*' COUNT='3' bundle exec rake resque:workers or is this redundant?
The command heroku ps:scale worker=3 will spin up 3 dynos each running the worker defined in your Procfile.
Your definition of worker is worker: env TERM_CHILD=1 QUEUE='*' COUNT='3' bundle exec rake resque:workers. This command will create 3 worker threads within a dyno. These worker threads will share the resources of the dyno (memory, cpu).
So if you make both changes then you'll end up with 3 dynos each with 3 worker threads - for a total of 9 workers threads.
Hope that clarifies things. I don't know the needs of your application I'll let you be the judge of that but if your jobs aren't that intensive then you may be able to get by with 1 dyno that has 3 worker threads. You can save some money this way. Goodluck
Related
My application is processing jobs with Resque.enqueue
I am starting the worker with rake resque:work QUEUE='converter'
To my understanding, in order to start 2 workers to process 2 requests for the same queue concurrently, I can start another worker with rake resque:work QUEUE='converter' in another terminal.
Is there a simple option to start 2 workers to work concurrently on the same queue without using the resque-pool gem, and without having to type in rake resque:work QUEUE='converter' twice?
It's advised within the same Resque codebase to only use COUNT=2 while on development environment. To manage more than one Resque worker, you'd need something like https://github.com/nevans/resque-pool
You can specify the worker count when running the command, like so:
$ COUNT=2 QUEUE='converter' rake resque:workers
I'm confused how heroku and sidekiq work. My Procfile looks like:
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -e $RAILS_ENV
Now inside my rails I run my sidekiq jobs in my code like:
SomeWorker.perform_async(some.id)
Now will this automatically somehow make this process run in the worker dyno?
If yes, how does it just know to run this out of process?
It is confusing because when I am in my main git folder I can run heroku commands and I know this are for my web dyno, but how do I then see the logs for my worker dyno or will these be in my same dyno logs?
When you setup your Procfile, you're telling Heroku to setup 2 types of dynos: web and worker. It's likely that these are using the same Rails app code but are starting up with different commands (bundle exec puma vs. bundle exec sidekiq). You then can scale however many VMs (dynos) that you need for each type of process.
The glue that holds the two together is Redis. When you run SomeWorker.perform_async(some.id) from your web process, you're adding a record to Redis describing a job to run. Your worker process watches Redis for new records and then processes them.
The Heroku logs show logs from all running dynos. So you should see logs from both your web and worker processes mixed in together.
Let's say my process will run 3 workers, and I want to dedicate 1 of them to process web requests, and 2 to handle Sidekiq background jobs, each process potentially being multi-threaded. Is there an easy or best-practices way to handle this? I've tried a few different configurations, but they either give me an error, or just don't process the jobs at all.
I'm using Rails 4 and ActiveJob, but I don't think those points are relevant.
You don't have to take care of sidekiq worker scheduling from you rails application. Sidekiq runs a separate process with the whole rails environment for managing the background workers. Each Worker is a separate Thread, backed by the Celluloid Actor framework. So for your setup you just do the following:
# start a single puma web server process with min 4 and max 16 Threads
$ bundle exec puma -t 4:16
# start a single multithreaded sidekiq process,
# with councurrency settings defined in sidekiq.yml
$ bundle exec sidekiq --pidfile tmp/pids/sidekiq_1.pid
# start another sidekiq process (if really necessary)
$ bundle exec sidekiq --pidfile tmp/pids/sidekiq_2.pid
I have been operating under 0 workers, but need to increase the speed at which I'm processing background tasks. I am using Sidekiq for all of my background workers.
When I increase the worker dyno count to 1, I keep getting this error in my heroku logs:
dont know how to build task 'jobs:work'
From researching this, it seems like the issue is that heroku worker dynos are reliant on delayed_job and I am not using delayed_job anywhere.
If I install delayed_job, what will I have to change to get sidekiq to work? Or do I even need delayed_job?
Update your projects Procfile to specify sidekiq for the worker:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq
Then redeploy your application.
I think Heroku defaults to trying to run delayed_job if you don't specify the worker in your Procfile.
I have these Resque workers defined in my Procfile
resque1: env QUEUE=mailer rake resque:work
resque2: env QUEUE=calculation rake resque:work
But seems non of them gets created, is it because the process type is wrong? If yes, how can I create two resque workers for my rails app?
[Problem Solved]
The reason been heroku only give one free dyno and the rails app already used one. Once I scale the web dyno to 0, and scale my resque dyno to 1 instance, the queued job gets executed immediately.
Thanks!