We've got a Rails app where certain requests trigger long-running tasks on a worker dyno with delayed_job, while the front-end polls until it receives a result.
Our traffic is small but growing, and the tasks generally take only a few seconds to complete, but can take up to a minute. Right now a single web and worker dyno each should be sufficient to handle our load.
The problem is, the delayed_job queue won't process jobs in parallel, so a longer task ends up holding up the tasks behind it.
What we're looking for is something like Unicorn for the backend, where a single worker dyno can process multiple tasks concurrently. And since it's Rails, we're looking for something multi-process, not multi-threaded.
(We tried creating multiple worker entries in our procfile- This worked on the local dev box, but not on Heroku)
Procfile:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
# Multiple workers defined here doesn't translate into multiple processes on single worker dyno:
worker: bundle exec rake jobs:work
worker: bundle exec rake jobs:work
worker: bundle exec rake jobs:work
This Heroku article proposes using the resque-pool gem as a solution. Is that the only solution, or can delayed_job be used for parallel background jobs as well?
According to #radiospiel's related post, you can use foreman to start multiple processes.
1) Add foreman to your Gemfile
2) Create two files:
Procfile:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec foreman start -f Procfile.workers
Procfile.workers:
dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work
I just deployed this to Heroku, works great.
Related
i'm using sidekiq in a rails DEVELOPMENT environment with rvm and passenger.
At app's boot, i need to manually start Sidekiq with:
bundle exec sidekiq --environment development -C sidekiq.yml
is ther a way to Autostart it on App start or restart (not server boot) ?
thanks
Use the foreman gem. With foreman, you declare all the processes your app requires to run in a Procfile.
See a sample Procfile definition of an app using puma and sidekiq below:
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq --environment development -C sidekiq.yml
The foreman start command starts your app.
I am currently running delayed::job on my Heroku instance for quite a few different types of jobs (exporting large lists etc.) and I'm using the awesome progress job gem that shows a progress bar of the job to the user who made the request.
I'd like to be able to run a sidekiq worker as well for other jobs, not involving the user, that I don't need a progress bar for, because of it's obvious memory improvements over Delayed::Job.
Is it possible for me to run both delayed job and sidekiq on the same heroku app? If so is there any examples I can follow? I'm confused on how I would setup the procfile or this.
Below is my Procfile. I don't see how to start both Delayed Job and Sidekiq? If I do something like just add bundle exec sidekiq on a worker line below, it seems to replace delayed job?
Procfile:
web: bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development}
web: bundle exec puma -C config/puma.rb
worker: bundle exec rake jobs:work
It is possible.
First, You should solve similar interfaces conflict problem like .delay. here: Run Delayed Jobs and Sidekiq at the same time
Second, you should set multiple active_job adapter configuration. here: http://edgeguides.rubyonrails.org/active_job_basics.html#backends
About Procfile, You can register multiple worker
For example)
web: bundle exec puma -t 5:5 -p ${PORT:-3000} -C config/puma.rb
worker: bundle exec sidekiq
delayedjobworker: bundle exec rake jobs:work
Check https://devcenter.heroku.com/articles/procfile#more-process-type-examples
I'm trying to deploy my production app on Heroku. I'm using both private_pub and Puma in my app.
I wanted to start both using this procfile :
web: bundle exec puma -C config/puma.rb
web: bundle exec rackup private_pub.ru -s thin -p $PORT -E production
But it fails, it seems like you can have only one web process running at a time on Heroku apps. Is there a way to initialize both ?
ok, so basically you cannot run the 2 processes on heroku at the same time. An easy alternative is to create a dedicated heroku app which sole purpose will be to maintain the Pubsub server running. To do so, just follow this tutorial
I have a Rails app which uses a Procfile to start sidekiq automatically on heroku. I'd like it to start sidekiq automatically on localhost (I currently just 'bundle exec sidekiq' in a separate window). Here's my procfile:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq
How would I do this? I do have foreman installed locally
You can create a Procfile.dev file that's meant to be used in development. To use it, just do 'foreman start -f Procfile.dev' from the terminal. Passing the -f option allows you to set the path the Procfile to use.
By the way, would probably be good to gitignore your Procfile.dev, as well. That way, others in your team could have their own Procfile.dev.
Hope that helps!
I'm using clockwork for the first time. I followed the Heroku guide, https://devcenter.heroku.com/articles/clock-processes-ruby, but when I run heroku ps:scale clock=1, I get this error:
Scaling web dynos... failed
! No such process type web defined in Procfile.
My Procfile looks like this:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec rake jobs:work
clock: bundle exec clockwork lib/clock.rb
The project had already been pushed to Heroku with the web and worker processes, and it worked fine.
I did foreman check and it says:
valid procfile detected (web, worker, clock)
I also did heroku run bash and the Procfile is there.
Any ideas what I may have missed?