I have an app that makes hundreds of thousands of calls to external APIs (via HTTP requests) each day.
Does the number of web dynos I have on my Heroku app affect the speed or number of calls I can make per second, or the ability to save the data retrieved?
No, your rake tasks will not slow down your web-dynos. Nor will your web dynos have an effect on your rake tasks. They run in separate processes (a dyno).
Heroku will end up billing you for your rake tasks though. (https://devcenter.heroku.com/articles/usage-and-billing)
One-off dynos
When executing a one-off dyno with heroku run, a dyno will be
provisioned for your command, and the time spent executing the command
will accrue usage.
Related
I guess I need a sanity check here because if I want to prevent any sidekiq jobs from ending prematurely, Heroku Redis should handle this for me?
When I want to push new changes to a production site, I put the application in maintenance mode: heroku maintenance:on. Now when I do this and run heroku ps I can see both my web process and my worker (i.e. sidekiq) are up still (makes sense because its just to prevent users having access to the site).
If I shut down the worker dyno with a command like this: heroku ps:stop worker after the site is in maintenance mode, will this safely stop sidekiq workers before it does down? Also, from Sidekiq's documentation:
https://github.com/mperham/sidekiq/wiki/Deployment#heroku
It mentions a -t N switch where N is a number in seconds but that Heroku has a hard limit of allowing a process 30 seconds to shut down on its own. Am I correct that if I stop the worker process with the heroku command, it will give any currently running jobs N seconds to finish before giving it a SIGTERM signal?
If not, what additional steps do I need to take to make sure Sidekiq has safely shut down?
Sounds like you are fine. Heroku sends SIGTERM when you call ps:stop. Sending SIGTERM tells Sidekiq to shut down within N seconds. Your worker dyno should be safely down within 30 seconds.
Currently our Heroku app has two dynos: web and worker. The worker dyno is set up to run bundle exec rake jobs:work, which starts up delayed_job. I have some new Sidekiq jobs that I also need to run. (I plan to convert our delayed_job jobs to Sidekiq soon, but haven't yet.) My question is: do I need to add and pay for a third Heroku dyno ("sidekiqworker"?), or is there a way for me to specify that my existing worker dyno run both delayed_job and Sidekiq?
You will need to pay for a third heroku dyno unfortunately. I've experimented with naming both processes as "Workers" but only one would be registered while the other one wouldn't be. When adding a new process name, heroku updates and will set that new process name to 0 dynos.
Refer to this for more details multiple worker/web processes on a single heroku app
I'm under the impression that free dynos will spin down after a while.
What happens to a script that's running currently with my main ruby server / fires off PhantomJS sraper every now and again?
Do I need a dedicated worker process for this or will Heroku Scheduler do just fine alongside a paid dyno?
I've no issues paying for it, the development always takes a hot second and their workers are a little pricey.
Thanks in advance.
If you want to periodically run a script, Heroku Scheduler is really the ideal way to do this. It'll use one-off dynos, which DO count towards your free dyno allocation each month, but only run during the duration of the task, and stop afterwards.
This is much cheaper, for instance, than running a dedicated worker dyno that is up 24x7, vs a one-off dyno (powered by Heroku Scheduler) which only runs for a few minutes per day.
I have about six Rake tasks I want to run around 4AM every morning. The issue is, they won't run at the same time.
I don't have a worker dyno switched on as I thought this was just wasting money. I'm not 100% sure why this work dyno actually exists.
How do I make the rake tasks all run at the same time? Would switching on the worker dyno make this work?
Having them all run concurrently is tough with just the Heroku scheduler. One dyno and Rake tasks won't do it. You'll need to use a threaded background job of some type. I have used sidekiq the most and like it the best.
There are a couple of moving parts to this but you'll basically need a worker dyno to run sidekiq and you can set the concurrency to 6 and then run them via a cron task such as whenever or clockwork.
I think this is the best way to handle your problem if you truly need them to run at very close to the same time. Exactly the same time isn't going to happen.
So right now I have an implementation of delayed_job that works perfectly on my local development environment. In order to start the worker on my machine, I just run rake jobs:work and it works perfectly.
To get delayed_job to work on heroku, I've been using pretty much the same command: heroku run rake jobs:work. This solution works, without me having to pay anything for worker costs to Heroku, but I have to keep my command prompt window open or else the delayed_job worker stops when I close it. Is there a command to permanently keep this delayed_job worker working even when I close the command window? Or is there another better way to go about this?
I recommend the workless gem to run delayed jobs on heroku. I use this now - it works perfectly for me, zero hassle and at zero cost.
I have also used hirefireapp which gives a much finer degree of control on scaling workers. This costs, but costs less than a single heroku worker (over a month). I don't use this now, but have used it, and it worked very well.
Add
worker: rake jobs:work
to your Procfile.
EDIT:
Even if you run it from your console you 'buy' worker dyno but Heroku has per second biling. So you don't pay because you have 750h free, and month in worst case has 744h, so you have free 6h for your extra dynos, scheduler tasks and so on.
I haven't tried it personally yet, but you might find nohup useful. It allows your process to run even though you have closed your terminal window. Link: http://linux.101hacks.com/unix/nohup-command/
Using heroku console to get workers onto the jobs will only create create a temporary dyno for the job. To keep the jobs running without cli, you need to put the command into the Procfile as #Lucaksz suggested.
After deployment, you also need to scale the dyno formation, as heroku need to know how many dyno should be put onto the process type like this:
heroku ps:scale worker=1
More details can be read here https://devcenter.heroku.com/articles/scaling