I just wanted to see what the best practice in the following situation would be.
I have setup scheduler in my heroku app to run two rake tasks, (performs a screen scrape), these are ran once a day, now from what i have read i have 750 hours free per month of dyno processes but you accrue usage even when the dyno is idle.. So do i need to run
heroku ps:scale web=0
so that the dyno doesnt accrue usage when not running or do i just leave it as it is?
What is the best thing to do here?
Thanks
If you haven't added any more web workers then you should be on the free tier. If you log into your Heroku account and go to the app's dashboard you'll see an estimated monthly cost for resources used, you can double check that it's on $0.
I tested both heroku ps:scale web=0 and heroku ps:scale web=1 on one of my apps. Both leave the cost at $0, and the app is still online even with 0 web workers, so I'm not sure how that works.
You will however pay for the scheduler, for the time it was up to call the rake task. Might be a few dollars per month, or perhaps less than a dollar, depends how long it was up for.
Related
I would like to use Rufus-Scheduler in order to send a mail daily.
I strictly followed the instructions given on GitHub here (the snippet, using Thin server, etc.); but nothnig happened (no mail sent) and I couldn't figured out the reason based on Heroku logs
my code
# config/initializers/scheduler.rb
require 'rufus-scheduler'
s = Rufus::Scheduler.singleton
# Here goes my mailing code (already tested and works well)
s.every '1d' do
Rails.logger.info "hello, it's #{Time.now}"
Rails.logger.flush
end
Is there some other points not mentionned on GitHub in order to make rufus-schedul work ? Many thanks
You can do this, but you have to understand how hobby dynos work. each free account is allocated hours like this.
Accounts are given a base of 550 hours each month in which your Free dynos can run. In addition to these base hours, accounts which verify with a credit card will receive an additional 450 hours of Free dyno quota.
Because rufus-scheduler runs as a thread in your ruby app process (this is why it starts running when you run rails console), as long as you have your server (I use puma) running, rufus scheduler will run just fine.
The downside is that if you run two processes in your server, say you run puma with 3-4 workers, you're going to have 3-4 of your schedulers running at the same time making it execute your scheduled events in triplicate/quadruplicate, so keep that in mind as well.
So the steps are simple
- make sure you have enough hours to run your dyno continuously all month
- use a service like pingdom, to ping your app every couple of minutes to keep the dyno active so that Heroku doesn't spin it down after 30 minutes of inactivity (it does that to free dynos)
- that should be all you need to do
Just remember that to run a dyno for a month you're going to need about 745 hours which your primary allocation covers (when you add a credit card). If for some reason you run out of hours (say you run two different apps on the account and use the method I describe below) then this could happen to you
A second notification will be sent when you reach 100% of your account quota, at which point your application’s dynos will be put to sleep for the remainder of that month. As a result, any apps using free dynos will not be accessible for the remainder of the month.
Seems like a lot of trouble to go to when you can just use the heroku scheduler to schedule rake tasks like everyone else does.
(Disclaimer, I'm not a Heroku expert, but I can google my way around and I can read documentation).
So, it's not your first Heroku - rufus-scheduler question... (Rails_using Rufus in order to schedule sending mails daily)
You say "in order to send a mail daily", so why don't you use the Heroku scheduler addon? https://devcenter.heroku.com/articles/scheduler It can schedule daily and even hourly (ironically, it may miss a schedule, so they recommend the below "custom clock process").
Do you realize that your dyno might be asleep when the time of the schedule comes? Heroku puts the dynos to "sleep" after a certain period of inactivity.
Heroku suggests to use a "custom clock process": https://devcenter.heroku.com/articles/scheduled-jobs-custom-clock-processes#custom-clock-processes
You have to get to know your target platform, Heroku, and adapt your system to it, with or without rufus-scheduler.
As a side-note, your previous post mentions Passenger, which is hard to tune to not kill rufus-scheduler's thread, but that wouldn't play a big role on Heroku where your dyno isn't supposed to live forever, rufus-scheduler can't outlive the dyno of its webapp, hence the "custom clock process" recommendation.
On a our Heroku Rails 4.2 web app running Unicorn workers, where each worker needs about 230MB, we can run 2 per '1X' Heroku dyno, or 4 per '2X' Heroku dyno. CPU requirements are quite low.
If we need 8 workers to handle our traffic, what are the technical / performance tradeoffs of using four '1X' dynos vs. two '2X' dynos to get 8 workers?
(Two years ago there was a well-publicized issue with the way Heroku was routing to dynos that as I recall suggested larger dynos with more workers worked best because there was less chance of a request being routed to a 'busy' dyno. But I'm not finding any current guidelines on when it is best to use larger dynos vs more dynos.)
As far as I know as you already mentioned Heroku routers don't know and care about load in a specific dyno and sends the request by a random selection algorithm. https://devcenter.heroku.com/articles/http-routing
So it is very likely to see many consecutive requests going to same dyno. I am observing that behaviour with my current applications.
But in your case if a worker need 230MB then you can't use 2 worker in a Standard-1X dyno because their max memory is 512M. So I think you should definitely go for 2X dynos with like 3-4 workers. 2X dynos can use up to 1GB memory.
I have a Rails app on Heroku and sometimes I use heroku run command that spawns one-off dyno and other times I want to run two 1x dyno so my app can run only 375 hrs for free. I don't want to spend money for my test app. I would rather prefer it to stop. Is there a way to achieve it? My credit card is linked to Heroku because I use some free addons that require its presence (such as sendgrid).
You can't turn them off automatically, but you can scale your dynos to 0 (zero) when you near the end of your 750 hours.
heroku ps:scale web=0 # Given you dynos are 'web' dynos
I have a Rails app hosted in Heroku. If I go the the site for the first time, it takes more than 10 seconds to start the app. After that, everything works fine and fast.
What should I do to boost first time loading? Should I add more Web Dynos, or more Worker Dynos, or do something with my configuration?
Thank you.
If you are on the free Heroku 1 dyno setup then the dyno will wind down after 1 hour of inactivity.
Should you pay for 1 or more extra dynos then none of you dynos will wind down and you will only ever face the bootup time when you push a new version.
You can choose to install the free New Relic add-on, and you can set this to ping your app every 2 minutes. This will stop it spinning down. http://addons.heroku.com/newrelic
Heroku unloads your app from memory if it is inactive for a while. See this answer for some suggestions: Why are my basic Heroku apps taking two seconds to load?
I just got the last month heroku bill, and the scheduled rake tasks were a relatively heavy burden. We are pretty early in our development process, so we just developed some rake tasks to get the job done recently, and didn't had much concern in theirs optimization.
Now we want to improve theirs performance and theirs heroku processing hours usage. We use New Relic to monitor the webapp performance, but apparently this type of rake tasks are ignored by default, and it's unclear how to override that.
Anyone had a similiar problem? How can I track the scheduled tasks in close to real time to monitor performance, optimize, and don't get suprise bills?
Whilst you can't really monitor rake tasks that well, there are a few little things you can do. One is the use of logging. Output start and end times of tasks to logs, and you can then see what's been happening duration wise. If you couple this with something like the Papertrail add-on then you can do additional interrogation later on.
As for running the jobs themselves, there's a couple of ways that you can run background processes which are dependant on how they need to run:
If you're needing to run jobs on a schedule, there's a few options available. Firstly there's the Heroku scheduler, which is pretty good, but doesn't guarantee executions will happen. Normally you would use this to kick off a rake task which will bring up a one-off dyno for the duration of the task - therefore you need to ensure in development that these tasks are as efficient as possible.
Alternatively, if you're looking at jobs that need a little more control or using a clock process. Essentially this is a dyno running 24/7 that does nothing but kick off other jobs at preset intervals and times. This would normally be done using the clockwork gem. The downside of this approach is that you need to pay for a clock process all the time.
A third approach, and one that might work is delayed job, with it's runat option, allowing you to queue a job to be run in the future (and jobs can re-queue themselves). There are a few issues with this in that a failure can kill the whole chain, and you need a full time worker running to process them all.
Therefore, in order to minimize your bills, ensure that your rake tasks are as performant and reliable, and then choose the scheduling option that suits you. If you're looking at schedules plus user created events, delayed_job might be the best option. If you're looking at a few tasks running periodically, then go scheduler. If you're looking at running lots of time critical jobs on a regular basis, go with clockwork.
Either way, you should be able to constrain a fair amount of processing into just one or two processes depending on your approach.
I know this question is almost 10 years old, but there is a new way!
You can now monitor your Heroku Scheduler jobs using One-off Dyno Metrics. This Heroku add-on gathers metrics for all detached one-off dynos running in your Heroku app. It was created to be an extension of Heroku's Application Metrics and works out of the box.
when you are running on heroku cedar there is a way to get a free setup for your workers. this is no answer to your monitoring question, but it might be interesting anyways: http://blog.nofail.de/2011/07/heroku-cedar-background-jobs-for-free/
You can force the New Relic agent to start in your rake tasks and report their performance data.
Not the answer to the specific question,but...
One method of reducing overhead is using Unicorn server to get multiple workers working on one dyno. It depends on your set up, but most people who've taken the time to test it can comfortably get 3 - 4 worker processes running concurrently. It's a huge boost in clearing cues or tasks. Just be careful not to max out the allocated memory for the dyno.