I have a production app on heroku with postgres, serving a Rails API. Every day between 7-8am, there will be one or more long-running requests (about 20s or more) and it only occurs then. According to logs the time spent is in the database
There are no scheduled jobs
Backups are not scheduled anywhere close to that time
Traffic is at its lowest at that time
Memory is stable
There are other requests between the daily restart and that, so the dynos are not "cold"
It is not always the same endpoint, and it doesn't occur any other time
I'm not sure if it means anything but timezone is Singapore GMT+8, so 7-8am is right before midnight.
Has anyone else experienced this or has ideas to troubleshoot?
EDIT to add details:
You can see here (Scout APM) that it precedes the high throughput, basically before the start of the work day, so it's not due to load.
In fact there is pretty much no load at all. Neither is it the first request since server restart (at 4am). The slow request at 7:46am here was repeated at night (same url, same query string) which finished in under a second
I'm working on a project and we are using Icinga to monitor some services. However, we need to get a downtime from some services, but I can't find it.
For an example:
My service is UP, running for 5 mins.
Suddenly, service is down.
After 10 minutes, service is up again.
Okay, how can I get the 10 minutes of down service? I mean,I know that I can get two times (last time it was up, and when it came back up), but can I get this information somewhere else?
Thanks.
You can look at the service history, either on the GUI (icingaweb2 or another one), or with a request against:
The core API ;
Livestatus.
There's a similar question about app harbor on StackOverflow, but the user didn't try to use new relic to overcome the problem.
I deployed my ASP.NET MVC project on App Harbor. It's very easy to configure and you can even set automatic deployments from Git. However, as my website is still mainly used only by me, I was getting very long cold starts (over 15 secs). To avoid it, I installed New Relic. The idea was to simultaneously to monitor the application but also to create periodic pings that, according to "a lot of people", would drastically reduce the loading time.
It's not working. I have New Relic correctly pinging my application every minute, but I still get very long cold starts. For instance, 5 min ago, I've got a cold start of 16 seconds. 1 minute after, I got the page loaded in less than a second.
I know I could have used Pingdom or StillAlive to achieve the same result:
How do I improve app performance on AppHarbor?
I wouldn't like to do it because I like New Relic and I don't want to have a lot of add-on's on app harbor as they will slow down my website. Do you have any idea what might be causing it?
I'm not familiar with AppHarbor's setup. But if it's using IIS, the pinging is just keeping the application pool from reaching the idle timeout. But there the default IIS setting for the application pool to be recycled every 29 hours no matter the number of requests. And it's normally in the best interest to let it recycle once in a while, so working around it may not be in your best interest.
Your best bet is to reduce the number of things happening on application start. Precompiling your views is a good place to start. And heck, Stack Exchange/Stack Overflow precompiles views to avoid the application start up cost.
Is there a way to see how many dyno hours you have used so far this month on Heroku? I know it's on the invoice you get, but I want some "live" data.
I use some "One off processes" (heroku run rake...) quite often and want to track this somehow.
It was a bug.
Contacted Heroku support and they fixed it. You can see current usage here: https://api.heroku.com/account
I have been hosting a site on Heroku for a few months that is very soon to go into production.
Since I began with them, there have been at least three significant outages, one of which was the disastrous Amazon outage last month and another of which is a multi-hour outage happening today.
I believe in Heroku's vision and I think they are a great company, but I am faced with the ultimate problem: if they can't keep sites up and running, everything I like about them doesn't really matter.
Is Heroku a reliable provider to run a production site on Rails?
Are there any other providers I might look into that have a better reputation for reliability than Heroku?
In my opinion, downtime can happen with almost any provider. What you need to see is how well or badly the host handles the downtime and the effort they make in keeping the customer updated about possible resolution.
In my opinion Heroku is a great place to host your app. The advantages and ease of deploying there covers up for the recent (and rare) downtime FOR ME.
I am user of Heroku with Amazon RDS plugin for the past 7-8 months and my conclusion is there is nothing to appreciate about Heroku except their architecture. Here is why I think:
Even though it is sold for $250 million+ they were still NOT using the Amazon multiple zones feature of Amazon. Below is the link how SmugMug survived amazon crash by using Amazon's multiple zones feature.
http://don.blogs.smugmug.com/2011/04/24/how-smugmug-survived-the-amazonpocalypse/
No phone contact support in the event of issues (not application but Heroku's), lot to learn from Rackspace
The application I am hosting, people will starve if it goes down for few hours on Friday forget about 60 hours downtime.
I see intermittent deployment and connectivity issues. Please visit this link for a confirmation:
http://status.heroku.com/
I know developers love it because they throw a cheap web process called 'dyno' for free.
So far Heroku does not offer multiple availability zone redundancy. If you want something more reliable than Heroku you can create your own EC2 instances in multiple availability zones. Of course this will require significantly more server upkeep, admin, and deployment time.
I have seem Heroku to be reliable. I highly recommended it for starting out and validating your idea. I believe when you start your project you want get it out quickly (to customer or to public).
As mentioned in other comments at some point you might need to switch over to EC2 as you might need zone redundancy and it might actually become cheaper to run of EC2 especially if you already have an SA in the company.
No. It is not. As a customer I've experienced multiple critical outages. These things happen and I get that. But what makes Heroku unreliable is their nearly non-existent support when things do go wrong. I would use caution when evaluating Heroku or any provider for that matter and really understand what you're paying for. Paying as much as I did for Heroku I expected more.
As an example one of their databases went offline early on a Sunday. I immediately was made aware, not from Heroku but from our customers and new relic alerts. I contacted Heroku support just to get the ball rolling as I began to troubleshoot. 24 hours later I had literally no responses from Heroku. I could not fork, follow, or take snapshot of the database as they suggest (because they were experiencing issues) so I basically sat on my hand and waited. Hoping that somebody would respond as I frantically attempt to recover somehow, someway.
Was this their fault. No. Not at all. I should/could have done something to mitigate this failure. But as much as I pay for their servies each month I expected something resembling a response to my critical issue.
Our our app is hosted by Heroku and went down mutliple times over the last 12 months.
Two times it was caused by one of the third-party apps that Heroku offers:
We used Zerigo (recommended by Heroku) for our DNS. This has caused our site to go down twice - one time it took over 12 hours te recover. This is absolutely crazy for something like DNS, so we have switched to a more reliable provider.
The Redistogo app went down once.
Heroku does bring some benefits, but be careful about the apps you select.
In my org i build simple SPA productivity apps, and have been using Heroku to host them for the last year after migrating away from a physical box server to cloud VMs.
I've had multiple days lost due to Heroku development hindering outages. Usually while running apps stay online, and work, when Heroku goes down you can't push updates or restart apps.
Lets also not forget the ridiculous times for scheduled maintenance (usually 2PM EST, midweek....REALLY?)
As of writing this, the Logging system for Heroku has now been acting up (more or less down) for over 24 hour.
Thankfully my apps aren't mission critical. While I like Heroku's ease of use, it's just not worth this much headache for what is nothing other than an AWS middle-man.
That said, I'm moving over to just pure AWS EC2 instances.