I'm using a paid Heroku ($35/mo 2x web dyno, $50/mo silver postgresdb) plan for a small internal app with typically 1-2 concurrent users. Database snapshot size is less than 1MB.
App is Rails 4.1. In the last two weeks, there's been a significant performance drop in production env, where Chrome dev tools reports response times like 8s
Typical:
Total 8.13s
Stalled 3.643 ms
DNS Lookup 2.637 ms
Initial connection 235.532 ms
SSL 133.738 ms
Request sent 0.546 ms
Waiting (TTFB) 3.43 s
Content Download 4.47 s
I'm using Nitrous dev environment and get sub-1s response on dev server with non-precompiled assets (with mirrored db).
I'm a novice programmer and am not clear on how to debug this. Why would I see 800% slower performance than the dev environment on an $85/mo+ Heroku plan? Given my current programming skill level, my app is probably poorly optimized (a few N+1 queries in there...) but how bad can it be when production has 1-2 concurrent users??
Sample from logs if it helps:
sample#current_transaction=16478 sample#db_size=18974904bytes sample#tables=28 sample#active-connections=6 sam
ple#waiting-connections=0 sample#index-cache-hit-rate=0.99929 sample#table-cache-hit-rate=0.99917 sample#load-avg-1m=0.365 sample#load-avg-5m=0.45 sample#load-avg-15m=0.445 sample#read-iops=
37.587 sample#write-iops=36.7 sample#memory-total=15405616kB sample#memory-free=1409236kB sample#memory-cached=12980840kB sample#memory-postgres=497784kB
Sample from server logs:
Completed 200 OK in 78ms (Views: 40.9ms | ActiveRecord: 26.6ms
I'm seeing similar numbers on dev server but the actual visual performance is night and day. The dev responds as you would expect - sub-1s response and render. The production server is 4-5s delay.
I don't think it's related to ISP as suggested because I've actually been traveling and seeing identical performance problems from USA and Europe.
Add NewRelic add-on to your app's ecosystem on Heroku and explore what's going on. You can do it from Heroku's dashboard. Choose free plan and you will be provided with 2 weeks trial period of full functionality. After trial period there will be some constraints but anyway it's enough for measuring performance of the small app.
Also you can add the Logentries add-on and you will get access to the app's log history via web interface.
Related
Our Ruby on Rails website on heroku shows a Waiting (Time to First Byte) in chrome inspector of 1000ms+ on most requests for all pages.
Heroku's logs and New Relic both show total response times of under 200ms however. (This includes request Queuing)
The heroku app has two dynos and doesn't go into idle.
What could account for 800ms on average of missing time?
I believe this discrepancy is due to the time it takes to leave the server vs. the time it takes to be received by the client.
I sent a support ticket to heroku support who were great enough to look into it show the latency was occurring before the request arrived at the heroku server. The application makes use of CloudFlare which was contributing to the latency. CloudFlare had a blog write up on this at https://blog.cloudflare.com/ttfb-time-to-first-byte-considered-meaningles/ which goes into full detail.
I'm running a Rails 2.3.11 app on Heroku Bamboo stack and I'm getting awful performance issues, as lots of pages take more than 15-30 seconds to load (resulting in frequent timeout errors).
The same app running on my local development environment, using the same database load, instead runs reasonably well (around 1000 milliseconds).
I've tried to use addons as NewRelic but I cannot make sense of it, I find it too difficult to understand.
Basically I need to understand if my bottleneck are slow db queries, slow remote urls (es: google maps api queries), some misconfiguration issue or the infamous Heroku random routing.
What would you suggest me to do?
UPDATE
As suggested I had a look at the logs and for instance when I call the homepage I get this result:
Completed in 8014ms (View: 437, DB: 698) | 200 OK
What does it mean that it is completed in 8014ms when the sum of the milliseconds of the DB and the View should be around 1 second?
I've a rails application which I am expecting to be low traffic, it is working fine on heroku free tier as of now.
Can I use Heroku free tier with my custom domain as my production environment? As of now I see 750 dyno hours will be fine for my website to work continuously but I want to know if there are any drawback of using free tier for production website.
For uploads I am already using amazon S3.
Thanks,
Yes, you can use custom domain for your production env on Heroku.
Heroku has database rows limitation for free tier; it takes away the INSERT privilege, if that limit is crossed and offers you to buy atleast basic (9$/month) database plan (10 million rows).
My service was shut down for same reason a while back; I got following email from them:
The database HEROKU_[hidden] on Heroku app [hidden] has
exceeded its allocated storage capacity. Immediate action is required.
The database contains 129,970 rows, exceeding the Dev plan limit of
10,000. INSERT privileges to the database will be automatically
revoked in 7 days. This will cause service failures in most
applications dependent on this database.
To avoid a disruption to your service, migrate the database to a Basic
($9/month) or Production plan:
Heroku free service is totally awesome for light production. Upgrade the performance by buying adequate dynos and database. You may need them when there are more incoming requests that they are getting queued up; consequently leading to occasional timeouts. John sufficiently answers when you may need more dynos here - https://stackoverflow.com/a/8428998/1376448
You will totally love it!
I have a Rails 3.2.12 app set up locally and on DotCloud. I'm seeing very slow ActiveRecord (postgres) performance on Dotcloud and can't work out why:
Page Load Macbook Air (rails app in development mode):
Completed 200 OK in 617ms (Views: 361.3ms | ActiveRecord: 39.1ms)
Page Load DotCloud (rails app in production mode, identical DB and page):
Completed 200 OK in 796ms (Views: 315.3ms | ActiveRecord: 329.4ms)
This is not an erratic time, but the standard performance delta on all page loads. My database is only 16MB, so not large. Memory allocation on the postgres service is sufficient (128MB), with only 30MB being utilized. I checked my local postgres.conf and the settings are default, untuned postgresql.app settings.
Is this poor performance just what to expect in the cloud? Is it network latency between the web server and db server?
Would very much appreciate thoughts on how to debug & fix this!
As a dotCloud employee, I have to tell you that if you want a thorough look at your application to see why it's not performing to your expectations, you have to file a support ticket. I should also tell you that if you're in sandbox mode, you can expect lesser performance compared to if you were in one of the paying tiers (such as Live or Enterprise).
As a fellow developer, however, I can quickly point out two key differences between your local dev environment and your dotCloud environment.
Your dev environment runs your DBMS (PostgreSQL) and your rails app on the same host, eliminating any noticeable latency caused by round-trips to the database. With dotCloud, these are likely to be on separate hosts and possibly even in separate datacenters.
Your dev environment probably uses a Solid State Drive (SSD) (depending on the age of your MacBook Air) and you just moved into EBS (Amazon Elastic Block Store), a form of network-attached storage (on hard disk drives).
Between these two changes, I'm not totally shocked by the increase in response time, but my curiosity is piqued by your ActiveRecord time increasing by 290ms while your overall response time only increased by 180ms.
I don't want to speculate where your response time is being eaten up, so once again I recommend you file a support ticket at support.dotcloud.com as soon as possible so that we can take a closer look. If and when you do, mention my name or this StackOverflow thread and include the URL for your app.
After performing load testing against an app hosted on Heroku, I am finding that the most DB intensive request takes 50-200ms depending upon load. It never gets slower, no matter the load. However, seemingly at random, the request will outright timeout (30s or more).
On Heroku, why might a relatively high performing query/request work perfectly 8 times out of 10 and outright timeout 2 times out of 10 as load increases?
If this is starting to seem like a question for Heroku itself, I'm looking to first answer the question of whether "bad code" could somehow cause this issue -- or if it is clearly a problem on their end.
A bit more info:
Multiple Dynos
Cedar Stack
Dedicated Heroku DB (16 connections, 1.7 GB RAM, 1 comp. unit)
Rails 3.0.7
Thanks in advance.
Since you have multiple dynos and a dedicated DB instance and are paying hundreds of dollars a month for their service, you should ask Heroku
Edit: I should have added that when you check your logs, you can look for a line that says "routing" That is the Heroku routing layer that takes HTTP request and sends them to your app. You can add those up to see how much time is being spent outside your app. Unfortunately I don't know how easy it is to get large volumes of those logs for a load test.