I'm running a Rails 2.3.11 app on Heroku Bamboo stack and I'm getting awful performance issues, as lots of pages take more than 15-30 seconds to load (resulting in frequent timeout errors).
The same app running on my local development environment, using the same database load, instead runs reasonably well (around 1000 milliseconds).
I've tried to use addons as NewRelic but I cannot make sense of it, I find it too difficult to understand.
Basically I need to understand if my bottleneck are slow db queries, slow remote urls (es: google maps api queries), some misconfiguration issue or the infamous Heroku random routing.
What would you suggest me to do?
UPDATE
As suggested I had a look at the logs and for instance when I call the homepage I get this result:
Completed in 8014ms (View: 437, DB: 698) | 200 OK
What does it mean that it is completed in 8014ms when the sum of the milliseconds of the DB and the View should be around 1 second?
Related
I'm using Heroku for a production Rails application.
I'm monitoring it with scoutapp and noticed that requests time can be 4 times slower or faster after a deploy in production.
I made some screenshots this time, but this happened multiple times, if I'm luky it will be fast after deploy.
The deployment just contains a css update
heroku stats also shows slower response time:
I'd guess that slower times are due to cache being cleared when you deploy and the server restarts. When each request is cached again you would expect it to speed back up.
I can't offer an explanation for the faster times though.
I'm using a paid Heroku ($35/mo 2x web dyno, $50/mo silver postgresdb) plan for a small internal app with typically 1-2 concurrent users. Database snapshot size is less than 1MB.
App is Rails 4.1. In the last two weeks, there's been a significant performance drop in production env, where Chrome dev tools reports response times like 8s
Typical:
Total 8.13s
Stalled 3.643 ms
DNS Lookup 2.637 ms
Initial connection 235.532 ms
SSL 133.738 ms
Request sent 0.546 ms
Waiting (TTFB) 3.43 s
Content Download 4.47 s
I'm using Nitrous dev environment and get sub-1s response on dev server with non-precompiled assets (with mirrored db).
I'm a novice programmer and am not clear on how to debug this. Why would I see 800% slower performance than the dev environment on an $85/mo+ Heroku plan? Given my current programming skill level, my app is probably poorly optimized (a few N+1 queries in there...) but how bad can it be when production has 1-2 concurrent users??
Sample from logs if it helps:
sample#current_transaction=16478 sample#db_size=18974904bytes sample#tables=28 sample#active-connections=6 sam
ple#waiting-connections=0 sample#index-cache-hit-rate=0.99929 sample#table-cache-hit-rate=0.99917 sample#load-avg-1m=0.365 sample#load-avg-5m=0.45 sample#load-avg-15m=0.445 sample#read-iops=
37.587 sample#write-iops=36.7 sample#memory-total=15405616kB sample#memory-free=1409236kB sample#memory-cached=12980840kB sample#memory-postgres=497784kB
Sample from server logs:
Completed 200 OK in 78ms (Views: 40.9ms | ActiveRecord: 26.6ms
I'm seeing similar numbers on dev server but the actual visual performance is night and day. The dev responds as you would expect - sub-1s response and render. The production server is 4-5s delay.
I don't think it's related to ISP as suggested because I've actually been traveling and seeing identical performance problems from USA and Europe.
Add NewRelic add-on to your app's ecosystem on Heroku and explore what's going on. You can do it from Heroku's dashboard. Choose free plan and you will be provided with 2 weeks trial period of full functionality. After trial period there will be some constraints but anyway it's enough for measuring performance of the small app.
Also you can add the Logentries add-on and you will get access to the app's log history via web interface.
So I got a rails app deployed on heroku. Nothing fancy going on, just a small custom cms, just used for uploading and editing static pages. Most of the time is just spent displaying pages, stored in the db.
There is a bit of a heavy load for each page display since the pages are stored as markdown which is then being rendered (and string-interpolatet). But even with the markdown rendering turned of, the server is still responding fairly slow (avg. 600 ms).
I'm trying to reduce the response time. The graph in New Relic tells me, that the app spends about 80 % or 90 % of its processing time, executing ruby code. I thought that I could make this faster by adding dynos, but no matter how many dynos I add, the ruby processing won't speed up. I'm currently running on just 1 dyno, so ramping it up to 4 should make it (roughly) 4 times faster, no? I tried 8 dynos but still no significant difference (not that I could afford 8 dynos anyways).
After performing load testing against an app hosted on Heroku, I am finding that the most DB intensive request takes 50-200ms depending upon load. It never gets slower, no matter the load. However, seemingly at random, the request will outright timeout (30s or more).
On Heroku, why might a relatively high performing query/request work perfectly 8 times out of 10 and outright timeout 2 times out of 10 as load increases?
If this is starting to seem like a question for Heroku itself, I'm looking to first answer the question of whether "bad code" could somehow cause this issue -- or if it is clearly a problem on their end.
A bit more info:
Multiple Dynos
Cedar Stack
Dedicated Heroku DB (16 connections, 1.7 GB RAM, 1 comp. unit)
Rails 3.0.7
Thanks in advance.
Since you have multiple dynos and a dedicated DB instance and are paying hundreds of dollars a month for their service, you should ask Heroku
Edit: I should have added that when you check your logs, you can look for a line that says "routing" That is the Heroku routing layer that takes HTTP request and sends them to your app. You can add those up to see how much time is being spent outside your app. Unfortunately I don't know how easy it is to get large volumes of those logs for a load test.
Ok. I know I don't have a lot of information. That is, essentially, the reason for my question. I am building a game using Flash/Flex and Rails on the back-end. Communication between the two is via WebORB.
Here is what is happening. When I start the client an operation calls the server every 60 seconds (not much, right?) which results in two database SELECTS and an UPDATE and a resulting response to the client.
This repeats every 60 seconds. I deployed a test version on heroku and NewRelic's RPM told me that response time degraded over time. One client with one task every 60 seconds. Over several hours the response time drifted from 150ms to over 900ms in response time.
I have been able to reproduce this in my development environment (my Macbook Pro) so it isn't a problem on Heroku's side.
I am not doing anything sophisticated (by design) in the server app. An action gets called, gets some data from the database, performs an AR update and then returns a response. No caching, etc.
Any thoughts? Anyone? I'd really appreciate it.
What does the development log say is slow for those requests? The view or db? If it's the db, check to see how many records there are in database and see how to optimize the queries. Maybe you need to index some fields.
Are you running locally in development or production mode? I've seen Rails apps performance degrade faster (memory usage) over time in development mode. I'm not sure if one can run an app on Heroku in development mode but if I were you I would check into that.