I have a Rails app on Heroku that is crashing with Error R15 (Memory quota vastly exceeded).
I've tracked this issue to pages that contain several asynchronous requests. The errors appears to coincide with ajax requests to build remote datatables.
The problem is, I can't figure out why these errors are being raised.
I thought perhaps the databse queries and controller actions behind the ajaxified datatables might be running slowly. But if I examine these in development using miniprofiler, the requests appear very efficent.
Then I thought, perhaps the server is receiving multiple simultaneous requests, and this is overloading the heroku dyno. But I ramped the dynos up to a very high number, and still see the error.
What would be a sensible way to start identfying and debugging what is causing this memory error? I've not had to solve a issue like this before.
Memory is allocated per-dyno on Heroku so adding more dynos will probably not actually solve the problem if it is code-level since it will cause each dyno to exceed its memory limit individually costing you lots of money and not actually solving the problem.
You're better off scaling horozontally and using Performance-L dynos. This will increase each dyno up to 14GB of memory. You can then use metrics to see how much memory is being used. If the amount of user memory manages to use up all 14GB then you may have a memory leak in one of your dependencies.
Related
I've developed web site using Ruby on rails.
But i have a serious problems.
Increasing continuously Heroku memory usage.
Heroku Response Time is too long sometimes.
I've tried a lot to solve this problem.
When visit user list page Without activerecord query execution, there is no memory increasing.
When visit user list page 20 times in normal, there is memory increasing.
So I added tunemygc gem for garbage collecting and tested. But no impaction.
So i think this Reasons of Memory issues are
Rails has such issue.
Dependencies of activerecord is not going well or there is bad dependency.
Does anyone knows the way to solve this problem.
Want to test the app by virtualizing requests ike as user does. Any idea?
Want to solve the memory issues clearly. Any ideas?
Does this solve by setting Heroku server configuration properly?
I launched an app today. After about 10 hours, once enough people had signed up and were using the application, the app came to a standstill.
I'm getting 3 types of errors:
H12: Request Timeout
R14: Memory Quota Exceeded
H13: Connection Closed Without Response
I've had 361 memory errors in the last 4 hours, and 674 Request Timeouts in the last 90 minutes.
I'm really quite unsure about how to tackle server side issues. I my Rails JSON Web API is on Heroku right now.
What's the best way to go about tackling these issues?
You have a few options:
Since your application is running out of memory, one strategy would be to run less processes-per-dyno. In your web sever configuration (Procfile), how are you running your app? A single thread? Multiple? If multiple, try lowering that number. This will use less memory, and will make your processes more stable.
To handle the 'Connection Closed Without Response' errors, you'll probably need to add more dynos to handle your increase in traffic. What's happening is that your requests are either taking too long to complete (> 30 seconds), or your app is just not finishing the processing of requests due to other issues (memory / cpu / etc).
My instinct tells me that by lowering your process concurrency per dyno, you'll likely get the performance benefits you need to solve all of these issues at once.
Heroku recently announced that cedar-10 will no longer be supported after this year in November. Switching to cedar-14 led to an increase in memory consumption until I experienced R14 "Memory Quota exceeded" errors and had to restart heroku. The same problem with increase in memory usage occured with unicorn before I started using unicorn_worker_killer gem. Is there a known issue with cedar-14 and unicorn/unicorn_worker_killer? I didn't find anything.
Here is a nice link for your 'problem' : http://blog.codeship.com/debugging-a-memory-leak-on-heroku/
It describe perfectly the continous increase in memory over time. The same 'problem' happen with Puma, there is also a Puma Worker Killer Gem
One thing to note is that you can tune your garbage collector Configuration to be more agressive. Just be careful, you can mess pretty everything with a single bad configuration.
There is -at the moment- no magic solution for this problem. We encounter it too in production, however the memory usage sometimes stabilize, just below the limit where swapping start.
As an immediate action, we choose to reduce the number of workers per dyno, reducing it to 2, and increasing the number of dyno dynamically with HireFire.
You have a loot of tools that can help, here is a list we use each days to track expensive queries / allocations :
https://github.com/brynary/rack-bug/
https://github.com/binarylogic/memorylogic
Good luck, it's not a simple problem to solve and I don't think that there is a universal true solution for it right now.
My users are seeing occasional request timeouts on Heroku. Unfortunately I can not consistently reproduce them which makes them really hard to debug. There's plenty of opportunity to improve performance - e.g. by reducing the huge number of database queries per request and by adding more caching - but without profiling that's a shot in the dark.
According to our New Relic analytics, many requests take between 1 and 5 seconds on the server. I know that's too slow, but it nowhere near the 30 seconds needed for the timeout.
The error tab on New Relic shows me several different database queries where the timeout occurs, but these aren't particularly slow queries and it can be different queries for each crash. Also for the same URL it sometimes does and sometimes does not show a database query.
How do I find out what's going on in these particular cases? E.g. how do I see how much time it was spending in the database when the timeout occurred, as opposed to the time it spends in the database when there's no error?
One hypothesis I have is that the database gets locked in some cases; perhaps a combination of reading and writing.
You may have already seen it, but Heroku has a doc with some good background about request timeouts.
If your requests are taking a long time, and the processes servicing them are not being killed before the requests complete, then they should be generating transaction traces that will provide details about individual transactions that took too long.
If you're using Unicorn, it's possible that this is not happening because the requests are taking long enough that they're hitting up against Unicorn's timeout (after which the workers servicing those requests will be forcibly killed, not giving the New Relic agent enough time to report back in).
I'd recommend a two-step approach:
Configure the rack-timeout middleware to have a timeout below Heroku's 30s timeout. If this works, it will terminate requests taking longer than the timeout by raising a Timeout::Error, and such requests should generate transaction traces in New Relic.
If that yields nothing (which it might, because rack-timeout relies on Ruby's stdlib Timeout class, which has some limitations), you can try bumping the Unicorn request handling timeout up from its default of 60s (assuming you're using Unicorn). Be aware that long-running requests will tie up a Unicorn worker for a longer period in this case, which may further slow down your site, so use this as a last resort.
Two years late here. I have minimal experience with Ruby, but for Django the issue with Gunicorn is that it does not properly handle slow clients on Heroku because requests are not pre-buffered, meaning a server connection could be left waiting (blocking). This might be a helpful article to you, although it applies primarily to Gunicorn and Python.
You're pretty clearly hitting the issue with long running requests. Check out http://artsy.github.com/blog/2013/02/17/impact-of-heroku-routing-mesh-and-random-routing/ and upgrade to NewRelic RPM 3.5.7.59 - the wait time measuring will be accurately reported.
We are using Rails 2.3.5 and have been experiencing seemingly random Timeout::Error: execution expired errors. The errors reported by Hoptoad are not consistently in any particular controller and show up everywhere from user sessions to account settings to some of our core functionality controllers.
The vast majority of requests do not Timeout but there are enough to cause concern.
Is this normal? If so, what are some things to look at to decrease the occurance? If not, has anyone run into this and what are some common problems that can trigger an error like this.
It is normal for requests to timeout, if your server is running under a heavy load. You should look to see if the timeouts are coincident with long-running SQL requests or some other activity that takes a lot of time. Often, you can decrease your timeouts by upgrading your hardware, or by optimizing your code in general. If you can't upgrade your hardware, try optimizing your longest running and most frequently accessed actions.