I have a Ruby on Rails site running on Heroku's performance-M dynamo, with autoscaling set up to 5 dynamos.
Recently, we have been receiving abrupt R15 and H12 errors on the site. During this, memory usage is shown well under the memory quota allowed for the dynamo.
Here are the errors shown in the log:
2019-09-16T10:12:08.523336+00:00 app[scheduler.2787]: Command :: identify -format '%wx%h,%[exif:orientation]' '/tmp/897302823996a945884a1d912c28d59520190916-4-1bn5w9k.jpg[0]' 2>/dev/null
2019-09-16T10:12:16.022212+00:00 heroku[scheduler.2787]: Process running mem=1022M(199.7%)
2019-09-16T10:12:16.022295+00:00 heroku[scheduler.2787]: Error R14 (Memory quota exceeded)
2019-09-16T10:12:16.365725+00:00 heroku[router]: at=info method=GET path="/favicon-16x16.png" host=www.site.com request_id=8755a947-ace9-471d-a192-a236785505b4 fwd="45.195.5.37" dyno=web.1 connect=1ms service=2ms status=200 bytes=928 protocol=https
2019-09-16T10:12:19.103405+00:00 heroku[scheduler.2787]: Process running mem=1279M(250.0%)
2019-09-16T10:12:19.103405+00:00 heroku[scheduler.2787]: Error R15 (Memory quota vastly exceeded)
2019-09-16T10:12:19.103405+00:00 heroku[scheduler.2787]: Stopping process with SIGKILL
2019-09-16T10:12:19.427029+00:00 heroku[scheduler.2787]: State changed from up to complete
2019-09-16T10:12:19.388039+00:00 heroku[scheduler.2787]: Process exited with status 137
2019-09-16T10:13:07.886016+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/favicon.ico" host=www.site.com request_id=c7cea0a2-7345-44c6-926e-3ad5a0eb2066 fwd="45.195.5.37" dyno=web.2 connect=1ms service=30000ms status=503 bytes=0 protocol=https
As you can see, just before the R15 error, paperclip was trying to compress an image.
The beginning of the graphs in the following screenshots show the status of Heroku Metrics for the affected period:
Heroku Metrics Part 1
Heroku Metrics Part 2
Can anyone please help me figure out how the R15 error, which is related to memory leakage occurring while the metrics show the memory well in the limit? Any help regarding how to stop this situation from repeating will be helpful.
Thanks.
Your R15 error occurred on a one-off dyno created by Heroku Scheduler, completely separate from your web dynos. Your request timeouts appear to be unrelated to the memory issues in your scheduled task.
The scheduled task appears to be running on a 1X dyno (mem=1022M(199.7%)). To change this, launch the Heroku Scheduler add-on and change the dyno type.
For your request timeouts, check out Scout or New Relic to find the problematic endpoint and where in the stack is taking so long.
Related
Converting a ~4.4MB quicktime (.MOV) video to .mp4 locally works fine, but on heroku it consumes over 1GB of memory and crashes the server.
Steps to reproduce
Take this video, place it in the root of a new or existing rails app.
Add gem 'streamio-ffmpeg', '~> 3.0', '>= 3.0.2' to your gemfile and bundle, push to heroku, bundle on heroku.
Run heroku run rails c to jump into the rails console and try to convert the movie:
require 'streamio-ffmpeg'
# Read video file
movie = FFMPEG::Movie.new("IMG_1459.MOV")
# Transcode video and save as an mp4
movie.transcode("IMG_1459.mp4")
The last line runs for a while then crashes.
Error logs
Converting the ~4.4MB video appears to consume over 1GB of memory (!!)
2022-03-05T13:00:24.678737+00:00 heroku[web.1]: Process running mem=1076M(210.3%)
2022-03-05T13:00:24.689256+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2022-03-05T13:00:24.691853+00:00 heroku[web.1]: Stopping process with SIGKILL
2022-03-05T13:00:24.810891+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=POST path="/posts" host=secure-inlet-07449.herokuapp.com request_id=7cd160ea-b332-4965-ae3f-563696dd054b fwd="1.145.252.113" dyno=web.1 connect=0ms service=29610ms status=503 bytes=0 protocol=https
2022-03-05T13:00:25.025812+00:00 heroku[web.1]: Process exited with status 137
2022-03-05T13:00:25.121873+00:00 heroku[web.1]: State changed from up to crashed
The logs are from similar code in an actual app.
Update
I think I seriously underestimated how much RAM it would consume. (I had assumed I had a memory leak or some unclosed connection somehow triggering a loop or similar). But it may be true that this is simply an incredibly memory-intensive task, and therefore is a bad idea to try to run in the app itself, and should be instead offloaded to an external processes (e.g. EC2's or AWS Lambdas etc).
I am using heroku to run my application and set the timeout = 12s (using racktimeout), using unicorn, sometimes I am getting the H13 issue
2010-10-06T21:51:37-07:00 heroku[router]: at=error code=H13 desc="Connection closed without response"
method=GET path="/" host=myapp.herokuapp.com fwd=17.17.17.17 dyno=web.1 connect=12610ms service=15882ms
status=503 bytes=0
So this breaks in between the code execution, can we handle this in the exception anyway? Am I doing anything wrong here?
Set the timeout to 30 seconds. And examine if the problem will recur. This should help.
The documentation says:
When a Unicorn web server is configured with a timeout shorter than 30s and a request has not been processed by a worker before the timeout happens.
In this case, Unicorn closes the connection before any data is written, resulting in an H13.
https://devcenter.heroku.com/articles/error-codes#h13-connection-closed-without-response
I have a very strange problem with my rails app on Heroku that I haven't been able to solve for nearly a month now. It's a problem that only occurs on the production server and I can't replicate in development and for which the logs report no errors accept H18 errors.
Here's what is happening. The application runs fine for about 12 hours, then at a certain point the number of requests spike a bit and Heroku starts reporting regular H18 errors
At this point the application doesn't completely fail, but all requests that invoke some kind of open-uri requests fail (basically request to an external webservice), return a 500 error. Normal requests that simply display a static view still seem to continue to work.
The logs are not particularly helpful.
Nearly every H18 error is associated with a request for /robots.txt but at least one error is associated with an assets request: "/assets/application-38a4580edd72e30f34ea76583ab7e1b1b5654c72a6313ece935177d23b0398d3.css"
Below is an excerpt
Oct 10 21:09:32 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/robots.txt" host=scta.lombardpress.org request_id=e0e344d1-0349-4b0a-8db0-3c6e3ad3e99f fwd="157.55.39.188" dyno=web.1 connect=0ms service=209ms status=503 bytes= protocol=http
Oct 10 21:09:41 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/assets/application-38a4580edd72e30f34ea76583ab7e1b1b5654c72a6313ece935177d23b0398d3.css" host=scta.lombardpress.org request_id=fdd88ca2-140e-4051-9011-4d81ca218f19 fwd="157.55.39.206" dyno=web.1 connect=0ms service=252ms status=503 bytes= protocol=http
Oct 10 21:33:55 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/robots.txt" host=scta.lombardpress.org request_id=5aa463f5-43ff-4b74-a2de-e944aa9d2387 fwd="46.229.168.147" dyno=web.1 connect=0ms service=254ms status=503 bytes= protocol=http
Oct 10 21:38:23 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/robots.txt" host=scta.lombardpress.org request_id=2300949d-8998-4ed0-a4ab-bf1975e93cc6 fwd="216.244.66.197" dyno=web.1 connect=0ms service=220ms status=503 bytes= protocol=http
Once the errors start, I simply have to restart the app and everything works fine again for another 12 or so errors. Even requests to robots.txt. But after approximately 12 hours the same problem occurs.
Due to web crawlers, I would estimate the application is getting hit a with a request every 2-4 seconds. Otherwise it is not a particularly high traffic site.
Heroku provided only the smallest feedback
The backend socket, belonging to your app’s web process was closed before the backend returned an HTTP response. This is happening at the application or web server level so there is not much additional insight we can provide. You can see that the request spends some time in your application service=1118ms before returning a status 500. I wonder if a middleware (maybe you have one making external requests?) is failing before it hits your actual rails stack. I would suggest starting by looking there.
The code is open source and available here if anyone is interested in poking around: https://github.com/lombardpress/lombardpress-web
I'd be grateful for any thoughts or suggestions. I've been struggling with this for a while now and I'm not sure how to solve the problem.
Heroku Rails site going down every day or two for 30 seconds max, alerts sent by uptime-robot.
Have tried to run some basic load testing, and doesn't seem take the site down when under solid traffic. Not running any expensive queries on the homepage.
Error logs look like the below. Running rails on hobby dev.
Not getting any errors through New Relic. Running Puma 'puma', '~> 2.15.3'.
Have this set on rack timeout initailizer: Rack::Timeout.timeout = 15
Dec 07 20:14:46 sleepy-wave-3748 heroku/router: at=error code=H12 desc="Request timeout" method=GET path="/" host=mySiteUrlHere request_id=274ed877-2edb-43f2-8b77-c7a82f17109a fwd="69.162.124.231,108.162.221.131" dyno=web.1 connect=1ms service=30007ms status=503 bytes=0
Dec 07 20:35:46 sleepy-wave-3748 heroku/router: at=error code=H12 desc="Request timeout" method=HEAD path="/" host=mySiteUrlHere request_id=1b43b249-f089-4a4a-a42b-b2886d607fa8 fwd="69.162.124.231,108.162.221.131" dyno=web.1 connect=0ms service=30003ms status=503 bytes=0
Any Suggestions ?
Heroku will typically disconnect after 30 seconds and return a 503 error. It appears that your service may be responding successfully but after 30 seconds have passed. This could explain why you are not seeing application errors but are seeing Heroku errors.
EDIT: I think I may have misread your question. Heroku goes through a cycling process on some of its dynos. During this time, there may be a possibility for it to appear that your application has gone down. If you are using free dynos, they require a period of downtime (per rolling 24hr period)
I think I have solved it. I found a pull request in the Puma github account relating to this issue and noticed the fix had been included in a later version of Puma. After upgrading my Puma version, the problem has stopped happening. Hopefully this can help someone else out.
Note, I upgraded to Puma 2.15.3 from 2.11.2 .
repo at: git#github.com:assafshomer/sample_app.git
This app is my learning app based on Michael Hartl's excellent railstutorial.
App runs smoothly in development and test suite is all green
When deploying to Heroku I get the following error immediatly: "Application Error. An error occurred in the application and your page could not be served"
The problem started after upgrading to Rails4 (but I know the problem is not Rails4 itself because, e.g. git://github.com/railstutorial/sample_app_rails_4.git deploys fine to Heroku)
I know you are going to ask "did you migrate the db on Heroku?". I did. I even ran pg:reset and then again rake db:migrate on Heroku just in case. They seemed to have run ok.
Heroku logs don't give any info either (at least, nothing I can see as useful)
2013-05-25T22:11:26.070468+00:00 heroku[web.1]: Process exited with
status 0
2013-05-25T22:11:26.083572+00:00 heroku[web.1]: State changed from
starting to crashed
2013-05-25T22:13:09.372294+00:00 heroku[run.4121]: State changed from up
to complete
2013-05-25T22:13:09.357146+00:00 heroku[run.4121]: Process exited with
status 1
2013-05-25T22:13:50.829887+00:00 heroku[router]: at=error code=H10
desc="App crashed" method=GET path=/ host=devsampleapp.herokuapp.com
fwd="93.172.16.63" dyno= connect= service= status=503 bytes=
also the rails console doesn't come up:
2013-05-26T06:41:34.942569+00:00 heroku[api]: Starting process with command `bin/rails console` by assafshomer#gmail.com
2013-05-26T06:41:38.312107+00:00 heroku[run.5325]: Awaiting client
2013-05-26T06:41:38.339816+00:00 heroku[run.5325]: Starting process with command `bin/rails console`
2013-05-26T06:41:39.112396+00:00 heroku[run.5325]: Client connection closed. Sending SIGHUP to all processes
2013-05-26T06:41:40.329832+00:00 heroku[run.5325]: Process exited with status 0
2013-05-26T06:41:40.357047+00:00 heroku[run.5325]: State changed from starting to complete
I submitted a support ticket with Heroku, but that will only be attended to earliest Tuesday, and I'm itching to get this "upgrade to rails4" working already, so, any help in resolving this mystery will be much appreciated.
obviously, the best way to reproduce is to clone the repo and deploy to heroku :)
Thanks!!
Assaf Shomer (assafshomer at gmail)
In rails4 several files (bin/rails, bin/rake and bin/bundle) responsible for the startup phase have been newly added or moved from different locations (e.g. script/rails).
Seems that they are missing in your application. Strange thing indeed that it works on your local machine but not on heroku - but well I tried and added them. It seems to work.
You can create them by simply running rake rails:update:bin.