A few hours ago we starting seeing a bunch of H10 errors in our logs and on our Heroku dashboard. Heroku says that H10 errors are due to web dyno crashes. Read more here. The strange part is that our 2 web dynos are running completely fine and not crashing at all.
Here's an example of what I'm seeing in the logs:
2021-08-12T20:50:35.934439+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=POST path="/v1/graphql" host=api.rize.io request_id=abcd-ace1-44e8-99f5-03d38d8eaa5a fwd="1.1.1.1" dyno= connect= service= status=503 bytes= protocol=https
When I search for that request ID, abcd-ace1-44e8-99f5-03d38d8eaa5a in this example, in the logs (as Heroku recommends), nothing shows up except the router error listed above.
I've restarted all the dynos and the issue persists.
Anyone have any idea what could be going on and how we can fix this?
I don't know if this is an answer for other people but we discovered that our DNS settings were not updated to match the changes documented here: https://devcenter.heroku.com/changelog-items/2151
We also needed to re-establish managed SSL certificates for our custom domains.
Our setup has some complications related to proxies we have in front of Heroku though and the fact that of the three custom domains we have configured on the app, only two of them had been migrated, which complicated the debugging of this situation.
This was due to a Heroku incident. Here is the incident status:
https://status.heroku.com/incidents/2311
I asked Heroku support and he rolled back the infrastructure upgrade (https://devcenter.heroku.com/changelog-items/2151) for our site.
After that, we don't have H10 errors.
Dunno if it helps, but I've cloned my app to a new one (letting all env vars on the old ones) and directed everything to the new Heroku app.
It seems to be working just good enough while waiting for the problem to be solved by the Heroku team. No H10 errors but some H31, nothing is broken anymore though.
Related
I have a very strange problem with my rails app on Heroku that I haven't been able to solve for nearly a month now. It's a problem that only occurs on the production server and I can't replicate in development and for which the logs report no errors accept H18 errors.
Here's what is happening. The application runs fine for about 12 hours, then at a certain point the number of requests spike a bit and Heroku starts reporting regular H18 errors
At this point the application doesn't completely fail, but all requests that invoke some kind of open-uri requests fail (basically request to an external webservice), return a 500 error. Normal requests that simply display a static view still seem to continue to work.
The logs are not particularly helpful.
Nearly every H18 error is associated with a request for /robots.txt but at least one error is associated with an assets request: "/assets/application-38a4580edd72e30f34ea76583ab7e1b1b5654c72a6313ece935177d23b0398d3.css"
Below is an excerpt
Oct 10 21:09:32 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/robots.txt" host=scta.lombardpress.org request_id=e0e344d1-0349-4b0a-8db0-3c6e3ad3e99f fwd="157.55.39.188" dyno=web.1 connect=0ms service=209ms status=503 bytes= protocol=http
Oct 10 21:09:41 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/assets/application-38a4580edd72e30f34ea76583ab7e1b1b5654c72a6313ece935177d23b0398d3.css" host=scta.lombardpress.org request_id=fdd88ca2-140e-4051-9011-4d81ca218f19 fwd="157.55.39.206" dyno=web.1 connect=0ms service=252ms status=503 bytes= protocol=http
Oct 10 21:33:55 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/robots.txt" host=scta.lombardpress.org request_id=5aa463f5-43ff-4b74-a2de-e944aa9d2387 fwd="46.229.168.147" dyno=web.1 connect=0ms service=254ms status=503 bytes= protocol=http
Oct 10 21:38:23 lombardpress-web heroku/router: sock=backend at=error code=H18 desc="Server Request Interrupted" method=GET path="/robots.txt" host=scta.lombardpress.org request_id=2300949d-8998-4ed0-a4ab-bf1975e93cc6 fwd="216.244.66.197" dyno=web.1 connect=0ms service=220ms status=503 bytes= protocol=http
Once the errors start, I simply have to restart the app and everything works fine again for another 12 or so errors. Even requests to robots.txt. But after approximately 12 hours the same problem occurs.
Due to web crawlers, I would estimate the application is getting hit a with a request every 2-4 seconds. Otherwise it is not a particularly high traffic site.
Heroku provided only the smallest feedback
The backend socket, belonging to your app’s web process was closed before the backend returned an HTTP response. This is happening at the application or web server level so there is not much additional insight we can provide. You can see that the request spends some time in your application service=1118ms before returning a status 500. I wonder if a middleware (maybe you have one making external requests?) is failing before it hits your actual rails stack. I would suggest starting by looking there.
The code is open source and available here if anyone is interested in poking around: https://github.com/lombardpress/lombardpress-web
I'd be grateful for any thoughts or suggestions. I've been struggling with this for a while now and I'm not sure how to solve the problem.
My Rails application on Heroku shows me this error screen:
When I run heroku logs from my terminal it only shows:
2016-03-03T22:30:11.442364+00:00 heroku[router]: at=info method=POST path="/posts" host=www.mysite.com request_id=9b490b19-ca40-45b9-b09f-16a5b064d174 fwd="23.248.115.183" dyno=web.1 connect=2ms service=2271ms status=500 bytes=1754
How can I get the logs to show some actual information about the error (as they would in the development environment).
This error appeared after and update involving a mailer and sendgrid smtp.
If you have complete control over the Heroku setup then I can recommend hooking up the Rollbar Add-on to keep track of all your errors and their tracebacks. There is a free tier so you can try it first.
https://elements.heroku.com/addons/rollbar
You can tail the logs with:
heroku logs -t
If you have multiple apps from that directory:
heroku logs -t --app the-name-of-your-app
Those will give you tailings logs.
Here's Heroku's article on logging (with anchor to --tail):
https://devcenter.heroku.com/articles/logging#real-time-tail
Heroku Rails site going down every day or two for 30 seconds max, alerts sent by uptime-robot.
Have tried to run some basic load testing, and doesn't seem take the site down when under solid traffic. Not running any expensive queries on the homepage.
Error logs look like the below. Running rails on hobby dev.
Not getting any errors through New Relic. Running Puma 'puma', '~> 2.15.3'.
Have this set on rack timeout initailizer: Rack::Timeout.timeout = 15
Dec 07 20:14:46 sleepy-wave-3748 heroku/router: at=error code=H12 desc="Request timeout" method=GET path="/" host=mySiteUrlHere request_id=274ed877-2edb-43f2-8b77-c7a82f17109a fwd="69.162.124.231,108.162.221.131" dyno=web.1 connect=1ms service=30007ms status=503 bytes=0
Dec 07 20:35:46 sleepy-wave-3748 heroku/router: at=error code=H12 desc="Request timeout" method=HEAD path="/" host=mySiteUrlHere request_id=1b43b249-f089-4a4a-a42b-b2886d607fa8 fwd="69.162.124.231,108.162.221.131" dyno=web.1 connect=0ms service=30003ms status=503 bytes=0
Any Suggestions ?
Heroku will typically disconnect after 30 seconds and return a 503 error. It appears that your service may be responding successfully but after 30 seconds have passed. This could explain why you are not seeing application errors but are seeing Heroku errors.
EDIT: I think I may have misread your question. Heroku goes through a cycling process on some of its dynos. During this time, there may be a possibility for it to appear that your application has gone down. If you are using free dynos, they require a period of downtime (per rolling 24hr period)
I think I have solved it. I found a pull request in the Puma github account relating to this issue and noticed the fix had been included in a later version of Puma. After upgrading my Puma version, the problem has stopped happening. Hopefully this can help someone else out.
Note, I upgraded to Puma 2.15.3 from 2.11.2 .
I have a snippet of code that takes quite a while to run - About two minutes. Locally, when this code is initiated through a button in my view, the page just appears to be loading until the two minutes is up.
When I deployed this to heroku, the code was erroring out. I changed my Unicorn timeout time to allow for the code to finish, and now when looking at my logs the code is completing just fine.
However, I am presented with this error in my heroku app's view:
An error occurred in the application and your page could not be served. Please try again in a few moments.
If you are the application owner, check your logs for details.
How can I get the same behavior I have locally on my heroku app? I would prefer that the page just continue to appear as if it were loading until the code is complete.
EDIT
I'm seeing this in my logs:
2014-01-14T17:11:49.388634+00:00 heroku[router]: at=error code=H12 desc="Request timeout"
method=POST path=/fluidsurveys/test_send_file host=glacial-spire-1431.herokuapp.com
fwd="66.195.31.22" dyno=web.1 connect=6ms service=30090ms status=503 bytes=0
repo at: git#github.com:assafshomer/sample_app.git
This app is my learning app based on Michael Hartl's excellent railstutorial.
App runs smoothly in development and test suite is all green
When deploying to Heroku I get the following error immediatly: "Application Error. An error occurred in the application and your page could not be served"
The problem started after upgrading to Rails4 (but I know the problem is not Rails4 itself because, e.g. git://github.com/railstutorial/sample_app_rails_4.git deploys fine to Heroku)
I know you are going to ask "did you migrate the db on Heroku?". I did. I even ran pg:reset and then again rake db:migrate on Heroku just in case. They seemed to have run ok.
Heroku logs don't give any info either (at least, nothing I can see as useful)
2013-05-25T22:11:26.070468+00:00 heroku[web.1]: Process exited with
status 0
2013-05-25T22:11:26.083572+00:00 heroku[web.1]: State changed from
starting to crashed
2013-05-25T22:13:09.372294+00:00 heroku[run.4121]: State changed from up
to complete
2013-05-25T22:13:09.357146+00:00 heroku[run.4121]: Process exited with
status 1
2013-05-25T22:13:50.829887+00:00 heroku[router]: at=error code=H10
desc="App crashed" method=GET path=/ host=devsampleapp.herokuapp.com
fwd="93.172.16.63" dyno= connect= service= status=503 bytes=
also the rails console doesn't come up:
2013-05-26T06:41:34.942569+00:00 heroku[api]: Starting process with command `bin/rails console` by assafshomer#gmail.com
2013-05-26T06:41:38.312107+00:00 heroku[run.5325]: Awaiting client
2013-05-26T06:41:38.339816+00:00 heroku[run.5325]: Starting process with command `bin/rails console`
2013-05-26T06:41:39.112396+00:00 heroku[run.5325]: Client connection closed. Sending SIGHUP to all processes
2013-05-26T06:41:40.329832+00:00 heroku[run.5325]: Process exited with status 0
2013-05-26T06:41:40.357047+00:00 heroku[run.5325]: State changed from starting to complete
I submitted a support ticket with Heroku, but that will only be attended to earliest Tuesday, and I'm itching to get this "upgrade to rails4" working already, so, any help in resolving this mystery will be much appreciated.
obviously, the best way to reproduce is to clone the repo and deploy to heroku :)
Thanks!!
Assaf Shomer (assafshomer at gmail)
In rails4 several files (bin/rails, bin/rake and bin/bundle) responsible for the startup phase have been newly added or moved from different locations (e.g. script/rails).
Seems that they are missing in your application. Strange thing indeed that it works on your local machine but not on heroku - but well I tried and added them. It seems to work.
You can create them by simply running rake rails:update:bin.