I'm making an expensive call to my Heroku Rails server. After 13-15 seconds the console in my browser reports a Service Unavailable 503 error. However when I check my heroku logs, it reports:
Completed 200 OK in 45592ms (Views: 220.3ms | ActiveRecord: 33457.5ms)
Other times the heroku logs report a memory exceeded quota. Here is an example of that below.
2015-06-11T15:17:20.238285+00:00 app[web.1]: Completed 200 OK in 81881ms (Views: 201.6ms | ActiveRecord: 18021.2ms)
2015-06-11T15:17:33.482930+00:00 heroku[web.1]: Process running mem=841M(164.4%)
2015-06-11T15:17:33.482930+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-06-11T15:17:53.147570+00:00 heroku[web.1]: Process running mem=841M(164.4%)
2015-06-11T15:17:53.147679+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2015-06-11T15:17:59.751540+00:00 app[web.1]: E, [2015-06-11T15:17:59.695813 #3] ERROR -- : worker=2 PID:13 timeout (121s > 120s), killing
2015-06-11T15:17:59.916750+00:00 app[web.1]: E, [2015-06-11T15:17:59.906435 #3] ERROR -- : reaped #<Process::Status: pid 13 SIGKILL (signal 9)> worker=2
2015-06-11T15:18:02.487428+00:00 app[web.1]: I, [2015-06-11T15:18:02.427293 #16] INFO -- : worker=2 ready
Why is it reporting a Completed 200 when the console is reporting a 503?
there are 2 different things here:
your app
the heroku load balancer
In this case the load balancer see that the request is taking too long and sends you the 503. In the back the service processes the request and returns completes with a 200.
See:
https://devcenter.heroku.com/articles/limits
https://devcenter.heroku.com/articles/request-timeout
Heroku does not gives us much information for error-code = H10. Simple put, something is going wrong with your application code / configuration. To see what's going wrong, run heroku run rails console and you will be able to see error details which will be help you in resolving the error. No need to use logs. heroku run rails console is a big help
This happened with me when I used Node. I put a timeout for 30 seconds and I used to get HTTP:503. I then came to know that it's the timeout that caused the issue. I changed the timeout to <30 seconds and it is working fine. Maybe it's because of the loadbalancer that Heroku uses.
Thanks.
Related
I'm running a Rails 4 app on a Digital Ocean VPS and I'm getting an Nginx 504 gateway time out error on a particular HTTP request. All requests succeed, but this particular one does not. I am running in production mode and there are no errors or exceptions in my Rails log.
The Nginx log looks like this:
2014/04/23 05:08:22 [error] 6946#0: *109805 upstream timed out
(110: Connection timed out) while reading response header from upstream, client:
174.126.176.107, server: new.pilotpro.com, request:
"GET /dropbox/sync?right_now=1398115006.949 HTTP/1.1", upstream:
"http://127.0.0.1:8080/dropbox/sync?right_now=1398115006.949",
host: "new.pilotpro.com"
And the Unicorn log looks like this:
E, [2014-04-23T04:52:06.285058 #5424] ERROR -- : worker=1 PID:5429 timeout (61s > 60s), killing
E, [2014-04-23T04:52:06.297947 #5424] ERROR -- : reaped #<Process::Status: pid 5429 SIGKILL (signal 9)> worker=1
I, [2014-04-23T04:52:06.298207 #5424] INFO -- : worker=1 spawning...
I, [2014-04-23T04:52:06.299481 #5457] INFO -- : worker=1 spawned pid=5457
As far as I can tell nothing is wrong other than the fact that the request is timing out. Is this simply a matter of increasing Nginx's timeout limits?
I'd appreciate any advice. Thanks!
Using carrier wave to upload a pdf document via CarrierWave. Then using RMagick to save the first page of the pdf as a thumbnail.jpg. I have this working correctly in my development environment (Mac OS X 10.9), however its failing in production (Nginx + Unicorn on Ubuntu 12.04 LTS).
It appears that the upload completes successfully, based on watching the upload percentage in the browser status bar. However, something errors in the processing, and I get Rails error page. The problem is, there is nothing in my production.log.
I get the following in unicorn.log:
I, [2013-12-12T01:22:55.232431 #2935] INFO -- : worker=1 ready
I, [2013-12-12T01:22:55.246992 #2932] INFO -- : worker=0 ready
E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing
E, [2013-12-12T01:26:00.676250 #2929] ERROR -- : reaped #<Process::Status: pid 2935 SIGKILL (signal 9)> worker=1
I, [2013-12-12T01:26:00.676382 #2929] INFO -- : worker=1 spawning...
I, [2013-12-12T01:26:00.681087 #3502] INFO -- : worker=1 spawned pid=3502
I, [2013-12-12T01:26:00.681399 #3502] INFO -- : Refreshing Gem list
SECURITY WARNING: No secret option provided to Rack::Session::Cookie.
This poses a security threat. It is strongly recommended that you
provide a secret to prevent exploits that may be possible from crafted
cookies. This will not be supported in future versions of Rack, and
future versions will even invalidate your existing user cookies.
Called from: /home/deployer/apps/______/shared/bundle/ruby/1.9.1/gems/actionpack-3.2.9/lib/action_dispatch/middleware/session/abstract_store.rb:28:in `initialize'.
And this in my nginx error.log
2013/12/12 01:26:00 [error] 2597#0: *9 upstream prematurely closed connection while reading response header from upstream, client: ###.###.###.###, server: ___.org, request: "POST /admin/explores HTTP/1.1", upstream: "http://unix:/tmp/unicorn.sock:/admin/explores", host: "___.org", referrer: "https://___.org/admin/explores/new"
Not sure how to get a better error description of what's going on. Would appreciate any help.
Thanks!
Ruby 1.9.3 + Rails 3.2.9
The line E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing from your unicorn.log tells you the problem: you have a timeout. Increase unicorn's timeout to 60 seconds by setting timeout 60 in your unicorn.rb. More information: http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-timeout
I manage a rails app for a client of mine and recently it went down. The site was down for 9 hours before I noticed. I checked the logs and every request for the past 9 hours is prepended with the following code:
at=error code=H10 desc="App crashed"
Before that, I see the following logs:
2012-11-16T00:55:46+00:00 heroku[web.1]: Idling
2012-11-16T00:55:50+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2012-11-16T00:55:51+00:00 app[web.1]: [2012-11-16 00:55:51] ERROR SignalException: SIGTERM
2012-11-16T00:55:51+00:00 app[web.1]: /usr/local/lib/ruby/1.9.1/webrick/server.rb:90:in `select'
2012-11-16T00:56:00+00:00 heroku[web.1]: Error R12 (Exit timeout) -> At least one process failed to exit within 10 seconds of SIGTERM
2012-11-16T00:56:00+00:00 heroku[web.1]: Stopping remaining processes with SIGKILL
2012-11-16T00:56:02+00:00 heroku[web.1]: State changed from up to down
2012-11-16T00:56:02+00:00 heroku[web.1]: Process exited with status 137
2012-11-16T01:03:55+00:00 heroku[web.1]: Unidling
2012-11-16T01:03:55+00:00 heroku[web.1]: State changed from down to starting
2012-11-16T01:03:59+00:00 heroku[web.1]: Starting process with command `bundle exec rails server -p 4303`
2012-11-16T01:04:00+00:00 heroku[nginx]: 98.139.241.251 - - [16/Nov/2012:01:04:00 +0000] "GET / HTTP/1.1" 499 0 "-" "YahooCacheSystem" domain.com
2012-11-16T01:04:22+00:00 app[web.1]: => Ctrl-C to shutdown server
2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Dispatcher: webrick
2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Application: acsolar
2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : New Relic Ruby Agent 3.4.0.1 Initialized: pid = 2
2012-11-16T01:04:22+00:00 app[web.1]: => Booting WEBrick
2012-11-16T01:04:22+00:00 app[web.1]: => Rails 3.1.1 application starting in production on http://0.0.0.0:4303
2012-11-16T01:04:22+00:00 app[web.1]: => Call with -d to detach
2012-11-16T01:04:25+00:00 app[web.1]: [DEPRECATION] Your applications public directory contains an assets/products and/or assets/taxons subdirectory.
2012-11-16T01:04:25+00:00 app[web.1]: Run `rake spree:assets:relocate_images` to relocate the images.
2012-11-16T01:04:34+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:32 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Reporting performance data every 60 seconds.
2012-11-16T01:04:34+00:00 app[web.1]: Connected to NewRelic Service at collector-5.newrelic.com
2012-11-16T01:05:00+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
2012-11-16T01:05:00+00:00 heroku[web.1]: Stopping process with SIGKILL
2012-11-16T01:05:02+00:00 heroku[web.1]: Process exited with status 137
2012-11-16T01:05:02+00:00 heroku[web.1]: State changed from crashed to down
2012-11-16T01:05:02+00:00 heroku[web.1]: State changed from starting to crashed
I'm guessing that it may have spun down and had an error booting back up, but how come it stayed in the crashed state without restarting itself? Is there anything I can do to have it automatically restart if this happens again in the future?
I've got NewRelic running on this too and it didn't notify me at all, but that's another problem I'll have to investigate.
Heroku's support answer suggests restarting your app manually with heroku restart. They're fixing the issue right now.
Hi, A process management error on our side caused some crashed apps
only running 1 web dyno to be reported as "idle" even though they were
actually crashed. This means that the crashed dyno was never
restarted, causing subsequent requests to fail. We've identified this
problem and are implementing a fix. If your app is still unresponsive,
please try restarting it with the heroku restart command. Please let
us know if you need more help. Thanks, Heroku Support
I have a Rails 3.1.4 app hosted on heroku. I have added the logentries add-on.
I didn't touch the default configuration for alerts, and I now receive every few hours an alert: 'Exit timeout':
Exit timeout: Heroku/myappname
2012-03-23 11:01:41.723
168 <40>1 2012-03-23T11:01:41+00:00 d. heroku web.1 - - Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
You are receiving this email because your Logentries alarm "Exit timeout"
has been triggered.
In context:
2012-03-23 11:01:28.878 95 <40>1 2012-03-23T11:01:28+00:00 d. heroku web.1 - - Idling
2012-03-23 11:01:31.740 118 <40>1 2012-03-23T11:01:31+00:00 d. heroku web.1 - - Stopping process with SIGTERM
2012-03-23 11:01:41.723 168 <40>1 2012-03-23T11:01:41+00:00 d. heroku web.1 - - Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
The logs nor this email alert do not give me anything useful to start looking for this issue.
I have a very low traffic (application still in private beta). I suspect it's heroku shutting down the application, and my application does not respond as expected to the SIGTERM signal.
I do not have anything fancy in the application (such as a rake task launched with Heroku Scheduler).
The only thing slightly out of the ordinary is the use of the gem apn_on_rails, which open a connection to Apple's APN servers.
Do you have any idea what is causing this? Should I just ignore this and turn the alert off?
If you're using webrick to run your application on Heroku, you should switch to using 'thin': See https://devcenter.heroku.com/articles/rails3#webserver
This removed all occurrences of the R12 error from my logs.
I seem to have a very odd issue.... I am trying to develop a Rails app to be deployed on Heroku using Unicorn - as such I'm using Foreman in my local development environment to try to replicate production as closely as possible.
As you'd expect, my web/worker processes output to the development.log file in path/to/app/log. If I navigate to the file, it contains everything you would expect.
However if I use the command
tail -f log/development.log
(from the app path), I get log output from HEROKU!! How is this possible? (e.g.):
app[web.1]: [Worker(host:xxxx-xxx-xxx pid:5)] Starting job worker
heroku[web.1]: Idling
heroku[web.1]: Stopping process with SIGTERM
app[web.1]: I, [2012-02-19xxx-xxx-xxx #1] INFO -- : reaped #<Process::Status: pid 7 exit 0> worker=0
app[web.1]: I, [2012-02-19xxx-xxx-xxx #1] INFO -- : reaped #<Process::Status: pid 11 exit 0> worker=1
app[web.1]: I, [2012-02-19xxx-xxx-xxx #1] INFO -- : reaped #<Process::Status: pid 14 exit 0> worker=2
app[web.1]: I, [2012-02-19xxx-xxx-xxx #1] INFO -- : master complete
heroku[web.1]: Process exited with status 0
heroku[web.1]: State changed from up to down
heroku[slugc]: Slug compilation started
heroku[api]: Release v22 created by brandon#example.com
heroku[api]: Deploy xxxx by randon#example.com
heroku[slugc]: Slug compilation finished
This is really annoying as I can't properly see my development log... help would be appreciated!
This isn't something that Foreman's doing.
What happens if you just look at log/development.log? Is it the same?
Do you have any wacky aliases setup that might be causing this?