Having experienced a few periods of downtime, we've recently upgraded to a production environment in Heroku (Crane database plus 2 x web dynos) however we've seen no improvement. In fact reliability seems to have decreased since upgrading.
The root cause seems to be the following exception:
PG::Error (SSL SYSCALL error: EOF detected
which causes the dyno to fail and - eventually - restart, but not before causing some downtime.
I've no idea what's causing it. Common culprits appear to be Resque and Unicorn, neither of which I'm using. We're on rails 3.2.11, on Heroku Cedar, using pg gem 1.14.1
Logs report the following at crash time:
2013-05-23T19:01:33+00:00 app[heroku-postgres]: source=HEROKU_POSTGRESQL_PINK measure.current_transaction=34490 measure.db_size=38311032bytes measure.tables=19 measure.active-connections=7 measure.waiting-connections=0 measure.index-cache-hit-rate=0.99438 measure.table-cache-hit-rate=0.8824
2013-05-23T19:01:35.123633+00:00 app[web.2]:
2013-05-23T19:01:35.123633+00:00 app[web.2]: PG::Error (SSL SYSCALL error: EOF detected
2013-05-23T19:01:35.123633+00:00 app[web.2]: ):
I have read the following: https://groups.google.com/forum/?fromgroups#!topic/heroku/a6iviwAFgdY but can't find anything that might help.
https://gist.github.com/ktopping/5657474
The above fixes the exception, which is useful (as it should declutter my logs, and even help speed up reconnecting to the database) but doesn't actually stop my main issue which is Heroku web dynos crashing more often than I would like.
Am investigating some other routes (Unicorn, rack-timeout).
Related
I build a backend service (nginx + passenger + rails) on AWS.
My service is running as expected most of the time but sometimes goes wrong when passenger creates a weird new process to handle request:
[INFO] Started POST "/apixxxxxx"
[FATAL] LoadError (Expected /xxxxxxxx/app/controllers/application_controller.rb to define ApplicationController):
[INFO] Started POST "/apixxxxxx"
[FATAL] ActionController::RoutingError (uninitialized constant ApplicationController):
[INFO] Started POST "/apioooooo"
[FATAL] ActionController::RoutingError (uninitialized constant ApplicationController):
...
It has some signature on this weird process:
Other ruby process still works fine when this specific weird ruby process keeps failing on every new request.
If error happens, it always starts by the "LoadError" in the beginning of the process's first request.
This kind of "uninitialized constant" error keeps forever as long as this process is alive.
*new added information:
According to rails log, the same api can reply without error in other 'normal' passenger process. I exclude the problem about typo or rails naming convention.
Here is some information of my machine:
ec2 type : m4.large
storage : general purpose SSD (use EBS)
system : ubuntu 14.04.2 (not AWS version)
application : version 1.8.1, passenger 5.0.26, Rails 3.2.22.2, ruby 2.2.4p230, sidekiq 4.1.1
I also googled some keyword like "AWS EBS volume's IOPS limit" or "sidekiq thread" with this LoadError issue but no luck. Is there any new direction to investigate this problem?
It was working just a couple of days ago and then I let my laptop's battery go dead over the weekend while the server was running.
Now when I try to start my rails server with the command
bundle exec rails s
It does a few things and then I get the message
Exiting
/Users/me/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1216:in `initialize': FATAL: no pg_hba.conf entry for host "fe80::1%lo0", user "city72", database "city72_development", SSL off (PG::ConnectionBad)
+ lots more stuff
... then tons more information about .rbenv and railties and active-record and stuff that I don't usually have to deal with.
Any ideas what I might have messed up when my server got shutdown by letting my battery die? Maybe it's just a coincidence that my battery died and something is all of a sudden wrong with my gems?
you probably got a new ip-adress after booting again and the security line in the pg_hba.conf line of the postgresql server no longer matches your address.
You can find this file in the main data folder of your postgresql server. (Well usually at least, the location is configurable)
This question has been asked before but no answer seems to work for me. I will break the problem down into its 3 components:
1) I receive a Heroku R14 memory (memory quota exceeded) occasionally (i.e. the site has been up 2 days on Heroku and I got this error twice for a period of about 10-15 mn [I was too emotional to count the time precisely]).
2) I installed the oink gem as advised by Heroku.
3) Oink definitely logs, as I get messages to that effect in heroku logs and in Webrick when I work locally. However, I am unable to access the logging summary that shows which functions exceed a memory threshold.
The only line that returns a result (but a wrong one) is :
oink --threshold=0 logfile_for_oink
But it returns empty lines as follows:
---- MEMORY THRESHOLD ----
THRESHOLD: 0 MB
-- SUMMARY --
Worst Requests:
Worst Actions:
Aggregated Totals:
Every other attempt - often copying advice already on StackOverflow - returns errors.
I will list the different attempts I have made (so no-one posts a suggestion I may have already tried) after this.
heroku run bundle exec oink --threshold=75 log/*
This line returns the following error:
/app/vendor/bundle/ruby/1.9.1/gems/oink-0.10.1/lib/oink/cli.rb:88:in `block in get_file_listing': Could not find "log/development.log" (RuntimeError)
Every variation on this, such as log/production.rb or /log/* or what have you has failed.
I also tried the advice on the following links to no avail:
Using oink gem with heroku
oink logs command not working on heroku
oink logs command not working on heroku
How can I run oink in heroku?
Can anyone help me?
Heroku prepends the log file with an additional timestamp so oink can't read it. You can use a regex though to fix it.
http://arches.io/2013/07/understand-memory-usage-on-heroku-rails-app-using-oink/
this is a follow up to my other question regarding unicorn. following problem:
i'm using ruby 1.8.7, rails 3.0, pg gem (0.13.0).
when i start the rails console in production mode, i can query records etc, so the
database connection and the pgsql adapter works. i can disconnect the AR connection via ActiveRecord::Base.connection.disconnect!, but as soon as i try to reconnect using ActiveRecord::Base.establish_connection, ruby just dies. no error, no log output, no seg fault, it just quits (with status 1). i tried setting the host in my database.yml to the postgres unix socket, as well as to 127.0.0.1, doesn't change a thing. what could be the problem, or what can i do to get to the root of this? i'm not sure how to debug this, with absolutely no output or error message.
i solved the problem (more or less) by downgrading pg to version 0.11.0. this works for me for now. cost me 1 and a 1/2 days ...
Why are you closing explicitely the connection to the database ? Rails/AR handles the connection pool for you. The rack middleware is supposed to release connections at the end of the request processing.
I'm deploying a Rails 3 app on Passenger 3.0.7 using capistrano and git-based deployment, similar to GitHub's setup:
https://github.com/blog/470-deployment-script-spring-cleaning -- this means the app operates entirely out of one directory, with no /releases/123456 and symlink switching.
If we've added any gems our app starts throwing 500 errors during deployment, during the "bundle:install" phase, but before deploy:restart. The code has been updated and it seems like passenger is already starting to use it, and required gems can't be found yet.
This is not caused by new workers being spun up, as I've tried setting the Passenger idle_time to 0 and max_instances and min_instances to the same value, so that workers are never spun down.
Running on Linux with ruby-ee 1.8.7-2011.03. Sample error from Passenger: https://gist.github.com/54794e85d2c799e4f697
I've also considered doing "two-directory" git-based deployment as a hack -- swapping in the new code once the bundle is complete. Ideas welcome.
Go with the two-directory deployment. Apart from avoiding the 500s during deployment, this will also act as a safety net if you need to rollback during/after deployment.