I'm looking to gracefully handle AWS Aurora's failover mechanism in a Rails 6 app.
Other solutions I've found swap out a specific Rack middleware adapter with one that will reset all ActiveRecord connections if a specific type of exception is raised. That middleware has been removed in Rails 6 so this solution no longer works.
The exception that gets raised is an ActiveRecord::StatementInvalid: "PG::ReadOnlySqlTransaction: ERROR: cannot execute UPDATE in a read-only transaction".
Would the following be an idiomatic way to rescue this exception and reset the connection pool as needed?
class ApplicationController < ActionController::Base
rescue_from ActiveRecord::StatementInvalid, with: :check_read_only_failover
private
def check_read_only_failover(exception)
if /Lost connection|gone away|read-only/.match?(exception.message)
ActiveRecord::Base.clear_all_connections!
end
end
(The regex above was lifted from the linked solution; I have no reason to believe the error condition has changed.)
Related
I'm getting an uninitialized constant error when my Rails app loads on Heroku, but it works just fine in development.
Heroku Logs (breaks on boot):
/app/app/controllers/application_controller.rb:19:in `<class:ApplicationController>': uninitialized constant ActiveRecord::DeleteRestrictionError (NameError)
Relevant lines in controllers/application_controller.rb:
class ApplicationController < ActionController::API
# ...
# Line 19
rescue_from ActiveRecord::DeleteRestrictionError, with: :not_processable
# ...
end
If I comment out the rescue_from, then I get Internal Server Errors from the uncaught exception if a record fails to save due to restrict_with_error dependencies. If I do rescue from it, then the server fails to boot, but only on production.
I'm guessing this is related to eager load and/or the changes with how zeitwerk loads constants, but I haven't been able to find anything with an answer on how to fix this.
I found a similar sounding issue / solution from 2008:
https://discuss.rubyonrails.org/t/rescue-from-issue-with-aws-uninitialized-constant/28759/2
Their solution was to wrap the constant in a string to avoid file parse time issues with loading, which appears to work for this scenario as well:
class ApplicationController < ActionController::API
# ...
rescue_from "ActiveRecord::DeleteRestrictionError", with: :not_processable
# ...
end
This issue was reproduced with a minimal app, and an issue opened on Github: https://github.com/rails/rails/issues/43666
I am working on a Rails 5.x application, and I use Postgres as my database.
I often run rake db:migrate on my production servers. Sometimes the migration will add a new column to the database, and this causes some controller actions to crash with the following error:
ActiveRecord::PreparedStatementCacheExpired: ERROR: cached plan must not change result type
This is happening in a critical controller action that needs to have zero downtime, so I need to find a way to prevent this crash from ever happening.
Should I catch the ActiveRecord::PreparedStatementCacheExpired error and retry the save? Or should I add some locking to this particular controller action, so that I don't start serving any new requests while a database migration is running?
What would be the best way to prevent this crash from ever happening again?
I was able to fix this issue in some places by using this retry_on_expired_cache helper:
class ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
class << self
# Retry automatically on ActiveRecord::PreparedStatementCacheExpired.
# (Do not use this for transactions with side-effects unless it is acceptable
# for these side-effects to occasionally happen twice.)
def retry_on_expired_cache(*_args)
retried ||= false
yield
rescue ActiveRecord::PreparedStatementCacheExpired
raise if retried
retried = true
retry
end
end
end
I would use it like this:
MyModel.retry_on_expired_cache do
#my_model.save
end
Unfortunately this was like playing "whack-a-mole", because this crash just kept happening all over my application during my rolling deploys (I'm not able to restart all the Rails processes at the same time.)
I finally learned that I can turn off prepared_statements to completely avoid this issue. (See this other question and answers on StackOverflow.)
I was worried about the performance penalty, but I found many reports from people who had set prepared_statements: false, and they hadn't noticed any problems. e.g. https://news.ycombinator.com/item?id=7264171
I created a file at config/initializers/disable_prepared_statements.rb:
db_configuration = ActiveRecord::Base.configurations[Rails.env]
db_configuration.merge!('prepared_statements' => false)
ActiveRecord::Base.establish_connection(db_configuration)
This allows me to continue setting the database configuration from the DATABASE_URL env variable, and 'prepared_statements' => false will be injected into the configuration.
This completely solves the ActiveRecord::PreparedStatementCacheExpired errors and makes it much easier to achieve high-availability for my service while still being able to modify the database.
I have a Rails 5 application (API) and a postgres DB that run on separate docker containers, all on the same AWS EC2 instance, and are controlled by an external manager (manage). manage needs to be able to make a request to the API and tell it to exit. I don't want to just kill the API externally or the docker container, as I want all API requests to complete. I want the API to exit gracefully and only it knows how to do that.
Ruby has exit, exit! and abort. All of them seem to be handled by Rails as exceptions and Rails just continues motoring on.
How can I terminate my rails application from within? Is there some sort of unhandleable exception that I can raise?
Most likely the ApplicationController is rescuing the SystemExit. Looking at the Rails source, there is a rescue Exception in ActionController, which includes ActiveSupport::Rescuable. That's where the controller methods like rescue_from are defined.
I tested this in a controller in a Rails API app, sent it a request, and Rails did indeed exit immediately, with an empty response to the caller:
class ProcessesController < ApplicationController
rescue_from SystemExit, with: :my_exit
def destroy
CleanupClass.method_that_exits
render json: { status: :ok }
end
def my_exit
exit!
end
end
class CleanupClass
def self.method_that_exits
exit
end
end
How can I use exception handling in Rails? Currently I have done following.
In each controller method I have added
begin
<myCode>
rescue
<exception handler>
But I think with Rails I should be able to define an exception handler method in Application controller and catch all the exceptions from there, without handling them from each method.
I have used 'rescue_action_in_public' in my application controller, but when I given a wrong database name in config/database.yml and load the application, it doesn't catch those exceptions.
My questions are
1 - Is it a recompilation practice to have one exception handler in the application controller and catch the exceptions ? If not, what is the standard way ?
2 - How can I handle the Exceptions like Databse not found, doesn't have permission to view table fields, etc kind of low level exceptions
I'm riding on Rails 3 and I have some projects in Rails 2.3.8 as well
With according to Advanced Rails Recipes book by PragProg, the general exception handling is the good approach.
rescue_action (all environments), and rescue_action_in_public (production) are using to caught any exceptions in abstract controller class. So you do it right way.
Boot application happens before the controllers are loaded, so you can't handle database.yml over there. If you still need to do it, put an initializer ruby file to check that the file exist and a valid, then initialize AR::Base connection to perform DESC table for instance.
For Rails 3 you can use rescue_from in your ApplicationController. If the exceptions you want to catch are at a lower level then you can configure a Rack Middleware layer will allow you to catch exceptions that the controller does not have access to. This is how Hoptoad works. I recently released a rails 3 gem that will catch common exceptions using rescue_from and provide well defined http status codes and error responses for html, json and xml. This may or may not fit your needs. https://github.com/voomify/egregious
In Rails, is there a way to display different error messages on a production rails server (a nice graphic) vs a development server (detailed error information, stack trace, etc.)?
Found one solution. In ApplicationController, add:
rescue_from Exception, :with => :rescue_all_exceptions if RAILS_ENV == 'production'
def rescue_all_exceptions(exception)
case exception
# do production-ey exception stuff
end
end
I'd be interested to hear other options...