Not sure if I am asking the right question, but here goes...
I am connected to postgres instance with auto commit turned off. What I am seeing is that if there is a syntax issue with a query:
select * fro table limit 400
, I have to first rollback before I can select again. If I don't I get the following error message:
"ERROR: current transaction is aborted, commands ignored until end of
transaction block"
Is there anywhere I can set to "auto rollback"
Maybe I am just using it incorrectly?
It's not an issue any more. Just check out the latest DataGrip version
Related
Some background
PG::InFailedSqlTransaction appears when some PG exception is rescued and prevents the transaction from rollback. Simple example:
ActiveRecord::Base.transaction do
ActiveRecord::Base.connection.execute('SELECT nothing FROM nowhere') rescue nil
binding.pry # <- let's assume we're here
User.count # raises PG::InFailedSqlTransaction
end
The first line inside the transaction silently breaks the transaction and proceeds, any SQL statements after the first line will raise PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block.
The question
Assuming we're on the binding.pry breakpoint above, are there any ways to debug it and get the initial exception or any other data?
Not quite sure how it is implemented under the hood, but it seems quite possible for PG to cache some metadata about errors.
EDIT: the code above is just an example to demonstrate the general issue, the question is how to get the error in the situation when we cannot easily find the place that rescues the exception
Per #engineersmnky comment, it is possible to get the last error message from the PG::Connection instance with #error_message method, the instance can be obtained from the ActiveRecord connection: ActiveRecord::Base.connection.raw_connection.error_message.
If we run the above command from the pry breakpoint in the example from the question, it will return the following message:
ERROR: relation "nowhere" does not exist
LINE 1: SELECT nothing FROM nowhere
^
I am trying to do some DB experiments in which I reset the DB a few times as part of Rails runner script. Here is a simple example:
`rake db:reset`
puts User.count
`rake db:reset`
This fails with:
PG::ObjectInUse: ERROR: database "my_db" is being accessed by other users
DETAIL: There is 1 other session using the database.
I understand why this happens. The User.count opens a connection so the reset cannot happen. I have tried a few things to close said connection, including calling close and reset_active_connections!, to no avail. Any idea how I can achieve this? Some other pursuits were not fruitful as well, such as trying to close the connection by passing the process ID to psql.
The right call is:
ApplicationRecord.connection_pool.connections.map &:disconnect!
I simply couldn't find the right method the first time, had to read the AR code in more detail to find it.
We are running two rails applications against the same database. When we deploy, we typically deploy to App A, then App B, restarting all rails processes during the deploy. App A runs on 7 servers with at least 20 processes connections to the database. App B runs on 4 servers with at least 8 connections to the database.
Today, when we deployed App A, we added a column to an existing table:
change_table :organizations do |t|
t.integer :users_count, default: 0
end
We expected this to be fine: its new column on an existing table and it has a default. Shortly after the migration ran, a number of errors showed up, from both App A (before it was restarted) and App B (before it was deployed to).
These errors were:
FATAL ActiveRecord::StatementInvalid error: PG::InFailedSqlTransaction:
ERROR: current transaction is aborted, commands ignored until end of
transaction block
In the postgres log, I have 58 errors like this:
postgres[12283]: ERROR: cached plan must not change result type
postgres[12283]: STATEMENT: SELECT "organizations".* FROM
"organizations" WHERE "organizations"."id" = $1 LIMIT $2
This repeats a number of times and goes away after all deploys have finished and all processes restarted.
It appeared that Rails bug #12330 and Rails PR 22170 addressed this in Rails 5.0, but I have that commit and am still seeing this error.
Relevant software versions
Rails 5.0.2
PG 0.19.0
Postgres 9.5
One comment on Rails bug #12330 suggests that I have to add the columns with null defaults. Another suggests performing multiple deploys, one to disable prepare statements, then another to perform the migration and and re-enable prepared statements.
Is there away to avoid this? It clears up when we restart the servers, but I feel like I'm missing something - like only using nullable columns perhaps that would avoid these errors all together. This doesn't happen on every deploy and I don't know how to reproduce it - but this wasn't the first time it has happened.
You have changed table structure and thus what your prepared SELECT returns, because you use "organizations".*, with is returning all columns. PostgreSQL apparently doesn't support updating of prepared statements, so you need to either create new session (reconnect) or use DEALLOCATE to remove that prepared statement.
EDIT: You could also stop using SELECT *.
I am trying to test an app that uses gem devise_token_auth, which basically includes a couple extra DB read/writes on almost every request (to verify and update user access tokens).
Everything is working fine, except when testing a controller action that includes several additional db read/writes. In these cases, the terminal locks up and I'm forced to kill the ruby process via activity monitor.
Sometimes I get error messages like this:
ruby /Users/evan/.rvm/gems/ruby-2.1.1/bin/rspec spec/controllers/api/v1/messages_controller_spec.rb(1245,0x7fff792bf310) malloc: *** error for object 0x7ff15fb73c00: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6
I have no idea how to interpret that. I'm 90% sure the problem is due to this gem and the extra DB activity it causes on each request because I when I revert to my previous, less intensive auth, all the issues go away. I've also gotten things under control by giving postgres some extra time on the offending tests:
after :each do
sleep 2
end
This works fine for all cases except one, which requires a timeout before the expect, otherwise it throws this error:
Failure/Error: expect(#user1.received_messages.first.read?).to eq true
ActiveRecord::StatementInvalid:
PG::UnableToSend: another command is already in progress
: SELECT "messages".* FROM "messages" WHERE "messages"."receiver_id" = $1 ORDER BY "messages"."id" ASC LIMIT 1
which, to me, points to the DB issue again.
Is there anything else I could be doing to track down/control these errors? Any rspec settings I should look into?
If you are running parallel rspec tasks, that could be triggering this. When we've run into issues like this, we have forced those tests to run in single, non-parallel instance of rspec in our CI using tags.
Try something like this:
context 'when both records get updated in one job', non_parallel do
it { is_expected.to eq 2 }
end
And then invoke rspec singularly on the non_parallel tag:
rspec --tag non_parallel
The bulk of your tests (not tagged with non_parallel) can still be run in parallel in your CI solution (e.g. Jenkins) for performance.
Of course, be careful applying this band-aid. It is always better to identify what is not race-safe in your code, since that race could happen in the real world.
I've been digging around stackoverflow trying to find others who get these prepared statements already exists errors.
In most cases configuring unicorn properly with the after/before fork resolves these issues.
However in my case we are still getting errors as such:
ActiveRecord::StatementInvalid: PG::Error: ERROR: prepared statement "a495" already exists: INSERT INTO "user_logins" ("account_id", "created_at", "ip_address", "user_agent", "user_id") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
This error gets thrown in different areas in our app but always seems to have the same statement number 'a495'.
We are on rails 3.2.17, using postgres and we are on heroku.
I really have no idea why this is happening, but its starting to happen more frequently now.
Any help would be greatly appreciated.
In the rails stack trace this error is being thrown in the .prepare call. I'm confused because its checking for the sql key in the statements collection. If it doesn't exist it prepares the new one....however when trying to prepare it, its throwing the error.
def prepare_statement(sql)
sql_key = sql_key(sql)
unless #statements.key? sql_key
nextkey = #statements.next_key
#connection.prepare nextkey, sql
#statements[sql_key] = nextkey
end
#statements[sql_key]
end
We had the same problem, and did very thorough investigation. We concluded that in our case, this error is caused by Rack::Timeout, that very occasionally interrupts code execution after the new statement has been already created, but before the counter is updated on Rails side. Next prepared statement then tries to use the same name (e.g. a494), and a collision occurred.
My belief is that Rails has not implemented prepared statements correctly. Instead of using the increasing counter (a001, a002, ...), they should have used GUIDs. This way, a race condition described above wouldn't be an issue.
We didn't find a workaround. Improving the performance of an app, and increasing the window for Rack::Timeout, made this problem nearly extinct, but it still happens from time to time.
This is typically not a Postgres issue, but an issue with sharing database connections in something like Unicorn:
Understanding Heroku Postgres Log Statements and Common Errors
Here's my solution for Heroku, which unfortunately is a little involved. On the plus side, though, you don't need to suffer from 100's of error notifications when this error starts happening. All that's needed is that the app/dyno is restarted.
The basic outline of the procedure is that when we detect a ActiveRecord::StatementInvalid exception, with an error message description that contains the words 'prepared statement', we run the heroku restart command using Heroku's platform-api gem.
Put the platform-api gem in your Gemfile, and run bundle install
Set the HEROKU_API_KEY to the correct value. (You can generate a key from your Heroku dashboard). Use heroku config:set HEROKU_API_KEY=whatever-the-value-is.
Set the HEROKU_APP_NAME to the correct value. You can get this information from the heroku CLI, but it's just whatever you called your app.
Add the following to your ApplicationController (/app/controllers/application_controller.rb):
...
class ApplicationController < ActionController::Base
rescue_from ActiveRecord::StatementInvalid do |exception|
# notify your error handler, or send an email, or whatever
# ...
if exception.message =~ /prepared statement/
restart_dyno
end
end
def restart_dyno
heroku = PlatformAPI.connect_oauth(ENV["HEROKU_API_KEY"])
heroku.dyno.restart(ENV["HEROKU_APP_NAME"], "web")
end
end
That's it. Hope this helps.