Deadlocks in PostgreSQL when running a simple UPDATE - ruby-on-rails

update cities set cdb_data = NULL, updated_at = now() where cities.id = 1;
We loop through cities and update cities with cdb_data as a part of rails code, however we keep getting the below error.
ActiveRecord::StatementInvalid: PG::TRDeadlockDetected: ERROR: deadlock detected
DETAIL: Process 26741 waits for ShareLock on transaction 2970537161; blocked by process 26818.
Process 26818 waits for ShareLock on transaction 2970537053; blocked by process 26741.
HINT: See server log for query details.
CONTEXT: while updating tuple (39,15) in relation "cities"
UPDATE "cities" SET "cdb_data" = $1, "updated_at" = $2 WHERE "cities"."id" = $3
Ruby code that updates the city object
city = City.find_or_create_by(uuid: city_data['uuid'])
city.name = city_data['name']
city.state_id = city_data['state_id']
city.cdb_data = city_data
city.save
I am clueless about to which record this error is happening and why?
Even with the production dump on local or in staging, this doesn't seem to happen.
Any help would be much appreciated.
I am running the server on heroku so I am not really sure I could see the postgres logs.

Two such transactions can easily deadlock.
To avoid that problem make sure that when you “loop through the cities”, you always do so in the same order, using something like:
FOR c IN
SELECT * FROM city
WHERE /* whatever */
ORDER BY city.id
LOOP
/* perform the update */
END LOOP;

To find what is locking the update query, one could use
SELECT pg_blocking_pids(<pid of the query that is locked>);

Related

Select 1 as one - Ruby Queries rolls back in staging and prod

I am trying to update the quantity of the products table after the payment has succeeded on my rails app.
This is how I am doing it,
#orders1 = Order.where("shoppingcart_id" => #cart.id)
#orders1.each do |order|
#oproduct = Product.where("id" => order.product.id).first
if(#oproduct.quantity >=1 )
#oproduct.quantity = #oproduct.quantity - 1;
#oproduct.save
end
end
When I am looking into the query logs, I see Ruby runs this select 1 as one query -
Product Exists (1.6ms) SELECT 1 AS one FROM "products" WHERE
"products"."name" = ? AND "products"."id" != ? LIMIT ?
On my local (SQLLite), it works fine without any rollbacks and my product quantity gets updated correctly.
But when I run the same on staging, it fails. The transaction rolls back.
On one other local machine, it fails as well. I am not able to reproduce this on my local and not sure why this is happening either. The other updates are working perfectly fine, like orders update. Only the product quantity update fails.
Is that product exists query mandatory? Not sure if I am missing any setting for my staging or is there any problem with the code.
Try this.
#product = Product.find(order.product.id)
if #product.quantity > 0
#product.update_attributes!(quantity: #product.quantity -1)
end

wrong number of arguments (1 for 2..3) for Active Record postgresql query (Rails 4/postgresql 9.4) [duplicate]

Right now I am in the middle of migrating from SQLite to Postgresql and I came across this problem. The following prepared statement works with SQLite:
id = 5
st = ActiveRecord::Base.connection.raw_connection.prepare("DELETE FROM my_table WHERE id = ?")
st.execute(id)
st.close
Unfortunately it is not working with Postgresql - it throws an exception at line 2.
I was looking for solutions and came across this:
id = 5
require 'pg'
conn = PG::Connection.open(:dbname => 'my_db_development')
conn.prepare('statement1', 'DELETE FROM my_table WHERE id = $1')
conn.exec_prepared('statement1', [ id ])
This one fails at line 3. When I print the exception like this
rescue => ex
ex contains this
{"connection":{}}
Executing the SQL in a command line works. Any idea what I am doing wrong?
Thanks in advance!
If you want to use prepare like that then you'll need to make a couple changes:
The PostgreSQL driver wants to see numbered placeholders ($1, $2, ...) not question marks and you need to give your prepared statement a name:
ActiveRecord::Base.connection.raw_connection.prepare('some_name', "DELETE FROM my_table WHERE id = $1")
The calling sequence is prepare followed by exec_prepared:
connection = ActiveRecord::Base.connection.raw_connection
connection.prepare('some_name', "DELETE FROM my_table WHERE id = $1")
st = connection.exec_prepared('some_name', [ id ])
The above approach works for me with ActiveRecord and PostgreSQL, your PG::Connection.open version should work if you're connecting properly.
Another way is to do the quoting yourself:
conn = ActiveRecord::Base.connection
conn.execute(%Q{
delete from my_table
where id = #{conn.quote(id)}
})
That's the sort of thing that ActiveRecord is usually doing behind your back.
Directly interacting with the database tends to be a bit of a mess with Rails since the Rails people don't think you should ever do it.
If you really are just trying to delete a row without interference, you could use delete:
delete()
[...]
The row is simply removed with an SQL DELETE statement on the record’s primary key, and no callbacks are executed.
So you can just say this:
MyTable.delete(id)
and you'll send a simple delete from my_tables where id = ... into the database.

ActiveRecord includes: How get access to multiple queries resuls that were performed by activerecord?

I have the following query that loads associations:
contacts = current_user.contacts.includes(:contact_lists).where(id: subscribed_contact_ids)
[DEBUG] Contact Load (2.6ms) SELECT "contacts".* FROM "contacts" WHERE "contacts"."user_id" = 7 AND "contacts"."id" IN (4273, 4275, 4277, 4278, 4281, 4285, 4297, 4305, 4307, 4308, 4315, 4318, 4323, 4326, 4331, 4333, 4344, 4349, 4359, 4361, 4368, 4372, 4373, 4378, 4382, 4389, 4392, 4394, 4404, 4428, 4450, 4469, 4473, 4489, 4490, 4495, 4497, 4498, 4501, 4505, 4514, 4520, 4525, 4536, 4545, 4554, 4555, 4561, 4568, 4572)
[DEBUG] Subscription Load (0.8ms) SELECT "subscriptions".* FROM "subscriptions" WHERE "subscriptions"."contact_id" IN (4273, 4275, 4277, 4278, 4281, 4285, 4297, 4305, 4307, 4308, 4315, 4318, 4323, 4326, 4331, 4333, 4344, 4349, 4359, 4361, 4368, 4372, 4373, 4378, 4382, 4389, 4392, 4394, 4404, 4428, 4450, 4469, 4473, 4489, 4490, 4495, 4497, 4498, 4501, 4505, 4514, 4520, 4525, 4536, 4545, 4554, 4555, 4561, 4568, 4572)
[DEBUG] ContactList Load (0.5ms) SELECT "contact_lists".* FROM "contact_lists" WHERE "contact_lists"."id" IN (9, 8)
I also need to get distinct contact_lists, basically i need to load the result of last query separately.
So the question is: is there way to load these contact lists without running the complex queries again?
The option to iterate through each record to get the contact_lists and then to remove duplicates is not attractive at all.
The other option is to run complex join query again (seems not really good either):
contact_list_ids = current_user.subscriptions.select('distinct contact_list_id').where(contact_id: contact_ids).pluck(:contact_list_id)
current_user.contact_lists.where(id: contact_list_ids)
Those queries are cached somewhere. Is there way to access the query cache directly?
Here is a solution for your problem. But it requires you to monkey patch rails which might not be a great idea.
How do I get the last SQL query performed by ActiveRecord in Ruby on Rails?
I would rather iterate over the contacts and store the contact_lists in a hash with their id as key. Then you do not need to worry about duplicates
contact_lists = {}
contacts.each { |c| c.contact_lists.each { |cl| contact_lists[cl.id] = cl } }
or something.

ActiveRecord SQL execution time

How can I get the SQL query execution time in rails?
I can see the time in logs, e.g.:
Posting Load (10.8ms) SELECT "postings".* FROM "postings" ORDER BY "postings"."id" ASC LIMIT 1
But how can I get that value (10.08) programmatically to use it further in my code?
You can use ActiveSupport::Notifications to retrieve the runtime of the SQL statements
You should be able to instrument sql.active_record to see how long the SQL call takes
ActiveSupport::Notifications.subscribe('sql.active_record') do |*args|
event = ActiveSupport::Notifications::Event.new(*args)
event.duration #how long it took to run the sql command
end
This code was not tested, so I can't guarantee it works, but it should
try this out
time_consumed = Benchmark.measure { Post.limit(1) }

"FOR UPDATE" clause is throwing error in esql program

We are developing a migrate program. There are nearly 80 million records are there in DB. The code is as follows:
static int mymigration(struct progargs *args)
{
exec sql begin declare section;
const char *selectQuery;
const char *updateQuery;
long cur_start;
long cur_end;
long serial;
long number;
char frequency[3];
exec sql end declare section;
selectQuery = "select * from mytable where number >= ? and number <= ? for update of frequency ,status";
updateQuery = "update mytable set frequency = ?, "
" status = ? "
" where current of my_cursor";
cur_start= args->start;
cur_end = args->end;
exec sql prepare my_select_query from :selectQuery;
/* Verify the sql code for error here */
exec sql declare my_select_cursor cursor with hold for my_select_query;
exec sql open my_select_cursor using :cur_start, :cur_end;
/* Verify the sql code for error here */
exec sql prepare my_update_query from :updateQuery;
/* Verify the sql code for error here */
while (1)
{
number = 0;
serial = 0;
memset(frequency,0,sizeof(frequency));
exec sql fetch my_select_cursor into number,:serial,:frequency;
if (sqlca.sqlcode != SQL_OK)
break;
exec sql execute my_update_query using :frequency, :frequency;
}
exec sql close my_select_trade_cursor;
}
While implementing this, we are getting the error message "-255". We found one solution as to add being work and commit work. Since we have large amount of data, this might clutter the transaction log.
Is there any other solution available for this problem? The IBM website for informix shows the usage is correct.
Appreciate the help in advance.
Thanks,
Mathew Liju
Error -255 is "Not in transaction".
I see no BEGIN WORK (or COMMIT WORK or ROLLBACK WORK) statements.
You need to add BEGIN WORK before you open the cursor with the FOR UPDATE clause. You then need to decide whether to commit periodically to avoid overlong transactions. The fact that you use a FOR HOLD cursor shows that you had thought about using sub-transactions; if you were not going to do so, you would not use that clause.
Note that Informix has 3 primary database logging modes:
Unlogged (no transaction support)
Logged (by default, each statement is a singleton transaction; an explicit BEGIN WORK starts a multi-statement transaction terminated by COMMIT WORK or ROLLBACK WORK).
Logged MODE ANSI (slightly simplistically, you are automatically in a transaction; you need an explicit COMMIT or ROLLBACK to terminate a transaction, and may then, optionally, use an explicit BEGIN, but the BEGIN is not actually necessary).
From the symptoms you describe, you have a logged but not MODE ANSI database. Therefore, you must explicitly code the BEGIN WORK statements.

Resources