I have this index in my Model's table:
UNIQUE KEY `index_panel_user_offer_visits_on_offer_id_and_panel_user_id` (`offer_id`,`panel_user_id`)
And this code:
def get_offer_visit(offer_id, panel_user_id)
PanelUserOfferVisit.where(:offer_id => offer_id, :panel_user_id => panel_user_id).first_or_create!
end
is randomly causing an ActiveRecord::RecordNotUnique exception in our application,
the issue is:
Duplicate entry for key
'index_panel_user_offer_visits_on_offer_id_and_panel_user_id'
I've read on the rails doc that first_or_create! is not an atomic method and can cause this kind of issues in a high concurrency environment and that a solution would be just to catch the exception and retry.
I tried different approaches, including Retriable gem to retry a certain number of times to repeat the operation but RecordNotUnique is still raised.
I even tried to change the logic:
def get_offer_visit(offer_id, panel_user_id)
begin
offer_visit = PanelUserOfferVisit.where(:offer_id => offer_id, :panel_user_id => panel_user_id).first_or_initialize
offer_visit.save!
offer_visit
rescue ActiveRecord::RecordNotUnique
offer_visit = PanelUserOfferVisit.where(:offer_id => offer_id, :panel_user_id => panel_user_id).first
raise Exception.new("offer_visit not found") if offer_visit.nil?
offer_visit
end
end
But the code still raise the custom exception I made and I really don't understand how it can fails to create the record on the first try because the record exists but then when it tries to find the record it doesn't find it again.
What am I missing?
You seem to be hitting rails query cache for PanelUserOfferVisit.where(...).first query: rails have just performed identical SQL query, it thinks that result will remain the same for the duration of the request.
ActiveRecord::Base.connection.clear_query_cache before the second call to db should help
Also you do not have to save! the record if it is not new_record?
If I understand properly your problem here is due to a race condition. I didn't found the exact piece of code from Github but lets say the method find_or_create is something like (edit: here is the code):
def first_or_create!(attributes = nil, &block)
first || create!(attributes, &block)
end
You should try to solve the race conditions using optimistic or pessimistic locking
Once locking implemented within your custom method, you should continue capturing possible NotUnique exceptions but try to clean cache or use uncached block before to perform the find again.
I experienced a similar issue but the root cause was not related to a race condition.
With first_or_create!, the INSERT remove surrounding spaces in the value but the SELECT don't.
So if you try this:
Users.where(email: 'user#mail.com ')
(mind the space at the end of the string), the resulting sql query will be:
SELECT "users".* FROM "users" WHERE "users"."email" = 'user#mail.com ' ORDER BY "users"."id";
which result in not finding the potentially existing record, then the insert query will be:
INSERT INTO "users" ("email") VALUES ('user#mail.com');
which can result in ActiveRecord::RecordNotUnique error if a record with 'user#mail.com' as an email already exists.
I do not have enough details about your problem so it might be unrelated but it might helps others.
Related
I'm trying to implement create_or_update method for User model, which will either create a new record or update the existing one.
def create_or_update_user(external_user_id, data)
user = User.find_or_initialize_by(external_user_id: external_user_id)
user.title = data[:title].downcase
# and so on - here we assign other attributes from data
user.save! if user.changed?
end
The pitfall here is that this table is being updated concurrently by different processes and when they are trying to modify user with the same external_user_id then race condition happens and ActiveRecord::RecordNotUnique is raised. I tried to use database lock to solve this problem, but it didn't work as expected - exception is still raised sometimes.
The table structure looks like this:
create_table :users do |t|
t.integer :external_user_id, index: { unique: true }
# ...
end
updated method - what I'm doing wrong here?:
def create_or_update_user(external_user_id, data)
user = User.find_or_initialize_by(external_user_id: external_user_id)
user.with_lock do
user.title = data[:title].downcase
# and so on - here we assign other attributes from data
user.save! if user.changed?
end
end
I can't use upsert, because I need model callbacks.
It probably can be fixed by adding
rescue ActiveRecord::RecordNotUnique
retry
end
but I want to use locks for best performance cause the race conditions is not a rare in this part of code.
UPD: added a gist to reproduce this race condition
https://gist.github.com/krelly/520a2397f64269f96489c643a7346d0f
If I'm understanding correctly, the error is likely still occurring in cases where two threads attempt to create a new User for an external id that doesn't exist.
In this situation, thread #1 would attempt to acquire a row lock on a row that does not exist - no locking would actually occur.
Thread #2 would be able to make the same find_or_initialize query while thread #1 is still building the record, and thus thread #2 would hit a uniqueness violation once it commits.
As you've already guessed, the only simple solution to this problem would be a rescue + retry. Thread #2 would just try again, and it would update the record created by thread #1.
An alternative solution would be an advisory lock - a lock with application-specific behavior. In this case, you would be saying that only one thread at a time can run create_or_update_user with a specific external ID.
Rails doesn't support advisory locks natively, but the linked article contains an example of how you would do it with Postgres. There's also the gem with_advisory_locks
Following the principle of fail-fast:
When querying the database where there should only ever be one record, I want an exception if .first() (first) encounters more than one record.
I see that there is a first! method that throws if there's less records than expected but I don't see anything for if there's two or more.
How can I get active record to fail early if there are more records than expected?
Is there a reason that active record doesn't work this way?
I'm used to C#'s Single() that will throw if two records are found.
Why would you expect activerecord's first method to fails if there are more than 1 record? it makes no sense for it to work that way.
You can define your own class method the count the records before getting the first one. Something like
def self.first_and_only!
raise "more than 1" if size > 1
first!
end
That will raise an error if there are more than 1 and also if there's no record at all. If there's one and only one it will return it.
It seems ActiveRecord has no methods like that. One useful method I found is one?, you can call it on an ActiveRecord::Relation object. You could do
users = User.where(name: "foo")
raise StandardError unless users.one?
and maybe define your own custom exception
If you care enough about queries performance, you have to avoid ActiveRecord::Relation's count, one?, none?, many?, any? etc, which spawns SQL select count(*) ... query.
So, your could use SQL limit like:
def self.single!
# Only one fast DB query
result = limit(2).to_a
# Array#many? not ActiveRecord::Calculations one
raise TooManySomthError if result.many?
# Array#first not ActiveRecord::FinderMethods one
result.first
end
Also, when you expect to get only one record, you have to use Relation's take instead of first. The last one is for really first record, and can produce useless SQL ORDER BY.
find_sole_by (Rails 7.0+)
Starting from Rails 7.0, there is a find_sole_by method:
Finds the sole matching record. Raises ActiveRecord::RecordNotFound if no record is found. Raises ActiveRecord::SoleRecordExceeded if more than one record is found.
For example:
Product.find_sole_by(["price = %?", price])
Sources:
ActiveRecord::FinderMethods#find_sole_by.
Rails 7 adds ActiveRecord methods #sole and #find_sole_by.
Rails 7.0 adds ActiveRecord::FinderMethods 'sole' and 'find_sole_by'.
Using ruby 2.1.2 and rails 4.1.4. I have a unique constraint on email column in my database. If there's a race condition where one thread is creating a user and the other attempts to create at the same time a record not unique exception is raised. I am trying to handle this exception by retrying so that when it retries it will find the customer during the SELECT in the find_or_create.
However, it does not seem to be working. It runs out of retries and then re-raises the exception.
Here's what I was doing originally:
retries = 10
begin
user = User.find_or_create_by(:email => email)
rescue ActiveRecord::RecordNotUnique
retry unless (retries-=1).zero?
raise
end
I then thought maybe the database connection was caching the SELECT query result causing it to think the user still does not exist and to continue to try to create it. To fix that I tried using Model.uncached to disable query caching:
retries = 10
begin
User.uncached do
user = User.find_or_create_by(:email => email)
end
rescue ActiveRecord::RecordNotUnique
retry unless (retries-=1).zero?
raise
end
`
This is also not working. I'm not sure what else to do to fix this issue? Should I increase retries? Add a sleep delay between retries? Is there a better way to clear query cache (if that's the issue?)
Any ideas?
Thank you!
If you are using Postgres there is an upsert (https://github.com/jesjos/active_record_upsert) gem that makes it really easy to add "ON CONFLICT DO NOTHING/UPDATE" in a scenario like this. You would be able to replace your code with
User.upsert(email: email)
If you want to add or update other data in the same operation as part of the same upsert call, you should first specify that email is the unique key for the model:
class User < ActiveRecord::Base
upsert_keys [:email]
end
Edit: To identify the problem with your approach try posting the relevant section of the log where we can see what SQL statements are being issued, including the transaction BEGIN/COMMITs.
This issue is fixed in rails 6+ with the method create_or_find_by.
Related PR: https://github.com/rails/rails/pull/31989
Documentation: https://apidock.com/rails/v6.0.0/ActiveRecord/Relation/create_or_find_by
This is similar to #find_or_create_by, but avoids the problem of stale reads between the SELECT and the INSERT, as that method needs to first query the table, then attempt to insert a row if none is found.
Warning: There are several drawbacks to #create_or_find_by, so consider reading the doc for your use cases.
I'm having a really confusing problem. Here it is:
[12] pry(EstimatedTime)> EstimatedTime.where(user_id: User.current.id, plan_on: date).pluck(:hours) => []
[13] pry(EstimatedTime)> EstimatedTime.where(user_id: User.current.id, plan_on: date).sum(:hours) => 3.0
What kind of magic is this?
This statement resides in model method, that is being called from view. Before that, in controller action, i'm calling another method of the same model, that is performing bulk create of records within transaction.
def self.save_all(records)
transaction do
records.each do |record|
record.save!
end
end
rescue ActiveRecord::RecordInvalid
return false
end
The exception is being thrown, method returns false, view is rendered and this happens.
UPD
I found a workaround, replacing .sum with .pluck(:hours).sum, but I still have no idea why my first way of doing this fails.
As David Aldrige pointed out in comments to the question, the problem was that .sum(:hours) was using cached data, while .pluck(:hours) was actually looking into database.
Why database and query cache contained different data? Well, seems like in Rails 3 (I should probably have specified my rails version when asking question) failed transaction would rollback the database, but leave query cache intact. This led to temporary data inconsistency between database and cache. This was corrected in Rails 4 as pointed out in this issue.
Solution 1 aka the one I chose for myself
Clear the query cache after transaction fails. So, the method performing the mass insert within transaction now looks like this:
def self.save_all(records)
transaction do
records.each do |record|
record.save!
end
end
rescue ActiveRecord::RecordInvalid
self.connection.clear_query_cache
return false
end
Solution 2
I personally find this much less elegant, though I cannot compare these solutions performance-wise, so I'll post them both. One can simply write the sum statement like this:
EstimatedTime.where(user_id: User.current.id, plan_on: date).pluck(:hours).sum
It will use the database data to calculate sum, bypassing the inconsistent cache problem.
I have a situation where I need to allow building up of SQL manually from a form. I have something like this:
SomeModel.where("id in (#{custom_sql})")
where custom_sql is a select statement like so:
SELECT u.id FROM some_model u WHERE country_iso = 'AU'
I want to be able to catch the StatementInvalid exception that is thrown when there is invalid SQL in the where clause, but I cannot figure out how to do so.
begin
SomeModel.where("id in (#{custom_sql})")
rescue
puts "Error"
end
But it keeps falling through without an error. Yet in rails c, when I do User.where("id in (#{custom_sql})"), it will correctly error. Any ideas?
Check the validity of your SQL statement using ActiveRecord::Base.connection.execute. It will throw an error if your statement is invalid.
begin
ActiveRecord::Base.connection.execute custom_sql
User.where("id in (#{custom_sql})")
rescue
puts "Error"
end
First of all your example is unclear or misleading (probably due to your efforts to mask your actual model names)
Because if you really wanted to run the scenario you are proposing, your resulting query would be
select * from some_model where id in (select u.id from some_model u where country_iso = "AU"
(ignoring the fact that your country_iso field is ambiguous) You are just running
SomeModel.where country_iso: "AU"
So you'll need to refine your example and properly illustrate your problem, because while #zeantsoi suggests a way to help you validate your statement(s), I agree with #phoet that you are probably approaching the situation incorrectly.
firstly, this is sql injection at it's best. you should reconsider whatever you want to accomplish.
secondly, User.where returns a relation. relations are lazy. lazy statements are not yet executed. so if you are doing User.where(...).all this should help.