To get a better understanding of pessimist locking (with InnoDB), I tried to run this code in my Rails application:
Thread.new do
Account.transaction do
account = Account.lock(true).first
account.balance += 250
account.save!
end
end
Thread.new do
Account.transaction do
account = Account.lock(true).first
account.balance += 500
account.save!
end
end
It actually works, account.balance then contains 750, then 1500 on the next hit. Without locking, it just takes the last thread into consideration and the result is 500.
Is that a dumb test to try out the difference between locking or not? I think I understand the principle of pessimist locking, but not sure though.
This test does allow you to observe pessimistic locking behaviour, but when you remove locks it doesn't switch to optimistic locking. It just does no locking at all.
Also, to be more idiomatic you could do this:
Thread.new do
account = Account.first
account.with_lock do
account.balance += 250
account.save!
end
end
with_lock starts a transaction and acquires a lock in one go.
To observe optimistic locking behaviour you need lock_version field to be present in your model. Rails will then do record version check automatically and throw ActiveRecord::StaleObjectError on conflict.
Related
I'm using the with_advisory_lock gem to try and ensure that a record is created only once. Here's the github url to the gem.
I have the following code, which sits in an operation class that I wrote to handle creating user subscriptions:
def create_subscription_for user
subscription = UserSubscription.with_advisory_lock("lock_%d" % user.id) do
UserSubscription.where({ user_id: user.id }).first_or_create
end
# do more stuff on that subscription
end
and the accompanying test:
threads = []
user = FactoryBot.create(:user)
rand(5..10).times do
threads << Thread.new do
subject.create_subscription_for(user)
end
end
threads.each(&:join)
expect(UserSubscription.count).to eq(1)
What I expect to happen:
The first thread to get to the block acquires the lock and creates a record.
Any other thread that gets to the block while it's being held by another thread waits indefinitely until the lock is released (as per docs)
As soon as the lock is released by the first thread that created the record, another thread acquires the lock and now finds the record because it was already created by the first thread.
What actually happens:
The first thread to get to the block acquires the lock and creates a record.
Any other thread that gets to the block while it's being held by another thread goes and executes the code in the block anyway and as a result, when running the test, it sometimes fails with a ActiveRecord::RecordNotUnique error (I have a unique index on the table that allows for a single user_subscription with the same user_id)
What is more weird is that if I add a sleep for a few hundred milliseconds in my method just before the find_or_create method, the test never fails:
def create_subscription_for user
subscription = UserSubscription.with_advisory_lock("lock_%d" % user.id) do
sleep 0.2
UserSubscription.where({ user_id: user.id }).first_or_create
end
# do more stuff on that subscription
end
My questions are: "Why is adding the sleep 0.2 making the tests always pass?" and "Where do I look to debug this?"
Thanks!
UPDATE: Tweaking the tests a little bit causes them to always fail:
threads = []
user = FactoryBot.create(:user)
rand(5..10).times do
threads << Thread.new do
sleep
subject.create_subscription_for(user)
end
end
until threads.all? { |t| t.status == 'sleep' }
sleep 0.1
end
threads.each(&:wakeup)
threads.each(&:join)
expect(UserSubscription.count).to eq(1)
I have also wrapped first_or_create in a transaction, which makes the test pass and everything to work as expected:
def create_subscription_for user
subscription = UserSubscription.with_advisory_lock("lock_%d" % user.id) do
UserSubscription.transaction do
UserSubscription.where({ user_id: user.id }).first_or_create
end
end
# do more stuff on that subscription
end
So why is wrapping first_or_create in a transaction necessary to make things work?
Are you turning off transactional tests for this test case? I'm working on something similar and that proved to be important to actually simulating the concurrency.
See uses_transaction https://api.rubyonrails.org/classes/ActiveRecord/TestFixtures/ClassMethods.html
If transactions are not turned off, Rails will wrap the entire test in a transaction and this will cause all the threads to share one DB connection. Furthermore, in Postgres a session-level advisory lock can always be re-acquired within the same session. From the docs:
If a session already holds a given advisory lock, additional requests
by it will always succeed, even if other sessions are awaiting the
lock; this statement is true regardless of whether the existing lock
hold and new request are at session level or transaction level.
Based on that I'm suspecting that your lock is always able to be acquired and therefore the .first_or_create call is always executed which results in the intermittent RecordNotUnique exceptions.
Context: Product is getting updated by multiple threads. So it leads the race condition. So I am using optimistic locking because it is frequently not updated. For latest update state of the product is determined by updated_at attribute.
For example, Prod (id=1) is updated at t1 on the machine:1. Same product(id=1) is updated at t2 on the machine:2. Now the state of Prod (id=1) on the machine:1 is stale.
Approach To determine stale: I will compare the value of updated_at on the machine with the store value of updated_atin the database value.
My main concern is setting the value of #original_updated_at. Should I use attr_writer :original_updated_at. Is this right way do optimistic locking.
attr_accessor : :original_updated_at
def original_updated_at
#original_updated_at || updated_at.to_f
end
def stale_object?
if updated_at.to_f > original_updated_at.to_f
#original_updated_at = nil
return true
else
#original_updated_at = updated_at.to_f
return false
end
end
def recalculate
tries = 0
begin
raise Product::StaleObjectError.new("StaleObjectError") if stale_object?
attributes["updated_at"] = Time.now.utc
Product.where(:id => self.id).update_all(attributes)
self.attributes = attributes
rescue Product::StaleObjectError => e
if tries < MAX_RETRIES
tries += 1
sleep(1 + tries)
reload
retry
else
raise Product::StaleObjectError("StaleObjectError")
end
end
end
It looks like you are using Rails? Not sure why you are trying to roll your own solution.
Just add a lock_version column to enable optimistic locking on DB level.
https://api.rubyonrails.org/classes/ActiveRecord/Locking/Optimistic.html
Also: since you are talking about multiple machines, concurrency could come from multiple processes and multiple threads as well.
Since your legacy system breaks when adding the lock column some other solutions:
Fix the issues with optimistic locking
use pessimistic locking. Depending on the load scenario - how many concurrent read/write - operations this might work as well
Fix your locking code (not entirely sure how it works because you are leaving out some details of the class in question)
no need to raise, catch and re-raise (just use a loop and breakon success)
you are calling to_f on floats
I don't know what self.attributes = attributes is supposed to do (from the code you show it is unclear if attributes is something else than self.attributes)
using a timestamp might or might not be good enough to determine concurrent updates (two exact same timestamps might be generated, clocks might be off on multiple machines, clock might get reset or adjusted in between)
using sleep is a code smell
When you write code that is so complicated that you have to ask on the internet if it works, then you are probably going the wrong way. Fix the problems with optimistic locking instead of adding such a workaround.
To allow users create balance withdrawal request, I have a WithdrawalsController#create action. The code checks if the user has sufficient balance before proceeding to creating the withdrawal.
def create
if amount > current_user.available_balance
error! :bad_request, :metadata => {code: "002", description: "insufficient balance"}
return
end
withdrawal = current_user.withdrawals.create(amount: amount, billing_info: current_user.billing_info)
exposes withdrawal
end
This can pose a serious issue in a multi-threaded server. When two create requests arrive simultaneously, and both requests pass the the balance check before creating the withdrawal, then both withdrawal can be created even if the sum of two could be exceeding the original balance.
Having a class variable of Mutex will not be a good solution because that will lock this action for all users, where a lock on a per-user level is desired.
What is the best solution for this?
The following diagram illustrates my suspected threading issue, could it be occurring in Rails?
As far as I can tell your code is safe here, mutlithreading is not a much of a problem. Even with more app instances generate by your app server, each instance will end up testing amount > current_user.available_balance.
If you are really paranoiac about it. you could wrap the all with a transacaction:
ActiveRecord::Base.transaction do
withdrawal = current_user.withdrawals.create!(
amount: amount,
billing_info: current_user.billing_info
)
# if after saving the `available_balance` goes under 0
# the hole transaction will be rolled back and any
# change happened to the database will be undone.
raise ActiveRecord::Rollback if current_user.available_balance < 0
end
My API allows users to buy certain unique items, where each item can only be sold to one user. So when multiple users try to buy the same item, one user should get the response: ok and the other user should get the response too_late.
Now, there seems to be bug in my code. A race condition. If two users try to buy the same item at the same time, they both get the answer ok. The issue is clearly reproducable in production. Now I have written a simple test that tries to reproduce it via rspec:
context "when I try to provoke a race condition" do
# ...
before do
#concurrent_requests = 2.times.map do
Thread.new do
Thread.current[:answer] = post "/api/v1/item/buy.json", :id => item.id
end
end
#answers = #concurrent_requests.map do |th|
th.join
th[:answer].body
end
end
it "should only sell the item to one user" do
#answers.sort.should == ["ok", "too_late"].sort
end
end
It seems like does not execute the queries at the same time. To test this, I put the following code into my controller action:
puts "Is it concurrent?"
sleep 0.2
puts "Oh Noez."
Expected output would be, if the requests are concurrent:
Is it concurrent?
Is it concurrent?
Oh Noez.
Oh Noez.
However, I get the following output:
Is it concurrent?
Oh Noez.
Is it concurrent?
Oh Noez.
Which tells me, that capybara requests are not run at the same time, but one at a time. How do I make my capabara requests concurrent?
Multithreading and capybara does not work, because Capabara uses a seperate server thread which handles connection sequentially. But if you fork, it works.
I am using exit codes as an inter-process communication mechanism. If you do more complex stuff, you may want to use sockets.
This is my quick and dirty hack:
before do
#concurrent_requests = 2.times.map do
fork do
# ActiveRecord explodes when you do not re-establish the sockets
ActiveRecord::Base.connection.reconnect!
answer = post "/api/v1/item/buy.json", :id => item.id
# Calling exit! instead of exit so we do not invoke any rspec's `at_exit`
# handlers, which cleans up, measures code coverage and make things explode.
case JSON.parse(answer.body)["status"]
when "accepted"
exit! 128
when "too_late"
exit! 129
end
end
end
# Wait for the two requests to finish and get the exit codes.
#exitcodes = #concurrent_requests.map do |pid|
Process.waitpid(pid)
$?.exitstatus
end
# Also reconnect in the main process, just in case things go wrong...
ActiveRecord::Base.connection.reconnect!
# And reload the item that has been modified by the seperate processs,
# for use in later `it` blocks.
item.reload
end
it "should only accept one of two concurrent requests" do
#exitcodes.sort.should == [128, 129]
end
I use rather exotic exit codes like 128 and 129, because processes exit with code 0 if the case block is not reached and 1 if an exception occurs. Both should not happen. So by using higher codes, I notice when things go wrong.
You can't make concurrent capybara requests. However, you can create multiple capybara sessions and use them within the same test to simulate concurrent users.
user_1 = Capybara::Session.new(:webkit) # or whatever driver
user_2 = Capybara::Session.new(:webkit)
user_1.visit 'some/page'
user_2.visit 'some/page'
# ... more tests ...
user_1.click_on 'Buy'
user_2.click_on 'Buy'
Assuming I'm doing something like this (from the Active Record Querying guide)
Item.transaction do
i = Item.first(:lock => true)
i.name = 'Jones'
i.save
end
Is the lock automatically released at the end of the transaction? I've looked at the Active Query guide and the ActiveRecord::Locking::Pessimistic docs, and couldn't find where it explicitly says where the lock is released.
Locking is not a function of rails, it is just adding the lock statement to the query, which will vary depending on the database that you are using. Pessimistic Locking takes a "pessimistic" view in thinking that every query is subject to corruption. So it is going to lock selected rows until you are finished with the transaction. so Lock > query > unlock. While these are fairly consistent database to database, it might be good to read up on the database documentation that you using for any database-specific things you should know.
Here is a good thread on optimistic vs. pessimistic locking that explains it better than I can. Optimistic vs. Pessimistic locking
Yes, the lock automatically released at the end of the transaction because this kind of lock is applicable to transactions only. It does not make sense to lock the record this way (pessimistic lock) outside the transaction.
Pessimistic locks are enforced on DB level.
Below is a description with examples for mysql:
http://dev.mysql.com/doc/refman/5.0/en/innodb-lock-modes.html
I acknowledged the problem with pessimistic lock within transaction during rspec tests.
For some reason on different systems (I found this because of CI failed to run spec) record is still locked and cannot be fetched.
So code and rspec example are below.
class FooBar
def foo
Model.with_lock do
model.update(bar: "baz")
end
end
end
red example
it "updates with lock" do
expect { Foobar.foo }.to change { model.reload.bar }.to("baz")
end
but correct green example should look like this
it "updates with lock" do
Foobar.foo
expect(model.reload.bar).to eq("baz")
end
I believe you'll want an "ensure" block to be certain the lock is released.
http://ruby-doc.org/core/classes/Mutex.src/M000916.html has:
def synchronize
lock
begin
yield
ensure
unlock
end
end
http://yehudakatz.com/2010/02/07/the-building-blocks-of-ruby/ seems to suggest, however, that the block structure of that method will automatically unlock.