I would expect the below code to force exclusive access to the code block within, but it isn't. In my test, each thread is able to run the block concurrently.
Assume a Rails environment and user is an activerecord object. Also note that this is a somewhat arbitrary test I wrote in order to resolve a concurrency issue I'm experiencing with web requests.
user = User.first
threads = []
3.times do |i|
threads << Thread.new do
user.with_lock do
puts "start: #{i}"
sleep 1
puts "stop: #{1}"
end
end
end
threads.each(&:join)
Expected output:
start: 1
stop: 1
start: 2
stop: 2
start: 3
stop: 3
Actual output:
start: 1
start: 2
start: 3
stop: 1
stop: 2
stop: 3
What am I missing? Does rails .with_lock not work within standard ruby threads? Or, is this possibly due to my test environment using sqlite3?
It appears to have been related to sqlite3.
You've evidently figured this out already, but to clarify for others encountering this problem:
Sqlite3 doesn't support row-level locking. Therefore, Arel simply ignores any locks when building the query:
https://github.com/rails/arel/blob/master/lib/arel/visitors/sqlite.rb#L7
Related
I have a rails application running with puma server. Is there any way, we can see how many number of threads used in application currently ?
I was wondering about the same thing a while ago and came upon this issue. The author included the code they ended up using to collect those stats:
module PumaThreadLogger
def initialize *args
ret = super *args
Thread.new do
while true
# Every X seconds, write out what the state of this dyno is in a format that Librato understands.
sleep 5
thread_count = 0
backlog = 0
waiting = 0
# I don't do the logging or string stuff inside of the mutex. I want to get out of there as fast as possible
#mutex.synchronize {
thread_count = #workers.size
backlog = #todo.size
waiting = #waiting
}
# For some reason, even a single Puma server (not clustered) has two booted ThreadPools.
# One of them is empty, and the other is actually doing work
# The check above ignores the empty one
if (thread_count > 0)
# It might be cool if we knew the Puma worker index for this worker, but that didn't look easy to me.
# The good news: By using the PID we can differentiate two different workers on two different dynos with the same name
# (which might happen if one is shutting down and the other is starting)
source_name = "#{Process.pid}"
# If we have a dyno name, prepend it to the source to make it easier to group in the log output
dyno_name = ENV['DYNO']
if (dyno_name)
source_name="#{dyno_name}."+source_name
end
msg = "source=#{source_name} "
msg += "sample#puma.backlog=#{backlog} sample#puma.active_connections=#{thread_count - waiting} sample#puma.total_threads=#{thread_count}"
Rails.logger.info msg
end
end
end
ret
end
end
module Puma
class ThreadPool
prepend PumaThreadLogger
end
end
This code contains logic that is specific to heroku, but the core of collecting the #workers.size and logging it will work in any environment.
I have the following code that I run in a rake task in rails:
10.times do |i|
Thread.new do
puts "#{i}"
end
end
When I run this locally, I get the following:
0
3
5
1
7
8
2
4
9
6 (with new lines)
However, when I run the same code in EC2 via the same rake task, it will print out maybe one or two lines, and then the task will terminate. I'm not sure why, but it seems my EC2 instance can't handle the multithreading for some reason.
Any insights why?
You've just been getting lucky locally - there is nothing that guarantees that your 10 threads will execute to completion before your program exits. If you want to wait for your threads then you must do so explicitly:
threads = 10.times.collect do |i|
Thread.new do
puts i
end
end
threads.each(&:join)
The join method blocks the calling thread until the specified thread has completed. It also returns the return value of that thread.
I've a rails app that I'm crashing on purpose.. it's local and I'm just hitting ctrl + c and killing it mid way through processing records..
To my mind the records in the block shouldn't have been committed.. Is this a postgres "error" or a rails "error", or a dave ERROR?
ActiveRecord::Base.transaction do
UploadStage.where("id in (#{ids.join(',')})").update_all(:status => 2);
records.each do |record|
record.success = process_line(record.id, klas, record.hash_value).to_s[0..250]
record.status = 1000
record.save();
end
end
I generate my ids by reading out all the records where the status is 1.
Nothing but this function sets the status to 1000..
If the action crashes for what ever reason, I'd expect there to be no records in the database with status = 2...
This is not what I'm seeing though. Half the records have status 1000, the other half have status 2.. .
Am I missing something?
How can I make sure there are no 2's if the app crashes?
EDIT:
I found this link http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/
As I suspected and as confirmed by dave's update, it looks like ActiveRecord will commit a half-finished transaction under some circumstances when you kill a thread. Woo, safe! See dave's link for detailed explanation and mitigation options.
If you're simulating hard crash (host OS crash or plug-pull), control-C is absolutely not the right approach. Use Control-\ to send a SIGQUIT, which is generally not handled, or use kill -KILL to hard-kill the process with no opportunity to do cleanup. Control-C sends SIGINT which is a gentle signal that's usually attached to a clean shutdown handler.
In general, if you're debugging issues like this, you should enable detailed query logging and see what Rails is doing. Use log_statement = 'all' in postgresql.conf then examine the PostgreSQL logs.
I have a Ruby process that listens on a given device. I would like to spin up/down instances of it for different devices with a rails app. Everything I can find for Ruby daemons seems to be based around a set number of daemons running or background processing with message queues.
Should I just be doing this with Kernel.spawn and storing the PIDs in the database? It seems a bit hacky but if there isn't an existing framework that allows me to bring up/down daemons it seems I may not have much choice.
Instead of spawning another script and keeping the PIDs in the database, you can do it all within the same script, using fork, and keeping PIDs in memory. Here's a sample script - you add and delete "worker instances" by typing commands "add" and "del" in console, exiting with "quit":
#pids = []
#counter = 0
def add_process
#pids.push(Process.fork {
loop do
puts "Hello from worker ##{#counter}"
sleep 1
end
})
#counter += 1
end
def del_process
return false if #pids.empty?
pid = #pids.pop
Process.kill('SIGTERM', pid)
true
end
def kill_all
while del_process
end
end
while cmd = gets.chomp
case cmd.downcase
when 'quit'
kill_all
exit
when 'add'
add_process
when 'del'
del_process
end
end
Of course, this is just an example, and for sending comands and/or monitoring instances you can replace this simple gets loop with a small Sinatra app, or socket interface, or named pipes etc.
I am using lock! in my code and want to catch the exception thrown if lock! fails for some reason (e.g. cannot get the lock). What kind of exceptions can lock! throw? I checked the ruby docs but couldn't find the specific Exception classes.
Thanks.
When in doubt, probe.
Consider the following pair of functions:
def long_hold
ActiveRecord::Base.transaction do
u = User.find(220)
u.lock!
sleep 100.seconds
u.email="foo#bar.com"
u.save!
end
end
def short_hold
ActiveRecord::Base.transaction do
u = User.find(220)
u.lock!
u.email="foo#bar.com"
u.save!
end
end
In my setup (OSX 10.11, ruby 2.2.4, rails 4.2, postgres 9.5), running long_hold in one rails console and then running short_hold in a second console, I observe short_hold blocks until long_hold completes; moreover, instrumenting the code with puts, we see that while long_hold is sleeping, short_hold is waiting to acquire the lock.
Assuming no caveats about the independence of rails consoles, this suggests that no exceptions are thrown if a second process tries to lock a row that is already locked, but that process blocks until the first completes.
Here is the source for that locking call. It calls reload and its source looks like this:
# File lib/active_record/base.rb, line 2333
2333: def reload(options = nil)
2334: clear_aggregation_cache
2335: clear_association_cache
2336: #attributes.update(self.class.find(self.id, options).instance_variable_get('#attributes'))
2337: #attributes_cache = {}
2338: self
2339: end
so when you call reload(:lock => lock) as the call to lock does it it really updating the attributes of that record.
There are a lot of different situations here. You could try to lock a record that dosnt exist, or lock one that has been locked elsewhere. What error are you interested in catching?