I have a rake task that uses some threads and now I'm getting to a really strange case...
some code didn't get executed so I started playing with simple puts statements...
Basically I have this:
Thread.new do
puts "hi"
puts "there"
[more code]
end
These are three consecutive runs of my rake task:
$ rake task:execute
hi
there
$ rake task:execute
[nothing!]
$ rake task:execute
hi
I tried Ruby 2.0 and 2.1.
I don't know if the problem is just in puts but I think not because the code didn't get executed and that's why I started debugging with printouts only to discover that even this doesn't get executed (always).
Strange?
You need to save the reference to all the threads and then call join on each of them to wait for them to complete. Ruby will not wait for other threads once the main thread exits.
threads = 3.times.map { Thread.new { puts "hello" } }
# do something else while threads run, if you want
threads.each(&:join)
Your main thread (the rake task itself) is probably completing before your subthread completes. You can do something like this:
t = Thread.new do
puts "hi"
puts "there"
[more code]
end
[do other stuff in the main thread]
t.join # Let the subthread catch up
Related
I have 2 Sidekiq workers:
Foo:
# frozen_string_literal: true
class FooWorker
include Sidekiq::Worker
sidekiq_options queue: :foo
def perform
loop do
File.open(File.join(Rails.root, 'foo.txt'), 'w') { |file| file.write('FOO') }
end
end
end
Bar:
# frozen_string_literal: true
class BarWorker
include Sidekiq::Worker
sidekiq_options queue: :bar
def perform
loop do
File.open(File.join(Rails.root, 'bar.txt'), 'w') { |file| file.write('BAR') }
end
end
end
Which has pretty the same functionality, both runs on different queues and the yaml file looks like this:
---
:queues:
- foo
- bar
development:
:concurrency: 5
The problem is, even both are running and showing in the Busy page of Sidekiq UI, only one of them will actually create a file and put contents in. Shouldn't Sidekiq be multi-threaded?
Update:
this happens only on my machine
i created a new project with rails new and same
i cloned a colleague project and ran his sidekiq and is working!!!
i used his sidekiq version, not working!
New Update:
this happens also on my colleague machine if he clone my project
if I run 2 jobs with a finite loop ( like 10 times do something with a sleep), first job will be executed and then the second, but after the second finishes and start again both will work on same time as expected -- everyone who cloned the project from: github.com/ArayB/sidekiq-test encountered the problem.
It's not an issue with Sidekiq. It's an issue somewhere in Ruby/MRI/Thread/GIL. Google for more info, but my understanding is that sometimes threads aren't real threads (see "green threads") so really just simulate threading. The important thing is that only one thread can execute at a time.
It's interesting that with only two threads the system isn't giving time to the second thread. No idea why, but it must realize it's mistake when you run it again.
Interestingly if you run your same app but instead fire off 10 TestWorkers (and tweak the output so you can tell the difference) sidekiq will run all 10 "at once".
10.times {|i| TestWorker.perform_async(i) }
Here is the tweaked worker. Be sure to flush the output cause that can also cause issues with threading and TTY buffering not reflecting reality.
class TestWorker
include Sidekiq::Worker
def perform(n)
10.times do |i|
puts "#{n} - #{i} - #{Time.current}"
$stdout.flush
sleep 1
end
end
end
Some interesting links:
https://en.wikipedia.org/wiki/Green_threads
http://ruby-doc.org/core-2.4.1/Thread.html#method-c-pass
https://github.com/ruby/ruby/blob/v2_4_1/thread.c
Does ruby have real multithreading?
I have the following code that I run in a rake task in rails:
10.times do |i|
Thread.new do
puts "#{i}"
end
end
When I run this locally, I get the following:
0
3
5
1
7
8
2
4
9
6 (with new lines)
However, when I run the same code in EC2 via the same rake task, it will print out maybe one or two lines, and then the task will terminate. I'm not sure why, but it seems my EC2 instance can't handle the multithreading for some reason.
Any insights why?
You've just been getting lucky locally - there is nothing that guarantees that your 10 threads will execute to completion before your program exits. If you want to wait for your threads then you must do so explicitly:
threads = 10.times.collect do |i|
Thread.new do
puts i
end
end
threads.each(&:join)
The join method blocks the calling thread until the specified thread has completed. It also returns the return value of that thread.
I have an operation that I need to execute in my rails application that before my Rails app dies. Is there a hook I can utilize in Rails for this? Something similar to at_exit I guess.
Ruby itself supports two hooks, BEGIN and END, which are run at the start of a script and as the interpreter stops running it.
See "What does Ruby's BEGIN do?" for more information.
The BEGIN documentation says:
Designates, via code block, code to be executed unconditionally before sequential execution of the program begins. Sometimes used to simulate forward references to methods.
puts times_3(gets.to_i)
BEGIN {
def times_3(n)
n * 3
end
}
The END documentations says:
Designates, via code block, code to be executed just prior to program termination.
END { puts "Bye!" }
Okay so I am making no guarantees as to impact because I have not tested this at all but you could define your own hook e.g.
ObjectSpace.define_finalizer(YOUR_RAILS_APP::Application, proc {puts "exiting now"})
Note this will execute after at_exit so the rails application server output will look like
Stopping ...
Exiting
exiting now
With Tin Man's solution included
ObjectSpace.define_finalizer(YOUR_RAILS_APP::Application, proc {puts "exiting now"})
END { puts "exiting again" }
Output is
Stopping ...
Exiting
exiting again
exiting now
Let's say we want to test that the database is being locked..
$transaction = Thread.new {
Rails.logger.debug 'transaction process start'
Inventory.transaction do
inventory.lock!
Thread.stop
inventory.units_available=99
inventory.save
end
}
$race_condition = Thread.new {
Rails.logger.debug 'race_condition process start'
config = ActiveRecord::Base.configurations[Rails.env].symbolize_keys
config[:flags] = 65536 | 131072 | Mysql2::Client::FOUND_ROWS
begin
connection = Mysql2::Client.new(config)
$transaction.run
$transaction.join
rescue NoMethodError
ensure
connection.close if connection
end
}
Rails.logger.debug 'main process start'
$transaction.join
Rails.logger.debug 'main process after transaction.join'
sleep 0.1 while $transaction.status!='sleep'
Rails.logger.debug 'main process after sleep'
$race_condition.join
Rails.logger.debug 'main process after race_condition.join'
In theory, I'd think it would do the transaction thread, then wait( Thread.stop ), then the main process would see that it's sleeping, and start the race condition thread(which will be trying to alter data in the locked table when it actually works). Then the race condition would continue the transaction thread after it was done.
what's weird is the trace
main process start
transaction process start
race_condition process start
Coming from nodejs, it seems like threads aren't exactly as user friendly.. though, there has to be a way to get this done.
Is there an easier way to lock the database, then try to change it with a different thread?
Thread.new automatically starts the Thread.
But that does not mean that it is executing.
That depends on Operations system, ruby or jruby, how many cores, etc.
In your example the main thread runs until
$transaction.join,
and only then your transaction thread starts, just by chance.
It runs still Thread.stop, then your '$race_condition' Thread starts, because both other are blocked (it might have started before)
So that explains your log.
You have two $transaction.join
they wait until the thread exits, but a thread can only exit once...
I don't know what is happen then, maybe the second call waits forever.
For your test, you need some sort of explicit synchronization, so that our race_thread writes exactly when the transaction_thread is in the middle of the transaction. You can do this with Mutex, but better would be some sort of message passing. The following blog post may help:
http://www.engineyard.com/blog/2011/a-modern-guide-to-threads/
For any resource to make it a "Mutually Exclusive", you need to use Mutex class and use a synchronize method to make the resources locked while one thread is using them. You have to do something like this:
semaphore = Mutex.new
and use it inside the Thread instance.
$transaction = Thread.new {
semaphore.synchronize{
# Do whatever you want with the *your shared resource*
}
}
This way you can prevent any deadlocks.
Hope this helps.
I have a long running rake task. Every now and then I print an update to the screen to let me know how far along the task has come.
puts "Almost there..."
My problem is all the puts statements seem to buffer somewhere and won't print to the screen until after the task is complete. At which point, they will be printed all at once.
Is there some way to force them to print as the task is running?
STDOUT.sync = true
May be you could flush the standart output:
STDOUT.flush