I am using OmniThread Parallel.foreach(). There are instances where the loop takes a long time or gets stuck.
I would like to know, is it possible to timeout each process in the Parallel.foreach() loop?
In short: Nope, there isn't.
Unless you program the timeout handling in your 'thread body' code (what gets called in the execute).
eg my database engine allows sending a CancelProcessing call to running queries from a different thread that runs the query, this would 'cleanly' end the running subthread.
'Dirty' end of the subthreads:
I added a FR to Omnithread's github site to add an (Dirty) Terminate method to the IOmniParallel interfaces (and alikes). Which has is drawback because killing subthreads will probably leave you with memory/resource leaks.
Meanwile you might use this dirty shutdown solution/workaround wich actually comes down fixing a similar problem (I had a deadlock in my parallel processed routine, so my parallel.Waitfor never returned true, and worse my IOmniParallelTask interface variable was never released causing the calling thread to block as well.
Related
I need a way shutting down the Xodus environment so that
It waits for all the writing transactions in all threads to finish (or be aborted after a timeout as an option).
It blocks starting new transactions (or throws an exception as an option).
Safely closes the environment.
So far we tried something like this
if (env.isOpen()) {
env.clear();
env.close();
}
but I am not sure that it does exactly the right thing, I still get exceptions thrown from env.close() from time to time. So what is the proper way to do that?
First of all, note that Environment#clear() just clears all the data in your environment.
Minor: you don't have to check whether your environment is open before closing it.
If you don't care much of the application state then you can set the exodus.env.closeForcedly option on Environment creation:
Environment env = Environments.newInstance("db path", new EnvironmentConfig().setEnvCloseForcedly(true));
In that case, close() method reports in logs the number of not finished at the moment transactions and does close the environment anyway.
I'm using rufus-scheduler to run a number of frequent jobs that do some various tasks with ActiveRecord objects. If there is any sort of network or postgresql hiccup, even after recovery, all the threads will throw the following error until the process is restarted:
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds (waited 5.000122687 seconds). The max pool size is currently 5; consider increasing it.
The error can easily be reproduced by restarting postgres. I've tried playing (up to 15) with the pool size, but no luck there.
That leads me to believe the connections are just in a stale state, which I thought would be fixed with the call to clear_stale_cached_connections!.
Is there a more reliable pattern to do this?
The block that is passed is a simple select and update active record call, and happens to matter what the AR object is.
The rufus job:
scheduler.every '5s' do
db do
DataFeed.update #standard AR select/update
end
end
wrapper:
def db(&block)
begin
ActiveRecord::Base.connection_pool.clear_stale_cached_connections!
#ActiveRecord::Base.establish_connection # this didn't help either way
yield block
rescue Exception => e
raise e
ensure
ActiveRecord::Base.connection.close if ActiveRecord::Base.connection
ActiveRecord::Base.clear_active_connections!
end
end
Rufus scheduler starts a new thread for every job.
ActiveRecord on the other hand cannot share connections between threads, so it needs to assign a connection to a specific thread.
When your thread doesn't have a connection yet, it will get one from the pool.
(If all connections in the pool are in use, it will wait untill one is returned from another thread. Eventually timing out and throwing ConnectionTimeoutError)
It is your responsibility to return it back to the pool when you are done with it, in a Rails app, this is done automatically. But if you are managing your own threads (as rufus does), you have to do this yourself.
Lucklily, there is an api for this:
If you put your code inside a with_connection block, it will get a connection form the pool, and release it when it is done
ActiveRecord::Base.connection_pool.with_connection do
#your code here
end
In your case:
def db
ActiveRecord::Base.connection_pool.with_connection do
yield
end
end
Should do the trick....
http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html#method-i-with_connection
The reason can be that you have many threads which are using all connections, if DataFeed.update method takes more than 5 seconds, than your block can be overlapped.
try
scheduler.every("5s", :allow_overlapping => false) do
#...
end
Also try release connection instead of closing it.
ActiveRecord::Base.connection_pool.release_connection
I don't really know about rufus-scheduler, but I got some ideas.
The first problem could be a bug on rufus-scheduler that does not checkout database connection properly. If it's the case the only solution is to clear stale connections manually as you already do and to inform the author of rufus-scheduler about your issue.
Another problem that could happen is that your DataFeed operation takes a really long time and because it is performed every 5 secondes Rails is running out of database connections, but it's rather unlikely.
I ran into this weird issue while working with ruby(on rails) time outs. This time out
timeout(10) do
//some code involving http calls that takes more than 10 seconds
end
is not working. But this timeout
timeout(20) do
timeout(10) do
//some code involving http calls that takes more than 10 seconds
end
end
times out after 20 seconds. I read that timeout in ruby wont work properly if it involves system calls. If that be the case then any number of nested timeout should also not work. Why would this work on the second timeout?
btw..the link I referred
http://ph7spot.com/musings/system-timer
Thanks in advance
You might have better luck using a combination of timeout and terminator to do this sort of thing.
One of the known deficiencies of the timeout method is it's not always strictly enforced and many things can block it.
Poking around in the rails code, I ran across with_scope.
From what I can tell, it takes the scope type and conditions, merges them into existing conditions for that scope type, yields to the block, then gets rid of the additional scope.
So my first thought is in a multithreaded environment (like jruby on rails), what happens if while thread 1 is executing the block, thread 2 decides to do a Model.find :all? It seems to me like a race condition waiting to happen.
Am I missing something?
So the trick in here is that if you trace deep enough, the scopes are getting set through Thread.current[method], which will execute method but only in the scope of the current thread. I didn't even know that was possible for ruby... guess you learn something new every day
When reading data from a potentially slow website, I want to ensure that get_response can not hang, and so added a timer to timeout after x seconds. So far, so good. I then read http://ph7spot.com/musings/system-timer which illustrates that in certain situations timer.rb doesn't work due to ruby's implementation of threads.
Does anyone know if this is one of these situations?
url = URI.parse(someurl)
begin
Timeout::timeout(30) do
response = Net::HTTP.get_response(url)
#responseValue = CGI.unescape(response.body)
end
rescue Exception => e
dosomething
end
well, first of all Timeout is not a class defined in Rails but in Ruby, second, Timeout is not reliable in cases when you make system calls.
Ruby uses what it's so called Green Threads. Let's suppose you have 3 threads, you think all of them will run in parallel but if one of the threads makes a syscall all the rest of the threads will be blocked until the syscall finishes, in this case Timeout won't work as expected, so it's always better to use something reliable like SystemTimer.