In my rails app, I use open-uri to open an external file which may take up to 10 minutes to load
Example:
dl_stream = open('http://wetten.overheid.nl/xml.php?regelingID=bwbr0020368')
Now, after 1 minute, Ruby will throw a timeout error. I gleaned this from the source code, in \net\protocol.rc:
#read_timeout = 60
def rbuf_fill
begin
#rbuf << #io.read_nonblock(BUFSIZE)
rescue IO::WaitReadable
if IO.select([#io], nil, nil, #read_timeout)
retry
else
raise Timeout::Error
end
rescue IO::WaitWritable
# OpenSSL::Buffering#read_nonblock may fail with IO::WaitWritable.
# http://www.openssl.org/support/faq.html#PROG10
if IO.select(nil, [#io], nil, #read_timeout)
retry
else
raise Timeout::Error
end
end
end
I'm guessing I can set this timeout value to something more amenable to my situation, like 15 minutes, in my app settings, but how and where?
You can add the timeout in seconds to the call to open with the :read_timeout option:
# timeout after 10 minutes
open('http://example.com', :read_timeout => 600).read
All the options are documented here.
Related
I just created this little currency converter for my Rails 4 app:
module Currency
def self.get_exchange_rate(from_curr = "EUR", to_curr = "USD")
if from_curr == to_curr
result = 1
else
begin
amount = 1
url = "http://www.google.com/finance/converter?a=#{amount}&from=#{from_curr}&to=#{to_curr}"
doc = Nokogiri::HTML(open(url))
result_span = doc.css('span.bld').text
result = result_span.tr('^0-9.', '')
rescue => e
puts e
result = 1
end
end
result
end
end
I haven't done this a lot, so my question would be: How can I deal with the (unlikely) event that Google Currency times out or is not available for some reason?
In that case I would like my result to be 1. How can this be achieved?
Thanks for any suggestions.
You can simply rescue any error that Nokogiri might raise (or OpenURI) like so:
require 'nokogiri'
require 'open-uri'
def currency(a)
Nokogiri::HTML(open(a))
rescue => e
puts e
1 # default value when error is raised
end
puts currency('https://www.somedomainthatdoesntexist.com')
That will print the backtrace of the error and then return the number 1
getaddrinfo: nodename nor servname provided, or not known
1
For a custom timeout actions you can use Timeout module
http://ruby-doc.org/stdlib-2.1.2/libdoc/timeout/rdoc/Timeout.html
It'd be worth looking at this answer to see how to get nokogiri to timeout : Adjusting timeouts for Nokogiri connections
The question makes use of the timeout module which is an option but I think it is better to get the connection to time out as per the accepted answer.
All that will mean that you'll get a Timeout::Error exception raised if the call times out which you then need to handle:
begin
doc = ... use nokogiri ...
result_span = doc.css('span.bld').text
result = result_span.tr('^0-9.', '')
rescue Timeout::Error
result = 1
end
result
I have a page that I'd like to raise error if it loads too slow.
Is there some method for Watir analogous to Watir-Webdriver's:
client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 10
#browser = Watir::Browser.new :firefox, http_client: client
Watir-Classic does not have an API for controlling how long to wait for a page to load.
When clicking a link or using the goto method, the Browser#wait method is called. This will block execution until the page is loaded. It is hard-coded to timeout if the page does not load in 5 minutes:
def wait(no_sleep=false)
#xml_parser_doc = nil
#down_load_time = 0.0
interval = 0.05
start_load_time = ::Time.now
Timeout::timeout(5*60) do
...
end
Solution 1 - Use Timeout
If you only need to change timeout for a small number of scenarios, the simplest option may be to use the Timeout library.
For example, www.cnn.com takes 9 seconds to load on my computer. However, to only wait up to 5 seconds, you can wrap the goto (or click) method in an extra timeout:
Timeout::timeout(5) do
browser.goto 'www.cnn.com'
end
#=> execution expired (Timeout::Error)
Solution 2 - Monkey patch Browser#wait
If you want the change to apply to all pages, you could overwrite the Browser#wait method to use a different timeout. For example, overwriting it to only be 5 seconds:
require 'watir-classic'
module Watir
class Browser
def wait(no_sleep=false)
#xml_parser_doc = nil
#down_load_time = 0.0
interval = 0.05
start_load_time = ::Time.now
# The timeout can be changed here (it is in seconds)
Timeout::timeout(5) do
begin
while #ie.busy
sleep interval
end
until READYSTATES.has_value?(#ie.readyState)
sleep interval
end
until #ie.document
sleep interval
end
documents_to_wait_for = [#ie.document]
rescue WIN32OLERuntimeError # IE window must have been closed
#down_load_time = ::Time.now - start_load_time
return #down_load_time
end
while doc = documents_to_wait_for.shift
begin
until READYSTATES.has_key?(doc.readyState.to_sym)
sleep interval
end
#url_list << doc.location.href unless #url_list.include?(doc.location.href)
doc.frames.length.times do |n|
begin
documents_to_wait_for << doc.frames[n.to_s].document
rescue WIN32OLERuntimeError, NoMethodError
end
end
rescue WIN32OLERuntimeError
end
end
end
#down_load_time = ::Time.now - start_load_time
run_error_checks
sleep #pause_after_wait unless no_sleep
#down_load_time
end
end
end
browser.goto 'www.cnn.com'
#=> execution expired (Timeout::Error)
You could put the timeout value into a variable so that it can be dynamically changed.
We have an application that makes hundreds of API calls to external services. Sometimes some calls take too much time to respond.
I am using the rake_timeout gem to find time consuming process, so, Timeout::Error will be thrown whenever some request is taking too long to respond. I am rescuing this error and doing a retry on that method:
def new
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
#Below code will be repeated in every method
tries = 0
rescue Timeout::Error => e
tries += 1
retry if tries <= 3
logger.error e.message
end
This lets the method fully re-run it. This is very verbose and I am repeating it every time.
Is there any way to do this so that, if the Timeout:Error occurrs, it will retry that method automatically three times?
I have a little module for that:
# in lib/retryable.rb
module Retryable
# Options:
# * :tries - Number of tries to perform. Defaults to 1. If you want to retry once you must set tries to 2.
# * :on - The Exception on which a retry will be performed. Defaults to Exception, which retries on any Exception.
# * :log - The log level to log the exception. Defaults to nil.
#
# If you work with something like ActiveRecord#find_or_create_by_foo, remember to put that call in a uncached { } block. That
# forces subsequent finds to hit the database again.
#
# Example
# =======
# retryable(:tries => 2, :on => OpenURI::HTTPError) do
# # your code here
# end
#
def retryable(options = {}, &block)
opts = { :tries => 1, :on => Exception }.merge(options)
retry_exception, retries = opts[:on], opts[:tries]
begin
return yield
rescue retry_exception => e
logger.send(opts[:log], e.message) if opts[:log]
retry if (retries -= 1) > 0
end
yield
end
end
and than in your model:
extend Retryable
def new
retryable(:tries => 3, :on => Timeout::Error, :log =>:error) do
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
end
...
end
You could do something like this:
module Foo
def self.retryable(options = {})
retry_times = options[:times] || 10
try_exception = options[:on] || Exception
yield if block_given?
rescue *try_exception => e
retry if (retry_times -= 1) > 0
raise e
end
end
Foo.retryable(on: Timeout::Error, times: 5) do
# your code here
end
You can even pass multiple exceptions to "catch":
Foo.retryable(on: [Timeout::Error, StandardError]) do
# your code here
end
I think what you need is the retryable gem.
With the gem, you can write your method like below
def new
retryable :on => Timeout::Error, :times => 3 do
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
end
end
Please read the documentation for more information on how to use the gem and the other options it provides
you could just write a helper-method for that:
class TimeoutHelper
def call_and_retry(tries=3)
yield
rescue Timeout::Error => e
tries -= 1
retry if tries > 0
Rails.logger.error e.message
end
end
(completely untested) and call it via
TimeoutHelper.call_and_retry { [your code] }
I'm using Timeout::timeout(1) for a process that takes longer than 1 second, though it only occasionally triggers a timeout. When it does, rescue captures it in different ways each time. Here's a sample of my code:
require 'timeout'
...
begin
status = Timeout::timeout(1) {
open(file_url) do |foo|
feed = RSS::Parser.parse(foo)
some_method_call(arg1, arg2)
#other stuff
end
}
rescue Timeout::Error
Rails.logger.debug "Timeout"
return nil
rescue Exception => ex
Rails.logger.debug "EXCEPTION - #{ex.message}"
return nil
end
Here are the three scenarios I encounter with the same input:
Process runs to completion and takes longer than 60 seconds
Process times out and hangs, printing only execution expired in development.log
Process times out, is rescued properly, prints "Timeout" in development.log, and returns nil
Why is this so inconsistent?
UPDATE
After reducing the timeout to 0.0001s, the process is timing out consistently and as expected. It seems that the open(file_url) block was opening faster than 1 second, and despite everything within the block taking more than 1 second, the Timeout was only triggered if the opening itself took longer than 1 second.
This however did not explain the execution expired exception. To test this, I moved the Timeout::timeout(0.0001) to within the open block. The code looks like the following:
require 'timeout'
...
begin
open(file_url) do |foo|
status = Timeout::timeout(0.0001) do
begin
feed = RSS::Parser.parse(foo)
some_method_call(arg1, arg2)
#other stuff
rescue Timeout::Error
Rails.logger.debug "Timeout 2"
rescue Exception => ex
Rails.logger.debug "EXCEPTION 2 - #{ex.message}"
end
end
end
rescue Timeout::Error
Rails.logger.debug "Timeout"
return nil
rescue Exception => ex
Rails.logger.debug "EXCEPTION - #{ex.message}"
return nil
end
Now, I'm consistently receiving the output EXCEPTION 2 - execution expired. Why is it that the Timeout::Error is not being triggered here?
Your inner
rescue Exception => ex
Rails.logger.debug "EXCEPTION 2 - #{ex.message}"
end
keeps the outer timeoutblock from raising Timeout::Error.
Removing that rescuestatement should do the trick.
If you really need to catch any exception, replace it with:
rescue StandardError => ex
Rails.logger.debug "EXCEPTION 2 - #{ex.message}"
end
Internally (within the Timeout block) Timeout does not use Timeout::Error. If it did, then every garden-variety rescue would catch it, and you don't want that. So it creates a new Exception and uses that, so that it hopefully blows through all normal error handling and actually makes the code stop running.
Check out the timeout.rb code in ruby200/lib/ruby/2.0.0. It's quite short, and pretty informative.
In particular, you can pass your own Exception in as the second parameter of Timeout::timeout, and Timeout will use that. So you can catch it inside your code, if you want.
Note that Logger currently traps all exceptions that happen while writing, and doesn't re-raise, so it breaks Timeout. I've filed a bug report.
I have a Rufus "every job" that runs periodically, but sometimes it may fail to perform it's task.
On failure, I would like to reschedule a retry soon rather than wait until the next cycle.
class PollProducts
def initialize()
end
def call(job)
puts "Updating products"
begin
# Do something that might fail
raise if rand(3) == 1
rescue Exception => e
puts "Request failed - recheduling: #{e}"
# job.in("5s") <-- What can I do?
end
end
end
scheduler.every '6h', PollProducts.new, :first_in => '5s', :blocking => true
Is this possible?
Ok this worked for me:
job.scheduler.in '5s', self