Errno::ETIMEDOUT: Connection timed out - connect(2) - ruby-on-rails

Question

In production, I'm occasionally getting the following error:
Errno::ETIMEDOUT: Connection timed out - connect(2)
It only seems to be happening when I generate a PDF using the prawn gem that contains an image that was uploaded by paperclip/aws-sdk to s3. This probably only happens a couple times a week on an action that is used hundreds of times a day (with no issues), but it results in a 500 error when it does fail.
The trace is:
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`initialize'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`open'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`connect'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/timeout.rb:53:in
`timeout'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/timeout.rb:101:in
`timeout'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:560:in
`connect'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:553:in
`do_start'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:542:in
`start'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:242:in
`open_http'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:616:in
`buffer_open'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:164:in
`open_loop'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:162:in
`catch'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:162:in
`open_loop'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:132:in
`open_uri'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:518:in
`open'
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/open-uri.rb:30:in
`open'
....rb:57:in `render_image_to_pdf'
Line 57:
pdf.image open(image.expiring_url(30.minutes, :full)), :width => 300, :position => 20
Setup:
Rails 3.0.10
Ruby 1.8.7EE
Prawn 0.11.1
AWS-SDK 1.3.3
Paperclip 2.5.2
What can I do to prevent this error?

Answer 1

One option is to catch the exception and try again:
begin
pdf.image open(image.expiring_url(30.minutes, :full)), :width => 300, :position => 20
rescue Errno::ETIMEDOUT
# try one more time, or use retry with a counter to attempt a limited number of times
pdf.image open(image.expiring_url(30.minutes, :full)), :width => 300, :position => 20
end
You may also want to put some diagnostics (either logging or generating an Airbrake) that notifies you so you can see if there is some sort of pattern to the failures.

Answer 2

I have been getting the same error in Amazon EC2 when using aws-sdk to spawn new server instances. Here's what I know:
I have a function (sleep_while_not_running) which tests the instance.status:
def sleep_while_not_running(instance)
print "waiting on system: "
$stdout.flush
while instance.status != :running
print "."
$stdout.flush
sleep 1
end
puts "resuming."
end
After sleep_while_not_running returns, I assume that we should be able to connect (as follows):
Net::SSH.start(instance.ip_address,'root',:key_data=>[key_pair.private_key]) do |ssh|
sleep_while_not_running(instance)
puts "testing SSH connectivity."
puts ssh.exec("uname -a")
puts " "
What happens is the same time-out error you describe. Investigating manually, I have found that--
a. If I wait until "Status Checks" (in the management console) shows that the new instance has "Passed 2/2 Checks", then I can SSH in manually just fine using the same key pair used by the script.
b. sleep_while_not_running checks instance.status (which says "running" long before the server is fully baked.
c. I can find no call in aws-sdk to check the "status checks" I would check in the management console, but I suspect that is where the solution exists.
P.S. I realize this isn't an "answer" but it may help identify a solution.