Heroku, Rails, Resque, Redis: All jobs failing because of Errno::ETIMEDOUT - ruby-on-rails

Question

I have a rails app on heroku that uses Resque.enqueu to schedule background jobs using Redis and redistogo.
Jobs used to work normally but I notice now that they have all been failing for some time (a few weeks, 100% of jobs).
The error I get:
Exception Errno::ETIMEDOUT Error Connection timed out - connect(2)
And the stack trace:
> /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `initialize'
> /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `open'
> /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `block in
> connect' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/timeout.rb:52:in
> `timeout' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:877:in
> `connect' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:862:in
> `do_start' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:851:in
> `start' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1373:in
> `request' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1131:in
> `get'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:121:in `block in request'
> /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.1/lib/active_support/notifications.rb:164:in
> `block in instrument'
> /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.1/lib/active_support/notifications/instrumenter.rb:20:in
> `instrument'
> /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.1/lib/active_support/notifications.rb:164:in
> `instrument'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:118:in `request'
> /app/vendor/bundle/ruby/2.0.0/gems/shopify_api-4.0.6/lib/active_resource/connection_ext.rb:13:in `request_with_detailed_log_subscriber'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:82:in
> `block in get'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:216:in `with_auth'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:82:in
> `get'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/custom_methods.rb:57:in
> `get'
> /app/vendor/bundle/ruby/2.0.0/gems/shopify_api-4.0.6/lib/shopify_api/countable.rb:4:in
> `count' /app/app/models/shop.rb:263:in `unscanned_customers'
> /app/app/models/shop.rb:230:in `shopify_customers'
> /app/app/models/shop.rb:144:in `bulk_scan'
> /app/app/workers/bulk_scanner.rb:16:in `perform'
I was thinking maybe it is something to do with not connecting to Redis or redistogo properly. Here are the relevant initializer code that I have:
resque.rb:
Resque.redis = RedisConnection.connection
redis_connection.rb:
class RedisConnection
def self.close
connection.quit
end
def self.connection
#connection ||= new_connection
end
def self.new_connection
uri = URI.parse(ENV['REDISTOGO_URL'] || 'redis://localhost:6379/')
Redis.new(host: uri.host,
port: uri.port,
password: uri.password,
thread_safe: true,
:timeout => 1)
end
end
If I call Resque.redis or #connection in my app, it returns a Redis instance. What else would be causing this connection error? Where can I troubleshoot?

Answer 1

Looking at the Backtrace, could it be a Shopify API connection timeout?
It looks like unscanned_customers is a supposed to be a countable Shopify collection and the problem occurs when an attempt to retrieve the customer is made.
I would start my debugging here:
> 'count' /app/app/models/shop.rb:263:in
'unscanned_customers'
Additionally, I would attempt to connect to Shopify from the server with a private app API key and password:
shop_url = "https://#{API_KEY}:#{PASSWORD}#SHOP_NAME.myshopify.com/admin"
ShopifyAPI::Base.site = shop_url
Refer to the Shopify API for more details if needed Shopify API Readme
It will help isolate the connection issue.
Cheers.