Redis::TimeoutError while using redis-rails on heroku - ruby-on-rails

Question

Updated to include Redis/Resque versions and stack trace (below):
redis (3.0.4)
redis-namespace (1.3.0)
redis (~> 3.0.0)
redis-store (1.1.2)
redis (>= 2.2.0)
resque (1.24.1)
mono_logger (~> 1.0)
multi_json (~> 1.0)
redis-namespace (~> 1.2)
sinatra (>= 0.9.2)
vegas (~> 0.1.2)
resque-scheduler (2.0.1)
redis (>= 2.0.1)
resque (>= 1.20.0)
rufus-scheduler
I'm seeing intermittent Redis::TimeoutError: Connection timed out on heroku while writing a moderately sized array (~200 Fixnums) to the Redis store using the Rails.cache.fetch command.
I'm also using Resque.
I see here that the Redis::Client can receive a timeout option, but I don't see where to pass initialization options to Redis.
I'm using the standard heroku resque.rb:
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
ENV['REDIS_URI'] = resque_config[rails_env]
Resque.redis = resque_config[rails_env]
Resque.inline = rails_env == 'test'
require 'resque_scheduler'
require 'resque/scheduler'
require 'resque_scheduler/server'
Resque.schedule = YAML.load_file(rails_root + '/config/resque-schedule.yml')
Resque.before_fork do
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
end
Resque.after_fork do
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end
I assume that a Redis client is instantiated here. Is this a different client from the one instantiated in production.rb:
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
config.cache_store = :redis_store, resque_config[rails_env], { expires_in: 14.days }
The options hash there is for Rails.cache options as far as I know. Is a new client instantiated here? How can I pass options to this one?
Updated to include this experiment in the heroku console implying that they are different client instances:
irb(main):002:0> Rails.cache
=> #<ActiveSupport::Cache::RedisStore:0x00000003860e18 #data=#<Redis client v3.0.4 for redis://spinyfin.redistogo.com:9485/0>, #options={:expires_in=>14 days}>
irb(main):003:0> Resque.redis.redis
=> #<Redis client v3.0.4 for redis://spinyfin.redistogo.com:9485/0>
irb(main):004:0> Rails.cache.instance_variable_get(:#data).object_id == Resque.redis.redis.object_id
=> false
Stack trace:
Redis::TimeoutError: Connection timed out
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:208:in `rescue in io'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:206:in `io'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:214:in `read'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:84:in `block in call'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:187:in `block (2 levels) in process'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:295:in `ensure_connected'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:177:in `block in process'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:256:in `logging'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:176:in `process'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:84:in `call'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:644:in `block in setex'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `block in synchronize'
from /app/vendor/ruby-1.9.3/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `synchronize'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-3.0.4/lib/redis.rb:643:in `setex'
from /app/vendor/bundle/ruby/1.9.1/gems/redis-store-1.1.2/lib/redis/store/interface.rb:17:in `setex'
... 11 levels...
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/cache.rb:299:in `fetch'
...SNIP...
...my code...
...SNIP...
from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.13/lib/active_record/relation/delegation.rb:6:in `each'
from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.13/lib/active_record/relation/delegation.rb:6:in `each'
...SNIP...
...my code...
...SNIP...
from (irb):5
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'irb(main):006:0> ! Heroku client internal error.
! Search for help at: https://help.heroku.com
! Or report a bug at: https://github.com/heroku/heroku/issues/new
Error: Operation timed out (Errno::ETIMEDOUT)
Backtrace: /usr/local/heroku/ruby/lib/ruby/1.9.1/openssl/buffering.rb:121:in `sysread'
/usr/local/heroku/ruby/lib/ruby/1.9.1/openssl/buffering.rb:121:in `readpartial'
/Users/me/.heroku/client/lib/heroku/client/rendezvous.rb:69:in `block in start'
/Users/me/.heroku/client/lib/heroku/client/rendezvous.rb:53:in `loop'
/Users/me/.heroku/client/lib/heroku/client/rendezvous.rb:53:in `start'
/Users/me/.heroku/client/lib/heroku/command/run.rb:132:in `rendezvous_session'
/Users/me/.heroku/client/lib/heroku/command/run.rb:119:in `run_attached'
/Users/me/.heroku/client/lib/heroku/command/run.rb:24:in `index'
/Users/me/.heroku/client/lib/heroku/command.rb:206:in `run'
/Users/me/.heroku/client/lib/heroku/cli.rb:28:in `start'
/usr/local/heroku/bin/heroku:24:in `<main>'
Command: heroku run rails c
Plugins: heroku-redis-cli
Version: heroku-toolbelt/2.39.4 (x86_64-darwin10.8.0) ruby/1.9.3

Answer 1

The "Connection timed out" message means that redis-rb is having trouble initiating a connection to your Redis server. Normally, you'd want to just initialize your Redis connection once on app boot. However, because Resque forks (rather than using threads like Sidekiq), it has to initialize a fresh Redis connection for every individual job.
Normally, that's not a problem, but Heroku has intermittent issues with creating new Redis connections. I've seen this issue across languages / client libraries / Redis hosts, and beyond reducing the number of Redis connections you create, the only way to mitigate the issue is to automatically retry connecting to Redis a few times in your Resque.after_fork block. (Eg. catch the timeout error and either retry or re-raise the exception if you've tried 3 times)

Answer 2

This isn't much of an answer, but at the recommendation of a colleague, I switched from RedisToGo to openredis and these problems went away immediately.

Answer 3

You can't pass both a URL and an options hash to config.cache_store, because of this bug:
config.cache_store = :redis_store, resque_config[rails_env], { expires_in: 14.days }
If you pass a URL and an options hash, it ignores the URL and defaults to localhost, which is probably why you're seeing a timeout (I just had the same issue).
Instead, do something like this:
redis_uri = URI.parse(ENV["REDISTOGO_URL"])
config.cache_store = :redis_store, {
host: redis_uri.host,
port: redis_uri.port,
password: redis_uri.password,
namespace: "cache",
expires_in: 7.days
}