Reconnect with Redis after Puma fork - ruby-on-rails

Question

I am using a global variable in a rails application to store a redis client using the redis gem. In a config/initializers/redis.rb, I have
$redis = Redis.new(host: "localhost", port: 6379)
Then in application code, I use $redis to work the data in the Redis store.
I also use puma as the web server in production environment, and capistrano to deploy code. In the deploy process, capistrano restarts puma.
Every time I start or restart the puma web servers, I always get an "Internal Server Error" when I first use $redis to access data in the Redis store. I saw errors like Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.)
Searching around with google and stackoverflow led me to think that I needed to reconnect to Redis after puma forks child processes. So, I added in my config/puma.rb:
on_worker_boot do
$redis.ping
end
But I was still getting the "Internal Server Error" caused by Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.).
I saw this post http://qiita.com/yaotti/items/18433802bf1720fc0c53. I then tried adding in config/puma.rb:
on_restart do
$redis.quit
end
That did not work.
I tried in config/initializers/redis.rb to $redis.ping right after Redis.new. That did not work either.
I got this error if puma was started with no puma processes running, or restarted when an instance of puma process was running.
Refreshing the page would get me past this error. But I want to get rid of this even on the first attempt to use $redis. I was thinking that I did not use the redis gem or configure its reconnection correctly. Could someone tell me:
Is that the right way to use redis gem in a rails application?
How should the redis connection be reconnected in puma?
puma gem documentation says, "You should place code to close global log files, redis connections, etc in this block so that their file descriptors don't leak into the restarted process. Failure to do so will result in slowly running out of descriptors and eventually obscure crashes as the server is restart many times." It was talking about the on_restart block. But it did not say how that should be done.

Answer 1

I was able to fix the error with a monkeypatch. This changes the behaviour so it just reconnects instead of throwing the Redis::InheritedError
###### MONKEYPATCH redis-rb
# https://github.com/redis/redis-rb/issues/364
# taken from https://github.com/redis/redis-rb/pull/389/files#diff-597c124889a64c18744b52ef9687c572R314
class Redis
class Client
def ensure_connected
tries = 0
begin
if connected?
if Process.pid != #pid
reconnect
end
else
connect
end
tries += 1
yield
rescue ConnectionError
disconnect
if tries < 2 && #reconnect
retry
else
raise
end
rescue Exception
disconnect
raise
end
end
end
end
## MONKEYPATCH end

Answer 2

I'm running a Rails Application with IdentityCache using Puma in clustered mode with workers=4.
It is essential that the reconnects happen in the on_worker_boot callback.
I have to reconnect both the Rails.cache and the IdentityCache to avoid restart errors. Here's what I got working:
puma-config.rb
on_worker_boot do
puts 'On worker boot...'
puts "Reconnecting Rails.cache"
Rails.cache.reconnect
begin
puts "Reconnecting IdentityCache"
IdentityCache.cache.cache_backend.reconnect
rescue Exception => e
puts "Error trying to reconnect identity_cache_store: #{e.message}"
end
end
Then I see the following in my logs, showing me the proof that it all works.
On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
On worker boot...
Reconnecting Rails.cache
Reconnecting IdentityCache
[7109] - Worker 7115 booted, phase: 0
[7109] - Worker 7123 booted, phase: 0
[7109] - Worker 7119 booted, phase: 0
[7109] - Worker 7127 booted, phase: 0
Sure enough, the first request problems that used to be there after server restart are gone. QED.

Answer 3

Upgrade redis-rb to 3.1.0 or above.
The detail https://github.com/redis/redis-rb/pull/414/files#
monkey patch
# https://github.com/redis/redis-rb/pull/414/files#diff-5bc007010e6c2e0aa70b64d6f87985c20986ee1b2882b63a89b52659ee9c91f8
class Redis
class Client
def ensure_connected
tries = 0
begin
if connected?
if Process.pid != #pid
raise InheritedError,
"Tried to use a connection from a child process without reconnecting. " +
"You need to reconnect to Redis after forking."
end
else
connect
end
tries += 1
yield
rescue ConnectionError, InheritedError
disconnect
if tries < 2 && #reconnect
retry
else
raise
end
rescue Exception
disconnect
raise
end
end
end
end

Answer 4

Here's what I did:
Redis.current.client.reconnect
$redis = Redis.current
($redis is my global instance of a redis client)

Answer 5

I've put this into my config/puma.rb file, works for me.
on_restart do
$redis = DiscourseRedis.new
Discourse::Application.config.cache_store.reconnect
end
on_worker_boot do
$redis = DiscourseRedis.new
Discourse::Application.config.cache_store.reconnect
end