Redis counter consistently being off - ruby-on-rails

Question

This is a long question, so apologies in advance. We're using Redis to track customer states for each company. The customer states are tracked because we have customers separated by states in tabs on our website.
Whenever a customer instance is saved (after_save), we look to see what has changed and increment/decrement the relevant counts. For some reason, the counts are off pretty frequently and we can't figure out why. We wrote tests and verified all logic looks correct.
One thing to note is that we use sidekiq for background processing quite often, not sure if that will affect it or not.
We have 3 counts that we're calculating: my_customers_length (customers that have a user_id, which means they are 'assigned' to that user), all_customers_length (customers that are either 'open' or 'assigned', open means aasm_state = 'open' and they don't have a user_id), and open_customers_length (customers that aasm_state = 'open' and don't have a user_id).
Here's the logic that increments/decrements the Redis counters (after_save in customer.rb):
def reset_stats
if user_id_changed?
if user_id_was == nil # open => assigned
# update company and user
user.redis_increment_my_customers_length
company.redis_decrement_open_customers_length
elsif user_id_was != nil && user_id != nil # assigned => assigned
# update users (assigner and assignee)
user_was = User.find(user_id_was)
user.redis_increment_my_customers_length
user_was.redis_decrement_my_customers_length
elsif user_id_was != nil && user_id == nil # assigned => closed
# update company and user
user_was = User.find(user_id_was)
user_was.redis_decrement_my_customers_length
company.redis_decrement_all_customers_length
end
else
if aasm_state_was == 'closed' && aasm_state == 'open' # closed => open
# update company
company.redis_increment_all_customers_length
company.redis_increment_open_customers_length
elsif aasm_state_was == 'open' && aasm_state == 'closed' # open => closed
# update company
company.redis_decrement_all_customers_length
company.redis_decrement_open_customers_length
end
end
end
and here are the redis functions:
user.rb:
def redis_length_key
"my_customers_length_for_#{id}"
end
def set_my_customers_length(l)
RED.set(redis_length_key, l)
l
end
def redis_increment_my_customers_length
set_my_customers_length(my_customers_length.to_i + 1)
end
def redis_decrement_my_customers_length
set_my_customers_length(my_customers_length.to_i - 1)
end
and company.rb:
def open_customers
customers.open
end
def redis_open_length_key
"open_customers_length_for_#{id}"
end
def set_open_customers_length(l)
RED.set(redis_open_length_key, l)
l
end
def redis_increment_open_customers_length
set_open_customers_length(open_customers_length.to_i + 1)
end
def redis_decrement_open_customers_length
set_open_customers_length(open_customers_length.to_i - 1)
end
def open_customers_length(reset = false)
l = RED.get(redis_open_length_key)
if l.present? && reset == false && l.to_i >=0
l
else
set_open_customers_length(open_customers.length)
end
end
def redis_all_length_key
"all_customers_length_for_#{id}"
end
def set_all_customers_length(l)
RED.set(redis_all_length_key, l)
l
end
def all_customers_length
RED.get(redis_all_length_key)
end
def redis_increment_all_customers_length
set_all_customers_length(all_customers_length.to_i + 1)
end
def redis_decrement_all_customers_length
set_all_customers_length(open_or_claimed_customers_length.to_i - 1)
end
def open_or_claimed_customers_length(reset = false)
l = RED.get(redis_all_length_key)
if l.present? && reset == false && l.to_i >=0
l
else
set_all_customers_length(open_or_claimed_customers.length)
end
end
def open_or_claimed_customers
customers.open_or_claimed
end
We noticed that the "open" customers count in Redis is always less than the actual count by hitting the database. For "my customers count" and "all customers count," Redis is usually lower but not always.
Is our logic wrong? Are we missing something? Could this be a Redis problem? Sidekiq problem?

Answer 1

Your Redis logic is not atomic, it's got a giant race condition in redis_increment_open_customers_length. Use the INCR command instead of GET + SET.
http://redis.io/commands/incr