We have a Rails 6.1 app and using Sidekiq for bacground jobs. Sometimes it happens that the sidekiq web UI resets to the initial state, showing the completed/failed job counters near zero, indicating a recent Redis reset. What could be causing this?
This was being caused by Rails using the same Redis as cache store and someone just ran Rails.cache.clear which in turns calls Redis command FLUSHDB which clears the whole DB.
There are two possible solutions:
use separate Redis instance for Rails cache and Sidekiq workers (IMO overkill for most cases)
use separate Redis DB for Rails.cache and Sidekiq. This still means that you need separare REDIS_URL config for Rails.cache and Sidekiq but you can just use different Redis DB (by default redis supports up to 16 separate DBs). To do this set
REIDS_URL_RAILS=redis://host:10000/0
REIDS_URL_SIDEKIQ=redis://host:10000/1
And adjust your configs accordingly. The number at the end of the url is the ID of the database to use (by default DB=0 is used).
Related
I am using SideKiq with Heroku Redis on a Rails 5 app. Over time, my memory utilization has grown linearly, causing me to have to upgrade my plan every month or so. My load has been relatively flat.
I have an Eviction Policy of noeviction, as recommended in the SideKiq wiki. Does that mean that my memory footprint will continue to grow unabated over time?
You should have 2 Redis instances running, one for Caching and another dedicated to Sidekiq.
Here's an entry from the Sidekiq wiki explaining this:
Many people use Redis as a cache (it works great as a Rails cache store) but it's important that Sidekiq be run against a Redis instance that is not configured as a cache but as a persistent store. I recommend using two separate Redis instances, each configured appropriately, if you wish to use Redis for caching and Sidekiq. Redis namespaces do not allow for this configuration and come with many other problems, so using discrete Redis instances is always preferred.
That way, the cache store Redis will be able to manage evictions and memory as needed, whilst the Sidekiq Redis will be able to keep jobs persistently without a risk of an eviction. As you said the Sidekiq Redis instance should have noeviction set.
Just a guess, but likely the reason you had a linear increase in memory is because in other places of your Rails app you'd use Redis and that data would not be cleared due to the eviction policy, perhaps this was even Rails under the hood doing this, although I believe Rails would set a TTL everywhere, but I'm not certain.
noeviction is a eviction policy setting in Redis. It results in Redis returning errors if the maximal memory usage is reached.
To limit the memory usage in Redis you can set in the redis.conf file and set the maxmemory setting.
Read here for more information about it.
Rails 5.0.1, Ruby 2.4.0, Sidekiq 4.2.9
I need count some specific data in background jobs. I implemented it already through Postgres, but I faced with problem: Sidekiq concurrency very loads DB connections and if I decrease concurrency number, jobs runing take a lot time.
I found that it's possible to use atomic counter and in some period save result to DB.
So can I share variable between threads in Sidekiq? If it is, how I should initialize shared variable?
Thanks for any advise
If you share a variable between threads, you need to worry about locking it with a Mutex and it only scales to a single process.
Instead, use Redis commands to increment counters.
https://redis.io/commands/incr
So, recently I moved all the session-related information in my app to the redis. Everything is running fine and now I am not facing the cookie-related issues (especially from IE).
In doing that, I read some blogs and all of them defined a redis-connector as a global variable in the config like
$redis = Redis.new(:host => 'localhost', :port => 6379)
Now there are a few things that bugging me:
Defining a global resource means that I have just a single connection to the redis. Will it create a bottleneck in my system when I have to serve multiple requests?
Also when multiple request arrives, will the Rails enqueue the requests for the redis as the connection is global resource, in case it is already in use?
Redis supports multiple instances. Wouldn't creating multiple instances boost the performance?
There are no standard connections pools included into Redis gem. If we consider Rails as a single threaded execution model it doesn't sound too problematic.
It might be evil when used in multi-threaded environment (think of background jobs as an example). So connection pooling is a good idea in general.
You can implement it for Redis using connection_pool gem.
Sidekiq also uses this gem for connecting to Redis. It can be seen here and here. Also, sidekiq author is the same person as connection_pool author, https://github.com/mperham.
As to your questions:
Multiple requests still don't mean multi-threading, so this approach might work well before you use threads;
Rails is not going to play the role of connection pool for your database;
It will boost performance (and avoid certain errors) if used in multi-threaded environment.
1) No it's not a bottleneck, opening TCP for Redis for every query/request cause leak of perfomance.
3) Yes if you have more then one core/thread.
Simply measure Redis connection number to see there is no new connection instantiated before each Rails request processed. The connection established on rails processor (Unicorn, Puma, Passenger etc) side during application load process.
echo info | redis-cli | grep connected_clients
Try to run the bash command before and during your application is being run locally.
I've looked around at other similar questions on SO but can't quite piece things together well enough. I have a Rails app (on Heroku) that uses Puma with both multiple processes and multiple threads. My app also uses Redis as a secondary data store (in addition to a SQL database), querying Redis directly (well, through the connection_pool gem). Here's my Puma config file:
workers Integer(ENV["WEB_CONCURRENCY"] || 4)
threads_count = Integer(ENV["MAX_THREADS"] || 5)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV["PORT"] || 3000
environment ENV["RACK_ENV"] || "development"
on_worker_boot do
# Worker specific setup for Rails 4.1+
ActiveRecord::Base.establish_connection
redis_connections_per_process = Integer(ENV["REDIS_CONNS_PER_PROCESS"] || 5)
$redis = ConnectionPool.new(size: redis_connections_per_process) do
Redis.new(url: ENV["REDIS_URL"] || "redis://localhost:6379/0")
end
end
My Redis instance has a connection limit of 20, and I find myself regularly going over this limit, despite having what should be (as far as I can tell) only 5 connections per process spread across 4 worker processes.
In fact, I even get max number of clients reached Redis errors when I set REDIS_CONNS_PER_PROCESS to 1. Is on_worker_boot called for each thread rather than each process?
I've also tried having a separate redis.rb initializer, which still gives me errors even when REDIS_CONNS_PER_PROCESS is 1. This seems odd since I should be able to have it up to 4 if I'm doing my math correctly (4 worker processes + 1 master process) * 4 connections per process. (Note that for the purposes of this question I'm ignoring errors that occur around deploying, since I'm assuming Heroku might be connecting both old and new processes during that process, even though I'm not using Preboot.)
Where am I misunderstanding how this all fits together?
I had similar problem. At first I was using redis-togo, and it has no problem. but After I changed from redis-togo to Heroku redis, I got "ERR max number of clients reached" erros.
My app's code is not changed, redis provider's changing was the only one.
I opened a ticket at Heroku support, and they advised me to change the default setting of timeout value.
https://devcenter.heroku.com/articles/heroku-redis#configuring-your-instance
after I changed the default timeout value of Heroku redis, everyting was solved.
I guess the default value of redis timeout is different by redis providers. and Heroku redis's default setting is 0.
"A value of zero means that connections will not be closed."
I wish my experience is helpful.
After further reading and testing, I ended up moving my Redis connection pool code into a separate initializer. Unfortunately, this didn't solve my problem at all—despite lots of tinkering with process and connection numbers, I was still getting max number of clients reached errors way before I should have been.
The answer, it turns out, was to switch Redis providers from Heroku Redis to Redis Cloud. I'm not sure why Heroku Redis wasn't allowing the number of connections it advertises, but upon some investigation Redis Cloud actually appears to allow more connections than advertised (or at least limit connections transparently and without errors) without any issues whatsoever. Wow. They've certainly earned my business.
I ran into this problem also, and while the Heroku Redis dashboard showed only a few connections, I was running out of connections.
Then I contacted Heroku support, and they told me that the dashboard only shows active clients/connections, and not the idle ones.
So because of the Redis timeout being 0 (never timeout), on reboot the Redis connections idle and new ones are opened. So the situation gets worse on every reboot.
A solution, as mentioned by others on this page, is to set the timeout to something else than 0:
heroku redis:timeout -s 10 -a APPLICATION_NAME
This makes the connections die after 10 seconds, which shouldn't be a problem because while it's being used it will stay open (no unnecessary closes).
When you have only little traffic, you might consider setting this to something a bit higher.
I am running a rails app and using redis for jbulder's cache and sidekiq queue. I use sidekiq to send emails asyncly, everytime when I try to send mass emails, say 20k emails in background using sidekiq, after a while, all the background jobs in sidekiq queue are cleared, left 0 jobs in queue.
I filed an issue on sidekiq github page(link), the author said it could be something or someone flushing my redis. There's no one flush redis manually and I wonder how can I find when and how redis gets flushed.
I've checked redis log file with nothing strange.
Here is the documentation on changing certain commands. Perhaps consider changing flushAll and flushDB to something abnormal.