Hey I am attempting to spawn a sidekiq worker that connects to a completely separate Redis database. I know with 3.0's connection pooling this is possible, and I have been able to successfully push a job onto the correct Redis DB, but the problem is the Sidekiq web UI is not showing these jobs in the queue (I have mounted a separate Rack app for this that points exclusively to the other Redis DB). The "Busy" tab in the admin interface also shows my sidekiq workers that I have pointed at this DB, with correct PIDs.
Here's my sidekiq.rb:
Sidekiq.configure_server do |config|
if ENV['REDIS_DB'] == "2"
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379/2", :namespace => 'drip' }
else
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379", :namespace => 'drip' }
end
end
Sidekiq.configure_client do |config|
if ENV['REDIS_DB'] == "2"
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379/2", :namespace => 'drip' }
else
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379", :namespace => 'drip' }
end
end
My use case is that I need to have fine grain control over the jobs that go into the second database, so I need the workers configured precisely so they are only using as many resources as I need them to. I only want the workers that are configured in this way to pick up these jobs.
The Web UI cannot be mounted multiple times to point to multiple Redises in a single process. You have to run multiple web processes with REDIS_DB set too.
Related
We've provisioned an Azure Redis Cache server using the Premium tier. The server is clustered, with 2 shards, and the server is configured to allow public access over the public internet through a firewall--allow-listing a set of known IPs.
Deploys of our Rails application two any of these known IPs fail with the error:
Redis::CannotConnectError: Redis client could not connect to any cluster nodes
Here is our Rails config:
# application.rb
if Rails.env.test?
config.cache_store = :redis_cache_store, {
url: config.nines[:redis_url],
expires_in: 90.minutes,
namespace: ENV['TEST_ENV_NUMBER'],
}
else
config.cache_store = :redis_store, {
namespace:ENV['TEST_ENV_NUMBER'],
cluster: [ Rails.application.config.nines[:redis_url] ],
replica: true, # allow reads from replicas
compress: true,
compress_threshold: 1024,
expires_in: 90.minutes
}
end
The config.nines[:redis_url] is set like this: rediss://:<pw>#<cache-name>.redis.cache.windows.net:6380/0
Then, we initialize the Redis connection in code like this:
if Rails.env.test?
::Redis.new :url => redis_url, :db => ENV['REDIS_DB']
else
::Redis.new(cluster: [ "#{redis_url}" ], db: ENV['REDIS_DB'])
end
We're using the redis-rb gem and redis-rails gem.
If anyone can point out what we're doing wrong, please do share!
Thank you User Peter Pan - Stack Overflow. Posting your suggestion as answer to help other community members.
You can try below code:
# Import the redis library for Ruby
require "redis"
# Create a redis client instance for connecting Azure Redis Cache
# At here, for enabling SSL, set the `:ssl` symbol with the
# symbol value `:true`, see https://github.com/redis/redis-rb#ssltls-support
redis = Redis.new(
:host => '<azure redis cache name>.redis.cache.windows.net',
:port => 6380,
:db => <the db index you selected like 10>,
:password => "<access key>",
:ssl => :true)
# Then, set key `foo` with value `bar` and return `OK`
status = redis.set('foo', 'bar')
puts status # => OK
# Get the value of key `foo`
foo = redis.get('foo')
puts foo # => bar
Reference: How to setup Azure Redis cache with Rails - Stack Overflow
I am using ruby 2.3.3 and Rails 4.2.8 with Puma (1 worker, 5 threads) and on my admin (i.e. not critical) page I want to show some stats (integer values) from my database. Some requests take quite a long time to perform so I decided to cache these values and use a rake task to re-write them every day.
Admin#index controller
require 'timeout'
begin
timeout(8) do
#products_a = Rails.cache.fetch('products_a', :expires_in => 24.hours) { Product.where(heavy_condition_a).size }
#products_b = Rails.cache.fetch('products_b', :expires_in => 24.hours) { Product.where(heavy_condition_b).size }
#products_c = Rails.cache.fetch('products_c', :expires_in => 24.hours) { Product.where(heavy_condition_c).size }
#products_d = Rails.cache.fetch('products_d', :expires_in => 24.hours) { Product.where(heavy_condition_d).size }
end
rescue Timeout::Error
#products_a = 999
#products_b = 999
#products_c = 999
#products_d = 999
end
Admin#index view
<li>Products A: <%= #products_a %></li>
<li>Products B: <%= #products_b %></li>
<li>Products C: <%= #products_c %></li>
<li>Products D: <%= #products_d %></li>
Rake task
task :set_admin_index_stats => :environment do
Rails.cache.write('products_a', Product.where(heavy_condition_a).size, :expires_in => 24.hours)
Rails.cache.write('products_b', Product.where(heavy_condition_b).size, :expires_in => 24.hours)
Rails.cache.write('products_c', Product.where(heavy_condition_c).size, :expires_in => 24.hours)
Rails.cache.write('products_d', Product.where(heavy_condition_d).size, :expires_in => 24.hours)
end
I am using this in production and use Memcachier (on Heroku) as a cache store. I also use it for page caching on the website and it works fine there. I have:
production.rb
config.cache_store = :dalli_store
The problem I am experiencing is that the cached values disappear almost instantly, and quite intermittently, from the cache. In the console I have tried:
I Rails.cache.write one value (e.g. product_a) and check it a minute later, it is still there. Although crude, I can see the "Set cmds" increments by one in Memcachier admin tool.
However, when I add the next value (e.g. product_b) the first one disappears (becomes nil)! Sometimes if I add all 4 values, 2 seems to stick. These are not always the same values. It is like whack-a-mole!
If I run the rake to write the values and then try to read the values typically only two values are left, whereas the others are lost.
I have seen a similar question related to this where the reason explained was the use of a multithread server. The cached value was saved in one thread and could not be reached in another, the solution was to use a memcache, which I do.
It is not only the console. If I just reload admin#index view to store the values or run the rake task, I experience the same problem. The values do not stick.
My suspicion is that I am either not using the Rails.cache-commands properly or that these commands do not in fact use Memcachier. I have not been able to determine whether my values are in fact stored in Memcachier but when I use my first command in the console I do get the following:
Rails.cache.read('products_a')
Dalli::Server#connect mc1.dev.eu.ec2.memcachier.com:11211
Dalli/SASL authenticating as abc123
Dalli/SASL authenticating as abc123
Dalli/SASL: abc123
Dalli/SASL: abc123
=> 0
but I do not get that for subsequent writes (which I assume is a matter of readability in the console and not a proof of Memcachier not being used.
What am I missing here? Why won't the values stick in my cache?
Heroku DevCenter states a little different cache config and gives some advice about threaded Rails app servers like Puma using connection_pool gem:
config.cache_store = :mem_cache_store,
(ENV["MEMCACHIER_SERVERS"] || "").split(","),
{ :username => ENV["MEMCACHIER_USERNAME"],
:password => ENV["MEMCACHIER_PASSWORD"],
:failover => true,
:socket_timeout => 1.5,
:socket_failure_delay => 0.2,
:down_retry_delay => 60,
:pool_size => 5
}
I planned to migrate redis server to a new one with sidekiq running, but doesn't want to stop the current application running. And I don't want to use redis cluster which is still alpha version.
My thought is to ask sidekiq to write to new redis server, but pulls from both of them, so that once the workers in old redis are completed, the new one can totally take over all workers. I think this solution is feasible, but I have no idea how to make it happen.
This is my sidekiq.rb:
sidkiq_config = YAML.load(ERB.new(Rails.root.join('config/redis.yml').read).result)
Sidekiq.configure_server do |config|
config.logger.level = Logger::ERROR
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
config.redis = { :url => "redis://redis.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
Sidekiq.configure_client do |config|
config.logger.level = Logger::ERROR
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
I believe the easiest solution is to run two instances of sidekiq - one that reads from the old cluster, and one that reads from the new cluster
sidkiq_config = YAML.load(ERB.new(Rails.root.join('config/redis.yml').read).result)
Sidekiq.configure_server do |config|
config.logger.level = Logger::ERROR
if ENV['read_from_new']
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
else
config.redis = { :url => "redis://redis.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
end
Sidekiq.configure_client do |config|
config.logger.level = Logger::ERROR
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
You can update to a recent version of sidekiq and use Sharding.
REDIS_A = ConnectionPool.new { Redis.new(...) }
REDIS_B = ConnectionPool.new { Redis.new(...) }
# To create a new connection pool for a namespaced Sidekiq worker:
ConnectionPool.new do
client = Redis.new(:url => "Your Redis Url")
Redis::Namespace.new("Your Namespace", :redis => client)
end
# Create a job in the default redis instance
SomeWorker.perform_async
# Push a job to REDIS_A using the low-level Client API
client = Sidekiq::Client.new(REDIS_A)
client.push(...)
client.push_bulk(...)
Sidekiq::Client.via(REDIS_B) do
# All jobs defined within this block will go to B
SomeWorker.perform_async
end
To quite sidekiq, click the quite button in the UI
So I have a rails app where I want my delayed job process to communicate with an SMPP server. But the problem occurs when I try to send the messages. My thread that I created in an initializer (delayed_job.rb):
if $0.ends_with?('/delayed_job')
require_relative '../../lib/gateway'
config = {
:host => 'SERVER.COM',
:port => 2345,
:system_id => 'USERNAME',
:password => 'PASSWORD',
:system_type => '', # default given according to SMPP 3.4 Spec
:interface_version => 52,
:source_ton => 0,
:source_npi => 1,
:destination_ton => 1,
:destination_npi => 1,
:source_address_range => '',
:destination_address_range => '',
:enquire_link_delay_secs => 60
}
Thread.new{
gw = Gateway.new
gw.start(config)
}
end
But checking my log file for the smpp server, it seems that the thread dies right after it starts. So I guess my question is how to persist the thread while the delayed_job daemon is running?
If I start my rails app in production and I try to send messages individually, it works without a problem, but because delayed_job is a separate process, I can't communicate with the the smpp thread in the rails app from my workers in the delayed_job queues.
Any ideas?
Sorted, decided to separate everything into their own daemons and each would communicate with the database independently as opposed to trying to work with pipes and signals.
Good afternoon,
I have two separate, but related apps. They should both have their own background queues (read: separate Sidekiq & Redis processes). However, I'd like to occasionally be able to push jobs onto app2's queue from app1.
From a simple queue/push perspective, it would be easy to do this if app1 did not have an existing Sidekiq/Redis stack:
# In a process, far far away
# Configure client
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end
# Push jobs without class definition
Sidekiq::Client.push('class' => 'Example::Workers::Trace', 'args' => ['hello!'])
# Push jobs overriding default's
Sidekiq::Client.push('queue' => 'example', 'retry' => 3, 'class' => 'Example::Workers::Trace', 'args' => ['hello!'])
However given that I would already have called a Sidekiq.configure_client and Sidekiq.configure_server from app1, there's probably a step in between here where something needs to happen.
Obviously I could just take the serialization and normalization code straight from inside Sidekiq and manually push onto app2's redis queue, but that seems like a brittle solution. I'd like to be able to use the Client.push functionality.
I suppose my ideal solution would be someting like:
SidekiqTWO.configure_client { remote connection..... }
SidekiqTWO::Client.push(job....)
Or even:
$redis_remote = remote_connection.....
Sidekiq::Client.push(job, $redis_remote)
Obviously a bit facetious, but that's my ideal use case.
Thanks!
So one thing is that According to the FAQ, "The Sidekiq message format is quite simple and stable: it's just a Hash in JSON format." Emphasis mine-- I don't think sending JSON to sidekiq is too brittle to do. Especially when you want fine-grained control around which Redis instance you send the jobs to, as in the OP's situation, I'd probably just write a little wrapper that would let me indicate a Redis instance along with the job being enqueued.
For Kevin Bedell's more general situation to round-robin jobs into Redis instances, I'd imagine you don't want to have the control of which Redis instance is used-- you just want to enqueue and have the distribution be managed automatically. It looks like only one person has requested this so far, and they came up with a solution that uses Redis::Distributed:
datastore_config = YAML.load(ERB.new(File.read(File.join(Rails.root, "config", "redis.yml"))).result)
datastore_config = datastore_config["defaults"].merge(datastore_config[::Rails.env])
if datastore_config[:host].is_a?(Array)
if datastore_config[:host].length == 1
datastore_config[:host] = datastore_config[:host].first
else
datastore_config = datastore_config[:host].map do |host|
host_has_port = host =~ /:\d+\z/
if host_has_port
"redis://#{host}/#{datastore_config[:db] || 0}"
else
"redis://#{host}:#{datastore_config[:port] || 6379}/#{datastore_config[:db] || 0}"
end
end
end
end
Sidekiq.configure_server do |config|
config.redis = ::ConnectionPool.new(:size => Sidekiq.options[:concurrency] + 2, :timeout => 2) do
redis = if datastore_config.is_a? Array
Redis::Distributed.new(datastore_config)
else
Redis.new(datastore_config)
end
Redis::Namespace.new('resque', :redis => redis)
end
end
Another thing to consider in your quest to get high-availability and fail-over is to get Sidekiq Pro which includes reliability features: "The Sidekiq Pro client can withstand transient Redis outages. It will enqueue jobs locally upon error and attempt to deliver those jobs once connectivity is restored." Since sidekiq is for background processes anyway, a short delay if a Redis instance goes down should not affect your application. If one of your two Redis instances goes down and you're using round robin, you've still lost some jobs unless you're using this feature.
As carols10cents says its pretty simple but as I always like to encapsulate the capability and be able to reuse it in other projects I updated an idea from a blog from Hotel Tonight. This following solution improves upon Hotel Tonight's that does not survive Rails 4.1 & Spring preloader.
Currently I make do with adding the following files to lib/remote_sidekiq/:
remote_sidekiq.rb
class RemoteSidekiq
class_attribute :redis_pool
end
remote_sidekiq_worker.rb
require 'sidekiq'
require 'sidekiq/client'
module RemoteSidekiqWorker
def client
pool = RemoteSidekiq.redis_pool || Thread.current[:sidekiq_via_pool] || Sidekiq.redis_pool
Sidekiq::Client.new(pool)
end
def push(worker_name, attrs = [], queue_name = "default")
client.push('args' => attrs, 'class' => worker_name, 'queue' => queue_name)
end
end
You need to create a initializer that sets redis_pool
config/initializers/remote_sidekiq.rb
url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'
redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))
RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }
EDIT by Aleks:
In never versions of sidekiq, instead of lines:
redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))
RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }
use lines:
redis_remote_options = {
namespace: "yournamespace",
url: ENV.fetch("REDISCLOUD_URL")
}
RemoteSidekiq.redis_pool = Sidekiq::RedisConnection.create(redis_remote_options)
You can then simply the include RemoteSidekiqWorker module wherever you want. Job done!
**** FOR MORE LARGER ENVIRONMENTS ****
Adding in RemoteWorker Models adds extra benefits:
You can reuse the RemoteWorkers everywhere including the system that has access to the target sidekiq workers. This is transparent to the caller. To use the "RemoteWorkers" form within the target sidekiq system simply do not use an initializer as it will default to using the local Sidekiq client.
Using RemoteWorkers ensure correct arguments are always sent in (the code = documentation)
Scaling up by creating more complicated Sidekiq architectures is transparent to the caller.
Here is an example RemoteWorker
class RemoteTraceWorker
include RemoteSidekiqWorker
include ActiveModel::Model
attr_accessor :message
validates :message, presence: true
def perform_async
if valid?
push(worker_name, worker_args)
else
raise ActiveModel::StrictValidationFailed, errors.full_messages
end
end
private
def worker_name
:TraceWorker.to_s
end
def worker_args
[message]
end
end
I came across this and ran into some issues because I'm using ActiveJob, which complicates how messages are read out of the queue.
Building on ARO's answer, you will still need the redis_pool setup:
remote_sidekiq.rb
class RemoteSidekiq
class_attribute :redis_pool
end
config/initializers/remote_sidekiq.rb
url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'
redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))
RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }
Now instead of the worker we'll create an ActiveJob Adapter to queue the request:
lib/active_job/queue_adapters/remote_sidekiq_adapter.rb
require 'sidekiq'
module ActiveJob
module QueueAdapters
class RemoteSidekiqAdapter
def enqueue(job)
#Sidekiq::Client does not support symbols as keys
job.provider_job_id = client.push \
"class" => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
"wrapped" => job.class.to_s,
"queue" => job.queue_name,
"args" => [ job.serialize ]
end
def enqueue_at(job, timestamp)
job.provider_job_id = client.push \
"class" => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
"wrapped" => job.class.to_s,
"queue" => job.queue_name,
"args" => [ job.serialize ],
"at" => timestamp
end
def client
#client ||= ::Sidekiq::Client.new(RemoteSidekiq.redis_pool)
end
end
end
end
I can use the adapter to queue the events now:
require 'active_job/queue_adapters/remote_sidekiq_adapter'
class RemoteJob < ActiveJob::Base
self.queue_adapter = :remote_sidekiq
queue_as :default
def perform(_event_name, _data)
fail "
This job should not run here; intended to hook into
ActiveJob and run in another system
"
end
end
I can now queue the job using the normal ActiveJob api. Whatever app reads this out of the queue will need to have a matching RemoteJob available to perform the action.