I am currently using Sidekiq with my Rails app in production along with an ElasticCache Redis database. I've noticed recently that when monitoring the CurrItems metric using the AWS tools, I see the number of items gradually increasing over time in an almost step-like way:
However, when I look at the jobs in queue in the Sidekiq dashboard, I don't see anything backing up at all. I see 0 jobs in queue, 0 busy, 0 scheduled.
The step-like increase seems to happen at a very particular time each day (right at the end of the day), which made me think it might be related to a chron job/clockwork process I have running. However, I only have 4 jobs that run once a day and none of them run during that time or even near that time. Just for good measure though, here is my clock.rb file (I have shorted all the job descriptions and class and method names for simplicity's sake):
module Clockwork
every(30.seconds, 'Task 1') { Class.method }
every(30.seconds, 'Task 2') { Class.method }
every(10.minutes, 'Task 3') { Class.method }
every(1.day, 'Task 4', :at => '06:00', :tz => 'EST') { Class.method }
every(10.minutes, 'Task 5') { Class.method }
every(1.day, 'Task 6', :at => '20:00', :tz => 'UTC') { Class.method }
every(1.day, 'Task 7', :at => '20:00', :tz => 'UTC') { Class.method }
every(1.day, 'Task 8', :at => '20:00', :tz => 'UTC') { Class.method }
every(1.hour, 'Task 9') {Class.method}
every(30.minutes, 'Task 10') {Class.method}
every(30.minutes, 'Task 11') {Class.method}
every(1.hour, 'Task 12') {Class.method}
end
I'm not quite sure where this is coming from. Maybe Sidekiq isn't removing the keys from the database once the job is complete?
Another potential helpful piece of information is that I'm running 4 workers/servers. Here is my Redis configuration:
if (Rails.env == "production" || Rails.env == "staging")
redis_domain = ENV['REDIS_DOMAIN']
redis_port = ENV['REDIS_PORT']
redis_url = "redis://#{redis_domain}:#{redis_port}"
Sidekiq.configure_server do |config|
ActiveRecord::Base.establish_connection(
adapter: "postgresql",
encoding: "unicode",
database: ENV["RDS_DB_NAME"],
pool: 25,
username: ENV["RDS_USERNAME"],
password: ENV["RDS_PASSWORD"],
host: ENV["RDS_HOST"],
port: 5432
)
config.redis = {
namespace: "sidekiq",
url: redis_url
}
end
Sidekiq.configure_client do |config|
config.redis = {
namespace: "sidekiq",
url: redis_url
}
end
end
Anyone know why this could be happening?
Historical job metrics are stored per-day, for the past 5 years. You are seeing those 4-6 keys/day. This gives you the nice metrics on the Web UI's Dashboard.
Related
We've provisioned an Azure Redis Cache server using the Premium tier. The server is clustered, with 2 shards, and the server is configured to allow public access over the public internet through a firewall--allow-listing a set of known IPs.
Deploys of our Rails application two any of these known IPs fail with the error:
Redis::CannotConnectError: Redis client could not connect to any cluster nodes
Here is our Rails config:
# application.rb
if Rails.env.test?
config.cache_store = :redis_cache_store, {
url: config.nines[:redis_url],
expires_in: 90.minutes,
namespace: ENV['TEST_ENV_NUMBER'],
}
else
config.cache_store = :redis_store, {
namespace:ENV['TEST_ENV_NUMBER'],
cluster: [ Rails.application.config.nines[:redis_url] ],
replica: true, # allow reads from replicas
compress: true,
compress_threshold: 1024,
expires_in: 90.minutes
}
end
The config.nines[:redis_url] is set like this: rediss://:<pw>#<cache-name>.redis.cache.windows.net:6380/0
Then, we initialize the Redis connection in code like this:
if Rails.env.test?
::Redis.new :url => redis_url, :db => ENV['REDIS_DB']
else
::Redis.new(cluster: [ "#{redis_url}" ], db: ENV['REDIS_DB'])
end
We're using the redis-rb gem and redis-rails gem.
If anyone can point out what we're doing wrong, please do share!
Thank you User Peter Pan - Stack Overflow. Posting your suggestion as answer to help other community members.
You can try below code:
# Import the redis library for Ruby
require "redis"
# Create a redis client instance for connecting Azure Redis Cache
# At here, for enabling SSL, set the `:ssl` symbol with the
# symbol value `:true`, see https://github.com/redis/redis-rb#ssltls-support
redis = Redis.new(
:host => '<azure redis cache name>.redis.cache.windows.net',
:port => 6380,
:db => <the db index you selected like 10>,
:password => "<access key>",
:ssl => :true)
# Then, set key `foo` with value `bar` and return `OK`
status = redis.set('foo', 'bar')
puts status # => OK
# Get the value of key `foo`
foo = redis.get('foo')
puts foo # => bar
Reference: How to setup Azure Redis cache with Rails - Stack Overflow
I took a shared server to deploy my app, to begin.
Everything works but I have some troubles to create my plans with stripe.
On localhost, I can use 'rails c' to create them but my server don't allow me to do it.
Here's the command to create plans from the console :
CreatePlan.call(stripe_id: 'test_plan', name: 'Test Plan', amount: 500, interval: 'month', description: 'Test Plan', published: false)
The create plan method is a service object : app/services/create_plan.rb
Here's my create_plan.rb :
class CreatePlan
def self.call(options={})
plan = Plan.new(options)
if !plan.valid?
return plan
end
begin
Stripe::Plan.create(
id: options[:stripe_id],
amount: options[:amount],
currency: 'usd',
interval: options[:interval],
name: options[:name],
)
rescue Stripe::StripeError => e
plan.errors[:base] << e.message
return plan
end
plan.save
return plan
end
end
How could I create my plans with no console ?
I tried with seeds.rb but it don't work.
To do stuff like this, make a rake task.
Like this:
namespace :stripe do
desc "Create stripe plans"
task :create_plans => :environment do
# Do the business
end
end
And then run rake stripe:create_plans on your server.
Hey I am attempting to spawn a sidekiq worker that connects to a completely separate Redis database. I know with 3.0's connection pooling this is possible, and I have been able to successfully push a job onto the correct Redis DB, but the problem is the Sidekiq web UI is not showing these jobs in the queue (I have mounted a separate Rack app for this that points exclusively to the other Redis DB). The "Busy" tab in the admin interface also shows my sidekiq workers that I have pointed at this DB, with correct PIDs.
Here's my sidekiq.rb:
Sidekiq.configure_server do |config|
if ENV['REDIS_DB'] == "2"
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379/2", :namespace => 'drip' }
else
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379", :namespace => 'drip' }
end
end
Sidekiq.configure_client do |config|
if ENV['REDIS_DB'] == "2"
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379/2", :namespace => 'drip' }
else
config.redis = { :url => "redis://#{SIDEKIQ_HOST}:6379", :namespace => 'drip' }
end
end
My use case is that I need to have fine grain control over the jobs that go into the second database, so I need the workers configured precisely so they are only using as many resources as I need them to. I only want the workers that are configured in this way to pick up these jobs.
The Web UI cannot be mounted multiple times to point to multiple Redises in a single process. You have to run multiple web processes with REDIS_DB set too.
I planned to migrate redis server to a new one with sidekiq running, but doesn't want to stop the current application running. And I don't want to use redis cluster which is still alpha version.
My thought is to ask sidekiq to write to new redis server, but pulls from both of them, so that once the workers in old redis are completed, the new one can totally take over all workers. I think this solution is feasible, but I have no idea how to make it happen.
This is my sidekiq.rb:
sidkiq_config = YAML.load(ERB.new(Rails.root.join('config/redis.yml').read).result)
Sidekiq.configure_server do |config|
config.logger.level = Logger::ERROR
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
config.redis = { :url => "redis://redis.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
Sidekiq.configure_client do |config|
config.logger.level = Logger::ERROR
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
I believe the easiest solution is to run two instances of sidekiq - one that reads from the old cluster, and one that reads from the new cluster
sidkiq_config = YAML.load(ERB.new(Rails.root.join('config/redis.yml').read).result)
Sidekiq.configure_server do |config|
config.logger.level = Logger::ERROR
if ENV['read_from_new']
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
else
config.redis = { :url => "redis://redis.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
end
Sidekiq.configure_client do |config|
config.logger.level = Logger::ERROR
config.redis = { :url => "redis://redis2.staging:6379", :namespace => "app_#{Rails.env}:sidekiq" }
end
You can update to a recent version of sidekiq and use Sharding.
REDIS_A = ConnectionPool.new { Redis.new(...) }
REDIS_B = ConnectionPool.new { Redis.new(...) }
# To create a new connection pool for a namespaced Sidekiq worker:
ConnectionPool.new do
client = Redis.new(:url => "Your Redis Url")
Redis::Namespace.new("Your Namespace", :redis => client)
end
# Create a job in the default redis instance
SomeWorker.perform_async
# Push a job to REDIS_A using the low-level Client API
client = Sidekiq::Client.new(REDIS_A)
client.push(...)
client.push_bulk(...)
Sidekiq::Client.via(REDIS_B) do
# All jobs defined within this block will go to B
SomeWorker.perform_async
end
To quite sidekiq, click the quite button in the UI
I'm building an app on Herokou and Redis that sends an SMS messages for every row in an input CSV file which contains the mobile phone number. The message is sent using Twilio in a sidekiq worker shown below. The problem is that even though the SMS is being sent for all the rows in the CSV, the database write (TextMessage.create) and log write (puts statement) only executes for one row in the CSV. There is one Sidekiq worker spawned for each row in the CSV file. It seems like only one Sidekiq worker has I/O (DB, file) access and it locks it from the other Sidekiq workers. Any help would be appreciated.
sidekiq worker:
require 'sidekiq'
require 'twilio-rb'
class TextMessage < ActiveRecord::Base
include Sidekiq::Extensions
def self.send_message(number, body, row_index, column_index, table_id)
puts "TextMessage#send_message: ROW INDEX: #{row_index} COLUMN INDEX: #{column_index} TABLEID: #{table_id} BODY: #{body} PHONE: #{number}"
Twilio::Config.setup :account_sid => 'obfuscated', :auth_token => '<obfuscated>'
sms = Twilio::SMS.create :to => number, :from => '+17085555555', :body => body + ' | Sent: ' + Time.now.in_time_zone('Central Time (US & Canada)').strftime("%m/%d/%Y %I:%M%p Central")
TextMessage.create :to => number, :from => '+17085555555'
ImportCell.add_new_column(table_id, row_index, column_index, "Time Sent", Time.now.in_time_zone('Central Time (US & Canada)').strftime("%m/%d/%Y %I:%M%p Central"))
end
end
call to sidekiq worker:
TextMessage.delay_until(time_to_send, :retry => 3).send_message(phone, 'Scheduled: ' + time_to_send.in_time_zone('Central Time (US & Canada)').strftime("%m/%d/%Y %I:%M%p Central"), row_index, column_index, table.id)
column_index += 1
Heroku Procfile
worker: bundle exec sidekiq -C config/sidekiq.yml
sidekiq.yml
:verbose: false
:concurrency: 3
:queues:
- [default, 5]
config/initializers/redis.rb:
uri = URI.parse(ENV["REDISTOGO_URL"])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Sidekiq.configure_server do |config|
database_url = ENV['DATABASE_URL']
if(database_url)
ENV['DATABASE_URL'] = "#{database_url}?pool=25"
ActiveRecord::Base.establish_connection
end
end
I am one of the people who commented on your question, just fixed it!
You are using .create which SideKiq seemed to not like, so I tried using .new and then .save which made it work! I think it has to do with .create not being thread safe or something of the sort, but I honestly have no idea.
Non Working code:
class HardWorker
include Sidekiq::Worker
def perform(name, count)
puts 'Doing some hard work!'
UserInfo.create(
:user => "someone",
:misc1 => 0,
:misc2 => 0,
:misc3 => 0,
:comment => "Made from HardWorker",
:time_changed => Time.now
)
puts 'Done with hard work!'
end
end
Working code:
class HardWorker
include Sidekiq::Worker
def perform(name, count)
puts 'Doing some hard work!'
a_row = UserInfo.new(
:user => "someone",
:misc1 => 0,
:misc2 => 0,
:misc3 => 0,
:comment => "Made from HardWorker",
:time_changed => Time.now
)
a_row.save
puts 'Done with hard work!'
end
end