I've this worker that runs for ever.
class Worker
include Sidekiq::Worker
sidekiq_options queue: "infinity", retry: true
def perform(params)
# ...
self.class.perform_in(30.seconds, params)
end
end
The problem is that I load workers on start up, like this. config/initializers/load_workers.rb
Rails.application.config.after_initialize do
if ENV["SIDEKIQ"] == "1"
Worker.perform_async({})
end
end
Using this to start sidekiq SIDEKIQ=1 sidekiq --verbose --environment production -C config/sidekiq.yml.
This means that old workers as to stop, both those currently running but also the ones being rescheduled.
I tried running this on start up (just before loading new works), but that didn't work.
q = []
q += Sidekiq::RetrySet.new.select { |job| job.klass.match(/Worker/) }
q += Sidekiq::Queue.new("infinity").select { |job| job.klass.match(/Worker/) }
q += Sidekiq::ScheduledSet.new.select { |job| job.klass.match(/Worker/) }
q.each(&:delete)
After 5-ish deploys there are bunch of duplicate workers in the queue scheduled for later. So, is there a way to clear everyting in one queue and prevent already running jobs from rescheduling?
I'm using sidekiq 3.0.
Deletes all Jobs in a Queue, by removing the queue.
require 'sidekiq/api' # for the case of rails console
Sidekiq::Queue.new("infinity").clear
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
This will clear all queues, schedules and retries:
require 'sidekiq/api'
Sidekiq::Queue.all.each(&:clear)
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Sidekiq::DeadSet.new.clear
Works for me for most sidekiq versions:
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Clear statistics (Optional)
Sidekiq::Stats.new.reset
There is one more convenient way to clear all Sidekiq queues and sets for local env: Sidekiq.redis(&:flushdb)
Basically, it just flushes Redis that is configured for Sidekiq, so I would avoid using it in a non-local environment, as it can also remove some data you store in Redis. Anyways, it can be useful for the development or when you have a separate Redis instance for Sidekiq. And it's just one line instead of four.
You can clear your queue by running this code although there would be built-in methods.
queue = Sidekiq::Queue.new
queue.each do |job|
job.delete
end
Related
I'm using delayed_job library as adapter for Active Jobs in Rails:
config.active_job.queue_adapter = :delayed_job
My delayed_job worker is configured to sleep for 60 seconds before checking the queue for new jobs:
Delayed::Worker.sleep_delay = 60
In my Rails code I add a new job to the queue like this:
MyJob.perform_later(data)
However, the job will not be picked up by the delayed_job worker immediately, even if there are no jobs in the queue, because of the sleep_delay. Is there a way to tell the delayed_job worker to wake up and start processing the job queue if it's sleeping?
There is a MyJob.perform_now method, but it blocks the thread, so it's not what I want because I want to execute a job asynchronously.
Looking at the delayed_job code and it appears that there's no way to directly control or communicate with the workers after they are daemonized.
I think the best you can do would be to start a separate worker with a small sleep_delay that only reads a specific queue, then use that queue for these jobs. A separate command is necessary because you can't start a worker pool where the workers have different sleep delays:
Start your main worker: bin/delayed_job start
Start your fast worker: bin/delayed_job start --sleep-delay=5 --queue=fast --identifier=999 (the identifier is necessary to differentiate the workers daemons)
Update your job to use that queue:
class MyJob < ApplicationJob
queue_as :fast
def perform...
end
Notes:
When you want to stop the workers you'll also have to do it separately: bin/delayed_job stop and bin/delayed_job stop --identifier=999.
This introduces some potential parallelism and extra load on the server when both workers are working at the same time.
The main worker will process jobs from the fast queue too, it's just a matter of which worker grabs the job first. If you don't want that you need to setup the main worker to only read from the other queue(s), by default that's only 'default', so: bin/delayed_job start --queue=default.
I have a rails app using an engine where Sidekiq workers are defined. The worker's perform_async is invoked in a controller within the engine. The worker's perform does the work on arguments passed in through that controller. The worker specific queue is defined in the worker class too. However, when a request comes in to that controller, it gets pushed to the redis server from 'perform_async', to the right queue. A bundle exec sidekiq starts up Sidekiq. However, the worker's perform never gets executed. Checking the sidekiq UI, I can see that the job is in the right queue.
This is how my worker looks like
require 'pando'
class PandoWorker
include Sidekiq::Worker
sidekiq_options :queue => :pando, :backtrace=> true
def perform(*args)
puts "in here"
puts args
end
end
So in this case the sidekiq UI shows that the args are queued in 'pando'. The sidekiq process never processes from that queue or even the default.
You have to tell sidekiq process which queue to look at:
bundle exec sidekiq -q pando
Otherwise process is only watching 'default' queue.
I am using sidekiq for background tasks in Rails application. Now the numbers of jobs becomes more, so I want to clear all the jobs. I tried the following command in console
Sidekiq::Queue.new.clear
but it was giving following error.
NameError: uninitialized constant Sidekiq::Queue
How do I clear all the jobs from sidekiq?
You can do as it says on the issue 1077 or as reported in this blog at noobsippets
Both suggest we do the following, and can be done on rails console:
Sidekiq.redis(&:flushdb)
Caution: This command will clear all redis records. I suggest not using it in production
another approach would be
redis-cli --scan --pattern users: * | xargs redis-cli del
according to this blog
Clear Sidekiq Jobs commands:
require 'sidekiq/api'
# Clear retry set
Sidekiq::RetrySet.new.clear
# Clear scheduled jobs
Sidekiq::ScheduledSet.new.clear
# Clear 'Dead' jobs statistics
Sidekiq::DeadSet.new.clear
# Clear 'Processed' and 'Failed' jobs statistics
Sidekiq::Stats.new.reset
# Clear all queues
Sidekiq::Queue.all.map(&:clear)
# Clear specific queue
stats = Sidekiq::Stats.new
stats.queues
# => {"main_queue"=>25, "my_custom_queue"=>1}
queue = Sidekiq::Queue.new('my_custom_queue')
queue.count
queue.clear
According to this issue on Github: https://github.com/mperham/sidekiq/issues/1732 you now need to
require 'sidekiq/api'
As of latest Sidekiq, just blow it up:
require 'sidekiq/api'
q = Sidekiq::Queue.new
q.💣
Yes, the command to clear all is literally a bomb emoji. Also works for Sidekiq::RetrySet.
Or if you're no fun you can use q.clear
redis-cli flushdb
You can also use redis-cli flushall
Use Rails runner in one line
rails runner 'Sidekiq.redis { |conn| conn.flushdb }'
You can use this for clearing all the jobs
require 'sidekiq/api'
Sidekiq::Queue.all.each(&:clear)
require 'sidekiq/api'
Sidekiq::Queue.all.each {|x| x.clear}
All Sidekiq tasks are saved in "Redis".
You can clean "Redis" by this command
redis-cli flushall
If you want to delete jobs from specific queues try:
queue = Sidekiq::Queue.new("default")
queue.each do |job|
job.klass # => 'TestWorker'
job.args # => ['easy']
job.delete if job.jid == 'abcdef1234567890' || job.klass == 'TestWorker'
end
Read all about sidekiq and important console commands- https://medium.com/#shashwat12june/all-you-need-to-know-about-sidekiq-a4b770a71f8f
While many ways for doing this are pretty well-documented by other answers here, Sidekiq::Queue#clear can't execute when you're running into Redis::CommandError (OOM command not allowed when used memory > 'maxmemory'.) When this happens, you can still individually delete records from your problem queue until there is enough memory for the clear method to work. For example, this worked for me in a Rails console:
problem_queue_name = 'my_big_queue'
queue = Sidekiq::Queue.new(problem_queue_name)
# Alternate between deleting jobs in a batch and attempting clear until it succeeds
queue.first(10000).each do |q| print('.') if q.delete end; nil
queue.clear
I realized that Sidekiq.redis { |conn| conn.flushdb } removes all keys from the redis database. There is a safer way to clear all sidekiq queues using redis-cli:
redis-cli keys "*queue:*" | xargs redis-cli del
The same can be achieved with Sidekiq API (see Ravi Prakash Singh answer)
I am using the Sidekiq Gem with Rails 3. Is there a way to see if the sidekiq workers are ready to process jobs? We want to make sure that stuff is not just getting enqueued and sitting there. I tried something like this...
stats = Sidekiq::Stats.new
puts stats.queues
But that will always return default => 0, if redis is up, even if sidekiq is not ready to process jobs.
You can access the Sidekiq Workers API to access the active set of workers:
worker_api = Sidekiq::Workers.new
worker_api.size
=> 1
worker_api.each {|worker| do_stuf(worker) }
The simplest solution would be to access the API and wait until the expected number of worker processes are up and running via the size attribute of the Worker API.
First of all, your worker processes should be running. Check this:
ps = Sidekiq::ProcessSet.new
ps.size # => 2
ps.each do |process|
p process['busy'] # => 3
p process['hostname'] # => 'myhost.local'
p process['pid'] # => 16131
end
ps.each(&:quiet!) # equivalent to the USR1 signal
ps.each(&:stop!) # equivalent to the TERM signal
From https://github.com/mperham/sidekiq/wiki/API#processes
So if they do - you can enqueue more work. The only case when tasks can remain in the queue while your worker processes are working is when the worker processes are processing jobs slower than they come in. Then your queue will grow and newly enqueued tasks will not be processed until all previous jobs are done. But this is your responsibility to figure out why your workers performing slowly.
Hope that helps!
For myself, I prefer to run the following in a console (either on my server or locally):
ps -ef | grep "sidekiq"
I am using Sidekiq for my background jobs:
I have a worker app/workers/data_import_worker.rb
class DataImportWorker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(job_id,file_name)
begin
#Some logic in it .....
end
end
Called from a file lib/parse_excel.rb
def parse_raw_data
#job_id and #filename are defined bfr
DataImportWorker.perform_async(job_id,filename)
end
As soon as i trigger it from my action the worker is not getting called.. Redis is running on localhost:6379
Any idea why this must be happening. The Environment is Linux.
I had a similar problem where Sidekiq was running but when I called perform_async it didn't do anything except return true.
The problem was rspec-sidekiq was added to my ":development, :test" group. I fixed the problem by moving rspec-sidekiq to the ":test" group only.
Start sidekiq from the root directory of your Rails app. For example,
bundle exec sidekiq -e staging -C config/sidekiq.yml
I encounter the same problem, it turns out that the argument I've passed in the function perform_async is not appropriate, it seems that one should not pass any query result in perform_async, you must do all the query in the function perform.
You need to specify the name of the queue that worker is for.
Example:
sidekiq_options retry: false, :queue => data_import_worker
data_import_worker can be any name you want to give it.
Then when you go to the web interface: yoursite.com/sidekiq, you'll be able to see the current workers for the queue "data_import_worker"
For me when doing a perform_later, it would enqueue but never remove from queue. I needed to add my queue name to the sidekiq.yml file
---
:concurrency: 25
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:queues:
- default
- my_queue
Lost a good 15 min on this. To check if Sidekiq is correctly loading your config file (with the queues names), go to the web interface in the Busy tab and you'll find your Process ID and below it you'll find your queues.
In our case, we had misspelled mailer (the correct ActiveJob queue for Mailers is mailers, in plural).
My issue was simply having the worker file in the wrong path.
Needs to be in "project_root/app/worker/worker.rb", not "project_root/worker/worker.rb"
Check the file path!
is it realy run multiple workers on standalone sidekiq?
for example I have 2 workers:
ProccessWorker
CallbackWorker
when I am runnigs sidekiq:
bundle exec sidekiq -r ./workers/proccess_worker.rb -C ./config/sidekiq.yml
only one worker in same time.
I was calling perform_async(23) in a production console, however my sidekiq was started in staging mode.
After I started the Sidekiq in production mode, things have started working very well.