I am using sidekiq for background tasks in Rails application. Now the numbers of jobs becomes more, so I want to clear all the jobs. I tried the following command in console
Sidekiq::Queue.new.clear
but it was giving following error.
NameError: uninitialized constant Sidekiq::Queue
How do I clear all the jobs from sidekiq?
You can do as it says on the issue 1077 or as reported in this blog at noobsippets
Both suggest we do the following, and can be done on rails console:
Sidekiq.redis(&:flushdb)
Caution: This command will clear all redis records. I suggest not using it in production
another approach would be
redis-cli --scan --pattern users: * | xargs redis-cli del
according to this blog
Clear Sidekiq Jobs commands:
require 'sidekiq/api'
# Clear retry set
Sidekiq::RetrySet.new.clear
# Clear scheduled jobs
Sidekiq::ScheduledSet.new.clear
# Clear 'Dead' jobs statistics
Sidekiq::DeadSet.new.clear
# Clear 'Processed' and 'Failed' jobs statistics
Sidekiq::Stats.new.reset
# Clear all queues
Sidekiq::Queue.all.map(&:clear)
# Clear specific queue
stats = Sidekiq::Stats.new
stats.queues
# => {"main_queue"=>25, "my_custom_queue"=>1}
queue = Sidekiq::Queue.new('my_custom_queue')
queue.count
queue.clear
According to this issue on Github: https://github.com/mperham/sidekiq/issues/1732 you now need to
require 'sidekiq/api'
As of latest Sidekiq, just blow it up:
require 'sidekiq/api'
q = Sidekiq::Queue.new
q.💣
Yes, the command to clear all is literally a bomb emoji. Also works for Sidekiq::RetrySet.
Or if you're no fun you can use q.clear
redis-cli flushdb
You can also use redis-cli flushall
Use Rails runner in one line
rails runner 'Sidekiq.redis { |conn| conn.flushdb }'
You can use this for clearing all the jobs
require 'sidekiq/api'
Sidekiq::Queue.all.each(&:clear)
require 'sidekiq/api'
Sidekiq::Queue.all.each {|x| x.clear}
All Sidekiq tasks are saved in "Redis".
You can clean "Redis" by this command
redis-cli flushall
If you want to delete jobs from specific queues try:
queue = Sidekiq::Queue.new("default")
queue.each do |job|
job.klass # => 'TestWorker'
job.args # => ['easy']
job.delete if job.jid == 'abcdef1234567890' || job.klass == 'TestWorker'
end
Read all about sidekiq and important console commands- https://medium.com/#shashwat12june/all-you-need-to-know-about-sidekiq-a4b770a71f8f
While many ways for doing this are pretty well-documented by other answers here, Sidekiq::Queue#clear can't execute when you're running into Redis::CommandError (OOM command not allowed when used memory > 'maxmemory'.) When this happens, you can still individually delete records from your problem queue until there is enough memory for the clear method to work. For example, this worked for me in a Rails console:
problem_queue_name = 'my_big_queue'
queue = Sidekiq::Queue.new(problem_queue_name)
# Alternate between deleting jobs in a batch and attempting clear until it succeeds
queue.first(10000).each do |q| print('.') if q.delete end; nil
queue.clear
I realized that Sidekiq.redis { |conn| conn.flushdb } removes all keys from the redis database. There is a safer way to clear all sidekiq queues using redis-cli:
redis-cli keys "*queue:*" | xargs redis-cli del
The same can be achieved with Sidekiq API (see Ravi Prakash Singh answer)
Related
I have a rails app that I want to use sidekiq on.
Before I figure out how to setup sidekiq on the server etc., is it possible for me to create a rake task that will run the sidekiq workers for any pending jobs?
This isn't a production solution I know, but I just want to make sure everything else is working on the server and for the time being me running a rake task on the server is fine as it is more for QA'ing at this point.
Log into the server, cd to the app's dir and run 'bundle exec sidekiq' manually.
Referencing from
SortedEntry and ScheduledSet, I managed to come up with the following:
# ScheduledSet returns "pending" jobs
scheduled_jobs = Sidekiq::ScheduledSet.new
# puts scheduled_jobs.class
# => Sidekiq::ScheduledSet
scheduled_jobs.each do |scheduled_job|
# puts scheduled_job.class
# => Sidekiq::SortedEntry
job_class = scheduled_job.klass.constantize
job_args = scheduled_job.args
# you can also filter out only jobs in certain queue, i.e.:
# if scheduled_job.queue == 'mailers' ......
# run the job inline
job_class.new.perform(*job_args)
end
Then just wrap this into a rake task.
I've this worker that runs for ever.
class Worker
include Sidekiq::Worker
sidekiq_options queue: "infinity", retry: true
def perform(params)
# ...
self.class.perform_in(30.seconds, params)
end
end
The problem is that I load workers on start up, like this. config/initializers/load_workers.rb
Rails.application.config.after_initialize do
if ENV["SIDEKIQ"] == "1"
Worker.perform_async({})
end
end
Using this to start sidekiq SIDEKIQ=1 sidekiq --verbose --environment production -C config/sidekiq.yml.
This means that old workers as to stop, both those currently running but also the ones being rescheduled.
I tried running this on start up (just before loading new works), but that didn't work.
q = []
q += Sidekiq::RetrySet.new.select { |job| job.klass.match(/Worker/) }
q += Sidekiq::Queue.new("infinity").select { |job| job.klass.match(/Worker/) }
q += Sidekiq::ScheduledSet.new.select { |job| job.klass.match(/Worker/) }
q.each(&:delete)
After 5-ish deploys there are bunch of duplicate workers in the queue scheduled for later. So, is there a way to clear everyting in one queue and prevent already running jobs from rescheduling?
I'm using sidekiq 3.0.
Deletes all Jobs in a Queue, by removing the queue.
require 'sidekiq/api' # for the case of rails console
Sidekiq::Queue.new("infinity").clear
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
This will clear all queues, schedules and retries:
require 'sidekiq/api'
Sidekiq::Queue.all.each(&:clear)
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Sidekiq::DeadSet.new.clear
Works for me for most sidekiq versions:
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Clear statistics (Optional)
Sidekiq::Stats.new.reset
There is one more convenient way to clear all Sidekiq queues and sets for local env: Sidekiq.redis(&:flushdb)
Basically, it just flushes Redis that is configured for Sidekiq, so I would avoid using it in a non-local environment, as it can also remove some data you store in Redis. Anyways, it can be useful for the development or when you have a separate Redis instance for Sidekiq. And it's just one line instead of four.
You can clear your queue by running this code although there would be built-in methods.
queue = Sidekiq::Queue.new
queue.each do |job|
job.delete
end
I am using the Sidekiq Gem with Rails 3. Is there a way to see if the sidekiq workers are ready to process jobs? We want to make sure that stuff is not just getting enqueued and sitting there. I tried something like this...
stats = Sidekiq::Stats.new
puts stats.queues
But that will always return default => 0, if redis is up, even if sidekiq is not ready to process jobs.
You can access the Sidekiq Workers API to access the active set of workers:
worker_api = Sidekiq::Workers.new
worker_api.size
=> 1
worker_api.each {|worker| do_stuf(worker) }
The simplest solution would be to access the API and wait until the expected number of worker processes are up and running via the size attribute of the Worker API.
First of all, your worker processes should be running. Check this:
ps = Sidekiq::ProcessSet.new
ps.size # => 2
ps.each do |process|
p process['busy'] # => 3
p process['hostname'] # => 'myhost.local'
p process['pid'] # => 16131
end
ps.each(&:quiet!) # equivalent to the USR1 signal
ps.each(&:stop!) # equivalent to the TERM signal
From https://github.com/mperham/sidekiq/wiki/API#processes
So if they do - you can enqueue more work. The only case when tasks can remain in the queue while your worker processes are working is when the worker processes are processing jobs slower than they come in. Then your queue will grow and newly enqueued tasks will not be processed until all previous jobs are done. But this is your responsibility to figure out why your workers performing slowly.
Hope that helps!
For myself, I prefer to run the following in a console (either on my server or locally):
ps -ef | grep "sidekiq"
As you can see from the attached image, I've got a couple of workers that seem to be stuck. Those processes shouldn't take longer than a couple of seconds.
I'm not sure why they won't clear or how to manually remove them.
I'm on Heroku using Resque with Redis-to-Go and HireFire to automatically scale workers.
None of these solutions worked for me, I would still see this in redis-web:
0 out of 10 Workers Working
Finally, this worked for me to clear all the workers:
Resque.workers.each {|w| w.unregister_worker}
In your console:
queue_name = "process_numbers"
Resque.redis.del "queue:#{queue_name}"
Otherwise you can try to fake them as being done to remove them, with:
Resque::Worker.working.each {|w| w.done_working}
EDIT
A lot of people have been upvoting this answer and I feel that it's important that people try hagope's solution which unregisters workers off a queue, whereas the above code deletes queues. If you're happy to fake them, then cool.
You probably have the resque gem installed, so you can open the console and get current workers
Resque.workers
It returns a list of workers
#=> [#<Worker infusion.local:40194-0:JAVA_DYNAMIC_QUEUES,index_migrator,converter,extractor>]
pick the worker and prune_dead_workers, for example the first one
Resque.workers.first.prune_dead_workers
Adding to answer by hagope, I wanted to be able to only unregister workers that had been running for a certain amount of time. The code below will only unregister workers running for over 300 seconds (5 minutes).
Resque.workers.each {|w| w.unregister_worker if w.processing['run_at'] && Time.now - w.processing['run_at'].to_time > 300}
I have an ongoing collection of Resque related Rake tasks that I have also added this to: https://gist.github.com/ewherrmann/8809350
Run this command wherever you ran the command to start the server
$ ps -e -o pid,command | grep [r]esque
you should see something like this:
92102 resque: Processing ProcessNumbers since 1253142769
Make note of the PID (process id) in my example it is 92102
Then you can quit the process 1 of 2 ways.
Gracefully use QUIT 92102
Forcefully use TERM 92102
* I'm not sure of the syntax it's either QUIT 92102 or QUIT -92102
Let me know if you have any trouble.
I just did:
% rails c production
irb(main):001:0>Resque.workers
Got the list of workers.
irb(main):002:0>Resque.remove_worker(Resque.workers[n].id)
... where n is the zero based index of the unwanted worker.
I had a similar problem that Redis saved the DB to disk that included invalid (non running) workers. Each time Redis/resque was started they appeared.
Fix this using:
Resque::Worker.working.each {|w| w.done_working}
Resque.redis.save # Save the DB to disk without ANY workers
Make sure you restart Redis and your Resque workers.
Started working on https://github.com/shaiguitar/resque_stuck_queue/ recently. It's not a solution to how to fix stuck workers but it addresses the issue of resque hanging/being stuck, so I figured it could be helpful for people on this thread. From README:
"If resque doesn't run jobs within a certain timeframe, it will trigger a pre-defined handler of your choice. You can use this to send an email, pager duty, add more resque workers, restart resque, send you a txt...whatever suits you."
Been used in production and works pretty well for me thus far.
Here's how you can purge them from Redis by hostname. This happens to me when I decommission a server and workers do not exit gracefully.
Resque.workers.each { |w| w.unregister_worker if w.id.start_with?(hostname) }
I ran into this issue and started down the path of implementing a lot of the suggestions here. However, I discovered the root cause that was creating this issue was that I was using the gem redis-rb 3.3.0. Downgrading to redis-rb 3.2.2 prevented these workers from getting stuck in the first place.
I've cleared them out from redis-cli directly. Luckily redistogo.com allows access from environments outside heroku.
Get dead worker ID from the list. Mine was
55ba6f3b-9287-4f81-987a-4e8ae7f51210:2
Run this command in redis directly.
del "resque:worker:55ba6f3b-9287-4f81-987a-4e8ae7f51210:2:*"
You can monitor redis db to see what it's doing behind the scenes.
redis xxx.redistogo.com> MONITOR
OK
1380274567.540613 "MONITOR"
1380274568.345198 "incrby" "resque:stat:processed" "1"
1380274568.346898 "incrby" "resque:stat:processed:c65c8e2b-555a-4a57-aaa6-477b27d6452d:2:*" "1"
1380274568.346920 "del" "resque:worker:c65c8e2b-555a-4a57-aaa6-477b27d6452d:2:*"
1380274568.348803 "smembers" "resque:queues"
Second last line deletes the worker.
In resque 2.0.0, here's one way that seems to work to remove only actually appearantly-dead workers in resque 2.0.0:
Resque::Worker.all_workers_with_expired_heartbeats.each { |w| w.unregister_worker }
I am not an expert in what's going, it's possible there's a better way to do this or that this will have problems. I'm just trying to figure this out too.
This seems to remove workers that haven't sent a "heartbeat" in much longer than expected from the resque worker list.
If the phantom worker was in a "running" state, then a new entry in the "failed" job queue will be created corresponding to phantom job.
I had stuck/stale resque workers here too, or should I say 'jobs', because the worker is actually still there and running fine, it's the forked process that is stuck.
I chose the brutal solution of killing the forked process "Processing" since more than 5min, via a bash script, then the worker just spawn the next in queue, and everything keeps on going
have a look at my script here: https://gist.github.com/jobwat/5712437
If you are using newer versions of Resque, you'll need to use the following command as the internal APIs have changed...
Resque::WorkerRegistry.working.each {|work| Resque::WorkerRegistry.remove(work.id)}
This avoids the problem as long as you have a resque version newer than 1.26.0:
resque: env QUEUE=foo TERM_CHILD=1 bundle exec rake resque:work
Keep in mind that it does not let the currently running job finish.
If you use Docker, you can also use this command:
<id> is the worker id.
docker stop <id>
docker start <id>
We have to use delayed_job (or some other background-job processor) to run jobs in the background, but we're not allowed to change the boot scripts/boot-levels on the server. This means that the daemon is not guaranteed to remain available if the provider restarts the server (since the daemon would have been started by a capistrano recipe that is only run once per deployment).
Currently, the best way I can think of to ensure the delayed_job daemon is always running, is to add an initializer to our Rails application that checks if the daemon is running. If it's not running, then the initializer starts the daemon, otherwise, it just leaves it be.
The question, therefore, is how do we detect that the Delayed_Job daemon is running from inside a script? (We should be able to start up a daemon fairly easily, bit I don't know how to detect if one is already active).
Anyone have any ideas?
Regards,
Bernie
Based on the answer below, this is what I came up with. Just put it in config/initializers and you're all set:
#config/initializers/delayed_job.rb
DELAYED_JOB_PID_PATH = "#{Rails.root}/tmp/pids/delayed_job.pid"
def start_delayed_job
Thread.new do
`ruby script/delayed_job start`
end
end
def process_is_dead?
begin
pid = File.read(DELAYED_JOB_PID_PATH).strip
Process.kill(0, pid.to_i)
false
rescue
true
end
end
if !File.exist?(DELAYED_JOB_PID_PATH) && process_is_dead?
start_delayed_job
end
Some more cleanup ideas: The "begin" is not needed. You should rescue "no such process" in order not to fire new processes when something else goes wrong. Rescue "no such file or directory" as well to simplify the condition.
DELAYED_JOB_PID_PATH = "#{Rails.root}/tmp/pids/delayed_job.pid"
def start_delayed_job
Thread.new do
`ruby script/delayed_job start`
end
end
def daemon_is_running?
pid = File.read(DELAYED_JOB_PID_PATH).strip
Process.kill(0, pid.to_i)
true
rescue Errno::ENOENT, Errno::ESRCH # file or process not found
false
end
start_delayed_job unless daemon_is_running?
Keep in mind that this code won't work if you start more than one worker. And check out the "-m" argument of script/delayed_job which spawns a monitor process along with the daemon(s).
Check for the existence of the daemons PID file (File.exist? ...). If it's there then assume it's running else start it up.
Thank you for the solution provided in the question (and the answer that inspired it :-) ), it works for me, even with multiple workers (Rails 3.2.9, Ruby 1.9.3p327).
It worries me that I might forget to restart delayed_job after making some changes to lib for example, causing me to debug for hours before realizing that.
I added the following to my script/rails file in order to allow the code provided in the question to execute every time we start rails but not every time a worker starts:
puts "cleaning up delayed job pid..."
dj_pid_path = File.expand_path('../../tmp/pids/delayed_job.pid', __FILE__)
begin
File.delete(dj_pid_path)
rescue Errno::ENOENT # file does not exist
end
puts "delayed_job ready."
A little drawback that I'm facing with this though is that it also gets called with rails generate for example. I did not spend much time looking for a solution for that but suggestions are welcome :-)
Note that if you're using unicorn, you might want to add the same code to config/unicorn.rb before the before_fork call.
-- EDITED:
After playing around a little more with the solutions above, I ended up doing the following:
I created a file script/start_delayed_job.rb with the content:
puts "cleaning up delayed job pid..."
dj_pid_path = File.expand_path('../../tmp/pids/delayed_job.pid', __FILE__)
def kill_delayed(path)
begin
pid = File.read(path).strip
Process.kill(0, pid.to_i)
false
rescue
true
end
end
kill_delayed(dj_pid_path)
begin
File.delete(dj_pid_path)
rescue Errno::ENOENT # file does not exist
end
# spawn delayed
env = ARGV[1]
puts "spawing delayed job in the same env: #{env}"
# edited, next line has been replaced with the following on in order to ensure delayed job is running in the same environment as the one that spawned it
#Process.spawn("ruby script/delayed_job start")
system({ "RAILS_ENV" => env}, "ruby script/delayed_job start")
puts "delayed_job ready."
Now I can require this file anywhere I want, including 'script/rails' and 'config/unicorn.rb' by doing:
# in top of script/rails
START_DELAYED_PATH = File.expand_path('../start_delayed_job', __FILE__)
require "#{START_DELAYED_PATH}"
# in config/unicorn.rb, before before_fork, different expand_path
START_DELAYED_PATH = File.expand_path('../../script/start_delayed_job', __FILE__)
require "#{START_DELAYED_PATH}"
not great, but works
disclaimer: I say not great because this causes a periodic restart, which for many will not be desirable. And simply trying to start can cause problems because the implementation of DJ can lock up the queue if duplicate instances are created.
You could schedule cron tasks that run periodically to start the job(s) in question. Since DJ treats start commands as no-ops when the job is already running, it just works. This approach also takes care of the case where DJ dies for some reason other than a host restart.
# crontab example
0 * * * * /bin/bash -l -c 'cd /var/your-app/releases/20151207224034 && RAILS_ENV=production bundle exec script/delayed_job --queue=default -i=1 restart'
If you are using a gem like whenever this is pretty straightforward.
every 1.hour do
script "delayed_job --queue=default -i=1 restart"
script "delayed_job --queue=lowpri -i=2 restart"
end