I have 20 Delayed job workers
I had search so many links and post for synchronous processing for that job but nothing works for me
Also assign QUEUE for perticular worker like this
script/delayed_job --queue=order_item_kit start -i 0
order.delay(:queue => "order_item_kit").update_order
But this will assign to this queue and other worker also works for it.
I am searching for one worker will run one job at a time.
for this particular case
Please give suggestion to work for this case, one method calls only one worker as a time.
Thanks in advance.
I can solve this by doing that way
When it assign next same job to another worker, we can add run_at one second ahead, so multiple workers does not save object same time and this fix our issue
latest_job = Delayed::Job.where(queue: "order_item_kit").last
run_at = latest_job ? latest_job.run_at + 1.seconds : 1.seconds.from_now
kit_sku.delay(:run_at => run_at,:queue =>"order_item_kit").update_ordere_item
According to the DJ documentation
Processes started without specifying a queue will run jobs from any queue.
You should run your workers with explicit queue name:
RAILS_ENV=production script/delayed_job --queue=tracking start
And with some other queue name for all other ('default') workers.
To define that name set Delayed::Worker.default_queue_name in initializer.
Related
I have around 10 queues in my code, each having separate workers. All have default priorities - 1.
Currently, the requests coming from user (UI) & rake tasks are going in same queue. Suppose, request coming for fetching data from user & rake task are going in same queue with priority 1 & both are executing one by one depends on request time.
But I want to run the request from user first & then the rake task. How can I manage this ?
I also want to understand if I create a separate queue for rake task & give it low priority, then how it will work. Will it run along with another queues or wait for them to execute first?
Delayed::Worker.queue_attributes = {
high_priority: { priority: -10 },
low_priority: { priority: 10 }
}
This article could help you further - it also has coding examples that you can use to try with your own code:
https://phil.tech/2016/tips-on-sidekiq-queues/
Basically, if you want to run the request from users first, you should make a queue for them and assign it a higher priority. No need to assign -10, just start from 0 and work your way up.
But also keep in mind that you should not have more than a handful queues, so it might make sense to bundle different priorities together:
I don't recommend having more than a handful of queues. Lots of queues makes for a more complex system and Sidekiq Pro cannot reliably handle multiple queues without polling. M Sidekiq Pro processes polling N queues means O(M*N) operations per second slamming Redis.
https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
Finally, if one part of your code has to run before another, you could also enqueue the rake task from the job that handles the user requests.
But I want to run the request from user first & then the rake task.
How can I manage this ?
Recommend change Delayed::Worker.default_priority to a positive number, for example: 5, then define a priority method in the user workers return a lower number
class UserRequestJob < ApplicationJob
def priority
1 # lower numbers run first
end
...
end
I also want to understand if I create a separate queue for rake task &
give it low priority, then how it will work. Will it run along with
another queues or wait for them to execute first?
It will run higher priority jobs first, and then lower priority jobs, in fact queue name doesn't matter after a job is enqueued, unless you have dedicated workers, when you define queues with priority:
Delayed::Worker.queue_attributes = {
high_priority: { priority: -10 },
low_priority: { priority: 10 }
}
and then you enqueue a job to a queue, delayed_job will get the priority by queue name, as you can see in job_preparer.rb
References:
https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html
https://github.com/collectiveidea/delayed_job#named-queues
Currently I am working with queue job on the ruby on rail with the Sidekiq. I have 2 jobs that are depend to each other and I want 1st job to finish first before starting the 2nd job, so is there any way to make it with Sidekiq.
Yes, you can use the YourSidekiqJob.new.perform(parameters_to_the_job) pattern. This will run your jobs in order, synchronously.
However, there are 2 things to consider here:
What happens if the first job fails?
How long does the each job run?
For #2, the pattern blocks execution for the length of time each job takes to run. If the jobs are extremely short in runtime, why use the jobs in the first place? If they're long, are you expecting the user to wait until they're done?
Alternatively, you can schedule the running of the second job as the last line in the body of the first one. You still need to account for the failure mode of job #1 or #2. Also, you need to consider that the job won't necessarily run when it's scheduled to run, due to the state of the queue at schedule time. How does this affect your business logic?
Hope this helps
--edit according to last comment
class SecondJob < SidekiqJob
def perform(params)
data = SomeData.find
return unless data.ready?
# do whatever you need to do with the ready data
end
end
I have several workers that are being run using Sidekiq and scheduled using Sidetiq. I'm looking for advice on the best way to wait for all workers to complete before executing a callback, similar to Sidekiq-Pro's batching functionality. Any advice on possible options would be greatly appreciated!
You can write a method:
def wait_for_sidekiq
sleep(1) until Sidekiq::Workers.new.size == 0 && Sidekiq::Queue.new.size == 0
end
I would also suggest that you make sure that the jobs got queued in the first place:
def wait_for_queuing
sleep(1) until Sidekiq::Queue.new.size > 0 || Sidekiq::Workers.new.size > 0
end
This is required, because sometimes the first method might get executed a few miliseconds before queuing the jobs, so that it won't wait at all.
For simple use cases you can poll stats to get a count of pending jobs for each queue.
So if you know that all your jobs go to a specific queue, and there are no other jobs going in that queue from elsewhere this would suffice. For more advanced/complex use cases you can refer the API source
Another simple solution is to use a Redis based counter (since you are already using Redis) and atomically decrement it from each job, and publish an event once the count reaches zero.
You should see this Sidekiq-batch gem, i already try it and it works just like the Sidekiq's Pro Batch, and it's free
Hope this could help
I am on Heroku and I got an error because my redis db got too full. The my sidekiq processes stopped working. It was like that for a day until I realized it. Now I have 600+ jobs that I have tried to process but they are just breaking everything now. How can I sound off the alarms when sidekiq can't process jobs or when the Enqueue starts to fill up?
You could set a rake task on a schedule to check Sidekiq stats, and then take the appropriate action ( like send an email ).
I've created my own module with helper methods for Sidekiq that serves a number of purposes, e.g deleting jobs, checking queues, retriving jobs by certain criteria etc. https://gist.github.com/blotto/10324119
For your purpose, grab the sidekiq stats as such :
def sidekiq_stats()
summary = Hash.new
stats = Sidekiq::Stats.new
summary = { processed: stats.processed,
failed: stats.failed,
enqueued: stats.enqueued,
queues: stats.queues}
end
And then evaluate the enqueued value, set a tolerance on what you think is too high, and then let loose the hounds.
If you're using zabbix for monitoring, you could use sidekiq_queue_zabbix template at https://github.com/hungntit/sidekiq_queue_zabbix. This template supports showing graph and sending alert when sidekiq queue size is higher than one specified limit number
I am using multiple heroku servers that share the same DB. I would like to have each server only process delayed jobs for the server that created the delayed job entry.
For example:
Server A only processes queue "server_a"
Server A only processes queue "server_b"
etc...
This is accomplishable by using Delayed Job 3 (https://github.com/collectiveidea/delayed_job)
However, for this to work I would need to manually assign a queue name for each delayed job created, which can be a pain. (for example: object.delay(:queue => 'tracking').method)
Instead I would like to be able to assign a "default queue" for all new jobs. Ideally, I put something like this in a delayed_job_config.rb & it works:
DEFAULT_QUEUE_NAME = ENV['APP_NAME']
... the idea being that I do nothing to existing delayed jobs & they automatically get assigned a queue with the same name as the app server.
I am looking for suggestions on how to accomplish this -- or if you want to give it a stab, throw some code my way.
Thanks in advance!
In config/initializers/delayed_job.rb
Delayed::Worker.default_queue_name = `hostname`.chomp