This question is about Grand Central Dispatch, and dispatch_group_wait() in particular.
Assume a dispatch_group called group with 10 tasks in it waiting to be performed.
Elsewhere I have a task that needs to wait for any tasks in group to finish before it executes. To make that happen, I use dispatch_group_wait(group, DISPATCH_TIME_FOREVER).
To distinguish it from the tasks in group I'll call it lonelyTask.
If another task gets added to group while lonelyTask is waiting, which gets executed first, lonelyTask or the task that was added to group? In other words, do tasks added to a group while another task is waiting to execute get to "cut in line" ahead of the waiting task, or is the order in which they were called maintained?
I have searched the documentation, but haven't been able to find an answer to this question...
dispatch_group_wait and dispatch_group_notify both wait for the number of items that have entered the group to transition to zero. So, if you add an eleventh task to the group before all of the original ten tasks complete, a call to dispatch_group_wait will wait for all eleven to complete before continuing.
Group is a +/- counter (semaphore) that fires each time it reaches zero. It is unaware of tasks, because dispatch_group_async() is just a wrapper that enter()s group before task submission and enqueues a new block that will call task's block and then leave() that group. That's all. Groups may be even used without queues as asynchronous retain-counters or sml that.
So in general "you can't".
But if you are able to [re]target (dispatch_set_target_queue()) all related queues to a single concurrent queue, then dispatch_barrier_async() on that queue may do what you want, if called in proper isolation. Note that group has nothing to do with this at all.
Also, original lonelyTask is not a task from group's point of view — it is unrelated thread of execution that waits until group is balanced. The synonym for "cutting in line" is a "barrier".
Another approach would be to create a private group for each taskset and wait for that specific group. That will not insert a barrier though — following tasks will not wait for completion of lonelyTask and any uncontrolled async()s may "break the rules", leading to funny debugging sessions.
Related
I have around 10 queues in my code, each having separate workers. All have default priorities - 1.
Currently, the requests coming from user (UI) & rake tasks are going in same queue. Suppose, request coming for fetching data from user & rake task are going in same queue with priority 1 & both are executing one by one depends on request time.
But I want to run the request from user first & then the rake task. How can I manage this ?
I also want to understand if I create a separate queue for rake task & give it low priority, then how it will work. Will it run along with another queues or wait for them to execute first?
Delayed::Worker.queue_attributes = {
high_priority: { priority: -10 },
low_priority: { priority: 10 }
}
This article could help you further - it also has coding examples that you can use to try with your own code:
https://phil.tech/2016/tips-on-sidekiq-queues/
Basically, if you want to run the request from users first, you should make a queue for them and assign it a higher priority. No need to assign -10, just start from 0 and work your way up.
But also keep in mind that you should not have more than a handful queues, so it might make sense to bundle different priorities together:
I don't recommend having more than a handful of queues. Lots of queues makes for a more complex system and Sidekiq Pro cannot reliably handle multiple queues without polling. M Sidekiq Pro processes polling N queues means O(M*N) operations per second slamming Redis.
https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
Finally, if one part of your code has to run before another, you could also enqueue the rake task from the job that handles the user requests.
But I want to run the request from user first & then the rake task.
How can I manage this ?
Recommend change Delayed::Worker.default_priority to a positive number, for example: 5, then define a priority method in the user workers return a lower number
class UserRequestJob < ApplicationJob
def priority
1 # lower numbers run first
end
...
end
I also want to understand if I create a separate queue for rake task &
give it low priority, then how it will work. Will it run along with
another queues or wait for them to execute first?
It will run higher priority jobs first, and then lower priority jobs, in fact queue name doesn't matter after a job is enqueued, unless you have dedicated workers, when you define queues with priority:
Delayed::Worker.queue_attributes = {
high_priority: { priority: -10 },
low_priority: { priority: 10 }
}
and then you enqueue a job to a queue, delayed_job will get the priority by queue name, as you can see in job_preparer.rb
References:
https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html
https://github.com/collectiveidea/delayed_job#named-queues
I have a task that I need to generate immediately after the request is created and get it done ASAP.
So for this purpose, I have created a /config/sidekiq.yml file where I defined this:
---
:queues:
- default
- [critical, 10]
And for the respective worker, I set this:
class GeneratePDFWorker
include Sidekiq::Worker
sidekiq_options queue: 'critical', retry: false
def perform(order_id)
...
Then, when I call this worker:
GeneratePDFWorker.perform_async(#order.id)
So I am testing this. But - I found this post, where is said that if I want to execute the tasks immediately, I should call:
GeneratePDFWorker.new.perform(#order.id)
So my question is - should I use the combination of a (critical) queue + the new (GeneratePDFWorker.new.perform) method? Does it make sense?
Also, how can I verify that the tasks is execute as critical?
Thank you
So my question is - should I use the combination of a (critical) queue + the new (GeneratePDFWorker.new.perform) method? Does it make sense?
Using GeneratePDFWorker.new.perform will run the code right there and then, like normal, inline code (in a blocking manner, not async). You can't define a queue, because it's not being queued.
As Walking Wiki mentioned, GeneratePDFWorker.new.perform(#order.id) will call the worker synchronously. So if you did this from a controller action, the request would block until the perform method completed.
I think your approach of using priority queues for critical tasks with Sidekiq is the way to go. As long as you have enough Sidekiq workers, and your queue isn't backlogged, the task should run almost immediately so the benefit of running your worker in-process is pretty much nil. So I'd say yes, it does make sense to queue in this case.
Also, you're probably aware of this, but sidekiq has a great monitoring UI: https://github.com/mperham/sidekiq/wiki/Monitoring. This should should make it easy to get reliable, detailed metrics on the performance of your workers.
should I use the combination of a (critical) queue?
Me:
Yes you can use critical queue if you feel so. A queue with a weight of 2 will be checked twice as often as a queue with a weight of 1.
Tips:
Keep the number of queues fewer as possible. Sidekiq is not designed to handler tremendous number of queues.
Also keep weights as simple as possible. If you want queues always processed in a specific order, just declare them in order without weights.
the new (GeneratePDFWorker.new.perform) method?
Me: No, using sidekiq in the same thread asynchronously is bad in the first place. This will hamper your application's performance as your application-server will be busy for longer. This will be very expensive for you. Then what will be the point of using sidekiq?
Currently I am working with queue job on the ruby on rail with the Sidekiq. I have 2 jobs that are depend to each other and I want 1st job to finish first before starting the 2nd job, so is there any way to make it with Sidekiq.
Yes, you can use the YourSidekiqJob.new.perform(parameters_to_the_job) pattern. This will run your jobs in order, synchronously.
However, there are 2 things to consider here:
What happens if the first job fails?
How long does the each job run?
For #2, the pattern blocks execution for the length of time each job takes to run. If the jobs are extremely short in runtime, why use the jobs in the first place? If they're long, are you expecting the user to wait until they're done?
Alternatively, you can schedule the running of the second job as the last line in the body of the first one. You still need to account for the failure mode of job #1 or #2. Also, you need to consider that the job won't necessarily run when it's scheduled to run, due to the state of the queue at schedule time. How does this affect your business logic?
Hope this helps
--edit according to last comment
class SecondJob < SidekiqJob
def perform(params)
data = SomeData.find
return unless data.ready?
# do whatever you need to do with the ready data
end
end
In my process, i need to allow a user to execute the same task ( with different params ) multiple times.
Showing TaskService class i observe that there is just complete method.
The complete method set that task to complete and user cannot execute it again.
Is there a solution to that?
Thanks!
A task can only be executed once. However, your process could allow the same task (node) being triggered multiple times, for example use an intermediate signal start event leading to your task, where depending on the data sent alongside the signal, the task will be triggered. The process could allow retriggering the same task multiple times (during a certain phase of the process for example).
Is there anyway to move a resque job between two different queues?
We sometimes get in the situation that we have a big queue and a job that is near the end we find a need to "bump up its priority." We thought it might be an easy way to simply move it to another queue that had a worker waiting for any high priority jobs.
This happens rarely and is usually a case where we get a special call from a customer, so scaling, re-engineering don't seem totally necessary.
There is nothing built-in in Resque. You can use rpoplpush like:
module Resque
def self.move_queue(source, destination)
r = Resque.redis
r.llen("queue:#{source}").times do
r.rpoplpush("queue:#{source}", "queue:#{destination}")
end
end
end
https://gist.github.com/rafaelbandeira3/7088498
If it's a rare occurrence you're probably better off just manually pushing a new job into a shorter queue. You'll want to make sure that your system has a way to identify that the job has already run and to bail out so that when the job in the long queue is finally reached it is not processed again (if double processing is a problem for you).