according to apple's doc, tasks are still started in the order in which they were added to a concurrent queue, so it seems to me that there is no difference between a concurrent queue vs a serial queue where all its task are run by async.
correct me if I miss something
read bunch of documents and did not find the answer
The difference is how many tasks may run at the same time:
A serial queue only processes (runs) one task at time, one after another. A concurrent queue may process multiple tasks (on multiple thread) at the same time.
You typically use a serial queue to ensure only one task is accessing a resource at the same time. These are scenarios where you would traditionally use mutexes.
If you have tasks that would benefit from (and are able to) running concurrently at the same time, or tasks that are completely independent and thus don't care, you usually use a concurrent queue.
Related
I've been using a DispatchGroup to facilitate a number of concurrent calls in my app.
My backend team noticed that when I tried to make eight concurrent calls, they were separated into two batches of four calls.
Is four concurrent calls the limit for GCD?
Is this a limitation of the GCD framework, or is this dependent on the hardware?
Is there a way to allow for more concurrent calls?
From the reference for GCD:
Concurrent queues (also known as a type of global dispatch queue)
execute one or more tasks concurrently, but tasks are still started in
the order in which they were added to the queue. The currently
executing tasks run on distinct threads that are managed by the
dispatch queue. The exact number of tasks executing at any given point
is variable and depends on system conditions.
The system automatically (and dynamically) determines how many tasks to execute simultaneously. Among other things it's based on battery state, # of cores, system load, etc.
See Dispatch Queues for more information.
Say for example, I have two queues and I want each queue to process only one job simultaneously.
Now I have one queue and I have it something like
bundle exec sidekiq -c 1 -q queue_name
I want two queues to process the job simultaneously and each queue should have concurrency 1. So, is that possible? If yes, how can I do that?
Queues are just places from where sidekiq takes jobs to perform. Concurrency allows to perform taken jobs simultaneously. I mean queues don't process jobs, they just collect them and determine their priority. Therefore the phrase set the concurrency for each queue doesn't make sense.
I have the following tasks to do in a rails application:
Download a video
Trim the video with FFMPEG between a given duration (Eg.: 00:02 - 00:09)
Convert the video to a given format
Move the converted video to a folder
Since I wanted to make this happen in background jobs, I used 1 resque worker that processes a queue.
For the first job, I have created a queue like this
#queue = :download_video that does it's task, and at the end of the task I am going forward to the next task by calling Resque.enqueue(ConvertVideo, name, itemId). In this way, I have created a chain of queues that are enqueued when one task is finished.
This is very wrong, since if the first job starts to enqueue the other jobs (one from another), then everything get's blocked with 1 worker until the first list of queued jobs is finished.
How should this be optimised? I tried adding more workers to this way of enqueueing jobs, but the results are wrong and unpredictable.
Another aspect is that each job is saving a status in the database and I need the jobs to be processed in the right order.
Should each worker do a single job from above and have at least 4 workers? If I double the amount to 8 workers, would it be an improvement?
Have you considered using sidekiq ?
As said in Sidekiq documentation :
resque uses redis for storage and processes messages in a single-threaded process. The redis requirement makes it a little more difficult to set up, compared to delayed_job, but redis is far better as a queue than a SQL database. Being single-threaded means that processing 20 jobs in parallel requires 20 processes, which can take a lot of memory.
sidekiq uses redis for storage and processes jobs in a multi-threaded process. It's just as easy to set up as resque but more efficient in terms of raw processing speed. Your worker code does need to be thread-safe.
So you should have two kind of jobs : download videos and convert videos and any download video job should be done in parallel (you can limit that if you want) and then each stored in one queue (the "in-between queue") before being converted by multiple convert jobs in parallel.
I hope that helps, this link explains quite well the best practices in Sidekiq : https://github.com/mperham/sidekiq/wiki/Best-Practices
As #Ghislaindj noted Sidekiq might be an alternative - largely because it offers plugins that control execution ordering.
See this list:
https://github.com/mperham/sidekiq/wiki/Related-Projects#execution-ordering
Nonetheless, yes, you should be using different queues and more workers which are specific to the queue. So you have a set of workers all working on the :download_video queue and then you other workers attached to the :convert_video queue, etc.
If you want to continue using Resque another approach would be to use delayed execution, so when you enqueue your subsequent jobs you specify a delay parameter.
Resque.enqueue_in(10.seconds, ConvertVideo, name, itemId)
The down-side to using delayed execution in Resque is that it requires the resque-scheduler package, so you're introducing a new dependency:
https://github.com/resque/resque-scheduler
For comparison Sidekiq has delayed execution natively available.
Have you considered merging all four tasks into just one? In this case you can have any number of workers, one will do the job. It will work very predictable, you can even know how much time will take to finish the task. You also don't have problems when one of the subtasks takes longer than all others and it piles up in the queue.
I have several concurrent asynchronous network operations, and want to be notified after all of them finished receiving the data.
The current thread shall not be blocked, synchronous connections aren't an option, and all operations shall be executed concurrently.
How can I achieve that? Or is it more performant to execute all network operations successively, particularly looking at mobile devices?
Is it possible to process jobs from a Sidekiq queue ONLY if all other queues are empty?
For example, say we have a photos queue and a updates queue. I only want to process photos if updates is free of pending jobs.
Is that possible?
Well, all you queues execute in parallel, so I don't get the idea of executing consequentially.
But you have several options to play with:
you can make more concurrent workers
you can set frequency higher to updates queue, so updates worker will check for updates more frequently then photo worker.
Take a look at this options in doc