fetchMessagesByNumberOperationWithFolder(path, requestKind: requestKind, numbers: indexSet)
From my understanding, the queue of "MCOIMAPFetchMessagesOperation" are managed by the imapSession.
Is it possible / recommended to have 2 MCOIMAPFetchMessagesOperation running concurrently to download messages more quickly?
I've read this:
Running simultaneous operations in Mailcore2
but it seems like the question is running simultaneous but different operations concurrently whereas what I want to do is run simultaneous operation of the same kind to speed up the whole process
Related
according to apple's doc, tasks are still started in the order in which they were added to a concurrent queue, so it seems to me that there is no difference between a concurrent queue vs a serial queue where all its task are run by async.
correct me if I miss something
read bunch of documents and did not find the answer
The difference is how many tasks may run at the same time:
A serial queue only processes (runs) one task at time, one after another. A concurrent queue may process multiple tasks (on multiple thread) at the same time.
You typically use a serial queue to ensure only one task is accessing a resource at the same time. These are scenarios where you would traditionally use mutexes.
If you have tasks that would benefit from (and are able to) running concurrently at the same time, or tasks that are completely independent and thus don't care, you usually use a concurrent queue.
For waiting from future completion in Dask distributed cluster, what's the difference between these two APIs? Are there any?
wait: https://docs.dask.org/en/latest/futures.html#waiting-on-futures
result(): tttps://docs.dask.org/en/latest/futures.html#distributed.Future.result
If there's any difference, what would be the more efficient way to block until result is available?
Thanks!
wait blocks further execution until the futures are completed, and once they are, the code proceeds. result transfers the result of the future from the worker to the client computer. In most cases, it’s probably more efficient to leave future with workers until the client needs them.
For example, imagine that you are coordinating calculations using a small laptop with 10GB ram which is connected to a cluster that has workers with memory of 50GB each. If the data you are processing is around 20GB, then the workers will have no problem doing calculations, however if you try to use .result() with the intention to just wait for execution to complete, then the workers will try to send to you 20GB of data each, which will crash your laptop session.
I've been using a DispatchGroup to facilitate a number of concurrent calls in my app.
My backend team noticed that when I tried to make eight concurrent calls, they were separated into two batches of four calls.
Is four concurrent calls the limit for GCD?
Is this a limitation of the GCD framework, or is this dependent on the hardware?
Is there a way to allow for more concurrent calls?
From the reference for GCD:
Concurrent queues (also known as a type of global dispatch queue)
execute one or more tasks concurrently, but tasks are still started in
the order in which they were added to the queue. The currently
executing tasks run on distinct threads that are managed by the
dispatch queue. The exact number of tasks executing at any given point
is variable and depends on system conditions.
The system automatically (and dynamically) determines how many tasks to execute simultaneously. Among other things it's based on battery state, # of cores, system load, etc.
See Dispatch Queues for more information.
I have the following tasks to do in a rails application:
Download a video
Trim the video with FFMPEG between a given duration (Eg.: 00:02 - 00:09)
Convert the video to a given format
Move the converted video to a folder
Since I wanted to make this happen in background jobs, I used 1 resque worker that processes a queue.
For the first job, I have created a queue like this
#queue = :download_video that does it's task, and at the end of the task I am going forward to the next task by calling Resque.enqueue(ConvertVideo, name, itemId). In this way, I have created a chain of queues that are enqueued when one task is finished.
This is very wrong, since if the first job starts to enqueue the other jobs (one from another), then everything get's blocked with 1 worker until the first list of queued jobs is finished.
How should this be optimised? I tried adding more workers to this way of enqueueing jobs, but the results are wrong and unpredictable.
Another aspect is that each job is saving a status in the database and I need the jobs to be processed in the right order.
Should each worker do a single job from above and have at least 4 workers? If I double the amount to 8 workers, would it be an improvement?
Have you considered using sidekiq ?
As said in Sidekiq documentation :
resque uses redis for storage and processes messages in a single-threaded process. The redis requirement makes it a little more difficult to set up, compared to delayed_job, but redis is far better as a queue than a SQL database. Being single-threaded means that processing 20 jobs in parallel requires 20 processes, which can take a lot of memory.
sidekiq uses redis for storage and processes jobs in a multi-threaded process. It's just as easy to set up as resque but more efficient in terms of raw processing speed. Your worker code does need to be thread-safe.
So you should have two kind of jobs : download videos and convert videos and any download video job should be done in parallel (you can limit that if you want) and then each stored in one queue (the "in-between queue") before being converted by multiple convert jobs in parallel.
I hope that helps, this link explains quite well the best practices in Sidekiq : https://github.com/mperham/sidekiq/wiki/Best-Practices
As #Ghislaindj noted Sidekiq might be an alternative - largely because it offers plugins that control execution ordering.
See this list:
https://github.com/mperham/sidekiq/wiki/Related-Projects#execution-ordering
Nonetheless, yes, you should be using different queues and more workers which are specific to the queue. So you have a set of workers all working on the :download_video queue and then you other workers attached to the :convert_video queue, etc.
If you want to continue using Resque another approach would be to use delayed execution, so when you enqueue your subsequent jobs you specify a delay parameter.
Resque.enqueue_in(10.seconds, ConvertVideo, name, itemId)
The down-side to using delayed execution in Resque is that it requires the resque-scheduler package, so you're introducing a new dependency:
https://github.com/resque/resque-scheduler
For comparison Sidekiq has delayed execution natively available.
Have you considered merging all four tasks into just one? In this case you can have any number of workers, one will do the job. It will work very predictable, you can even know how much time will take to finish the task. You also don't have problems when one of the subtasks takes longer than all others and it piles up in the queue.
I have several concurrent asynchronous network operations, and want to be notified after all of them finished receiving the data.
The current thread shall not be blocked, synchronous connections aren't an option, and all operations shall be executed concurrently.
How can I achieve that? Or is it more performant to execute all network operations successively, particularly looking at mobile devices?