I need to make some API calls and I want to ensure that they come back in the order that they went out. Is this the proper flow to have that happen?
Create NSOperationQueue, set max concurrent operations to 1
Create URL String to API
Create NSOperation block, call method to call API, pass URL string
Add NSOperation to NSOperationQueue
This is where I get confused. Setting the max concurrent operations to 1 essentially makes NSOperationQueue into a synchronous queue, only 1 operation gets called at a time. However, each operation is going to make a NSURLSession call, which is async. How can I ensure that the next operation doesn't run until I have finished with the first? (By finish I want to store the returned JSON in a NSArray, adding each additional returned JSON to that array).
The proper way to ensure that NSOperations run in order is to add dependencies. Dependencies are powerful as they allow ordering of different operations on different queues. You can make an API call or data processing on a background queue; when complete, a dependent operation can update the UI on the main thread.
let operation1 = NSBlockOperation {
print("Run First - API Call")
}
let operation2 = NSBlockOperation {
print("Run Second - Update UI")
}
operation2.addDependency(operation1)
let backgroundQueue = NSOperationQueue()
backgroundQueue.addOperation(operation1)
NSOperationQueue.mainQueue().addOperation(operation2)
// operation1 will finish before operation2 is called, regardless of what queue they're in
See Apple Docs on addDependency in NSOperation here: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/NSOperation_class/index.html#//apple_ref/occ/cl/NSOperation
Also, be careful with assuming that maxConcurrentOperationCount = 1 as all that it does is ensure that only 1 operation will run at a time. This does NOT ensure the order of the queue. An operation with a higher priority will likely run first.
I'm writing an app where I've got a long running server-syncronization task running in the background, and I'd like to use NSOperation and NSOperationQueue for this. I'm leaning this way, since I need to ensure only one synchronisation operation is running at once.
My question arises since my architecture is built around NSNotifications; my synchronisation logic proceeds based on these notifications. From what I can see, NSOperation logic needs to be packed into the main method. So what I'm wondering is if there is any way to have an NSOperation finish when a certain notification is received. I suspect this is not the case, since I haven't stumbled upon any examples of this usage, but I figured I'd ask the gurus in here. Does an NSOperation just finish when the end of the main method is reached?
There is no reason a NSOperation cannot listen for a notification on the main thread, but either the finish logic must be thread safe, or the operation must keep track of its current thread.
I would recommend a different approach. Subclass NSOperation to support a method like -finishWithNotification: Have a queue manager that listens for the notification. It can iterate through its operations finishing any operations which respond to -finishWithNotification:.
- (void)handleFinishNotification:(NSNotification *)notification
{
for (NSOperation *operation in self.notificationQueue) {
if ([operation isKindOfClass:[MYOperation class]]) {
dispatch_async(self.notificationQueue.underlyingQueue), ^{
MYOperation *myOperation = (MYOperation *)operation;
[myOperation finishWithNotification:notification];
});
}
}
}
If I understood you correctly concurrent NSOperation is what you need.
Concurrent NSOperation are suitable for long running background/async tasks.
NSOperation Documentation
See: Subclassing Notes & Operation Dependencies Section
EDIT:(Adding more explanation)
Basically concurrent operations do not finish when main method finishes. Actually what concurrent operation mean is that the control will return to calling code before the actual operation finishes. The typical tasks that are done in start method of concurrent operation are: Mark operation as executing, start the async work(e.g. NSURLConnection async call) or spawn a new thread which will perform bulk of the task. And RETURN.
When the async task finishes mark the operation as finished.
I'm using the AFNetworking library, which is excellent, however I'm having trouble keeping track of operations in the NSOperationQueue. I am adding NSOperation objects to the NSOperationQueue, and I need to keep track of progress - so update a UIProgressView to show how far the queue is to completion and then also execute a block of code once the queue is complete.
I've tried KVO - using the answer here: Get notification when NSOperationQueue finishes all tasks however I come across the problem (elaborated on the second answer down there) where sometimes operations in the queue may complete fast enough to temporarily decrement the operationCount property to 0 - which then cause issues with the code in the accepted answer - i.e. prematurely execute the code to be executed after all objects in the queue have finished and progress tracking will not be accurate as a result.
A variation I've tried is checking for operationCount == 0 in the success block of each NSOperation that I add to the NSOperationQueue and then executing code based on that, e.g.
[AFImageRequestOperation *imgRequest = [AFImageRequestOperation imageRequestOperationWithRequest:urlRequest success:^(UIImage *image) {
//Process image & save
if(operationQ.operationCount == 0){
// execute completion of Queue code here
}
else {
// track progress of the queue here and update UIProgressView
}
}];
However, I come up with the same issue as I do with KVO.
I've thought about using GCD with a dispatch queue using a completion block - so asynchronously dispatch an NSOperationQueue and then execute the completion block but that doesn't solve my issue with regard to keeping track of the queue progress to update UIProgressView.
Also not used
AFHttpClient enqueueBatchOfHTTPRequestOperations:(NSArray *) progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations)progressBlock completionBlock:^(NSArray *operations)completionBlock
since my images are coming from a few different URLs (rather than one base url).
Any suggestions or pointers will be appreciated. Thanks.
Just a final update:
Solved this issue using the AFHTTPClient enqueueBatchOfHTTPRequestOperations in the end with the help of Matt (see accepted answer) and note the comments as well.
I did come across another solution that does not make use of AFHTTPClient but just NSOperationQueue on its own. I've included this as well in case it's of any use to anyone, but if you're using the AFNetworking Library I'd recommend the accepted answer (since it's most elegant and easy to implement).
AFHTTPClient -enqueueBatchOfHTTPRequestOperations:progressBlock:completionBlock: is the correct way to do this. The method takes an array of request operations, which can be constructed from any arbitrary requests—not just ones sharing a domain.
Another (not as elegant) solution, if you're only using NSOperationQueue and not the AFHTTPClient, is the following (assuming the following code will be in some loop to create multiple requests and add to the NSOperationQueue).
[AFImageRequestOperation *imgRequest = [AFImageRequestOperation imageRequestOperationWithRequest:urlRequest success:^(UIImage *image) {
//Process image & save
operationNum++
//initially operationNum set to zero, so this will now increment to 1 on first run of the loop
if(operationNum == totalNumOperations){
//totalNumOperations would be set to the total number of operations you intend to add to the queue (pre-determined e.g. by [array count] property which would also be how many times the loop will run)
// code to execute when queue is finished here
}
else {
// track progress of the queue here and update UIProgressView
float progress = (float)operationNum / totalNumOperations
[progView setProgress:progress] //set the UIProgressView.progress property
}
}];
Adding these NSOperation objects to the NSOperationQueue will ensure the success block of each operation will complete before executing the queue completion code which is embedded in the success block of each NSOperation object. Note NSOperationQueue.operationCount property isn't used since it is not reliable on fast operations since there may be an state in between an operation exiting a queue and just before the next one is added where the operationCount is zero and so if we compared NSOperationQueue.operationCount = 0 instead then the completion code for the queue would execute prematurely.
I have a method that at times can be invoked throughout my code. Below is a very basic example, as the code processes images and files off of the iphone photo gallery and marks them already processed when done with the method.
#property (nonatomic, assign) dispatch_queue_t serialQueue;
....
-(void)processImages
{
dispatch_async(self.serialQueue, ^{
//block to process images
NSLog(#"In processImages");
....
NSLog(#"Done with processImages");
});
}
I would think that each time this method is called I would get the below output...
"In processImages"
"Done with processImages"
"In processImages"
"Done with processImages"
etc...
but I always get
"In processImages"
"In processImages"
"Done with processImages"
"Done with processImages"
etc...
I thought a serial queue would wait till the first block is done, then start. To me it seems it is starting the method, then it gets called again and starts up before the first call even finishes, creating duplicates of images that normally would not be processed due to the fact that if it really executed serially the method would know they were already processed. Maybe my understanding of serial queues is not concrete. Any input? Thank you.
EDIT:MORE Context below, this is what is going on in the block...Could this cause the issue???
#property (nonatomic, assign) dispatch_queue_t serialQueue;
....
-(void)processImages
{
dispatch_async(self.serialQueue, ^{
//library is a reference to ALAssetsLibrary object
[library enumerateGroupsWithTypes:ALAssetsGroupSavedPhotos usingBlock:^(ALAssetsGroup *group, BOOL *stop)
{
[group enumerateAssetsUsingBlock:^(ALAsset *asset, NSUInteger index, BOOL *stop)
{
....
//Process the photos here
}];
failureBlock:^(NSError *error) { NSLog(#"Error loading images from library");
}];
});
}
-(id)init
{
self = [super init];
if(self)
{
_serialQueue = dispatch_queue_create("com.image.queue",NULL);
}
return self;
}
this object is only created once, and as far as I can tell can never be created again based off my code...I will run tests to make sure though.
UPDATE 2: WHAT I THINK IS HAPPENING, please comment on this if you agree/disagree....
Obviously my main issue is that it seems this block of code is being executed concurrently, creating duplicate entries (importing the same photo twice) when it wouldn't normally do this if it was run serially. When a photo is processed a "dirty" bit is applied to it ensuring the next time the method is invoked it skips this image, but this is not happening and some images are processed twice. Could this be due to the fact I am enumerating the objects in a second queue using enumerategroupswithtypes: within that serialQueue?
call processImages
enumerateObjects
immediately return from enumerateObjects since it is async itself
end call to processImages
processImages is not really done though due to the fact that enumerategroups is probably still running but the queue might thing it is done since it reaches the end of the block before enumerategroups is finished working. This seems like a possibility to me?
Serial Queues ABSOLUTELY will perform serially. They are not guaranteed to perform on the same thread however.
Assuming you are using the same serial queue, the problems is that NSLog is NOT guaranteed to output results in the proper order when called near simultaneously from different threads.
here is an example:
SQ runs on thread X, sends "In processImages"
log prints "In proc"
SQ on thread X, sends "Done with processImages"
SQ runs on thread Y, sends "In processImages"
log prints "essImages\n"
After 5., NSLog doesn't necessarily know which to print, 3. or 4.
If you absolutely need time ordered logging, You need a dedicated queue for logging. In practice, I've had no problems with just using the main queue:
dispatch_async(dispatch_get_main_queue(), ^{
NSLog(#"whatever");
});
If all NSlog calls are the on the same queue, you shouldn't have this problem.
enumerateGroupsWithTypes:usingBlock:failureBlock: does its work asynchronously on another thread and calls the blocks passed in when it's done (on the main thread I think). Looking at it from another perspective, if it completed all the synchronously by the time the method call was complete, it could just return an enumerator object of the groups instead, for instance, for a simpler API.
From the documentation:
This method is asynchronous. When groups are enumerated, the user may be asked to confirm the application's access to the data; the method, though, returns immediately. You should perform whatever work you want with the assets in enumerationBlock.
I'm not sure why you're trying to accomplish by using the serial queue, but if you just want to prevent simultaneous access, then you could just add a variable somewhere that keeps track of whether we're currently enumerating or not and check that at first, if you don't have to worry about synchronization issues. (If you do, perhaps you should look into using a GCD group, but it's probably overkill for this situation.)
If the question is "Can serial queue perform tasks asynchronously?" then the answer is no.
If you think that it can, you should make sure that all tasks are really performing on the same queue. You can add the following line in the block and compare the output:
dispatch_async(self.serialQueue, ^{
NSLog(#"current queue:%p current thread:%#",dispatch_get_current_queue(),[NSThread currentThread]);
Make sure that you write NSLog in the block that performs on your queue and not in the enumerateGroupsWithTypes:usingBlock:failureBlock:
Also you can try to create your queue like this
dispatch_queue_create("label", DISPATCH_QUEUE_SERIAL);
but I don't think that will change anything
EDIT:
By the way, method
enumerateGroupsWithTypes:usingBlock:failureBlock:
is asynchronous, why do you call it on another queue?
UPDATE 2:
I can suggest something like this:
dispatch_async(queue, ^{
NSLog(#"queue");
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER, *pmutex = &mutex;
pthread_mutex_lock(pmutex);
ALAssetsLibraryGroupsEnumerationResultsBlock listGroupBlock = ^(ALAssetsGroup *group, BOOL *stop) {
NSLog(#"block");
if (group) {
[groups addObject:group];
} else {
[self.tableView performSelectorOnMainThread:#selector(reloadData) withObject:nil waitUntilDone:NO];
dispatch_async(dispatch_get_current_queue(), ^{
pthread_mutex_unlock(pmutex);
});
}
NSLog(#"block end");
};
[assetsLibrary enumerateGroupsWithTypes:groupTypes usingBlock:listGroupBlock failureBlock:failureBlock];
pthread_mutex_lock(pmutex);
pthread_mutex_unlock(pmutex);
pthread_mutex_destroy(pmutex);
NSLog(#"queue end");
});
I hit an issue like this, and the answer for me was to realize that asynchronous calls from a method on the serialized queue goes to another queue for processing -- one that is not serialized.
So you have to wrap all the calls inside the main method with explicit dispatch_async(serializedQueue, ^{}) to ensure that everything is done in the correct order...
Using Swift and semaphores to illustrate an approach to serialization:
Given: a class with an asynchronous ‘run’ method that will be run on multiple objects at once, and the objective is that each not run until the one before it completes.
The issue is that the run method allocates a lot of memory and uses a lot of system resources that can cause memory pressure among other issues if too many are run at once.
So the idea is: if a serial queue is used then only one will run at a time, one after the other.
Create a serial queue in the global space by the class:
let serialGeneratorQueue: DispatchQueue = DispatchQueue(label: "com.limit-point.serialGeneratorQueue", autoreleaseFrequency: DispatchQueue.AutoreleaseFrequency.workItem)
class Generator {
func run() {
asynchronous_method()
}
func start() {
serialGeneratorQueue.async {
self.run()
}
}
func completed() {
// to be called by the asynchronous_method() when done
}
}
The ‘run’ method of this class for which very many objects will be created and run will be processed on the serial queue:
serialGeneratorQueue.async {
self.run()
}
In this case an autoreleaseFrequency is .workItem to clean up memory after each run.
The run method is of some general form:
func run() {
asynchronous_method()
}
The problem with this: the run method exits before the asynchronous_method completes, and the next run method in the queue will run, etc. So the objective is not being achieved because each asynchronous_method is running in parallel, not serially after all.
Use a semaphore to fix. In the class declare
let running = DispatchSemaphore(value: 0)
Now the asynchronous_method completes it calls the ‘completed’ method:
func completed() {
// some cleanup work etc.
}
The semaphore can be used to serialized the chain of asynchronous_method’s by add ‘running.wait()’ to the ‘run’ method:
func run() {
asynchronous_method()
running.wait()
}
And then in the completed() method add ‘running.signal()’
func completed() {
// some cleanup work etc.
running.signal()
}
The running.wait() in ‘run’ will prevent it from exiting until signaled by the completed method using running.signal(), which in turn prevents the serial queue from starting the next run method in the queue. This way the chain of asynchronous methods will indeed be run serially.
So now the class is of the form:
class Generator {
let running = DispatchSemaphore(value: 0)
func run() {
asynchronous_method()
running.wait()
}
func start() {
serialGeneratorQueue.async {
self.run()
}
}
func completed() {
// to be called by the asynchronous_method() when done
running.signal()
}
}
I thought a serial queue would wait [until] the first block is done ...
It does. But your first block simply calls enumerateGroupsWithTypes and the documentation warns us that the method runs asynchronously:
This method is asynchronous. When groups are enumerated, the user may be asked to confirm the application's access to the data; the method, though, returns immediately.
(FWIW, whenever you see a method that has a block/closure parameter, that’s a red flag that the method is likely performing something asynchronously. You can always refer to the relevant method’s documentation and confirm, like we have here.)
So, bottom line, your queue is serial, but it is only sequentially launching a series of asynchronous tasks, but obviously not waiting for those asynchronous tasks to finish, defeating the intent of the serial queue.
So, if you really need to have each tasks wait for the prior asynchronous task, there are a number of traditional solutions to this problem:
Use recursive pattern. I.e., write a rendition of processImage that takes an array of images to process and:
check to see if there are any images to process;
process first image; and
when done (i.e. in the completion handler block), remove the first image from the array and then call processImage again.
Rather than dispatch queues, consider using operation queues. Then you can implement your task as an “asynchronous” NSOperation subclass. This is a very elegant way of wrapping an asynchronous task This is illustrated in https://stackoverflow.com/a/21205992/1271826.
You can use semaphores to make this asynchronous task behave synchronously. This is also illustrated in https://stackoverflow.com/a/21205992/1271826.
Option 1 is the simplest, option 2 is the most elegant, and option 3 is a fragile solution that should be avoided if you can.
You might have more than one object, each with its own serial queue. Tasks dispatched to any single serial queue are performed serially, but tasks dispatched to different serial queues will absolutely be interleaved.
Another simple bug would be to create not a serial queue, but a concurrent queue...
I've got a bunch of NSOperations added to a NSOperationQueue. The operation queue has the maxConcurrentOperationCount set to 1, so that the NSOperations run one after the other.
Now, in the completionBlock of a NSOperation I want to cancel all pending NSOperations by calling cancelAllOperations on the NSOperationQueue.
Is it safe to do this? Can I be sure that the start-method of the next operation is called only after the completionBlock of the previous operation has been fully executed? Or do the completionBlock of the previous operation and the task of the current operation run concurrently?
The reason why I'm asking: I use AFNetworking to execute a batch of AFHTTPRequestOperations and want to perform one request only if all previous requests of the batch were successful.
My findings below no longer seem to be true. I've re-run the tests on iOS 8 and iOS 9 and the completion block of an operation always runs concurrently with the next operation. Currently, I don't see a way to make an operation wait for the previous completion block to finish.
I just tried this scenario in a sample project. Here is the result:
If the NSOperationQueue's maxConcurrentOperationCount is set to 1, an NSOperation's completionBlock and the next NSOperation in the queue run simultaneously.
But, if every NSOperation is linked to its previous operation by calling addDependency:, the execution of an operation waits until the previous operation's completionBlock has finished.
So, if you want to cancel the next operation in the completionBlock of the current operation and be sure that it is cancelled before it is started, you have to set dependencies between the NSOperations by calling addDependency:
NSOperation establishes dependency only based on the completion states of operations, and not on the results of completed operations.
However, most of the scenarios that I encounter are such that, the execution of operations depend not only on the completion of some other operations, but also based on the results obtained from the completed operations.
I ended up doing like the below method, but still exploring if there is a better way:
1) Operation-A runs
2) Operation-A compeletes and its completionBlock runs
3) In the OperationA's completion block, check for the result obtained from Operation-A.
If result is X, create Operation-B and add to the queue.
If result is Y, create Operation-C and add to the queue.
If result is error, create Operation-D (usually an alert operation) and add to the queue
So, this ends up as a sequence of operations, that are dynamically added to the queue, depending on the result of completed operations.
I came up with another seemingly better way to ensure that an operaion is executed only if certain conditions (based on the results of previously finished operations) are met, else, the operation is cancelled.
One important consideration here is that the condition check for running an operation should not be coded inside the operation subclass, thus allowing the operation subclass to be poratble across different scenarios and apps.
Solution:
- Have a condition block property inside the subclass, and set whatever condition form where the operation is instantiated.
- Override "isReady" getter of the NSOperation subclass, check the condition there, and thus determine if its ready for execution.
- If [super isReady] is YES, which means the dependent operations are all finished, then evaluate the necessary condition.
- If the condition check is passed, return YES. Else, set isCancelled to YES and return YES for isReady
Code:
In the interface file have the block property:
typedef BOOL(^ConditionBlock)(void);
#property (copy) ConditionBlock conditionBlock;
In the implementation, override isReady, and cancelled:
#implementation ConditionalOperation
- (BOOL)isReady {
if([super isReady]) {
if(self.conditionBlock) {
if(!self.conditionBlock()) {
[self setCancelled:YES];
}
return YES;
} else {
return YES;
}
} else {
return NO;
}
}