I have 100+ request.I need send a new request when the last one is done,so the server will not return error code - 429.
How to make this by afnetworking 3.0?
I'm not very familiar with the specific APIs of AFNetworking, but you could setup:
A variable array containing all your pending requests,
A method called (e.g.) sendNext() that removes the first entry of the array, performs the request asynchronously, and inside the completion block, calls itself.
Of course, you will need a terminating condition, and that is simply stop when the array becomes empty.
There are 2 approaches that can deal with your problem.
Firstly, create an operation queue and add all requests to the queue. After that, create an operation of your new request, then add the dependency to all requests in the queue. As a result, your new operation (will execute the new request) will be performed after the last request is done.
Secondly, you can use dispatch_barrier_async, which will create a synchronized point on your concurrent queue. That means you should create a concurrency queue to execute your 100+ request, and that dispatch_barrier_async blocks in your custom queue will execute the new request.
Thanks Sendoa for the link to the GitHub issue where Mattt explains why this functionality is not working anymore. There is a clear reason why this isn't possible with the new NSURLSession structure; Tasks just aren't operations, so the old way of using dependencies or batches of operations won't work.
I've created this solution using a dispatch_group that makes it possible to batch requests using NSURLSession, here is the (pseudo-)code:
// Create a dispatch group
dispatch_group_t group = dispatch_group_create();
for (int i = 0; i < 10; i++) {
// Enter the group for each request we create
dispatch_group_enter(group);
// Fire the request
[self GET:#"endpoint.json"
parameters:nil
success:^(NSURLSessionDataTask *task, id responseObject) {
// Leave the group as soon as the request succeeded
dispatch_group_leave(group);
}
failure:^(NSURLSessionDataTask *task, NSError *error) {
// Leave the group as soon as the request failed
dispatch_group_leave(group);
}];
}
// Here we wait for all the requests to finish
dispatch_group_notify(group, dispatch_get_main_queue(), ^{
// Do whatever you need to do when all requests are finished
});
I want to look write something that makes this easier to do and discuss with Matt if this is something (when implemented nicely) that could be merged into AFNetworking. In my opinion it would be great to do something like this with the library itself. But I have to check when I have some spare time for that.
This question possible duplicate of AFNetworking 3.0 AFHTTPSessionManager using NSOperation . You can follow #Darji comment for few call, For 100+ call add these utility classes https://github.com/robertmryan/AFHTTPSessionOperation/ .
It is very impractical approach to sent 100+ request operation concurrently. If possible try to reduce it.
Related
I am having trouble coming up with an efficient way of deleting information from a SQLite DB after a JSON response.
Basically how what I am trying to do is:
I have a DB with 100 records.
I read the first record
Build a JSON package and send it to my API.
WAIT for the API to respond Ok or Fail
The delete the record from the DB on OK.
Move to the next record.
There is some time between waiting on a response before I can delete and go on.
A simple FOR doesn't seem to work as it can't wait for the 'OK' from the API.
I looked into dispatch_async(dispatch_get_main_queue()); but if the OK comes and the DB is processing it will lock the DB while it is being accessed and the logic fails because the db is locked.
I need a mixture of the two. Does anyone know of a tutorial or where to start with something like this?
Use an operation queue and your own custom operation. Setup the operation queue to be serial or set it to allow some small number of concurrent operations.
Loop through each record creating an operation for each record. Put each operation on the queue.
The implementation of the operation will be to make the API call and then delete the one record if that API call indicates it should be deleted. Ensure that the operation's main doesn't reach its end until the API call is complete.
Create an array where you will hold temporarily your data to delete
NSMutableArray* itemsToDelete = [[NSMutableArray alloc] init]
Once you received a response with success you append the items to be deleted to this array. do the same process in all calls with success. once your done with uploading your data at your last response of success or failure check your itemsToDelete array inside a for loop and delete everything from your DB
I suggest using blocks to receive the response from async calls. In that case, you can perform the DB operation within the block.
dispatch_async(dispatch_get_main_queue()); is not for the async calls. It is mostly used to update the UI.
All the async calls needs to be done on background thread.
Coming to your problem, you can do something like this:
-(void)checkToDeleteWithCompletion:(MyCompletion)completion{
__block NSMutableArray *responseArray = [NSMutableArray array];
// Create a dispatch group
dispatch_group_t group = dispatch_group_create();
for (int i = 0; i < yourArray.count; i++) {
// Enter the group for each request we create
dispatch_group_enter(group);
[self ..methodToPErformAsynccallWithCOmpleton success:^(BOOL success, id response) {
dispatch_group_leave(group);
[responseArray addObject:response];
} failure:^(NSError *error, NSInteger statusCode) {
dispatch_group_leave(group);
}];
}
// Here we wait for all the requests to finish
dispatch_group_notify(group, dispatch_get_main_queue(), ^{
// Do whatever you need to do when all requests are finished
completion(YES,[responseArray copy]);
});
}
And finally you can use a for loop on responseArray, to delete the unwanted objects.
You can also limit the number of concurrent calls by using :
dispatch_semaphore_create(maxConcurrencyValue)
Or you can use NSOperation as suggested by rmaddy!
I have been using the following structure in my projects for consuming API data:
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(queue, ^{
// SYNCHRONOUS network request
// Data processing
dispatch_async(dispatch_get_main_queue(), ^{
// UI update
});
});
On the other hand, I have seen quite frequently another structure where the network request is asynchronous (i.e. using AFNetworking) and then the data processing and UI update are handled in the completion block (which is not async - I think).
Here is an example of what I am saying:
NSURLRequest *request = [NSURLRequest requestWithURL:url];
AFHTTPRequestOperation *operation = [[AFHTTPRequestOperation alloc] initWithRequest:request];
operation.responseSerializer = [AFJSONResponseSerializer serializer];
[operation setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
// Data processing
// UI update
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
// Error handling
}];
[operation start];
So, my questions are:
In the second structure there the data processing is not being run asynchronously, it is?
Why are we generally encouraged to run async network requests instead of sync ones from async blocks/threads.
Why the second structure is much more known and spread than the first?
Is there something that I am missing?
In my opinion async calls are much more elegant solutions for networking in any language.
Yes data processing is done in main thread in that example. Parsing strings and creating objects in UI isn't really a cpu cycle consuming job. However lets say you download an image and you want to process it, you shouldn't do it in this block (maybe spawn another thread). But image processing is another scope.it isn't related to the network code so network code is working as intended.
With async network calls, you can cancel/pause the request. But in sync network calls you cannot do that.
First of all gcd is new to the objective-c (i am not really sure but it was available after iOS 4. correct me if i am wrong). Before that we were using delegation and it was a lot of boilerplate codes. But with the second approach you can easily manage the networking code.
I hope this helps.
In the second structure there the data processing is not being run
asynchronously, it is?
No its not. Its run on the main thread. But the time consuming task of fetching data from remote server is already done on another thread. Once the data is retrieved, data processing will not take much time generally and can be run on the main thread. In rare cases(i have not come across any such) if data processing itself takes time and appears to block UI, then we can use NSOperations or GCD queues to process them.
This is achieved by making use of blocks which is just like a callback function.
Why the second structure is much more known and spread than the first?
Its easier to implement and also to understand once you know the syntax for blocks ;)
I'm using RestKit 0.20.3 and have some REST operations that needs to be done in a certain order (the response from one REST operation needs to be included in the request parameter mapping for the next).
I tried setting up the queue to handle one operation at a time like this:
RKObjectManager.sharedManager.operationQueue.maxConcurrentOperationCount = 1;
And adding the operations like this:
for (id insertedObject in insertedObjects) {
[RKObjectManager.sharedManager postObject:insertedObject path:nil parameters:nil success:nil failure:nil];
}
But I get an error, because the first operation is not fully completed before the other start.
When inspecting the logs, it seems like it is executed like this:
REST operation 1 - Request mapping
REST operation 2 - Request mapping
REST operation 3 - Request mapping
REST operation 1 - HTTP call and response mapping
REST operation 2 - HTTP call and response mapping
REST operation 3 - HTTP call and response mapping
I have already tried setting operation dependencies, but that does not make a difference.
I need one REST operation to be completed at a time. How do I do this in RestKit?
PROBLEM
RestKit uses multiple NSOperation for one REST operation, so all request mappings will be queued first with the code in the question. So when the first request mapping is executed and queuing the actual HTTP request, it gets queued behind the first two request mapping operations.
SOLUTION
Queue the next operation after the first one finishes.
Example with recursion:
- (void)sync {
NSArray *objectsToPostInOrder = ...;
for (id objectToPost in objectsToPostInOrder) {
[RKObjectManager.sharedManager postObject:objectToPost path:nil parameters:nil success:^(RKObjectRequestOperation *operation, RKMappingResult *mappingResult) {
// Proceed with next if everything went OK
[self sync];
} failure:^(RKObjectRequestOperation *operation, NSError *error) {
// Don't continue posting as they are dependent on each other
[MyHUD showErrorWithStatus:error.localizedDescription];
}];
return;
}
}
}
I managed to make it work by defining a custom asynchronous NSOperation, which uses RestKit's object get and does not indicate it is finished before RestKit's success/failure block is executed. These custom operations are then added into a separate queue (not RestKit's operation's queue) with maxConcurrentOperationCount set to 1, or you can define inter-operation dependencies as you want.
An older question, but a common recurring problem:
This kind of asynchronous problem can be easily solved with a "Promise" (please read more about here in this wiki:Futures and Promises for a general instruction).
A "Promise" represents the eventual result of an asynchronous method. Eventual it becomes either the value you waiting for or an error.
You can "chain" such asynchronous methods in a way that guarantees that the "next" method is only called when the first has been finished successfully. And, there's also a way to "catch" errors that might have been "thrown" from an asynchronous method.
First, you need to wrap your NSOperationthat invokes the request into an asynchronous method that returns a promise:
- (Promise*) performRequestWithParams(NSDictionary* params);
Note, this is an asynchronous method. It returns immediately returning a "pending" promise. The promise is "resolved" eventually when the operation finishes by the NSOperation which you have to implement in the wrapper method.
Now, in order to "continue" with the next operation when the first finishes, you define a continuation using then as shown below:
Promise* promise = [self performRequestWithParams:params];
promise.then(^id(NSArray* result) {
// We enter here, when the 1. REST op finished successfully:
// note: here, `result` is the _result_ of the async operation above
// which was a JSON returned from the service and parsed into an array.
// Obtain a value from the result:
id x = result[0][#"someKey"];
// Create a new JSON representation which is the input for the next REST operation:
NSDictionary* params = ...;
// Now, invoke the 2. REST op asynchronously:
return [self performRequestWithParams:params];
}, nil)
.then(^id(NSArray* result) {
// We enter here, when the 2. REST op finished successfully:
id x = result[0][#"someKey"]; // obtain a value from some dictionary
NSDictionary* params = ...; // create a new JSON
// Now, invoke the next async method:
return [self performRequestWithParams:params];
}, nil)
... // 3., 4., ... add more REST operations
// In case of an error in *any* of the operations above, let us "catch" it here:
.then(nil, ^id(NSError* error){
NSLog(#"Error: %#", error);
});
Note: you can use a NSOperationQueue to control how many requests your program should execute concurrently. The continuations defined with the promise above are orthogonal to the constraints of the NSOperationQueue. That is, you can use the same NSOperationQueue which is configured to execute for example four concurrent operations, and which executes any other unrelated REST operation in parallel without disrupting the control flow of the "serialized" continuations above.
There are a couple of Promise libraries around. I'm the author of one of it. The project is open source available on GitHub: RXPromise.
Works well for me so far.
AFRKHTTPClient *client = [[AFRKHTTPClient alloc] initWithBaseURL:self.baseUrl];
client.operationQueue.maxConcurrentOperationCount = 1;
RKObjectManager *objectManager = [[RKObjectManager alloc] initWithHTTPClient:client];
objectManager.operationQueue.maxConcurrentOperationCount = 1;
[RKObjectManager setSharedManager:objectManager];
I'm using the AFNetworking library, which is excellent, however I'm having trouble keeping track of operations in the NSOperationQueue. I am adding NSOperation objects to the NSOperationQueue, and I need to keep track of progress - so update a UIProgressView to show how far the queue is to completion and then also execute a block of code once the queue is complete.
I've tried KVO - using the answer here: Get notification when NSOperationQueue finishes all tasks however I come across the problem (elaborated on the second answer down there) where sometimes operations in the queue may complete fast enough to temporarily decrement the operationCount property to 0 - which then cause issues with the code in the accepted answer - i.e. prematurely execute the code to be executed after all objects in the queue have finished and progress tracking will not be accurate as a result.
A variation I've tried is checking for operationCount == 0 in the success block of each NSOperation that I add to the NSOperationQueue and then executing code based on that, e.g.
[AFImageRequestOperation *imgRequest = [AFImageRequestOperation imageRequestOperationWithRequest:urlRequest success:^(UIImage *image) {
//Process image & save
if(operationQ.operationCount == 0){
// execute completion of Queue code here
}
else {
// track progress of the queue here and update UIProgressView
}
}];
However, I come up with the same issue as I do with KVO.
I've thought about using GCD with a dispatch queue using a completion block - so asynchronously dispatch an NSOperationQueue and then execute the completion block but that doesn't solve my issue with regard to keeping track of the queue progress to update UIProgressView.
Also not used
AFHttpClient enqueueBatchOfHTTPRequestOperations:(NSArray *) progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations)progressBlock completionBlock:^(NSArray *operations)completionBlock
since my images are coming from a few different URLs (rather than one base url).
Any suggestions or pointers will be appreciated. Thanks.
Just a final update:
Solved this issue using the AFHTTPClient enqueueBatchOfHTTPRequestOperations in the end with the help of Matt (see accepted answer) and note the comments as well.
I did come across another solution that does not make use of AFHTTPClient but just NSOperationQueue on its own. I've included this as well in case it's of any use to anyone, but if you're using the AFNetworking Library I'd recommend the accepted answer (since it's most elegant and easy to implement).
AFHTTPClient -enqueueBatchOfHTTPRequestOperations:progressBlock:completionBlock: is the correct way to do this. The method takes an array of request operations, which can be constructed from any arbitrary requests—not just ones sharing a domain.
Another (not as elegant) solution, if you're only using NSOperationQueue and not the AFHTTPClient, is the following (assuming the following code will be in some loop to create multiple requests and add to the NSOperationQueue).
[AFImageRequestOperation *imgRequest = [AFImageRequestOperation imageRequestOperationWithRequest:urlRequest success:^(UIImage *image) {
//Process image & save
operationNum++
//initially operationNum set to zero, so this will now increment to 1 on first run of the loop
if(operationNum == totalNumOperations){
//totalNumOperations would be set to the total number of operations you intend to add to the queue (pre-determined e.g. by [array count] property which would also be how many times the loop will run)
// code to execute when queue is finished here
}
else {
// track progress of the queue here and update UIProgressView
float progress = (float)operationNum / totalNumOperations
[progView setProgress:progress] //set the UIProgressView.progress property
}
}];
Adding these NSOperation objects to the NSOperationQueue will ensure the success block of each operation will complete before executing the queue completion code which is embedded in the success block of each NSOperation object. Note NSOperationQueue.operationCount property isn't used since it is not reliable on fast operations since there may be an state in between an operation exiting a queue and just before the next one is added where the operationCount is zero and so if we compared NSOperationQueue.operationCount = 0 instead then the completion code for the queue would execute prematurely.
I have an NSOperationQueue that handles importing data from a web server on a loop. It accomplishes this with the following design.
NSURLConnect is wrapped in an NSOperation and added to the Queue
On successful completion of the download (using a block), the data from the request is wrapped in another NSOperation that adds the relevant data to Core Data. This operation is added to the queue.
On successful completion (using another block), (and after a specified delay) I call the method that started it all and return to step 1. Thus, i make another server call x seconds later.
This works great. I'm able to get data from the server and handle everything on the background. And because these are just NSOperations I'm able to put everything in the background, and perform multiple requests at a time. This works really well.
The ONLY problem that I currently have is that I'm unable to successfully cancel the operations once they are going.
I've tried something like the following :
- (void)flushQueue
{
self.isFlushingQueue = YES;
[self.operationQueue cancelAllOperations];
[self.operationQueue waitUntilAllOperationsAreFinished];
self.isFlushingQueue = NO;
NSLog(#"successfully flushed Queue");
}
where self.isFlushingQueue is a BOOL that I use to check before adding any new operations to the queue. This seems like it should work, but in fact it does not. Any ideas on stopping my Frankenstein creation?
Edit (Solved problem, but from a different perspective)
I'm still baffled about why exactly I was unable to cancel these operations (i'd be happy to keep trying possible solutions), but I had a moment of insight on how to solve this problem in a slightly different way. Instead of dealing at all with canceling operations, and waiting til queue is finished, I decided to just have a data structure (NSMutableDictionary) that had a list of all active connections. Something like this :
self.activeConnections = [NSMutableDictionary dictionaryWithDictionary:#{
#"UpdateContacts": #YES,
#"UpdateGroups" : #YES}];
And then before I add any operation to the queue, I simply ask if that particular call is On or Off. I've tested this, and I successfully have finite control over each individual server request that I want to be looping. To turn everything off I can just set all connections to #NO.
There are a couple downsides to this solution (Have to manually manage an additional data structure, and every operation has to start again to see if it's on or off before it terminates).
Edit -- In pursuit of a more accurate solution
I stripped out all code that isn't relevant (notice there is no error handling). I posted two methods. The first is an example of how the request NSOperation is created, and the second is the convenience method for generating the completion block.
Note the completion block generator is called by dozens of different requests similar to the first method.
- (void)updateContactsWithOptions:(NSDictionary*)options
{
//Hard coded for ease of understanding
NSString *contactsURL = #"api/url";
NSDictionary *params = #{#"sortBy" : #"LastName"};
NSMutableURLRequest *request = [self createRequestUsingURLString:contactsURL andParameters:params];
ConnectionCompleteBlock processBlock = [self blockForImportingDataToEntity:#"Contact"
usingSelector:#selector(updateContactsWithOptions:)
withOptions:options andParsingSelector:#selector(requestUsesRowsFromData:)];
BBYConnectionOperation *op = [[BBYConnectionOperation alloc] initWithURLRequest:request
andDelegate:self
andCompletionBlock:processBlock];
//This used to check using self.isFlushingQueue
if ([[self.activeConnections objectForKey:#"UpdateContacts"] isEqualToNumber:#YES]){
[self.operationQueue addOperation:op];
}
}
- (ConnectionCompleteBlock) blockForImportingDataToEntity:(NSString*)entityName usingSelector:(SEL)loopSelector withOptions:(NSDictionary*)options andParsingSelector:(SEL)parseSelector
{
return ^(BOOL success, NSData *connectionData, NSError *error){
//Pull out variables from options
BOOL doesLoop = [[options valueForKey:#"doesLoop"] boolValue];
NSTimeInterval timeInterval = [[options valueForKey:#"interval"] integerValue];
//Data processed before importing to core data
NSData *dataToImport = [self performSelector:parseSelector withObject:connectionData];
BBYImportToCoreDataOperation *importOperation = [[BBYImportToCoreDataOperation alloc] initWithData:dataToImport
andContext:self.managedObjectContext
andNameOfEntityToImport:entityName];
[importOperation setCompletionBlock:^ (BOOL success, NSError *error){
if(success){
NSLog(#"Import %#s was successful",entityName);
if(doesLoop == YES){
dispatch_async(dispatch_get_main_queue(), ^{
[self performSelector:loopSelector withObject:options afterDelay:timeInterval];
});
}
}
}];
[self.operationQueue addOperation:importOperation];
};
}
Cancellation of an NSOperation is just a request, a flag that is set in NSOperation. It's up to your NSOperation subclass to actually action that request and cancel it's work. You then need to ensure you have set the correct flags for isExecuting and isFinished etc. You will also need to do this in a KVO compliant manner. Only once these flags are set is the operation finished.
There is an example in the documentation Concurrency Programming Guide -> Configuring Operations for Concurrent Execution. Although I understand that this example may not correctly account for all multi-threaded edge cases. Another more complex example is provided in the sample code LinkedImageFetcher : QRunLoopOperation
If you think you are responding to the cancellation request correctly then you really need to post your NSOperation subclass code to examine the problem any further.
Instead of using your own flag for when it is ok to add more operations, you could try the
- (void)setSuspended:(BOOL)suspend
method on NSOperationQueue? And before adding a new operation, check if the queue is suspended with isSuspended?