Trying to implement an app which sends offline data stored on local db to web server when connected to internet. I use the code shown below. As far I have tested it works fine, not sure it will work fine for huge number of records. I would like to know whether any tweaking on this code may increase the performance???
NOTE
I know this would be a worst code for offline sync purpose, so trying
to tweak it better.
Its a single way synchronization, from app to server.
-(void)FormatAnswersInJSON {
DMInternetReachability *checkInternet = [[DMInternetReachability alloc] init];
if ([checkInternet isInternetReachable]) {
if ([checkInternet isHostReachable:#"www.apple.com"]) {//Change to domain
responseArray = [[NSMutableArray alloc] init];
dispatch_async(backgroundQueue, ^(void) {
NSArray *auditIDArray = [[NSArray alloc] initWithArray: [self getUnuploadedIDs]];
for (int temp = 0; temp < [auditIDArray count]; temp ++) {
// Code to post JSON to server
NSURLResponse *response;
NSData *urlData=[NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];
if (!error) {
NSString *responseID = [[NSString alloc]initWithData:urlData encoding:NSUTF8StringEncoding];
if ([responseID isEqualToString:#"ERROR"]) {
//Error uploading records
} else {
[responseArray addObject:responseID];
}
} else {
//Error
return;
}
}
dispatch_async( backgroundQueue, ^{
/* Based on return code update local DB */
for (int temp = 0; temp < [responseArray count]; temp ++) {
[self updateRecordsForID:[auditIDArray objectAtIndex:temp] withID:[responseArray objectAtIndex:temp]];
}
});
});
}
}
}
- (void)upload { //Called when internet connection available
if(backgroundQueue){
dispatch_suspend(backgroundQueue);
dispatch_release(backgroundQueue);
backgroundQueue = nil;
}
backgroundQueue = dispatch_queue_create("com.XXXX.TestApp.bgqueue", NULL);
dispatch_async(backgroundQueue, ^(void) {
[self FormatAnswersInJSON];
});
}
If this code were sitting in front of me, my approach would be:
Look at the use cases and define 'huge number of records': Will 50 record updates at a time occur regularly? Or will it be in 1s and 2s? Do my users have wifi connections or is it over the paid network?, etc.
If possible, test in the wild. If my user base was small enough, gather real data and let that guide my decisions, or only release the feature to a subset of users/beta tests and measure.
If the data tells you to, then optimize this code to be more efficient.
My avenue of optimization would be doing group processing. The rough algorithm would be something like:
for records in groups of X
collect
post to server {
on return:
gather records that updated successfully
update locally
}
This assumes you can modify the server code. You could do groups of 10, 20, 50, etc. all depends on the type of data being sent, and the size.
A group algorithm means a bit more pre-processing client side, but has the pro of reducing HTTP requests. If you're only ever going to get a small number of updates, this is YAGNI and pre-mature optimization.
Don't let this decision keep you from shipping!
Your code has a couple of issues. One convention is to always check the return value before you test the error parameter. The error parameter might be set - even though the method succeeded.
When using NSURLConnection for anything else than a quick sample or test, you should also always use the asynchronous style with handling the delegate methods. Since using NSURLConnection properly may become quickly cumbersome and error prone, I would suggest to utilize a third party framework which encapsulates a NSURLConnection object and all connection related state info as a subclass of NSOperation. You can find one example implementation in the Apple samples: QHTTPOperation. Another appropriate third party framework would be AFNetworking (on GitHub).
When you use either the async style with delegates or a third party subclass, you can cancel the connection, retrieve detailed error or progress information, perform authentication and much more - which you can't with the synchronous API.
I think, once you have accomplished this and your approach works correctly, you may test whether the performance is acceptable. But unless you have large data - say >2 MByte - I wouldn't worry too much.
If your data becomes really large, say >10 MByte you need to consider to improve your approach. For example, you could provide the POST data as file stream instead a NSData object (see NSURLRequest's property HTTPBodyStream). Using a stream avoids to load all the POST data into RAM which helps alleviate the limited RAM problem.
If you have instead smaller POST data, but possibly many of them, you might consider to use a NSOperationQueue where you put your NSOperation connection subclass. Set the maximum number of concurrent operations to 2. This then may leverage HTTP pipelining - if the server supports this, which in effect reduces latency.
Of course, there might be other parts in your app, for example you create or retrieve the data which you have to send, which may affect the overall performance. However, if your code is sound and utilizes dispatch queues or NSOperations which let things perform in paralel there aren't many more options to improve the performance of the connection.
Related
In my app, I am using ReactiveCocoa to return signals to notify me when async api calls are completed (successfully or not). On the POST for saving the data, it only takes one object at a time:
- (RACSignal *)postJSONData:(NSDictionary *)dict toRelativeURL:(NSString *)urlString;.
The function that returns a RACSignal sends the subscriber a next:
[subscriber sendNext:json]or an Error: [subscriber sendError:jsonError].
This works great when saving a single object but I also have a scenario where I have to save multiple objects. These objects can be saved in any order (i.e. they are not dependent on one another) or sequentially - it doesn't matter to me.
I need to update the UI indicating the overall progress (Saving 1 of 4, Saving 2 of 4....) as well as a final progress update (Completed 4 of 4) and specific action to take when all have been processed (successful or not).
There are a number of ways to do this, but I'd like to do this the proper way using ReactiveCocoa. I'm thinking I either can do this with a concat: or then: with a rac_sequence map:^, but I'm not sure. On their github page, they show an example of addressing parallel work streams, but they use 2 discretely defined signals. I won't have my signals until I loop through each object I need to save. Would love some guidance or an example (even better!). Thanks in advance.
I'm doing something similar in my app where I start 3 different async network calls and combine them all into one signal that I can listen to. Basically I loop through all my objects and store the network signal in an array. I then call merge: and pass it the array of network signals.
NSMutableArray *recievedNames = [NSMutableArray new];
NSMutableArray *signals = [NSMutableArray new];
//go though each database that has been added and grab a signal for the network request
for (GLBarcodeDatabase *database in self.databases) {
[signals addObject:[[[[self.manager rac_GET:[database getURLForDatabaseWithBarcode:barcode] parameters:nil] map:^id(RACTuple *value) {
return [((NSDictionary *)value.second) valueForKeyPath:database.path];
}] doError:^(NSError *error) {
NSLog(#"Error while fetching name from database %#", error);
}]
}
//forward all network signals into one signal
return [[[RACSignal merge:signals] doNext:^(NSString *x) {
[recievedNames addObject:x];
}] then:^RACSignal *{
return [RACSignal return:[self optimalNameForBarcodeProductWithNameCollection:recievedNames]];
}];
Feel free to ask me questions about any of the operators I have used and I will do my best to explain them.
I am just learning ReactiveCocoa myself but I wanted to point out something important which also agrees with lightice11's answer. concat combines signals sequentially. Meaning you won't get anything from #2 or #3 until #1 completes. merge on the other hand interleaves the responses returning whatever comes next regardless of order. So for your scenario, it sounds like you really do want merge.
To quote the man, Justin Spahr-Summers:
concat joins the streams sequentially, merge joins the signals on an as-soon-as-values-arrive basis, switch only passes through the events from the latest signal.
I have a UITableView with cells to display images and text from a largish (5000 items) JSON file. I want to stream the JSON in and start updating the UITableView immediately, but can't seem to work out the plumbing for this.
- (NSArray *)parseJSONIntoImageObjectsFromData:(NSData *)rawJSONData {
NSError *error;
NSMutableArray *arrayOfImageObjects = [[NSMutableArray alloc] init];
NSURL *myURL = [[NSURL alloc] initWithString:self.urlString];
NSData *objects = [NSData dataWithContentsOfURL:myURL];
NSInputStream *stream = [[NSInputStream alloc] initWithData:objects];
[stream open];
NSMutableArray *arrayFromStream = [NSJSONSerialization JSONObjectWithStream:stream options:NSJSONReadingAllowFragments error:&error];
for (NSDictionary *JSonDictionary in arrayFromStream) {
NSLog(#"Count is %lu", (unsigned long)arrayOfImageObjects.count);
NSInteger imgID = (NSInteger)JSonDictionary[#"id"];
ImageObject *newImageObject = [[ImageObject alloc] initWithID:imgID andTitle:JSonDictionary[#"title"] andThumbnailURL:JSonDictionary[#"thumbnailUrl"]];
[arrayOfImageObjects addObject:newImageObject];
}
return arrayOfImageObjects;
}
This definitely gets them as a stream, as the NSLog reveals in the debug window. But since it waits for the return it has to complete. I'm a little puzzled at going about this and can't find a good code sample. Do I perhaps return a stream?
EDIT: I am not terribly concerned about the brief delay I am encountering and I am sure the delay is more on the retrieval than in the parsing, I just want to learn to retrieve the data as a stream and update the UITableView incrementally as a way to do this better. I enjoy working on data retrieval and manipulation and am trying to improve my skills by knowing more.
Also, the images are retrieved asynchronously at display time using an NSOperationQueue and don't really matter for this task.
If you benchmark this, I think you'll find that the parsing time of the JSON is inconsequential. The slow parts are going to be the download of the original JSON (and possibly the creation of the ImageObject objects). You should benchmark this in Instruments (use the "time profiler" tool) and use the "record waiting threads" option. See WWDC video Building Concurrent User Interfaces on iOS for a demonstration on how to use Instruments to diagnose these sorts of issues.
I would first retire the dataWithContentsOfURL, as that runs synchronously. I would advise using an asynchronous technique such as NSURLSession method dataWithURL (or if you need support for pre-iOS 7, NSURLConnection method sendAsynchronousRequest).
Usually in these cases, the JSON is small enough, that the biggest delay stems from the network latency in making the initial request. I mention that so that you don't bother embarking on some major refactoring of the code for paging/streaming approaches without confirming that this will solve the problem.
Also, you haven't shared this ImageObject logic, but if that is synchronously loading images, that's a likely candidate for refactoring for asynchronous retrieval, too. Without knowing more about that class, it's hard to advise you further on that point.
Define NSMutableArray *arrayOfImageObjectsas a property or variable outside this method and then in your for loop, call [self.tableView reloadData] after maybe every 100 objects.
That's assuming that your numberOfRowsInSection is keying off of arrayOfImageObjects as well and cellForRowAtIndexPath is using it to populate the table data.
But also consider 'paging' your data, so as to only load 50 objects or so, at once (assuming your API supports this like 'http://example.com/imagedata?page=1'). Then if the user flicks or scrolls the tableview you can do another api call, increasing the page number and adding that new set of data to your current set and calling reloadData.
EDIT: also I'm assuming your "parseJSONIntoImageObjectsFromData" is running asynchronously. If not then use something like AFNetworking (or sendAsynchronousRequest:queue:completionHandler: in NSURLConnection) and in the completion block you can start adding to your array.
I am fetching data from a server in JSON format. It's only about 150 records and I was not using GCD initially but every now and again when I hit the button in the app to view the table with data it would delay for about a couple of seconds before switching to the table view and displaying the data. So I implemented GCD and now when I hit the button it switches to the tableview immediately but then there is a few seconds delay in loading the data, which seems longer than the pre-GCD implementation. So I'm not sure if I am using GCD correctly, or if it's my server causing the delay (which I think is the culprit). Here is the implementation of GCD in a method called retrieveData which I call in viewDidLoad as [self retrieveData]:
- (void)retrieveData
{
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0),^{
NSURL *url = [NSURL URLWithString:#"http://MY_URL/JSON/document.json"];
NSData * data = [NSData dataWithContentsOfURL:url];
dispatch_async(dispatch_get_main_queue(), ^{
json = [NSJSONSerialization JSONObjectWithData:data options:kNilOptions error:nil];
//Set up our exhibitors array
exhibitorsArray = [[NSMutableArray alloc] init];
for (int i = 0; i < json.count; i++) {
//create exhibitors object
NSString * blabel = [[json objectAtIndex:i] objectForKey:#"BoothLabel"];
NSString * bName = [[json objectAtIndex:i] objectForKey:#"Name"];
NSString * bURL = [[json objectAtIndex:i] objectForKey:#"HyperLnkFldVal"];
exhibitors * myExhibitors = [[exhibitors alloc] initWithBoothName: bName andboothLabel: blabel andBoothURL: bURL];
//Add our exhibitors object to our exhibitorsArray
[exhibitorsArray addObject:myExhibitors];
//Sort by name
NSSortDescriptor *sort = [NSSortDescriptor sortDescriptorWithKey:#"name" ascending:YES];
[exhibitorsArray sortUsingDescriptors:[NSMutableArray arrayWithObject:sort]];
}
[self.myTableView reloadData];
});
});
}
This is basically correct. Dispatch the data retrieval to the background queue, and then dispatch the model and UI update back to the main queue. Well done.
In terms of its being slower, I don't see anything there that would account for that. GCD introduces some overhead, but generally not observable. It may be a bit of a "watched kettle never boils" issue.
A couple of unrelated thoughts, though:
I might suggest moving the sort to outside of the for loop, but before the reloadData. You're sorting it 150 times. If doing an insertion sort, you could do it within the loop, but I don't think that's happening here. I'd move the sort to the end of the loop. I'm not sure if the performance gain will be observable, but there should be some modest improvement.
You might want to make sure data is not nil (e.g. no network, or some other network issue), because if it is, JSONObjectWithData will crash.
Your json object is an external variable. It should probably be a local variable of your retrieveData method. There's no need to make it an instance variable. It's cleaner to make it a local variable if appropriate.
You probably should adopt the naming convention whereby class names start with uppercase letters (e.g. Exhibitor instead of exhibitors).
Very minor point, but your blabel variable should probably be bLabel. Even better, I might rename these three variables boothLabel, boothName, and boothUrlString.
You're using instance variable for exhibitorsArray. I presume you're doing this elsewhere, too. You might want to consider using declared properties instead.
You might want to turn on the network activity indicator before dispatching your code to the background, and turning it back off when you perform reloadData.
[[UIApplication sharedApplication] setNetworkActivityIndicatorVisible:YES];
If you wanted to get really sophisticated, you might reconsider whether you want to use GCD's global queues (because if you refresh 10 times quickly, all ten will run, whereas you probably only want the last one to run). This is a more complicated topic, so I won't cover that here, but if you're interested, you might want to refer to the discussion of operation queues the Concurrency Programming Guide, in which you can create operations that are cancelable (and thus, when initiating a new operation, cancel the prior one(s)).
You might also want to refer to the Building Concurrent User Interfaces on iOS WWDC 2012 video.
But this is all tangential to your original question: Yes, you have tackled this appropriately.
I have an NSOperationQueue that handles importing data from a web server on a loop. It accomplishes this with the following design.
NSURLConnect is wrapped in an NSOperation and added to the Queue
On successful completion of the download (using a block), the data from the request is wrapped in another NSOperation that adds the relevant data to Core Data. This operation is added to the queue.
On successful completion (using another block), (and after a specified delay) I call the method that started it all and return to step 1. Thus, i make another server call x seconds later.
This works great. I'm able to get data from the server and handle everything on the background. And because these are just NSOperations I'm able to put everything in the background, and perform multiple requests at a time. This works really well.
The ONLY problem that I currently have is that I'm unable to successfully cancel the operations once they are going.
I've tried something like the following :
- (void)flushQueue
{
self.isFlushingQueue = YES;
[self.operationQueue cancelAllOperations];
[self.operationQueue waitUntilAllOperationsAreFinished];
self.isFlushingQueue = NO;
NSLog(#"successfully flushed Queue");
}
where self.isFlushingQueue is a BOOL that I use to check before adding any new operations to the queue. This seems like it should work, but in fact it does not. Any ideas on stopping my Frankenstein creation?
Edit (Solved problem, but from a different perspective)
I'm still baffled about why exactly I was unable to cancel these operations (i'd be happy to keep trying possible solutions), but I had a moment of insight on how to solve this problem in a slightly different way. Instead of dealing at all with canceling operations, and waiting til queue is finished, I decided to just have a data structure (NSMutableDictionary) that had a list of all active connections. Something like this :
self.activeConnections = [NSMutableDictionary dictionaryWithDictionary:#{
#"UpdateContacts": #YES,
#"UpdateGroups" : #YES}];
And then before I add any operation to the queue, I simply ask if that particular call is On or Off. I've tested this, and I successfully have finite control over each individual server request that I want to be looping. To turn everything off I can just set all connections to #NO.
There are a couple downsides to this solution (Have to manually manage an additional data structure, and every operation has to start again to see if it's on or off before it terminates).
Edit -- In pursuit of a more accurate solution
I stripped out all code that isn't relevant (notice there is no error handling). I posted two methods. The first is an example of how the request NSOperation is created, and the second is the convenience method for generating the completion block.
Note the completion block generator is called by dozens of different requests similar to the first method.
- (void)updateContactsWithOptions:(NSDictionary*)options
{
//Hard coded for ease of understanding
NSString *contactsURL = #"api/url";
NSDictionary *params = #{#"sortBy" : #"LastName"};
NSMutableURLRequest *request = [self createRequestUsingURLString:contactsURL andParameters:params];
ConnectionCompleteBlock processBlock = [self blockForImportingDataToEntity:#"Contact"
usingSelector:#selector(updateContactsWithOptions:)
withOptions:options andParsingSelector:#selector(requestUsesRowsFromData:)];
BBYConnectionOperation *op = [[BBYConnectionOperation alloc] initWithURLRequest:request
andDelegate:self
andCompletionBlock:processBlock];
//This used to check using self.isFlushingQueue
if ([[self.activeConnections objectForKey:#"UpdateContacts"] isEqualToNumber:#YES]){
[self.operationQueue addOperation:op];
}
}
- (ConnectionCompleteBlock) blockForImportingDataToEntity:(NSString*)entityName usingSelector:(SEL)loopSelector withOptions:(NSDictionary*)options andParsingSelector:(SEL)parseSelector
{
return ^(BOOL success, NSData *connectionData, NSError *error){
//Pull out variables from options
BOOL doesLoop = [[options valueForKey:#"doesLoop"] boolValue];
NSTimeInterval timeInterval = [[options valueForKey:#"interval"] integerValue];
//Data processed before importing to core data
NSData *dataToImport = [self performSelector:parseSelector withObject:connectionData];
BBYImportToCoreDataOperation *importOperation = [[BBYImportToCoreDataOperation alloc] initWithData:dataToImport
andContext:self.managedObjectContext
andNameOfEntityToImport:entityName];
[importOperation setCompletionBlock:^ (BOOL success, NSError *error){
if(success){
NSLog(#"Import %#s was successful",entityName);
if(doesLoop == YES){
dispatch_async(dispatch_get_main_queue(), ^{
[self performSelector:loopSelector withObject:options afterDelay:timeInterval];
});
}
}
}];
[self.operationQueue addOperation:importOperation];
};
}
Cancellation of an NSOperation is just a request, a flag that is set in NSOperation. It's up to your NSOperation subclass to actually action that request and cancel it's work. You then need to ensure you have set the correct flags for isExecuting and isFinished etc. You will also need to do this in a KVO compliant manner. Only once these flags are set is the operation finished.
There is an example in the documentation Concurrency Programming Guide -> Configuring Operations for Concurrent Execution. Although I understand that this example may not correctly account for all multi-threaded edge cases. Another more complex example is provided in the sample code LinkedImageFetcher : QRunLoopOperation
If you think you are responding to the cancellation request correctly then you really need to post your NSOperation subclass code to examine the problem any further.
Instead of using your own flag for when it is ok to add more operations, you could try the
- (void)setSuspended:(BOOL)suspend
method on NSOperationQueue? And before adding a new operation, check if the queue is suspended with isSuspended?
This is my problem, I'm building an iOS with some JSON returs from my server, here, no problems, all works fine.
The problem is that when I run the program, it take a long long time to parse the result into a NSMutableArray: this is the log
2013-01-10 12:03:48.627 project[5917:907] <- time begin parser
2013-01-10 12:03:48.755 project[5917:907] <- time finish parser
2013-01-10 12:03:48.756 project[5917:907] <- time begin implement arrays
2013-01-10 12:03:58.522 project[5917:907] <- time finish implement array
As you can see, implementing the arrays is really long.
I know that I have to use queueing and grand central dispatch to make my UI responsive, but I don't know how to, could you please help me to do that ?
This is my viewDidLoad Method
- (void)viewDidLoad
{
[super viewDidLoad];
if debug
NSLog(#"<- time begin parser");
endif
NSString *URLStr = #"http://myJsonFile.json";
NSDictionary *myDictwithReturn = [TOCJSONParser awesomeParserWithStringOfYourJSONFile:URLStr]; //really cool parser, i can put it on gitHub if you want
NSArray *speakersArray = [myDictwithReturn objectForKey:#"speakers"];
myArray = [[NSMutableArray alloc]init];
NSLog(#"<- time finish parser");
NSLog(#"<- time begin implement arrays");
for (NSDictionary *myDict in speakersArray) {
_nextSpeaker = [[TOCSpk alloc]init];
[_nextSpeaker setName:[myDict objectForKey:#"name"]];
[_nextSpeaker setBusiness:[myDict objectForKey:#"business"]];
[_nextSpeaker setDesc:[myDict objectForKey:#"desc"]];
[_nextSpeaker setTwitter:[NSURL URLWithString:[myDict objectForKey:#"twitter"]]];
[_nextSpeaker setPicture:[_nextSpeaker retrieveImageFromServer:[myDict objectForKey:#"picture"]]];
[myArray addObject:_nextSpeaker];
}
NSLog(#"<- time finish implement array");
}
I suspect that the slowness comes from calling retrieveImageFromServer, which lets me think that you are accessing the network. If that access is synchronous, as it seems from the fact that you are assigning the image in the same statement, than this is bound to be slow.
You should review your code and make it run on a separate thread or use asynchronous network access.
EDIT:
After your comment about using, dataWithContentsOfURL, my above hypothesis is confirmed.
You can read this S.O. post about a way to download images asynchronously, or you might use any of various networking frameworks available out there.
Possibly, the easiest path forward is using SDWebImage, which is a class that offers async download for images, so you donĀ“t have to bother yourself with thread management or NSURLConnection:
Just #import the UIImageView+WebCache.h header, and call the setImageWithURL:placeholderImage: method from the tableView:cellForRowAtIndexPath: UITableViewDataSource method. Everything will be handled for you, from async downloads to caching management.