I got an app which stores data about restaurants. Their menu, schedule and so on.
Theres ~200 restaurants in the DB by now. The app used to retrieve those places by a single shot, but it was taking too much time to load, so i decided to load the data one by one. On the start app asks the server for an array of place's ids, then get the data via API.
I set completion blocks for RK's operations to background queues, but it didn't help. Scrolls are running not smoothy enough and sometimes the app even deadlocks(sic!) or crashes with no error output in the console. I've tried using Instruments to locate what makes UI go jerky, but didn't succeed. I even turned off image loading and sorting functions. I used to have RK's requests calls wrapped in #synchronized, so i removed it, but no effect.
Never thought i'd be facing this kind of issue after few weeks of experience with objective-c. I've tried every possible way that came to my mind, so now im kinda giving up.
Please help me :)
The code below gets called 200 times right after the app start.
NSURLRequest *request = [[DEAPIService sharedInstance].manager requestWithObject:nil
method:RKRequestMethodGET
path:path
parameters:params];
//DLog(request.URL.absoluteString);
NSManagedObjectContext *context = [NSManagedObjectContext MR_contextWithParent:[NSManagedObjectContext MR_contextForCurrentThread]];
RKManagedObjectRequestOperation *operation = [[DEAPIService sharedInstance].manager managedObjectRequestOperationWithRequest:request
managedObjectContext:context
success:success
failure:failure];
// dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0);
dispatch_queue_t backgroundQueue = dispatch_queue_create("com.name.bgqueue", NULL);
operation.successCallbackQueue = backgroundQueue;
[[DEAPIService sharedInstance].manager enqueueObjectRequestOperation:operation];
Response descriptors are set before that. And also there's many relations in the DB. I've looked up the DB's file size - it's ~1.5Mb. I wonder what will happen if there would be over 1k restaurants. Is it a good way to load this kind of data? What are the best practises?
Ok, from the information provided you should be able to simplify a lot. At the moment your code is dropping down to a low level and getting involved in threading and context management that you want to leave for RestKit to sort out. You have a properly configured object manager and core data stack by the looks of things so you should let it do the work.
This means removing the request, context and request operation code and simply calling `getObjectsAtPath:parameters:success:failure:'. RestKit will then deal with all of the download and mapping in the background and save the context.
You should also really be using fetched results controllers throughout your app, and if you are they will automatically detect the changes that RestKit has saved and update your UI.
Once you have that, any blocking of the UI is unrelated to RestKit and your download requirement and should be related to subsequent image management / downloads.
Related
Intro
I've read alot of tutorials and articles on Core Data concurrency, but I'm having an issue that is not often covered, or covered in a real-world way that I am hoping someone can help with. I've checked the related questions in SO and none give an answer to this particular question that I can find.
Background
We have an existing application which fetches data from an API (in the background thread) and then saves the records returned into core data. We also need to display these records in the application at the time.
So the process we currently go through is to:
Make a network request for data (background)
Parse the data and map the objects to NSManagedObjects and save (background)
In the completion handler (main thread) we fetch records from core data with the same order and limit that we requested from the API.
Most tutorials on core data concurrency follow this pattern of saving in one thread and then fetching in another, but most of them give examples like:
NSArray *listOfPeople = ...;
[NSManagedObjectHelper saveDataInBackgroundWithContext:^(NSManagedObjectContext *localContext){
for (NSDictionary *personInfo in listOfPeople)
{
PersonEntity *person = [PersonEntity createInContext:localContext];
[person setValuesForKeysWithDictionary:personInfo];
}
} completion:^{
self.people = [PersonEntity findAll];
}];
Source
So regardless of the amount of records you get back, you just fetch all content. This works for small datasets, but I want to be more efficient. I've read many times not to read/write data across threads, so fetching afterwards gets around this issue, but I don't want to fetch all, I just want the new records.
My Problem
So, for my real world example. I want to make a request to my API for the latest information (maybe anything older than my oldest record in core data) and save it, them I need the exact data returned from the API in the main thread ready for display.
So my question is, When I reach my completion handler, how do I know what to fetch? or what did the API return?. A couple of methods I've considered so far:
after saving each record, store the ID in a temporary array and then perform some fetch where id IN array_of_ids.
If I am asking for the latest records, I could just use the count of records returned, then use an order by and limit in my request to the latest x records.
My Question
I realize that the above could be answering my own question but I want to know if there is a better way, or is one of those methods much better to use than the other? I just have this feeling that I am missing something
Thanks
EDIT:
Neither answer below actually addresses the question, This is to do with fetching and saving data in the background and then using the returned data in the main thread. I know it's not a good idea to pass data between threads, so the common way around this is to fetch from core data after inserting. I want to work out the more efficient way.
Have you checked NSFetchedResultsController? Instead of fetching presented objects into array, you will use fetched controller in similar fashion. Through NSFetchedResultsControllerDelegate you would be notified about all the changes performed in background (rows added, removed, changed) and no manual tracking would be needed.
I feel You missing case with two silmultaneous API calls. Both storring ids and counting created enities wont work for that case. Consider adding timestamp property for each PersonEntity.
Assuming that Your intention is to display recently updated persons.
The calcutation of the oldest timestamp to display can look like this:
#property NSDate *lastViewRefreshTime;
#property NSDate *oldestEntityToDisplay;
(...)
if (self.lastViewRefreshTime.timeIntervalSinceNow < -3) {
self.oldestEntityToDisplay = self.lastViewRefreshTime;
}
self.lastViewRefreshTime = [NSDate date];
[self displayPersonsAddedAfter: self.oldestEntityToDisplay];
Now, if two API responses returns in period shorter than 3s their data will be displayed together.
I am having a major problem with my application speed in processing updates on a background thread. Instruments shows that almost all of this time is spend inside performBlockAndWait where I am fetching out the objects which need updating.
My updates may come in by the hundreds depending on the amount of time offline and the approach I am currently using is to process them individually; ie fetch request to pull out the object, update, then save.
It sounds slow and it is. The problem I have is that I don't want to load everything into memory at once, so need to fetch them individually as we go, also I save as I go to ensure that if there is an issue with a single update it won't mess up the rest.
Is there a better approach?
I hit similar slow performance when upserting a large collection of objects. In my case I'm willing to keep the full change set in memory and perform a single save so the large volume of fetch requests dominated my processing time.
I got a significant performance improvement from maintaining an in memory cache mapping my resources' primary keys to NSManagedObjectIDs. That allowed me to use existingObjectWithId:error: rather than a fetch request for an individual object.
I suspect I might do even better by collecting the primary keys for all resources of a given entity description, issuing a single fetch request for all of them at once (batching those results as necessary), and then processing the changes to each resource.
You may benefit from using NSBatchUpdateRequest assuming you're targeting iOS 8+ only.
These guys have a great example of it but the TLDR is basically:
Example: Say we want to update all unread instances of MyObject to be marked as read:
NSBatchUpdateRequest *req = [[NSBatchUpdateRequest alloc] initWithEntityName:#"MyObject"];
req.predicate = [NSPredicate predicateWithFormat:#"read == %#", #(NO)];
req.propertiesToUpdate = #{
#"read" : #(YES)
};
req.resultType = NSUpdatedObjectsCountResultType;
NSBatchUpdateResult *res = (NSBatchUpdateResult *)[context executeRequest:req error:nil];
NSLog(#"%# objects updated", res.result);
Note the above example is taken from the aforementioned blog, I didn't write the snippet.
So I have a bunch of objects in Core Data and want them to auto delete after X amount of days (this would be based off of an NSDate). I did some searching and it seems that you can only delete one core data object at a time, not a group of them, let alone ones that are based off of a certain date. I'm thinking maybe to have a loop running going through each object - but that seems like it would be very processor heavy. Any ideas on where I should be looking to do this? Thanks.
A loop deleting objects one by one is the correct approach.
Deleting objects in Core Data is extremely processor heavy. If that's a problem, then Core Data is not suitable for your project, and you should use something else. I recommend FCModel, as a light weight alternative that is very efficient.
If you are going to stick with Core Data, it's a good idea to perform large operations on a background NSOperationQueue, so the main application is not locked up while deleting the objects. You need to be very careful with Core Data across multiple threads, the approach is to have a separate managed object context for each thread, both using the same persistent store coordinator. Do not ever share a managed object across threads, but you can share the objectID, to fetch a second copy of the same database record on the other managed object context.
Basically your background thread creates a new context, deletes all the objects in a loop, then (on the main thread preferably, see documentation) save the background thread context. This will merge your changes unless there is a conflict (both contexts modify the same object) — in that scenario you have a few options, I'd just abort the entire delete operation and start again.
Apple has good documentation available for all the issues and sample code available here: https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/CoreData/Articles/cdConcurrency.html
It's a bit daunting, you need to do some serious homework, but the actual code is very simple once you've got your head around how everything works. Or just use FCModel, which is designed for fast batch operations.
It's not as processor heavy as you may think :) (of course it depends of data amount)
Feel free to use loop
- (void)deleteAllObjects
{
NSArray *allEntities = self.managedObjectModel.entities;
for (NSEntityDescription *entityDescription in allEntities)
{
NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] init];
[fetchRequest setEntity:entityDescription];
fetchRequest.includesPropertyValues = NO;
fetchRequest.includesSubentities = NO;
NSError *error;
NSArray *items = [self.managedObjectContext executeFetchRequest:fetchRequest error:&error];
for (NSManagedObject *managedObject in items) {
[self.managedObjectContext deleteObject:managedObject];
}
if (![self.managedObjectContext save:&error]) {
NSLog(#"Error occurred");
}
}
}
As others have noted, iterating over the objects is the only way to actually delete the objects in Core Data. This is one of those use cases where Core Data's approach kind of falls down, because it's just not optimized for that kind of use.
But there are ways to deal with it to avoid unwanted delays in your app, so that the user doesn't have to wait while your code chugs over a ton of delete requests.
If you have a lot of objects that need to be deleted and you don't want to have to wait until the process is complete, you can fake the initial delete at first and then later do the actual delete when it's convenient. Something like:
Add a custom boolean attribute to the entity type called something like toBeDeleted with a default value of NO.
When you have a bunch of objects to delete, set toBeDeleted to YES on all of them in a single step by using NSBatchUpdateRequest (new in iOS 8). This class is mostly undocumented, so look at the header file or at the BNR blog post about it. You'll specify the property name and the new attribute value, and Core Data will do a mass, quick update.
Make sure your fetch requests all check that toBeDeleted is NO. Now objects marked for deletion will be excluded when fetching even though they still exist.
At some point-- later on, but soon-- run some code in the background that fetches and deletes objects that have toBeDeleted set to YES.
I've learned that generally, intensive tasks should take place on background threads, as if they're on the main thread they'll block user interaction and interface updates.
Does Core Data fall under that umbrella? I received a great answer to my question about loading images asynchronously in a UITableView, but I'm curious how to then work with Core Data as the backend.
I know Apple has a Core Data Concurrency guide, but I'm curious in which cases one is supposed to use Core Data in the background.
As a quick example, say I'm making a Twitter client and want to get all the tweet information (tweet text, username, user avatar, linked images, etc.). I asynchronously download that information and receive some JSON from the Twitter API that I then parse. Should I then do a dispatch_async(dispatch_get_main_queue()...) when I add the information to Core Data?
I also then want to download the user's avatar, but I want to do that separately from presenting the tweet, so I can present the tweet quickly as possible, then present the image when ready. So I asynchronously download the image. Should I then update that Core Data item asynchronously?
Am I in a situation where I don't need multi-threaded Core Data at all? If so, when would be a situation where I need it?
I've learned that generally, intensive tasks should take place on background threads, as if they're on the main thread they'll block user interaction and interface updates.
Does Core Data fall under that umbrella?
Yes, actually.
Core Data tasks can and should - where possible - be executed on background threads or on non-main queues.
It's important to note though, that each managed object is associated to a certain execution context (a thread or a dispatch queue). Accessing a managed object MUST be executed only from within this execution context. This association comes from the fact that each managed object is registered with a certain Managed Object Context, and this managed object context is associated to a certain execution context when it is created.
See also:
[NSManagedObjectContext initWithConcurrencyType](),
[NSManagedObjectContext performBlock]
[NSManagedObjectContext performBlockAndWait]
Consequently, when displaying properties of managed objects, this involves UIKit, and since UIKit methods MUST be executed on the main thread, the managed object's context must be associated to the main thread or main queue.
Otherwise, Core Data and user code can access managed objects from arbitrary execution contexts, as long as this is the one to which the managed object is associated with.
The below picture is an Instrument Time Profile which shows quite clearly how Core Data tasks can be distributed on different threads:
The highlighted "spike" in the CPU activity shows a task which performs the following:
Load 1000 JSON objects from origin (this is just a local JSON file),
Create managed objects in batches of 100 on a private context
Merge them into the Cora Data Stack main context
Save the context (each batch)
Finally, fetch all managed objects into the main context
As we can see, only 26.4% of the CPU load will be executed on the main thread.
Core Data does indeed fall under that umbrella. Particularly for downloading and saving data, but also possibly for fetching depending on the number of objects in the data store and the predicate to be applied.
Generally, I'd push all object creation and saving which is coming from a server onto a background thread. I'd only update and save objects on the main thread if they're user generated (because it would only be one, updated slowly and infrequently).
Downloading your twitter data is a good example as there will potentially be a good amount of data to process. You should process and save the data on a background thread and save it up to the persistent store there. The persistent store should then merge the changes down to the main thread context for you (assuming you have the contexts configured nicely - use a 3rd party framework for that like MagicalRecord).
Again, for the avatar update, you're already on a background thread so you might as well stay there :-)
You might not need to use multiple threads now. If you only download the 5 most recent tweets then you might not notice the difference. But using a framework can make the multi-threading relatively easy. And using a fetched results controller can make knowing when the UI should be updated on the main thread very easy. So, it's worthwhile taking the time to understand the setup and using it.
The best way to handle behavior like this is to use multiple NSManagedObjectContexts. The "main" context you create like so:
_mainManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType];
_mainManagedObjectContext.persistentStoreCoordinator = [self mainPersistentStoreCoordinator];
You're going to want to do any heavy writes on a different NSManagedObjectContext to avoid blocking your UI thread as you import (which can be quite noticeable on large or frequent operations to your Main context).
To achieve this, you would do something like this:
Retrieve your data asynchronously from the network
Spin up a temporary (or perhaps a permanent) background NSManagedObjectContext and set the "parent" to the main ManagedObjectContext (so it will merge the data you import when you save)
Use the performBlock: or performBlockAndWait: APIs to write to that context on its own private queue (in the background)
Example (uses AFNetworking):
[_apiClient GET:[NSString stringWithFormat:#"/mydata"]
parameters:nil
success:^(AFHTTPRequestOperation *operation, id responseObject) {
NSError* jsonError;
id jsonResponse = [NSJSONSerialization JSONObjectWithData:operation.responseData options:kNilOptions error:&jsonError];
if (!jsonError) {
NSArray* parsedData = (NSArray*)jsonResponse;
completionBlock(parsedShares, nil);
} else {
NSManagedObjectContext *context = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
context.parentContext = YOUR_MAIN_CONTEXT; // when you save this context, it will merge its changes into your main context!
// the following code will occur in a background queue for you that CoreData manages
[context performBlock:^{
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
if (![context save:&saveError]) {
NSLog(#"Error saving context: %#", saveError);
} else {
NSLog(#"Saved data import in background!");
}
}];
}
}
failure:^(AFHTTPRequestOperation *operation, NSError *error) {
NSLog(#"There was an error: %#", error)
}];
Docs: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/CoreDataFramework/Classes/NSManagedObjectContext_Class/NSManagedObjectContext.html#//apple_ref/occ/instm/NSManagedObjectContext/performBlockAndWait:
As I alluded to in your other question, it's largely going to be a matter of how long the manipulations are going to take. If the computation required doesn't have a noticeable delay, by all means be lazy about and do your core data on the main thread.
In your other example, you're requesting 20 items from twitter, parsing 20 items and sticking them into CoreData isn't going to be noticeable. The best approach here is to probably continue to just fetch 20 at a time and update in the foreground as each chunk becomes available.
Downloading all items from twitter in one request will take a significant amount of time and computation and it's probably worth creating a separate ManagedObjectModel and synchronizing it with the foreground model. Since you really have one-way data (it always flows twitter->core data->user interface) the likelihood of clashes is minimal, so you can easily use NSManagedObjectContextDidSaveNotification and mergeChangesFromContextDidSaveNotification:
I'm building an interface to allow users to filter a set of Photos. The data model is above. I'd like the disable any controls that would result in 0 results.
To accomplish this, I'm running new fetch requests for every control that is off/not selected each time the user makes their own change. I add the data that the control represents to my NSCompoungPredicate, then remove it after I get the result. If the count of the result is 0, I disable that control.
I'm doing this all on the main thread so in some cases there is a bit of a lag in the app. Is there a better way to do this type of filtering with less overhead? Should I run these filter fetches on their own thread? I've never done that with CoreData and worm what I've read I need a separate context for that and I'm not sure how to go about setting up code for that.
A bit of code would help. Aside from that, here are a few suggestions.
First, on your fetch request, use countForFetchRequest:error: because it will just query the database, and return the count instead of object information.
Second, if you don't want to use threads, and the search is still too slow, you can do the initial query when the app starts. This will then enable/disable the various controls.
You can simple catch the context notifications that tell when data has changed, and update that information accordingly. Then, you don't have to do any queries at all. Just initialize, and update the status as objects are added/removed from the database.
If you want to use threads, then that's not really all that difficult.
It sounds like all you want is a thread that is just running queries. You setup a MOC...
NSManagedObjectContext *checkerMoc = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
checkerMoc.persistentStoreCoordinator = MyCurrentMoc.persistentStoreCoordinator;
Now, whenever you want to check the database...
[checkerMoc performBlock:^{
NSFetchRequest *fetchRequest = ...
// Do your fetch request... this block of code is running in the other thread
[checkerMoc fetch...];
// When the fetch request is done, do whatever you want in your UI...
dispatch_async(dispatch_get_main_queue(), ^{
// Now this code is running in the main thread... access your UI
self.myControl.enabled = fetchResultCount > 0;
});
}];
Note, you are using the same persistent store coordinator, so if the main thread tries to access the database, it will get stacked behind this request. You can also use a separate persistentStoreCoordinator for checkerMoc is this is an issue.