NSFetchedResultsController feeding table view while background update of same persistent store causes deadlock - ios

Still working on converting an app over from downloading information every time it uses or displays it, to caching it on-phone using CoreData (courtesy of MagicalRecord). This is on iOS 7
Because we don't have a data-push system set up to automatically update the phone's cached data whenever some data changes on the backend, I've been thinking over the last many months (as we worked on other aspects of the app) how to manage keeping a local copy of the data on the phone and being able to have the most up to date data in the cache.
I realized that as long as I still fetch the data every time :-( I can use the phone's CoreData backed cache of data to display and use, and just use the fetch of the data to update the on-phone database.
So I have been converting over the main data objects from being downloaded data making up a complete object, to these main data objects being light stand-in objects for CoreData objects.
Basically, each of the normal data objects in the app, instead of containing all the properties of the object internally, contains only the objectIDof the underlying CoreData object and maybe the app specific ID internally, and all other properties are dynamic and gotten from the CoreData object and passed through (most properties are read-only and updates are done through bulk-rewriting of the core data from passed in JSON)
Like this:
- (NSString *)amount
{
__block NSString *result = nil;
NSManagedObjectContext *localContext = [NSManagedObjectContext MR_newContext];
[localContext performBlockAndWait:^{
FinTransaction *transaction = (FinTransaction *)[localContext existingObjectWithID:[self objectID] error:nil];
if (nil != transaction)
{
result = [transaction.amount stringValue];
}
}];
return result;
}
Occasionally there is one that needs to be set and those look like this:
- (void)setStatus:(MyTransactionStatus)status
{
[MagicalRecord saveWithBlock:^(NSManagedObjectContext *localContext) {
FinTransaction *transaction = (FinTransaction *)[localContext existingObjectWithID:[self objectID] error:nil];
if (nil != transaction)
{
transaction.statusValue = status;
}
} completion:^(BOOL success, NSError *error){}];
}
Now, my issue is that I have a view controller that basically uses an NSFetchedResultsController to display stored data from the local phone's CoreData database in a table view. At the same time as this is happening, and the user may start to scroll through the data, the phone spins off a thread to download updates to the data and then starts updating the CoreData data store with the updated data, at which point it then runs an asynchronous GCD call back on the main thread to have the fetched results controller refetch its data and and tells the table view to reload.
The problem is that if a user is scrolling through the initial fetched results controller fetched data and table view load, and the background thread is updating the same Core Data objects in the background, deadlocks occur. It is not the exact same entities being fetched and rewritten (when a deadlock occurs), i.e., not that object ID 1 is being read and written, but that the same persistent data store is being used.
Every access, read or write, happens in a MR_saveWithBlock or MR_saveWithBlockAndWait (writes/updates of data) as the case may be, and a [localContext performBlock:] or [localContext performBlockAndWait:] as may be appropriate. Each separate read or write has its own NSManagedObjectContext. I have not seen any where there are stray pending changes hanging around, and the actual places it blocks and deadlocks is not always the same, but always has to do with the main thread reading from the same persistent store as the background thread is using to update the data.
The fetched results controller is being created like this:
_frController = [[NSFetchedResultsController alloc] initWithFetchRequest:fetchRequest
managedObjectContext:[NSManagedObjectContext MR_rootSavingContext]
sectionNameKeyPath:sectionKeyPath
cacheName:nil];
and then an performFetch is done.
How can I best structure this sort of action where I need to display the extent data in a table view and update the data store in the background with new data?
While I am using MagicalRecord for most of it, I am open to comments, answers, etc with or without (straight CD) using MagicalRecord.

So the way I'd handle this is to look at having two managed object contexts each with its own persistent store coordinator. Both of the persistent store coordinators talk to the same persistent store on disk.
This approach is outlined in some detail in Session 211 from WWDC 2013 — "Core Data Performance Optimization and Debugging", which you can get to on Apple's Developer Site for WWDC 2013.
In order to use this approach with MagicalRecord, you will need to look at using the upcoming MagicalRecord 3.0 release, with the ClassicWithBackgroundCoordinatorSQLiteMagicalRecordStack (yes, that name needs work!). It implements the approach outlined in the WWDC session, although you need to be aware that there will be changes needed to your project to support MagicalRecord 3, and that it's also not quite released yet.
Essentially what you end up with is:
1 x Main Thread Context: You use this to populate your UI, and for your fetched results controllers, etc. Don't ever make changes in this context.
1 x Private Queue Context: Make all of your changes using the block-based saved methods — they automatically funnel through this context and save to disk.
I hope that makes sense — definitely watch the WWDC session — they use some great animated diagrams to explain why this approach is faster (and shouldn't block the main thread as much as the approach you're using now).
I'm happy to go into more detail if you need it.

Related

What is the best practice to pass object references between two queues with CoreData?

I am facing a decision to be made for an applications architecture design.
The Application uses CoreData to persist user information, the same information is also stored on a remote server accessible by a REST-Interface. When the Application starts I provide the cached information from CoreData to be displayed, while I fetch updates from the server. The fetched information is persisted automatically as well.
All of these tasks are performed in background queues as to not block the main thread. I am keeping a strong reference to my persistenContainer and my NSManagedObject called User.
#property (nonatomic, retain, readwrite) User *fetchedLoggedInUser;
As I said the User is populated performing a fetch request via
[_coreDataManager.persistentContainer performBackgroundTask:^(NSManagedObjectContext * _Nonnull context) {
(...)
NSArray <User*>*fetchedUsers = [context executeFetchRequest:fetchLocalUserRequest error:&fetchError];
(...)
self.fetchedLoggedInUser = fetchedUsers.firstObject;
//load updates from server
Api.update(){
//update the self.fetchedLoggedInUser properties with data from the server
(...)
//persist the updated data
if (context.hasChanges) {
context.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy;
NSError *saveError = nil;
BOOL saveSucceeded = [context save:&saveError];
if (saveSucceeded) {
//notify the App about the updates
//**here is the problem**
}
};
}];
So the obvious thing about this is, that after performing the backgroundTask, my self.fetchedLoggedInUser is not in memory anymore, because of its weak reference to the NSManagedObjectContext provided by the performBackgroundTask() of my PersistentContainer.
Therefore, if I try to access the information from another Model, the values are nil.
What would be the best practice to keep the fetched ManagedObject in Memory and not have to fetch it again, every time I want to access its values?
A) In the Documentation, Apple suggests using the objectID of an ManagedObject, to pass objects between queues
Passing References Between Queues
NSManagedObject instances are not intended to be passed between
queues. Doing so can result in corruption of the data and termination
of the application. When it is necessary to hand off a managed object
reference from one queue to another, it must be done through
NSManagedObjectID instances.
You retrieve the managed object ID of a managed object by calling the
objectID method on the NSManagedObject instance.
The perfectly working code for that situation would be to replace the if(saveSucceeded) Check with this Code:
if (saveSucceeded) {
dispatch_async(dispatch_get_main_queue(), ^{
NSError *error = nil;
self.fetchedLoggedInUser = [self.coreDataManager.persistentContainer.viewContext existingObjectWithID:_fetchedLoggedInUser.objectID error:&error];
(...)
//notify the App about the updates
});
}
But I think this may not be the best solution here, as this needs to access the mainContext (in this case the persistentContainer's viewContext) on the mainQueue. This is likely contradictory to what I am trying to do here (performing on the background, to achieve best performance).
My other options (well, these, that I came up with) would be
B) to either store the user information in a Singleton and update it every time the information is fetched from and saved to CoreData. In this scenario I wouldn't need to worry about keeping the NSManagedObject context alive. I could perform any updates on a private background context provided by my persistentContainer's performBackgroundTask and whenever I'd need to persist new / edited user information I could refetch the NSManagedObject from the database, set the properties, save my context and then update the Singleton. I don't know if this is elegant though.
C) edit the getter Method of my self.fetchedLoggedInUser to contain a fetch request and fetch the needed information (this is probably the worst, because of the overhead when accessing the database) and I am not even sure if this would work at all.
I hope that one of these solutions is actually best practice, but I'd like to hear your suggestions why/how or why/how not to handle the passing of the information.
TL:DR; Whats the best practice to keep user information available throughout the whole app, when loading and storing new information is mostly done from backgroundQueues?
PS: I don't want to fetch the information every time I need to access it in one of my ViewControllers, I want to store the data on a central knot, so that it is accessible from every ViewController with ease. Currently the self.fetchedLoggedInUser is a property of a singleton used throughout the application. I find that this saves a lot of redundant code, using the Singleton makes loading and storing the information clearer and reduces the access count to the database. If this is considered bad practice I'd be happy to discuss about that with you.
Use a NSFetchedResultsController - they are very efficient and you can use them even for one object. A FetchedResultsController does a fetch once and then monitors core data for changes. When it changes you have a callback that it has changed. It also works perfectly for ANY core-data setup. So long as the changes are propagated to the main context (either with newBackgroundContext or performBackgroundTask or child contexts or whatever) the fetchedResultsController will update. So you are free to change your core-data stack without changes your monitoring code.
In general I don't like keeping pointers to ManagedObjects. If the entry is deleted from database then the managedObject will crash when you try to access it. A fetchedResultsController is always safe to read fetchedObjects as it tracks deletions for you.
Obviously attach the NSFetchedResultsController to the viewContext and only read it from the main thread.
I came up with a very elegant solution in my opinion.
From the beginning I was using a Singleton called sharedCoreDataManager, I added a property backgroundContext that is initialized like so
self.backgroundContext = _persistentContainer.newBackgroundContext;
_backgroundContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy;
_backgroundContext.retainsRegisteredObjects = YES;
and is retained by sharedCoreDataManager. I am using this context to perform any tasks. Through calling _backgroundContext.retainsRegisteredObjects my NSManagedObject is retained by the backgroundContext, which is itself (like I said) retained by my Singleton sharedCoreDataManager.
I think this is an elegant solution as I can access the ManagedObject threadsafe from the background anytime. I also won't need any extra class that holds the user information on top. And on top of that I can easily edit the user information at anytime and then call save() on my backgroundContext if needed.
Maybe I am going to add it as a child to my viewContext in the future, I'll evaluate the performance and eventually update this answer.
You are still welcome to propose a better solution or discuss this topic.

Deleting Core Data after X amount of days

So I have a bunch of objects in Core Data and want them to auto delete after X amount of days (this would be based off of an NSDate). I did some searching and it seems that you can only delete one core data object at a time, not a group of them, let alone ones that are based off of a certain date. I'm thinking maybe to have a loop running going through each object - but that seems like it would be very processor heavy. Any ideas on where I should be looking to do this? Thanks.
A loop deleting objects one by one is the correct approach.
Deleting objects in Core Data is extremely processor heavy. If that's a problem, then Core Data is not suitable for your project, and you should use something else. I recommend FCModel, as a light weight alternative that is very efficient.
If you are going to stick with Core Data, it's a good idea to perform large operations on a background NSOperationQueue, so the main application is not locked up while deleting the objects. You need to be very careful with Core Data across multiple threads, the approach is to have a separate managed object context for each thread, both using the same persistent store coordinator. Do not ever share a managed object across threads, but you can share the objectID, to fetch a second copy of the same database record on the other managed object context.
Basically your background thread creates a new context, deletes all the objects in a loop, then (on the main thread preferably, see documentation) save the background thread context. This will merge your changes unless there is a conflict (both contexts modify the same object) — in that scenario you have a few options, I'd just abort the entire delete operation and start again.
Apple has good documentation available for all the issues and sample code available here: https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/CoreData/Articles/cdConcurrency.html
It's a bit daunting, you need to do some serious homework, but the actual code is very simple once you've got your head around how everything works. Or just use FCModel, which is designed for fast batch operations.
It's not as processor heavy as you may think :) (of course it depends of data amount)
Feel free to use loop
- (void)deleteAllObjects
{
NSArray *allEntities = self.managedObjectModel.entities;
for (NSEntityDescription *entityDescription in allEntities)
{
NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] init];
[fetchRequest setEntity:entityDescription];
fetchRequest.includesPropertyValues = NO;
fetchRequest.includesSubentities = NO;
NSError *error;
NSArray *items = [self.managedObjectContext executeFetchRequest:fetchRequest error:&error];
for (NSManagedObject *managedObject in items) {
[self.managedObjectContext deleteObject:managedObject];
}
if (![self.managedObjectContext save:&error]) {
NSLog(#"Error occurred");
}
}
}
As others have noted, iterating over the objects is the only way to actually delete the objects in Core Data. This is one of those use cases where Core Data's approach kind of falls down, because it's just not optimized for that kind of use.
But there are ways to deal with it to avoid unwanted delays in your app, so that the user doesn't have to wait while your code chugs over a ton of delete requests.
If you have a lot of objects that need to be deleted and you don't want to have to wait until the process is complete, you can fake the initial delete at first and then later do the actual delete when it's convenient. Something like:
Add a custom boolean attribute to the entity type called something like toBeDeleted with a default value of NO.
When you have a bunch of objects to delete, set toBeDeleted to YES on all of them in a single step by using NSBatchUpdateRequest (new in iOS 8). This class is mostly undocumented, so look at the header file or at the BNR blog post about it. You'll specify the property name and the new attribute value, and Core Data will do a mass, quick update.
Make sure your fetch requests all check that toBeDeleted is NO. Now objects marked for deletion will be excluded when fetching even though they still exist.
At some point-- later on, but soon-- run some code in the background that fetches and deletes objects that have toBeDeleted set to YES.

Is asynchronous Core Data needed in most cases?

I've learned that generally, intensive tasks should take place on background threads, as if they're on the main thread they'll block user interaction and interface updates.
Does Core Data fall under that umbrella? I received a great answer to my question about loading images asynchronously in a UITableView, but I'm curious how to then work with Core Data as the backend.
I know Apple has a Core Data Concurrency guide, but I'm curious in which cases one is supposed to use Core Data in the background.
As a quick example, say I'm making a Twitter client and want to get all the tweet information (tweet text, username, user avatar, linked images, etc.). I asynchronously download that information and receive some JSON from the Twitter API that I then parse. Should I then do a dispatch_async(dispatch_get_main_queue()...) when I add the information to Core Data?
I also then want to download the user's avatar, but I want to do that separately from presenting the tweet, so I can present the tweet quickly as possible, then present the image when ready. So I asynchronously download the image. Should I then update that Core Data item asynchronously?
Am I in a situation where I don't need multi-threaded Core Data at all? If so, when would be a situation where I need it?
I've learned that generally, intensive tasks should take place on background threads, as if they're on the main thread they'll block user interaction and interface updates.
Does Core Data fall under that umbrella?
Yes, actually.
Core Data tasks can and should - where possible - be executed on background threads or on non-main queues.
It's important to note though, that each managed object is associated to a certain execution context (a thread or a dispatch queue). Accessing a managed object MUST be executed only from within this execution context. This association comes from the fact that each managed object is registered with a certain Managed Object Context, and this managed object context is associated to a certain execution context when it is created.
See also:
[NSManagedObjectContext initWithConcurrencyType](),
[NSManagedObjectContext performBlock]
[NSManagedObjectContext performBlockAndWait]
Consequently, when displaying properties of managed objects, this involves UIKit, and since UIKit methods MUST be executed on the main thread, the managed object's context must be associated to the main thread or main queue.
Otherwise, Core Data and user code can access managed objects from arbitrary execution contexts, as long as this is the one to which the managed object is associated with.
The below picture is an Instrument Time Profile which shows quite clearly how Core Data tasks can be distributed on different threads:
The highlighted "spike" in the CPU activity shows a task which performs the following:
Load 1000 JSON objects from origin (this is just a local JSON file),
Create managed objects in batches of 100 on a private context
Merge them into the Cora Data Stack main context
Save the context (each batch)
Finally, fetch all managed objects into the main context
As we can see, only 26.4% of the CPU load will be executed on the main thread.
Core Data does indeed fall under that umbrella. Particularly for downloading and saving data, but also possibly for fetching depending on the number of objects in the data store and the predicate to be applied.
Generally, I'd push all object creation and saving which is coming from a server onto a background thread. I'd only update and save objects on the main thread if they're user generated (because it would only be one, updated slowly and infrequently).
Downloading your twitter data is a good example as there will potentially be a good amount of data to process. You should process and save the data on a background thread and save it up to the persistent store there. The persistent store should then merge the changes down to the main thread context for you (assuming you have the contexts configured nicely - use a 3rd party framework for that like MagicalRecord).
Again, for the avatar update, you're already on a background thread so you might as well stay there :-)
You might not need to use multiple threads now. If you only download the 5 most recent tweets then you might not notice the difference. But using a framework can make the multi-threading relatively easy. And using a fetched results controller can make knowing when the UI should be updated on the main thread very easy. So, it's worthwhile taking the time to understand the setup and using it.
The best way to handle behavior like this is to use multiple NSManagedObjectContexts. The "main" context you create like so:
_mainManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType];
_mainManagedObjectContext.persistentStoreCoordinator = [self mainPersistentStoreCoordinator];
You're going to want to do any heavy writes on a different NSManagedObjectContext to avoid blocking your UI thread as you import (which can be quite noticeable on large or frequent operations to your Main context).
To achieve this, you would do something like this:
Retrieve your data asynchronously from the network
Spin up a temporary (or perhaps a permanent) background NSManagedObjectContext and set the "parent" to the main ManagedObjectContext (so it will merge the data you import when you save)
Use the performBlock: or performBlockAndWait: APIs to write to that context on its own private queue (in the background)
Example (uses AFNetworking):
[_apiClient GET:[NSString stringWithFormat:#"/mydata"]
parameters:nil
success:^(AFHTTPRequestOperation *operation, id responseObject) {
NSError* jsonError;
id jsonResponse = [NSJSONSerialization JSONObjectWithData:operation.responseData options:kNilOptions error:&jsonError];
if (!jsonError) {
NSArray* parsedData = (NSArray*)jsonResponse;
completionBlock(parsedShares, nil);
} else {
NSManagedObjectContext *context = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
context.parentContext = YOUR_MAIN_CONTEXT; // when you save this context, it will merge its changes into your main context!
// the following code will occur in a background queue for you that CoreData manages
[context performBlock:^{
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
if (![context save:&saveError]) {
NSLog(#"Error saving context: %#", saveError);
} else {
NSLog(#"Saved data import in background!");
}
}];
}
}
failure:^(AFHTTPRequestOperation *operation, NSError *error) {
NSLog(#"There was an error: %#", error)
}];
Docs: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/CoreDataFramework/Classes/NSManagedObjectContext_Class/NSManagedObjectContext.html#//apple_ref/occ/instm/NSManagedObjectContext/performBlockAndWait:
As I alluded to in your other question, it's largely going to be a matter of how long the manipulations are going to take. If the computation required doesn't have a noticeable delay, by all means be lazy about and do your core data on the main thread.
In your other example, you're requesting 20 items from twitter, parsing 20 items and sticking them into CoreData isn't going to be noticeable. The best approach here is to probably continue to just fetch 20 at a time and update in the foreground as each chunk becomes available.
Downloading all items from twitter in one request will take a significant amount of time and computation and it's probably worth creating a separate ManagedObjectModel and synchronizing it with the foreground model. Since you really have one-way data (it always flows twitter->core data->user interface) the likelihood of clashes is minimal, so you can easily use NSManagedObjectContextDidSaveNotification and mergeChangesFromContextDidSaveNotification:

How to avoid the UI freeze while a Managed Object Context is saving?

I want to implement a UI-responsive downloading and parsing of a large data set, saving it with Core Data.
My setup:
I display the downloaded content in a custom view controller. I don't use a NSFetchedResultsController.
There are 3 MOCs:
masterMOC (responsible for saving to disk, NSPrivateQueueConcurrencyType)
mainMOC (used by UI, NSMainQueueConcurrencyType, a child of the masterMOC)
backgroundMOC (responsible for the import from JSON, created in a separate thread, a child of the masterMOC)
I am importing in batches - every 50 items I perform the MOC saving in the following way:
NSError *error;
[backgroundMOC save:&error];
NSManagedObjectContext *masterMOC = backgroundMOC.parentContext; //set during initialization
[masterMOC performBlock:^{
NSError *parentContextError = nil;
[masterMOC save:&parentContextError];
}];
I expect the changes in the mainMOC to be made after the masterMOC is saved. If I try to access some relationship of a random managed object while the masterMOC is saving (saving takes some time), the UI hangs until the saving is completed.
Question: how to avoid the UI freeze while the masterMOC is saving?
Your problem probably is that the data store is blocking while you are writing to it. So, either make the data store non-blocking (this may or may not be possible in your case) or if not viable, make the accessor non-blocking. In the latter case the GUI will not hang, but it also will not update either until the result of the access comes back.

Implementing CoreData using the parent/child approach with temporary objects

Question: What will be the best way to handle bunch of data that just some of the objects should be saved to disk in coredata?
This is the app/problem:
The app will provide users with the possibility to search for different items on the internet.
The search will return a number of objects that will be displayed to the user.
The user should be able to favorites any of these object at any time. Objects that has been favored should be connected to the current logged in user and live after the app has quit.
The app will have iOS6 as base.
I have been using using these resources
Apple's Core Data Programming Guide: Efficiently Importing Data
Implementing Fast and Efficient Core Data Import on iOS 5
iDeveloperTV CoreData performance course
Im currently looking into the parent/child approach with use of 3 Contexts: Master, Main and Confinement context types.
Current possible solution:
MasterContext that perform save on disk (Has the persistentStoreCoordinator)
MainContext that is used by the UI (child of the masterContext)
BackgroundContext to handle new objects from searches. (child of the mainContext)
So the user may do a search that will return 100 objects (imported on the background context and saved up to the main context).
2 of these objects are favored by the user (on the maincontext). The object will be added to the user and set as "should be saved". (On a save the objects will be pushed up to the master context)
When I save the mastercontext I dont want to save all the 100 objects to disk. Just the two objects the user have favored.
So im think about deleting the object that should not be saved to disk just before I do a save on the mastercontext.
- (void) mainContextHasSaved: (NSNotification *) notification {
NSLog(#"Lets save the master");
[_masterManagedObjectContext performBlock:^{
//Loop through all inserted object and check if they should be saved to disk
[self removeObjectThatShouldNotBeSavedToDisk];
NSError *error = nil;
BOOL saveSuccess = [_masterManagedObjectContext save:&error];
if(saveSuccess) {
//Do something
}
}];
}
But after what I understood is that when a save is performed on a parent context, all the changes will be propagated to the children. Then I will loose all the objects except the two that has been stored.
So does anyone know how to solve this kind of problem? Is it something I can do in the example presented above? Or should I create multiple persistentStores and move objects between contexts?
Thanks to all that is willing to help and if more information is needed just ask :)
In a similar project I used this solution which was also favored by the users:
Keep a time stamp attribute in the downloaded items and delete them when the time stamp is older than a certain threshold and they are not marked as favorite.

Resources