I want to implement a UI-responsive downloading and parsing of a large data set, saving it with Core Data.
My setup:
I display the downloaded content in a custom view controller. I don't use a NSFetchedResultsController.
There are 3 MOCs:
masterMOC (responsible for saving to disk, NSPrivateQueueConcurrencyType)
mainMOC (used by UI, NSMainQueueConcurrencyType, a child of the masterMOC)
backgroundMOC (responsible for the import from JSON, created in a separate thread, a child of the masterMOC)
I am importing in batches - every 50 items I perform the MOC saving in the following way:
NSError *error;
[backgroundMOC save:&error];
NSManagedObjectContext *masterMOC = backgroundMOC.parentContext; //set during initialization
[masterMOC performBlock:^{
NSError *parentContextError = nil;
[masterMOC save:&parentContextError];
}];
I expect the changes in the mainMOC to be made after the masterMOC is saved. If I try to access some relationship of a random managed object while the masterMOC is saving (saving takes some time), the UI hangs until the saving is completed.
Question: how to avoid the UI freeze while the masterMOC is saving?
Your problem probably is that the data store is blocking while you are writing to it. So, either make the data store non-blocking (this may or may not be possible in your case) or if not viable, make the accessor non-blocking. In the latter case the GUI will not hang, but it also will not update either until the result of the access comes back.
Related
Still working on converting an app over from downloading information every time it uses or displays it, to caching it on-phone using CoreData (courtesy of MagicalRecord). This is on iOS 7
Because we don't have a data-push system set up to automatically update the phone's cached data whenever some data changes on the backend, I've been thinking over the last many months (as we worked on other aspects of the app) how to manage keeping a local copy of the data on the phone and being able to have the most up to date data in the cache.
I realized that as long as I still fetch the data every time :-( I can use the phone's CoreData backed cache of data to display and use, and just use the fetch of the data to update the on-phone database.
So I have been converting over the main data objects from being downloaded data making up a complete object, to these main data objects being light stand-in objects for CoreData objects.
Basically, each of the normal data objects in the app, instead of containing all the properties of the object internally, contains only the objectIDof the underlying CoreData object and maybe the app specific ID internally, and all other properties are dynamic and gotten from the CoreData object and passed through (most properties are read-only and updates are done through bulk-rewriting of the core data from passed in JSON)
Like this:
- (NSString *)amount
{
__block NSString *result = nil;
NSManagedObjectContext *localContext = [NSManagedObjectContext MR_newContext];
[localContext performBlockAndWait:^{
FinTransaction *transaction = (FinTransaction *)[localContext existingObjectWithID:[self objectID] error:nil];
if (nil != transaction)
{
result = [transaction.amount stringValue];
}
}];
return result;
}
Occasionally there is one that needs to be set and those look like this:
- (void)setStatus:(MyTransactionStatus)status
{
[MagicalRecord saveWithBlock:^(NSManagedObjectContext *localContext) {
FinTransaction *transaction = (FinTransaction *)[localContext existingObjectWithID:[self objectID] error:nil];
if (nil != transaction)
{
transaction.statusValue = status;
}
} completion:^(BOOL success, NSError *error){}];
}
Now, my issue is that I have a view controller that basically uses an NSFetchedResultsController to display stored data from the local phone's CoreData database in a table view. At the same time as this is happening, and the user may start to scroll through the data, the phone spins off a thread to download updates to the data and then starts updating the CoreData data store with the updated data, at which point it then runs an asynchronous GCD call back on the main thread to have the fetched results controller refetch its data and and tells the table view to reload.
The problem is that if a user is scrolling through the initial fetched results controller fetched data and table view load, and the background thread is updating the same Core Data objects in the background, deadlocks occur. It is not the exact same entities being fetched and rewritten (when a deadlock occurs), i.e., not that object ID 1 is being read and written, but that the same persistent data store is being used.
Every access, read or write, happens in a MR_saveWithBlock or MR_saveWithBlockAndWait (writes/updates of data) as the case may be, and a [localContext performBlock:] or [localContext performBlockAndWait:] as may be appropriate. Each separate read or write has its own NSManagedObjectContext. I have not seen any where there are stray pending changes hanging around, and the actual places it blocks and deadlocks is not always the same, but always has to do with the main thread reading from the same persistent store as the background thread is using to update the data.
The fetched results controller is being created like this:
_frController = [[NSFetchedResultsController alloc] initWithFetchRequest:fetchRequest
managedObjectContext:[NSManagedObjectContext MR_rootSavingContext]
sectionNameKeyPath:sectionKeyPath
cacheName:nil];
and then an performFetch is done.
How can I best structure this sort of action where I need to display the extent data in a table view and update the data store in the background with new data?
While I am using MagicalRecord for most of it, I am open to comments, answers, etc with or without (straight CD) using MagicalRecord.
So the way I'd handle this is to look at having two managed object contexts each with its own persistent store coordinator. Both of the persistent store coordinators talk to the same persistent store on disk.
This approach is outlined in some detail in Session 211 from WWDC 2013 — "Core Data Performance Optimization and Debugging", which you can get to on Apple's Developer Site for WWDC 2013.
In order to use this approach with MagicalRecord, you will need to look at using the upcoming MagicalRecord 3.0 release, with the ClassicWithBackgroundCoordinatorSQLiteMagicalRecordStack (yes, that name needs work!). It implements the approach outlined in the WWDC session, although you need to be aware that there will be changes needed to your project to support MagicalRecord 3, and that it's also not quite released yet.
Essentially what you end up with is:
1 x Main Thread Context: You use this to populate your UI, and for your fetched results controllers, etc. Don't ever make changes in this context.
1 x Private Queue Context: Make all of your changes using the block-based saved methods — they automatically funnel through this context and save to disk.
I hope that makes sense — definitely watch the WWDC session — they use some great animated diagrams to explain why this approach is faster (and shouldn't block the main thread as much as the approach you're using now).
I'm happy to go into more detail if you need it.
I'm trying to implement some basic UIManagedDocument import/export functionality into my app, mainly for dev so that I can easily inspect the document contents and more crucially preserve a set of test data when I start trying to iterate on my CoreData models.
All I am trying to do is load some JSON data from a local file and inject it into my apps UIManagedDocument. The UIManagedDocument's ManagedObjectContext contents are visualised in my app using some Core Data Table View Controllers from the Stanford iOS courses.
I thought I'd try to write this with some threading to keep the UI responsive and to learn how to do it. So I've done something like this
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
// read JSON data file
// parse JSON into dictionaries etc
// iterate over dictionaries
// try to create a new core data entity with my document's ManagedObjectContext
} );
At first I thought this was working. No errors, asserts, crashes triggered and upon looking at my CoreData TableViews I could see the newly added objects in my UI. Unfortunately the newly added objects were seemingly never saved back to the store. I even hooked up to listen to the NSManagedObjectContextDidSaveNotification from my UIDocument's managedObjectContext and saw it wasn't triggering on pressing the home button, like it usually does if it has some changes performed in the app with my UI pending. Infact even doing these operations in the UI wouldn't cause the notification and saving to occur so it was clearly not happy.
I unrolled the code from within the background queue and ran it on the main thread synchronously and everything worked ok, the new data was saved correctly.
I started reading about the complexities of threading and coredata, the documentation seemed to suggest using the UIDocument's ManagedObjectContext's parent ManagedObjectContext to perform operations on in the background so I tried doing the same code again using this parent context, so as follows
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
// read JSON data file
// parse JSON into dictionaries etc
// iterate over dictionaries
// try to create a new core data entity with my document's ManagedObjectContext parent ManagedObjectContext
} );
This time for some reason the CoreData TableView controllers no longer updated to show the newly injected objects. Even after explicitly calling save on the parent context, nothing appeared. However on quitting the app and reloading the app the newly injected objects did seem to be added correctly. Interestingly at this point i'd left a fetchrequest with a cachename specified and that threw up an error on this first run of the app after injecting the objects this way. I guess somehow the way the object had come from the parent context directly perhaps invalidated the cache somehow, that's still something I don't fully understand. Even changing the cache to nil didn't fix the issue of the table views not updated the same session as when the objects were injected into the parent context.
Looking elsewhere I've seen some uses of the managedObjectContext performBlock suggested. Another case where someone has said you must call
[document updateChangeCount:UIDocumentChangeDone]
after all changes to ensure the saving is performed, or perhaps using
- (void)autosaveWithCompletionHandler:(void (^)(BOOL success))completionHandler
instead. Though elsewhere I've seen mentioned that saving should be enough to push context contents through the hierarchy. Does saving only work from child -> parent and not from parent -> child.
Or am I just doing it wrong?
Anyone's time and help is really appreciated! Cheers.
please look at the 'Parent/Child Contexts' section in the Multi-Context CoreData.
Whenever a child MOC saves the parent learns about these changes and this causes the fetched results controllers to be informed about these changes as well. This does not yet persist the data however, since the background MOCs don’t know about the PSC. To get the data to disk you need an additional saveContext: on the main queue MOC.
I do saving to the PSC n following way:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
{
// read JSON data file
// parse JSON into dictionaries etc
// iterate over dictionaries
NSError* error;
if (self.managedObjectContext.hasChanges)
{
[self.managedObjectContext save: &error];
}
// Save main context
dispatch_sync(dispatch_get_main_queue(), ^
{
[[AXContextHelper sharedInstance] saveMainContext];
});
}
After looking into all the suggestions I could find for similar problems, none of them seemed to help.
The scenario i was describing in the end had the store stuff to file handled ok, but the UI not updating. At the time when I do this import/export I'm in a view ontop of the core data table view controller that doesn't update when I inject all these JSON objects, so in the end all I did was force that controller to re-fetch it's data. I believe the NSFetchedResultsController is meant to monitor the managedObjectContext and update the fetch request as required. For whatever reason, this wasn't working.
With the force re-fetch called, everything seems to work ok. The newly injected entities appear in my tables, and are saved to the store files.
Either there are some bugs in these objects, or I'm still using it wrong but I see other problems... my import/export data both work now on background threads with the parent context. What I've noticed today is that I can use the UI to properly insert and edit some objects using the child context on the main thread. I don't deliberately call anything at the moment after making these edits so i guess the edits are are still pending till core data decides to save. If i then go to my export feature and use it using the parent context then i find the newly edited or inserted objects aren't there.
Clearly there's no messaging happening in time from child -> parent, and messaging from background thread edits on the the parent itself just doesn't seem to work.. I never get any notifications for the edits i perform in a thread on the parent context, even though they seem to work. If I force the save to happen via the documents then the passed in dictionaries show no edits being made, even though the edits are saved correctly to store files somehow.
I guess i now need to force my document to save after every operation, or at least before every operation i'm about to do on another thread to ensure the parent/child are both in sync.
Still the problem I originally described I've managed to get around, but I'm still left wondering quite how I should be using the parent and child context of a UIManagedDocument to avoid encountering these issues.
I've learned that generally, intensive tasks should take place on background threads, as if they're on the main thread they'll block user interaction and interface updates.
Does Core Data fall under that umbrella? I received a great answer to my question about loading images asynchronously in a UITableView, but I'm curious how to then work with Core Data as the backend.
I know Apple has a Core Data Concurrency guide, but I'm curious in which cases one is supposed to use Core Data in the background.
As a quick example, say I'm making a Twitter client and want to get all the tweet information (tweet text, username, user avatar, linked images, etc.). I asynchronously download that information and receive some JSON from the Twitter API that I then parse. Should I then do a dispatch_async(dispatch_get_main_queue()...) when I add the information to Core Data?
I also then want to download the user's avatar, but I want to do that separately from presenting the tweet, so I can present the tweet quickly as possible, then present the image when ready. So I asynchronously download the image. Should I then update that Core Data item asynchronously?
Am I in a situation where I don't need multi-threaded Core Data at all? If so, when would be a situation where I need it?
I've learned that generally, intensive tasks should take place on background threads, as if they're on the main thread they'll block user interaction and interface updates.
Does Core Data fall under that umbrella?
Yes, actually.
Core Data tasks can and should - where possible - be executed on background threads or on non-main queues.
It's important to note though, that each managed object is associated to a certain execution context (a thread or a dispatch queue). Accessing a managed object MUST be executed only from within this execution context. This association comes from the fact that each managed object is registered with a certain Managed Object Context, and this managed object context is associated to a certain execution context when it is created.
See also:
[NSManagedObjectContext initWithConcurrencyType](),
[NSManagedObjectContext performBlock]
[NSManagedObjectContext performBlockAndWait]
Consequently, when displaying properties of managed objects, this involves UIKit, and since UIKit methods MUST be executed on the main thread, the managed object's context must be associated to the main thread or main queue.
Otherwise, Core Data and user code can access managed objects from arbitrary execution contexts, as long as this is the one to which the managed object is associated with.
The below picture is an Instrument Time Profile which shows quite clearly how Core Data tasks can be distributed on different threads:
The highlighted "spike" in the CPU activity shows a task which performs the following:
Load 1000 JSON objects from origin (this is just a local JSON file),
Create managed objects in batches of 100 on a private context
Merge them into the Cora Data Stack main context
Save the context (each batch)
Finally, fetch all managed objects into the main context
As we can see, only 26.4% of the CPU load will be executed on the main thread.
Core Data does indeed fall under that umbrella. Particularly for downloading and saving data, but also possibly for fetching depending on the number of objects in the data store and the predicate to be applied.
Generally, I'd push all object creation and saving which is coming from a server onto a background thread. I'd only update and save objects on the main thread if they're user generated (because it would only be one, updated slowly and infrequently).
Downloading your twitter data is a good example as there will potentially be a good amount of data to process. You should process and save the data on a background thread and save it up to the persistent store there. The persistent store should then merge the changes down to the main thread context for you (assuming you have the contexts configured nicely - use a 3rd party framework for that like MagicalRecord).
Again, for the avatar update, you're already on a background thread so you might as well stay there :-)
You might not need to use multiple threads now. If you only download the 5 most recent tweets then you might not notice the difference. But using a framework can make the multi-threading relatively easy. And using a fetched results controller can make knowing when the UI should be updated on the main thread very easy. So, it's worthwhile taking the time to understand the setup and using it.
The best way to handle behavior like this is to use multiple NSManagedObjectContexts. The "main" context you create like so:
_mainManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType];
_mainManagedObjectContext.persistentStoreCoordinator = [self mainPersistentStoreCoordinator];
You're going to want to do any heavy writes on a different NSManagedObjectContext to avoid blocking your UI thread as you import (which can be quite noticeable on large or frequent operations to your Main context).
To achieve this, you would do something like this:
Retrieve your data asynchronously from the network
Spin up a temporary (or perhaps a permanent) background NSManagedObjectContext and set the "parent" to the main ManagedObjectContext (so it will merge the data you import when you save)
Use the performBlock: or performBlockAndWait: APIs to write to that context on its own private queue (in the background)
Example (uses AFNetworking):
[_apiClient GET:[NSString stringWithFormat:#"/mydata"]
parameters:nil
success:^(AFHTTPRequestOperation *operation, id responseObject) {
NSError* jsonError;
id jsonResponse = [NSJSONSerialization JSONObjectWithData:operation.responseData options:kNilOptions error:&jsonError];
if (!jsonError) {
NSArray* parsedData = (NSArray*)jsonResponse;
completionBlock(parsedShares, nil);
} else {
NSManagedObjectContext *context = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
context.parentContext = YOUR_MAIN_CONTEXT; // when you save this context, it will merge its changes into your main context!
// the following code will occur in a background queue for you that CoreData manages
[context performBlock:^{
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
// THIS IS WHERE YOU IMPORT YOUR DATA INTO CORE DATA
if (![context save:&saveError]) {
NSLog(#"Error saving context: %#", saveError);
} else {
NSLog(#"Saved data import in background!");
}
}];
}
}
failure:^(AFHTTPRequestOperation *operation, NSError *error) {
NSLog(#"There was an error: %#", error)
}];
Docs: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/CoreDataFramework/Classes/NSManagedObjectContext_Class/NSManagedObjectContext.html#//apple_ref/occ/instm/NSManagedObjectContext/performBlockAndWait:
As I alluded to in your other question, it's largely going to be a matter of how long the manipulations are going to take. If the computation required doesn't have a noticeable delay, by all means be lazy about and do your core data on the main thread.
In your other example, you're requesting 20 items from twitter, parsing 20 items and sticking them into CoreData isn't going to be noticeable. The best approach here is to probably continue to just fetch 20 at a time and update in the foreground as each chunk becomes available.
Downloading all items from twitter in one request will take a significant amount of time and computation and it's probably worth creating a separate ManagedObjectModel and synchronizing it with the foreground model. Since you really have one-way data (it always flows twitter->core data->user interface) the likelihood of clashes is minimal, so you can easily use NSManagedObjectContextDidSaveNotification and mergeChangesFromContextDidSaveNotification:
I have an application which continuously receives a stream of XML messages from a TCP/IP endpoint. Upon receipt of each message, the application digests it's contents into a set of core data entities. This is accomplished via a three context structure:
Master (Private queue)
Main (Main queue -> Master)
Stream (Private queue -> Main)
This arrangement keeps the stream processing off the main thread. The application typically receives anywhere from 10 - 150 messages every second or two. Saving of the Stream context takes place after each message is deconstructed and persisted. CPU usage is typically short of 15% on an A6 level device.
My problem however is memory. If I hook up an NSFetchedResultsController to the Main context I get a nice flow of the messages as they arrive. However, if I profile I notice that my NSManagedObject count gradually increases. Eventually memory pressure will cause the app to terminate.
After 12 minutes of profiling, the app has consumed 6300 XML messages and parsed 121,000 properties. This consumes 7.8MB for the properties, 438KB for the messages and the total app size is now 54MB. Obviously this isn't sustainable.
Instruments notes that all of the objects are still live. Trolling around the interwebs leads me to believe I might have a retain cycle causing the objects to not be faulted. However, the suggestion of using "refreshObject" isn't clear in the documentation that it would apply here.
Once the XML has been received, a Message entity is created. Next a Type entity is created using the root node of the XML as it's name - and associated bits. Similarly for each element and sub element of those elements and any inline properties of the XML a property element is created. This is the fun part as it has a reference to the message (for a flat representation of all properties) as well as a hierarchal childProperties relationship to itself. At the end of this process the context is saved and the Main context picks it up and the FRC displays the new row.
One thought was to reset the Stream context after a save every few hundred of messages persisted. If I disconnect the FRC, I can stay basically level - however this feels wrong and doesn't solve the problem when I wire the FRC back up.
Any thoughts would be appreciated.
I would suggest configuring your Stream context with the same persistent store coordinator that is used for the Master context. And maybe periodically reset the stream context.
In the current configuration Stream context fill put additional pressure to its parents. And if big updates are happening in the Stream context, this pressure becomes more visible.
First, when the Stream context needs to do something that requires a lock, it will lock both parents.
Second, when save happens in the Stream context, all the changes are pushed back to the parent, the Main context. And you don’t have control over it. If there is a fetched results controller in the Main context, then on save it will replay all the changes one-by-one. And if the update is big, it will bring a big overhead. Definitely in CPU and probably in memory.
I think the best pattern for handling big updates in the background and refreshing the UI (especially with the fetched results controller) is to configure the context that does big updates directly with persistent store coordinator. And then, when big a update happens, just refetch in the UI context. And don’t forget to set fetch batch size on the fetch request to some meaningful to your case value. You could start with the number of cells visible on the screen.
This pattern is more efficient but comes with complexity cost. You need to think how to refresh the data in other contexts. You need to take care of this because Core Data doesn’t touch objects that are fully realized. (Setting setShouldRefreshRefetchedObjects doesn’t help either because of the bug confirmed to me by Apple.)
For example, you fetched some object in the Main context, accessed its property for displaying it on the screen. This object is not a fault any more. Then your Stream context (now configured with the persistent store coordinator directly) updated the same property. Even if you refetch in the Main context and the object will be in the search results, object properties will not be updated.
So you could use something like this:
- (void)refreshObjectsOnContextDidSaveNotification:(NSNotification *)notification {
NSSet *updatedObjects = notification.userInfo[NSUpdatedObjectsKey];
NSSet *updatedObjectIDs = [updatedObjects valueForKey:#"objectID"];
[self.mainContext performBlock:^{
for (NSManagedObject *object in [self.mainContext registeredObjects]) {
if (![object isFault] && [updatedObjectIDs containsObject:[object objectID]]) {
[self.mainContext refreshObject:object mergeChanges:YES];
}
}
}];
[self.masterContext performBlock:^{
for (NSManagedObject *object in [self.masterContext registeredObjects]) {
if (![object isFault] && [updatedObjectIDs containsObject:[object objectID]]) {
[self.masterContext refreshObject:object mergeChanges:YES];
}
}
}];
}
This will refresh updated objects in main and master contexts.
When the save in the Stream context is not huge you could simply merge changes using the standard merge method into other two contexts. When fetched results controller is used, you’ll be able to see nice cell animations on object deletion and insertion. The number of objects affected in a save you can get from NSInsertedObjectsKey, NSUpdatedObjectsKey, and NSDeletedObjectsKey keys of the user info in the context-did-save notification.
And after each big save you could reset the Stream context. Just don’t forget that you can’t access any previously fetched objects in this context after the reset.
Last year I used RestKit 0.10 to seamlessly download and save core data objects in background. However, when I tried to use restkit in 2013, I noticed that they have taken out the ActiveRecord pattern, which I relied upon to abstract away all the unpleasantness of background saving.
I found that the ActiveRecord pattern exists in MagicalRecord framework, but most of the documentation I could find is for version 2.x, while my cocoapods install 3.x.
I spent the last 2 hours searching, and find a lot of answers that are really out of date and no longer work for these new frameworks.
This poses the question: what the standard/easiest way to deal with saving core data objects in background using frameworks available in 2013? Should I try some other framework?
If you don't use any external library like Magical Record or RestKit, but simply go for the all manual stuff, you can take advantage of the new NSManagedObjectContext APIs.
You can now have contexts nested with a parent-child relationship and you can also tell each context to perform a block in it's own thread. My advice, therefore is to have the following structure for your application:
1) A background saving context. This will be the only context that saves and reads data directly to/from the database.
2) A context initalized on the main thread that will be your point of access for everything you need to do in the application, especially updating the UI. This context will be a child of the saving context.
3) As needed, you'll create background contextes that perform work on background threads, e.g. loading data from the network and serialize this data in NSManagedObject instances. This contextes will be children of the main context.
4) Every time you call -[NSManagedObjectContext save:] on a context, you should also call the same method on it's parentContext. To do this you could have a convenience method in a category on NSManagedObjectContext that reads something like this:
- (void)saveSelfAndParent {
[self save:NULL];
[self.parentContext performBlock:^{
[self.parentContext saveSelfAndParent];
}];
}
This is already a thread safe configuration and your changes will propagate the changes up to the database. Note that as the saving context will have no parent (and thus self.parentContext will be nil), the performBlock: won't crash the app.
Here's an example what you need to do to create a new entity assuming you kick off your background work using Grand Central Dispatch (GCD):
dispatch_async(dispatch_async_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
NSManagedObjectContext *context = [[NSManagedObjectContext alloc] initWithConcurrencyType: NSPrivateQueueConcurrencyType];
context.parentContext = mainContext;
// do some expensive job
...
// initialize a new NSManagedObject instance using the information we calculated
NSManagedObject *myObject = ...;
// once we're done, let's save the context
[context saveSelfAndParent];
});
Note that we initialized the context with a private queue concurrency type (NSPrivateQueueConcurrencyType) which tells the context that he's a background context. That is very important!
That's all! :)
For more information refer to the NSManagedObjectContext Class Reference.