MagicalRecord: saving context after deleting is too long - ios

I have next problem: I have database with about 7000 of entities, when I need to update them (I have XML file which I parse) for first I delete all entities, after it I parse XML file, later I create new entities and save context. Earlier all worked perfect: no freezes, no crashes - all was fast on iOS 7.
But with release iOS 8 there were problems:
I resolved this problem by providing one context for all operations: deleting, creating and saving.
BUT! What I've got:
When I just install app on my device all goes well: there are no deleting, only creating entities, 7000 terms and 7 groups are parsed so fast (about 4 seconds on iPhone 6), saving goes fast too.
When I changing version of DB in my plist file (increase) my parser start this algorthm:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
[MagicalRecord saveWithBlock:^(NSManagedObjectContext *localContext) {
[Term MR_truncateAllInContext:localContext];
[Group MR_truncateAllInContext:localContext];
} completion:^(BOOL success, NSError *error) {
[self parseTermsInContext:[NSManagedObjectContext contextForCurrentThread] from:self.count];
}];
});
"saveWithBlock" method blocks Thread 1 (in profiler), my CPU loaded on 99-108(error apparently) percents (with every next update saving operation takes more and more seconds, from 20 and more, more than 120 seconds).
I've tried this way (I gathered all operations in one method for you):
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
NSManagedObjectContext *localContext = [NSManagedObjectContext contextForCurrentThread];
NSMutableArray *objects = [NSMutableArray arrayWithArray:[Group MR_findAllInContext:localContext]];
[objects addObjectsFromArray:[Term MR_findAllInContext:localContext]];
if (objects && [objects count] > 0) {
for (NSManagedObject *object in objects) {
[object MR_deleteInContext:localContext];
}
[localContext MR_saveToPersistentStoreAndWait];
}
[self parseTermsInContext:localContext from:self.count];
});
Here operation "MR_saveToPersistentStoreAndWait" take a long time too like "saveWithBlock".
I tried way without saving context after deleting, that is line "[localContext MR_saveToPersistentStoreAndWait];" does not exists. In this way Groups and Terms was deleted so fast too, later they was parsed so fast too but saving context was so long.
And I don't know why but even if I start deleting and saving processes in background thread saving operation froze UI thread (in UI thread I show progress from 0 to 100). When I parse XML in this thread I send message to view that one term is parsed and setting progress in percents, delegate calls method for setting progress in ProgressView in main queue.
I have not another threads that can operate core data objects.
There is link with work of app: http://rghost.ru/60274051
After 6 seconds: for test purposes I start NSTimer that updated progress every 0.3 second with fake data to fill progress for 50% before starting deleting and saving operation (updating progress goes in main queue). Timer fires several times then saving process starts in background thread but blocks main thread (as I understand) and moves setting progress operation to end (if I understand correctly).
1:08 : then after saving ends I start parsing xml-file. This is thread where I saved context after deleting. You can see progress updating. In this video it works with bugs because of a lot of manipulations, but You can believe me that it works and looks fine. After parsing 7000 objects I save context AGAIN and saving operation does not block UI thread.
Additional info:
Relations:

} completion:^(BOOL success, NSError *error) {
[self parseTermsInContext:[NSManagedObjectContext contextForCurrentThread] from:self.count];
}];
block above called in Main Thread (block UI), so contextForCurrentThread == UI Thread
instead of iterate an delete all objects you can use TruncateAll
My suggestion is:
// saveWithBlock - already perform block in background thread
[MagicalRecord saveWithBlock:^(NSManagedObjectContext *localContext) {
[Term MR_truncateAllInContext:localContext];
[Group MR_truncateAllInContext:localContext];
} completion:^(BOOL success, NSError *error) {
[MagicalRecord saveWithBlock:^(NSManagedObjectContext *localContext) {
// self must be weak!
[self parseTermsInContext:localContext from:self.count];
} completion:^(BOOL success, NSError *error) {
// Update UI
}];
}];

Related

NSManagedObjects created on background turning into faults on main thread

I'm dealing with a strange case. I'm getting some data from a API, transform the response JSON into NSManagedObjects and saving them, all that in a background thread and using a NSPrivateQueueConcurrencyType context whose parent is a NSMainQueueConcurrencyType.
At the same time I'm creating the NSManagedObjects, I'm placing them inside an array to use it in my view controller via a completion block executed on the main therad. Here's a simplified version of what I'm doing.
- (void)getObjectFromAPIWithCompletion:((^)(NSArray *objects, NSError *error))completion
{
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
NSManagedObjectContext *backgroundMoc = [[CoreDataStack sharedManager] newChildContextWithConcurrencyType:NSPrivateQueueConcurrencyType];
NSArray *items = [self processToParseResponseFromAPIAndSaveInCoreDataUsingContext:backgroundContext];
[backgroundMoc save:nil];
dispatch_async(dispatch_get_main_queue(), ^{
//Beyond this point, all properties of items are nil or 0 (if integer)
if(completion) completion(items, error);
});
});
}
The weird thing happens inside the completion block. I put a breakpoint on the first line of that block and print the items array on the console. Everything looks fine, with all the items (50 in this case) and their properties are OK.
The problem is when that after that line (where I'm not changing anything on the items array) all objects turn into faults and I'm not getting any data on the properties.
Don't know what is the cause of this rare behaviour. Any idea?
Thanks.
EDIT: Hal's answer put me on the right track so here's what I did to fix this problem. Just before calling the block that will be executed on the main thread I "move" the managed objects from the background context to the main context, fetching with their objectIDs. Like this:
NSArray *objects = //Objects created on background thread with Private queue
NSMutableArray *objectsIDs = [NSMutableArray array];
for (Object *object in object) {
[objectsIDs addObject:object.objectID];
}
//Save on private managed object context and on completion...
[self.managedObjectContext saveWithCompletion:^(NSError *error) {
NSManagedObjectContext *mainMOC = [[CoreDataManager sharedManager] mainContext];
NSMutableArray *fetchedObjects = [NSMutableArray array];
for (NSManagedObjectID *objectID in objectsIDs) {
[fetchedArticles addObject:[mainMOC objectWithID:objectID]];
}
if (completion) completion(fetchedObjects, pagination, nil);
}];
This way, all objects are not faults on the main thread.
If those objects (the contents of items) have been created on the background thread, you can't use them in the main thread. You can't pass NSManagedObject instances between threads.
You'll have to pass NSManagedObjectIDs (which won't be permanent until you've saved the background MOC), or use an NSFetchedResultsController (pointing to the main MOC) on the main thread. This is one of the cases that NSFetchedResultsController was designed for, so I recommend you go that route.

Core Data Concurrency Import Performance

I’m working on an app connected to a web service which retrieves lot of data during app launch. I use concurrency to avoid UI blocking. I choosed the following Core Data Stack pattern : background private moc —> main moc —> writer private moc —> coordinator —> file.
The problem occures when operations are being imported. The CPU is 100% used and the app gets slow along the process. I work with batches of 300 objects for a total import of about 10,000 objects.
For each batch, an NSOperation is created with an associated temporary moc, child of the background one. Operation is enqueue in an NSOperationQueue.
When the importing jobs are done, the app get even slower, depending on the number of jobs running. I also note that when the app is killed, and relaunched, it’s really way more usable and fast.
My memory footprint changes between 40Mo and 60Mo when importing. Do you think it’s too much?
Do you think my stack pattern is appropriate for my needs? Should I migrate to a stack with 2 coordinators?
Moreover, when fetching data to display in tableView, should I use performBlockAndWait to get data immediately before displaying the view ?
Thanks for your help
Your stack as described is fine.
CPU usage can be misleading. You want to make sure you are not on the main thread as that will cause most of your slowness and/or stuttering in the app.
When you watch your app in Instruments, what is taking the most time? How much time is spent on the main queue?
In general, imports shouldn't be causing the CPU to sit at 100%. If you are doing that from a background thread there is most likely some performance gains to be made.
If should share your import code and or Instruments trace so that I can see what is going on.
I think your setup is problematic. You state that the child of the background managed object context is main thread and that you create such children to import. This is bound to cause UI glitches.
Also, I believe that relying on NSOperation is unnecessary over-engineering. You should use the NSManagedObjectContext block APIs instead.
My recommended setup would be:
RootContext (background, writing to persistent store) -> parent of
MainContext (foreground, UI) -> parent of
WorkerContext(s) (background, created and discarded ad hoc)
You can create worker contexts in the callbacks of your web calls to do the heavy lifting for the import. Make sure you are using the block APIs and confine all objects to the local context. You save the context to push the changes up to the main thread (which can already start displaying data before it is saved to the store), and periodically, you save the main context and the writer context, always using the bock APIs.
A typical such saveContext function that can be called thread safe (here self refers to the data manager singleton or app delegate):
func saveContext () {
if self.managedObjectContext.hasChanges {
self.managedObjectContext.performBlocAndWait {
do { try self.managedObjectContext.save() }
catch let error as NSError {
print("Unresolved error while saving main context \(error), \(error.userInfo)")
}
}
self.rootContext.performBlockAndWait {
do { try self.rootContext.save() }
catch let error as NSError {
print("Unresolved error while saving to persistent store \(error), \(error.userInfo)")
}
}
}
}
After few days of test and instruments trace, I can give you more details. The following snippet shows how I save my context (based on parent/child pattern) from a shared instance :
- (void)save {
[self.backgroundManagedObjectContext performBlockAndWait:^{
[self saveContext:self.backgroundManagedObjectContext];
[self.mainManagedObjectContext performBlock:^{
[self saveContext:self.mainManagedObjectContext];
[self.writerManagedObjectContext performBlock:^{
[self saveContext:self.writerManagedObjectContext];
}];
}];
}];
}
- (void)saveContext:(NSManagedObjectContext*)context {
NSError *error = nil;
if ([context hasChanges] && ![context save:&error]) {
NSLog(#"Unresolved error %#, %#", error, [error userInfo]);
}
}
Then each import job is perform on the background context thank to a synchronous operation. The following method is fired in operation main.
- (void)operationDidStart
{
NSManagedObjectContext *moc = self.context;
NSMutableArray *insertedOrUpdatedObjects = [NSMutableArray array];
NSMutableArray *subJSONs = [NSMutableArray array];
NSUInteger numberOfJobs = ceil((double)self.JSONToImport.count/self.batchSize);
for (int i = 0; i < numberOfJobs; i++) {
NSUInteger startIndex = i * self.batchSize;
NSUInteger count = MIN(self.JSONToImport.count - startIndex, self.batchSize);
NSArray *arrayRange = [self.JSONToImport subarrayWithRange:NSMakeRange(startIndex, count)];
[subJSONs addObject:arrayRange];
}
__block NSUInteger numberOfEndedJobs = 0;
for (NSArray *subJSON in subJSONs) {
[moc performBlock:^{
[self startJobWithJSON:subJSON context:moc completion:^(NSArray *importedObjects, NSError *error) {
numberOfEndedJobs++;
if (!error && importedObjects && importedObjects.count > 0) {
[insertedOrUpdatedObjects addObjectsFromArray:importedObjects];
}
if (numberOfEndedJobs == numberOfJobs) {
[[CoreDataManager manager] save];
if (self.operationCompletion) {
self.operationCompletion(self, insertedOrUpdatedObjects, error);
}
}
}];
}];
}
}
As you can see, I segment my import in batches (of 500). The operation perform each batch on the background context queue and I save my stack when all batches are ended.
It seems the save method take 23% of CPU usage for each thread thanks to Time Profiler.
Hope to be as clear as possible.

Magical Record background save seems to be blocking UI

I have a NSOperation that I put in a queue. The NSOperation does some long running photo processing then I save the information/meta data in core data for that photo. In the main method of my custom NSOperation class is where I execute the below block of code
-(void)main
{
//CODE ABOVE HANDLES PHOTO PROCESSING...
//........
//THEN I SAVE ALL DATA BELOW LIKE SO
[MagicalRecord saveWithBlock:^(NSManagedObjectContext *localContext) {
Post *post = [Post createInContext:localContext];
//set about 15 pieces of data, all strings and floats
post.XXXX = XXXXX;
post.DDDD = DDDDD;
etc...
} completion:^(BOOL success, NSError *error) {
NSLog(#"Done saving");
}];
}
My issue is that even with only 3 photos when it saves it really freezes my UI. I would have thought executing this in the NSOperation I would be fine.
I should add that each NSOperation processes one photo, so at times the queue could have 5-10 photos, but I would not think this would make any difference, even with just three like I said its freezing the UI.
Thank you for the help.
UPDATE:------------*--------------
I switched to version 2.2 but that seems to be blocking the UI even more...also now I'm using
-(void)main
{
NSManagedObjectContext *localContext = [NSManagedObjectContext contextForCurrentThread];
//CODE BELOW HANDLES PHOTO PROCESSING...
//........
//THEN I SAVE ALL DATA BELOW LIKE SO
Post *post = [Post createInContext:localContext];
//set about 15 pieces of data, all strings and floats
post.XXXX = XXXXX;
post.DDDD = DDDDD;
[localContext saveToPersistentStoreWithCompletion:^(BOOL success, NSError *error) {
}];
}
This is all done in my NSOperation class, am I doing something wrong?
Don't put the saveWithBlock calls in a background thread. You're effectively creating a background thread from a background thread, which, in this case, is just slowing you down. You should just be able to call saveWithBlock and it should put all your saving code in the background. However, I'm also noticed that you make all your changes in the main UI page of the code, and only call save afterward. This is the wrong usage of this method. You want to do something more like this:
[MagicalRecord saveWithBlock:^(NSManagedObjectContext *localContext) {
Post *post = [Post createInContext:localContext];
//photo processing
//update post from photo processing
} completion:^(BOOL success, NSError *error) {
//This is called when data is in the store, and is called on the main thread
}];
If you do need an NSOperation, I suggest a different pattern:
- (void) main {
NSManagedObjectContext *localContext = [NSManagedObjectContext confinementContext];
// Do your photo stuff here
Post *post = [Post createInContext:localContext];
//more stuff to update post object
[localContext saveToPersistentStoreAndWait];
}
Be careful in how you start the operation.
[operation start]
will start the operation on the current thread, so if you call it from the main thread (which is the UI one) it will block the interface.
You should add the operation to a queue, so that it runs in background without hogging the main thread
[[NSOperationQueue new] addOperation:operation];

How to create multiple objects in background?

I'm using MagicalRecord 2.0.3 and I can't really figure out how to save data in the background.
According to the documentation, something like this should work:
[MagicalRecord saveInBackgroundWithBlock:^(NSManagedObjectContext *localContext) {
// Do this hundreds of times
[MyObject createInContext:localContext];
}];
However, nothing is saved to the database. I've seen multiple people posting solutions similar to this:
[MagicalRecord saveInBackgroundWithBlock:^(NSManagedObjectContext *localContext) {
// Do this hundreds of times
[MyObject createInContext:localContext];
} completion:^{
[[NSOperationQueue mainQueue] addOperationWithBlock:^{
[[NSManagedObjectContext defaultContext] saveNestedContexts];
}];
}];
This does save my data in the database, however since the save happens on the main thread, my application is unresponsive for a while (with my dataset, about 3 seconds which is way too long).
I've also tried this, but it also blocks up while saving:
self.queue = [[NSOperationQueue alloc] init];
[self.queue addOperationWithBlock:^{
NSManagedObjectContext *localContext = [NSManagedObjectContext contextForCurrentThread];
// Do this hundreds of times
[MyObject createInContext:localContext];
[localContext saveNestedContexts];
}];
And lastly, same blocking effect with this code:
dispatch_queue_t syncQueue = dispatch_queue_create("Sync queue", NULL);
dispatch_async(syncQueue, ^{
NSManagedObjectContext *localContext = [NSManagedObjectContext contextForCurrentThread];
// Do this hundreds of times
[MyObject createInContext:localContext];
[[NSManagedObjectContext contextForCurrentThread] saveNestedContexts];
});
So, what is the best way to solve this? I need to create hundreds of objects in the background and the app needs to remain responsive.
MagicalRecord uses a child context when doing work in the background. This works fine for small changes, but will create excessive main thread blocking when importing large amounts of data.
The way to do it is to use a parallel NSManagedObjectContext and to do the merging yourself with the NSManagedObjectContextDidSaveNotification notification and the mergeChangesFromContextDidSaveNotification method. See performance tests here: http://floriankugler.com/blog/2013/5/11/backstage-with-nested-managed-object-contexts
When saving a nested contexts everything has to be copied to the parent context. As opposed to this, objects that have not been fetched (in the context into which you are merging) will not be merged by mergeChangesFromContextDidSaveNotification. This is what makes it faster.
You might encounter problems if you want to display these results right away after saving in batches and using an NSFetchResultsController. See the following question for a solution:
NSFetchedResultsController with predicate ignores changes merged from different NSManagedObjectContext
For more performance tips take a look at this question: Implementing Fast and Efficient Core Data Import on iOS 5
Create your own context.
NSManagedObjectContext *importContext = [[NSManagedObjectContext alloc]
initWithConcurrencyType:NSPrivateQueueConcurrencyType];
[importContext setPersistentStoreCoordinator:yourPersistentStoreCoordinator];
[importContext setUndoManager:nil]; // For importing you don't need undo: Faster
// do your importing with the new importContext
// …
NSError* error = nil;
if(importContext.hasChanges) {
if(![importContext save:&error]) {
NSLog(#"Unresolved error %#, %#", error, [error userInfo]);
}
}
Make sure you are listening for the saves to managed object contexts.
[[NSNotificationCenter defaultCenter]
addObserver:singleton
selector:#selector(contextDidSave:)
name:NSManagedObjectContextDidSaveNotification object:nil];
In the contextDidSave:you merge the change yourself.
- (void) contextDidSave:(NSNotification*) notification
{
if(![notification.object isEqual:self.mainContext]) {
dispatch_async(dispatch_get_main_queue(), ^{
[self.mainContext mergeChangesFromContextDidSaveNotification:notification];
});
}
}
Managed object contexts are not thread safe so if you ever need to do any kind of background work with your Coredata objects (i.e. a long running import/export function without blocking the main UI) you will want to do that on a background thread.
In these cases you will need to create a new managed object context on the background thread, iterate through your coredata operation and then notify the main context of your changes.
You can find an example of how this could work here
Core Data and threads / Grand Central Dispatch

Deadlock when saving context with concurrency type NSPrivateQueueConcurrencyType

Since two days I'm trying to get Core Data to work with multiple threads. I tried standard thread confinement method with NSOperations, merging notifications, using objectWithId, dictionaries of contexts per thread and still I get strange deadlocks, inconsistency exceptions and a bunch of other nasty stuff. It's driving me crazy... moreover I can't find a single example or explanation on how to manage context in two threads when both threads may make changes to the shared persistent store...
I tried to use new iOS 5 method, that supposed to be easier, but still I get errors. The first problem is the deadlock when saving context. I removed all the unnecessary code and stil get deadlocks when executing this code fast enough (by quickly tapping a button):
NSManagedObjectContext *context = [StoreDataRetriever sharedRetriever].managedObjectContext;
for (int i = 0; i < 5; i++) {
NSError *error = nil;
NSLog(#"Main thread: %#, is main? %d", [NSThread currentThread], [NSThread isMainThread]);
BOOL saveOK = [context save:&error];
if (!saveOK) {
NSLog(#"ERROR!!! SAVING CONTEXT IN MAIN");
}
[context performBlock:^{
NSLog(#"Block thread: %#", [NSThread currentThread]);
NSError *error = nil;
BOOL savedOK = NO;
savedOK = [context save:&error];
if (!savedOK) {
NSLog(#"ERROR!!! SAVING CONTEXT IN BLOCK");
}
}];
}
There are no other changes to the database, nothing, only saving context. What is wrong with this code? How should it look like?
Note: [StoreDataRetriever sharedRetriever].managedObjectContext is created in appDelegate using initWithConcurrencyType:NSPrivateQueueConcurrencyType.
What's going on with that code? You are saving the context on a thread synchronously, then you schedule a save on the context private queue. 5 times. So basically, you may well have two save operations, one synchronous and one asynchronous, colliding with each other.
This is clearly an issue. You aren't supposed to save a context with a private queue outside of that queue. It will work with the current context implementation provided there is no scheduled block on the context queue. But this is wrong nevertheless.
…
for (int i = 0; i < 5; i++) {
NSLog(#"Main thread: %#, is main? %d", [NSThread currentThread], [NSThread isMainThread]);
__block NSError *error = nil;
__block BOOL saveOK = YES;
[context performBlockAndWait: ^{
saveOK = [context save: &error];
}];
if (!saveOK) {
NSLog(#"ERROR!!!");
}
…
With that code, you execute the save operation synchronously and most certainly on the same thread - thanks GCD - sparing context switches and synchronization stuff, and without any risk of having two operations running on that context at the same time.
The same rule applies when using NSMainQueueConcurrencyType, with an exception. That queue is bound to the main thread and the main thread only. You can schedule blocks on a context using the main queue from any thread with performBlock and performBlockAndWait like NSPrivateQueueConcurrencyType, and (the exception:) you can use the context directly on the main thread.
NSConfinementConcurrencyType binds the context to a specific thread and you cannot use GCD or blocks to deal with such a context, only the bound thread. There is very little reasons to use that concurrency model as of today. If you have to, use it, but if you do not absolutely have to, don't.
edit
Here is a very nice article about multi-contextes setups: http://www.cocoanetics.com/2012/07/multi-context-coredata/

Resources