How to correctly add large dataset in CoreData? - ios

Question

I have a huge NSArray (4.000.000 objects) that I want to save into Core Data.
Because I use ARC and the autorelease pool may get too big, I partitioned the process into multiple loops (so autorelease pools can have the chance to drain themselves).
In the following code I use a manager (clMan) to add items from the dictionaries inside the array(regions). The dictionaries contain two string fields which are parsed into scalar integers.
Code for partitioning the data into multiple loops
int loopSize = 50000;
int loops = 0;
int totalRepetitions = regions.count;
loops = totalRepetitions / loopSize;
int remaining = totalRepetitions % loopSize;
loops += 1;
for (int i = 0; i < loops; i++) {
int k = 0;
if (i == 0) k = 1;
if (i == (loops - 1))
{
// Last loop
for (long j = i * loopSize + k; j < i * loopSize + remaining; j++) {
[clMan addItemWithData:[regions objectAtIndex:j]];
}
[clMan saveContext];
break;
}
// Complete loops before the last one
for (long j = i * loopSize + k; j < (i + 1) * loopSize; j++) {
[clMan addItemWithData:[regions objectAtIndex:j]];
}
[clMan saveContext];
NSLog(#"Records added : %d", i * loopSize);
}
NSLog(#"Finished adding into core data");
Code for adding the data into core data:
-(void)addItemWithData:(NSDictionary *)data
{
MyRegion *region = [NSEntityDescription
insertNewObjectForEntityForName:#"MyRegion"
inManagedObjectContext:self.context];
region.index = [((NSString *)[data objectForKey:REGION_INDEX]) intValue];
region.id = [((NSString *)[data objectForKey:REGION_ID]) intValue];
}
The program crashes when it reaches the 1 500 000 index. The crash does not seem to happen because of parsing issues / logic.
Can anyone tell me if my logic is bad or what is the correct way to add this amount of data in CoreData?

Answer 1

After each loop, try calling NSManagedObjectContext.reset to "forget" the local copies in the MOC. Otherwise these might not be cleared and are causing a problem.
The WWDC 2012 code samples on iCloud have a method called seedStore where they migrate a local core data SQL database to the iCloud one - using a batch size of 5000 records and there it explicitly states that:
if (0 == (i % batchSize)) {
success = [moc save:&localError];
if (success) {
/*
Reset the managed object context to free the memory for the inserted objects
The faulting array used for the fetch request will automatically free
objects with each batch, but inserted objects remain in the managed
object context for the lifecycle of the context
*/
[moc reset];
} else {
NSLog(#"Error saving during seed: %#", localError);
break;
}
}
(Here i is the current index of the batch, thus i % batchSize == 0 if we start a new batch)