I'm trying to execute a fetch against a fairly large data set (~60000 rows) in the background of my app. Despite using a separate thread, the UI noticeably hangs for a second or so whenever the fetch executes. Is my approach correct?
- (id)init
{
if(self = [super init])
{
ABAppDelegate *appDelegate = (ABAppDelegate *)[[UIApplication sharedApplication] delegate];
_rootManagedObjectContext = appDelegate.managedObjectContext;
_backgroundContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
[_backgroundContext setPersistentStoreCoordinator:_rootManagedObjectContext.persistentStoreCoordinator];
}
return self;
}
- (void)fetch {
[_backgroundContext performBlock:^{
NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:#"ItemPhoto"];
NSPredicate *pred = [NSPredicate predicateWithFormat:#"full_uploaded_to_server == 0 OR thumb_uploaded_to_server == 0"];
fetchRequest.predicate = pred;
NSSortDescriptor *sort = [NSSortDescriptor sortDescriptorWithKey:#"modified" ascending:YES]; //Start at the back of the queue
fetchRequest.sortDescriptors = [NSArray arrayWithObject:sort];
fetchRequest.fetchBatchSize = 1;
fetchRequest.fetchLimit = 1;
NSError *error;
NSArray *photos = [_backgroundContext executeFetchRequest:fetchRequest error:&error];
if(error != nil) {
NSLog(#"Fetch error: %#", error.localizedDescription);
}
}];
}
Looking in Instruments, it's definitely the call to executeFetchRequest: that's taking a long time to complete, and it does seem to be running on its own thread. Why would this be causing the UI to hang?
Thanks!
Any activity against the NSPersistentStoreCoordinator is going to cause a block of other activities against it. If you are fetching on one thread and another thread attempts to access the NSPersistentStoreCoordinator it is going to be blocked.
This can be solved in one of two ways:
Reduce the fetches so that the block is not noticeable. Fetching in chunks, for example, can help to reduce this issue.
Make sure the data the UI is exposing is fully loaded into memory. This will stop the main thread from trying to hit the NSPersistentStoreCoordinator and getting blocked.
Depending on the application and the situation, one or the other (or both) of these implementations will remove the issue.
At the end of the day, background threads are not a silver bullet to solve fetch times. If you are planning on fetching on a background thread just to put them onto the main thread you are going to be disappointed with the performance.
If you are fetching on the background thread for a non-UI purpose then consider reducing the size of each fetch or change the batch size, etc. to decrease the fetch time itself.
Update
Warming up the cache doesn't work on iOS like it used to on OS X. You can get the objects fully loaded into memory by configuring your NSFetchRequest so that the objects are fully loaded, relationships are loaded (if needed), your batch size and fetch size are large enough. Naturally this needs to be balanced against memory usage.
Related
I've got CoreData concurrency violation assertions enabled in my project, which have been great for tracking down and fixing errors in our usage of it. However, I occasionally get issues on fetch requests where I can't determine what we're doing wrong.
All of our managed contexts are private queue concurrency types other than one that is a main queue concurrency type. This is a block of code that occasionally triggers the concurrency violation:
- (NSArray *)fetchFromCacheEntitiesWithClass:(Class)entityClass
predicate:(NSPredicate *)predicate
sortDescriptors:(NSArray *)sortDescriptors
managedContext:(NSManagedObjectContext *)managedContext
{
NSFetchRequest *request = [NSFetchRequest fetchRequestWithEntityName:[entityClass entityName]];
request.predicate = predicate;
request.sortDescriptors = sortDescriptors;
request.fetchBatchSize = 30;
__block NSArray *results = nil;
[managedContext performBlockAndWait:^{
NSError *error = nil;
results = [managedContext executeFetchRequest:request error:&error];
NSAssert(!error, #"fetch had an error, investigate why");
}];
return results;
}
The entityName method exists on all of our CoreData classes, is generated from MOGenerator, and I've double checked that it's correct for all of our classes. The violation in this snippet happens on the executeFetchRequest call.
One thing to note is that we've only seen this occur in the simulator, never on a device. It has me leaning towards a simulator bug, though I don't normally like to assign blame to tools since developer error is so much more common.
I am experiencing issues with Core Data which I cannot resolve. I've learned about concurrency issues in Core Data the hard way, thus I am really careful and only perform any core data operations in performBlock: and performBlockAndWait: blocks.
Here goes my code:
/// Executes a fetch request with given parameters in context's block.
+ (NSArray *)executeFetchRequestWithEntityName:(NSString *)entityName
predicate:(NSPredicate *)predicate
fetchLimit:(NSUInteger)fetchLimit
sortDescriptor:(NSSortDescriptor *)sortDescriptor
inContext:(NSManagedObjectContext *)context{
NSCAssert(entityName.length > 0,
#"entityName parameter in executeFetchRequestWithEntityName:predicate:fetchLimit:sortDescriptor:inContext:\
is invalid");
__block NSArray * results = nil;
NSPredicate * newPredicate = [CWFCoreDataUtilities currentUserPredicateInContext:context];
if (predicate){
newPredicate = [NSCompoundPredicate andPredicateWithSubpredicates:#[newPredicate, predicate]];
}
[context performBlockAndWait:^{
NSFetchRequest * request = [NSFetchRequest fetchRequestWithEntityName:entityName];
request.fetchLimit = fetchLimit;
request.predicate = newPredicate;
if (sortDescriptor) {
request.sortDescriptors = #[sortDescriptor];
}
NSError * error = nil;
results = [context executeFetchRequest:request error:&error];
if (error){
#throw [NSException exceptionWithName:NSInternalInconsistencyException
reason:#"Fetch requests are required to succeed."
userInfo:#{#"error":error}];
NSLog(#"ERROR! %#", error);
}
NSCAssert(results != nil, #"Fetch requests must succeed");
}];
return results;
}
When I enter this method concurrently from two different threads and pass two different contexts, I result in a deadlock on this row: results = [context executeFetchRequest:request error:&error];
Which is interesting: it seems like both threads cannot acquire some lock on the Persistent Store Coordinator in order to execute a fetch request.
All of my contexts are NSPrivateQueueConcurrencyType.
I can't put my finger on, why am I locking the app and what should I do differently. My research on Stack Overflow gave me nothing, since most of the people fixed all the locks by dispatching the fetch requests on the MOC's queue, which I already do.
I will appreciate any information on this issue. Feel free to provide documentation links and other long reads: I am eager to learn more about all kind of concurrency problems and strategies.
Which is interesting: it seems like both threads cannot acquire some lock on the Persistent Store Coordinator in order to execute a fetch request.
The persistent store coordinator is a serial queue. If one context is accessing it another context will be blocked.
From Apple Docs:
Coordinators do the opposite of providing for concurrency—they serialize operations. If you want to use multiple threads for different write operations you use multiple coordinators. Note that if multiple threads work directly with a coordinator, they need to lock and unlock it explicitly.
If you need to perform multiple background fetch request concurrently you will need multiple persistent store coordinators.
Multiple coordinators will make your code only slightly more complex but should be avoided if possible. Do you really need to do multiple fetches at the same time? Could you do one larger fetch and then filter the in-memory results?
If you are interested in learning more about Core Data (and threading), the following website will be extremely helpful. I attended Matthew Morey's talk at the Atlanta CocoaConf 2013.
High Performance Core Data (http://highperformancecoredata.com)
The companion sample code to the website is available at: https://github.com/mmorey/MDMHPCoreData
All you need to do in your app, is to have a Singleton instance (somewhere) of the MDMPersistenceStack class.
As far as your problem/issue, even though the Apple documentation for NSManagedObjectContext class, does allow for code to be written the way that you have (with the Core Data operations performed on an NSManagedObjectContext instance in a block) and to some extent encourages that - I would like to point out that is not the only way.
Whenever I have fixed apps that have Core Data concurrency issues (locking), the simplest thing, in my opinion, is to create a private NSManagedObjectContext inside the thread or block that I want to execute Core Data operations on.
It may not be the most elegant approach, but it's never failed me. One is always guaranteed that the NSManagedObjectContext is created and executed in the same thread, because the NSManagedObjectContext is explicitly created.
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(queue, ^{
/*
Create NSManagedObjectContext
Concurrency of Managed Object Context should be set to NSPrivateQueueConcurrencyType
If you use the MDMPersistenceStack class this is handled for you.
*/
NSManagedObjectContext *managedObjectContext = ....
/*
Call the method that you have listed in your code.
Let's assume that this class method is in MyClass
Remove the block that you have in your method, as it's not needed
*/
[MyClass executeFetchRequestWithEntityName: ......] // rest of parameters
});
I did it this way. It fixed the issue for me. I was experiencing a lot of deadlocks too. See if it works for you.
+ (NSArray *)getRecordsForFetchRequest:(NSFetchRequest *)request inContext:(NSManagedObjectContext *)context
{
#try
{
__weak __block NSError *error = nil;
__block __weak NSArray *results = nil;
[context performBlockAndWait:^{
[context lock];
results = [context executeFetchRequest:request error:&error];
[context processPendingChanges];
[context unlock];
}];
[self handleErrors:error];
request = nil;
context = nil;
return results;
}
#catch (NSException *exception)
{
if([exception.description rangeOfString:#"Can only use -performBlockAndWait: on an NSManagedObjectContext that was created with a queue"].location!=NSNotFound)
{
NSError *error = nil;
[context lock];
__weak NSArray *results = [context executeFetchRequest:request error:&error];
[context processPendingChanges];
[context unlock];
[self handleErrors:error];
request = nil;
context = nil;
return results;
}
return nil;
}
}
I am inserting cca 100 000 records in core data database. Database contains 3 entities.
Player, Club, PlayerClub
Entities are in relations:
Player<-->>PlayerClub<<-->Club
While inserting in PlayerClub I have noticed a lot of memory consumption and loss of speed after about 50 000 records have been inserted. Records are never updated cause PlayerClub has only unique values.
For test reasons I have emptied Player and Club tables and run the program again. There was no speed drop but memory consumption was huge again. This is the code:
NSManagedObjectContext *context = [[NSManagedObjectContext alloc] init];
[context setPersistentStoreCoordinator:ap.persistentStoreCoordinator];
[context setUndoManager:nil];
NSEntityDescription *entity = [NSEntityDescription entityForName:#"PlayerClub"
inManagedObjectContext:context];
NSFetchRequest *request = [[NSFetchRequest alloc] init];
[request setEntity:entity];
NSEntityDescription *entityKlubovi = [NSEntityDescription entityForName:#"Club" inManagedObjectContext:context];
NSFetchRequest *requestKlubovi = [[NSFetchRequest alloc] init];
[requestKlubovi setEntity:entityKlubovi];
[requestKlubovi setFetchLimit:1];
NSEntityDescription *entityIgraci = [NSEntityDescription entityForName:#"Player" inManagedObjectContext:context];
NSFetchRequest *requestIgraci = [[NSFetchRequest alloc] init];
[requestIgraci setFetchLimit:1];
[requestIgraci setEntity:entityIgraci];
for (int i=0; i<[parsedKlubIgraci count]; i++) {
#autoreleasepool {
KlubIgrac *klubIgrac = [[KlubIgrac alloc] initWithEntity:entity insertIntoManagedObjectContext:context];
klubIgrac.iD = p.id;
//... add some othe properties
variables = [NSDictionary dictionaryWithObject:p.igracID forKey:#"ID"];
localPredicate = [predicateIgraci predicateWithSubstitutionVariables:variables];
[requestIgraci setPredicate:localPredicate];
klubIgrac.igrac = [self DohvatiIgracZaIgracId:p.igracID cntx:context request:requestIgraci predicate:localPredicate];
variables = [NSDictionary dictionaryWithObject:p.klubID forKey:#"ID"];
localPredicate = [predicateKlubovi predicateWithSubstitutionVariables:variables];
[requestKlubovi setPredicate:localPredicate];
klubIgrac.klub = [self DohvatiKlubZaKlubId:p.klubID cntx:context request:requestKlubovi predicate:localPredicate];
}
}
+(Club *)DohvatiKlubZaKlubId:(int)klubid cntx:(NSManagedObjectContext *)context
request:(NSFetchRequest *)request predicate:(NSPredicate *)predicate
{
#autoreleasepool {
NSError *error;
NSArray *arTmp;
arTmp = [context executeFetchRequest:request error:&error];
Club *klub;
if(arTmp != nil && [arTmp count]){
return [arTmp objectAtIndex:0];
}
}
}
DohvatiIgracZaIgracId method is basically the same method as DohvatiKlubZaKlubId so I dinnt post it.
This code is called about 100 000 times. The memory consumption before is about 150 MB. After it finishes its 650 MB. So it consumes 500 MB, without saving the context and without fetching anything from the database cause the tables are empty. If I comment
arTmp = [context executeFetchRequest:request error:&error];
line in DohvatiIgracZaIgracId and DohvatiKlubZaKlubId methods the memory consumption falls to 200 MB. So the diference is 400MB for a line of code that does nothing. This cant be. Anyone has some ideas. After a while my app consumes more than 2.5 GB.
The mesuments were done on simulator cause I need to build a database that I will preload later, so the speed is of importance :)... Thx in advance
without saving the context
This is the part of the question where experienced Core Data developers say "oh holy crap". That's your biggest problem right there. Save changes at regular intervals-- every 50 entries, or every 100, but whatever you do, don't wait until you're finished. You're forcing Core Data to keep all of those objects in memory as unsaved changes. This is the biggest reason you're having memory problems.
Some other things you should consider:
Don't fetch your objects one at a time. Fetches are relatively expensive. If you run through 100k instances and fetch each of them one at a time, your code will be spending almost all of its time executing fetches. Fetch in batches of 50-100 (you can tune the number to get the best balance of speed vs. memory use). Process one batch, then save changes at the end of the batch.
When you're done with a fetched object, tell the managed object context that you're done. Do this by calling refreshObject:mergeChanges: with NO as the second argument. That tells the context it can free up any internal memory it's using for the object. This loses any unsaved changes on the objects, but if you haven't made any changes then there's nothing to lose.
Consider getting rid of the PlayerClub entity completely. Core Data supports many-to-many relationships. This kind of entity is almost never useful. You're using Core Data, not SQL, so don't design your entity types as if you were.
I am using a NSFetchResultsController to display 100,000 + records in a UITableView. This works but it is SLOW, especially on an iPad 1. It can take 7 seconds to load which is torture for my users.
I'd also like to be able to use sections but this adds at least another 3 seconds onto the laod time.
Here is my NSFetchResultsController:
- (NSFetchedResultsController *)fetchedResultsController {
if (self.clientsController != nil) {
return self.clientsController;
}
NSFetchRequest *request = [[NSFetchRequest alloc] init];
NSEntityDescription *entity = [NSEntityDescription entityForName:#"Client" inManagedObjectContext:self.managedObjectContext];
[request setEntity:entity];
[request setPredicate:[NSPredicate predicateWithFormat:#"ManufacturerID==%#", self.manufacturerID]];
[request setFetchBatchSize:25];
NSSortDescriptor *sort = [[NSSortDescriptor alloc] initWithKey:#"UDF1" ascending:YES];
NSSortDescriptor *sort2= [[NSSortDescriptor alloc] initWithKey:#"Name" ascending:YES];
[request setSortDescriptors:[NSArray arrayWithObjects:sort, sort2,nil]];
NSArray *propertiesToFetch = [[NSArray alloc] initWithObjects:#"Name", #"ManufacturerID",#"CustomerNumber",#"City", #"StateProvince",#"PostalCode",#"UDF1",#"UDF2", nil];
[request setPropertiesToFetch:propertiesToFetch];
self.clientsController =
[[NSFetchedResultsController alloc] initWithFetchRequest:request
managedObjectContext:self.managedObjectContext sectionNameKeyPath:nil
cacheName:nil];
return self.clientsController;
}
I have an index on ManufacturerID which is used in my NSPredicate. This seems like a pretty basic NSFetchRequest - anything I can do to speed this up? Or have I just hit a limitation? I must be missing something.
First: you can use the NSFetchedResultsController's cache to speed up display after the first fetch. This should quickly go down to a fraction of a second.
Second: you can try to display the only the first screenful and then fetch the rest in the background. I do this in the following way:
When the view appears, check if you have the first page cache.
If not, I fetch the first page. You can accomplish this by setting the fetch request's fetchLimit.
In case you are using sections, do two quick fetches to determine the first section headers and records.
Populate a second fetched results controller with your long fetch in a background thread.
You can either create a child context and use performBlock: or
use dispatch_async().
Assign the second FRC to the table view and call reloadData.
This worked quite well in one of my recent projects with > 200K records.
I know the answer #Mundi provided is accepted, but I've tried implementing it and ran into problems. Specifically the objects created by the second FRC will be based on the other thread's ManagedObjectContext. Since these objects are not thread safe and belong to their own MOC on the other thread, the solution I found was to fault the objects as they are being loaded. So in cellForRowAtIndexPath I added this line:
NSManagedObject *object = [self.fetchedResultsController objectAtIndexPath:indexPath];
object = (TapCellar *)[self.managedObjectContext existingObjectWithID:[object objectID] error:nil];
Then you have an object for the correct thread you are in. One further caveat is that the changes you make to the objects won't be reflected in the background MOC so you'll have to reconcile them. What I did was make the background MOC a private queue MOC and the foreground one is a child of it like this:
NSPersistentStoreCoordinator *coordinator = [self persistentStoreCoordinator];
if (coordinator != nil) {
_privateManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
[_privateManagedObjectContext setPersistentStoreCoordinator:coordinator];
_managedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType];
[_managedObjectContext setParentContext:_privateManagedObjectContext];
}
Now when I make changes in the main thread, I can reconcile them easily by doing this:
if ([self.managedObjectContext hasChanges]) {
[self.managedObjectContext performBlockAndWait:^{
NSError *error = nil;
ZAssert([self.managedObjectContext save:&error], #"Error saving MOC: %#\n%#",
[error localizedDescription], [error userInfo]);
}];
}
I wait for it's return since I'm going to reload the table data at this point, but you can choose not to wait if you'd like. The process is pretty quick even for 30K+ records since usually only one or two are changed.
Hope this helps those who are stuck with this!
Good Morning.
I have a problem when I run my app on my device, it lags/stutters when I scroll in the main tableView.
I've narrowed the problem down to a call to core data from inside my tableCell
--In cell for row at indexPath
person is a custom class and contact manager is my file with all my calls to core data and manipulating data
person.contactSelected = [contactManager checkContactSelectedStatus:person];
--In my contactManager file the call goes to this function.
and just updates the contacts selected status (when the user presses a button to change from being allowed in the call or not in the call)
-(NSNumber *) checkContactSelectedStatus:(ContactPerson *)person{
SWNAppDelegate *delegate = [[UIApplication sharedApplication]delegate];
NSManagedObjectContext *context = [delegate managedObjectContext];
NSEntityDescription *entity = [NSEntityDescription entityForName:#"Contact" inManagedObjectContext:context];
NSFetchRequest *req =[[NSFetchRequest alloc]init];
[req setEntity:entity];
NSPredicate *pred =[NSPredicate predicateWithFormat:#"(recordID like %#)",[person.recordID stringValue]];
[req setPredicate:pred];
NSError *error;
NSManagedObject *checkStatus = [[context executeFetchRequest:req error:&error] objectAtIndex:0];
person.contactSelected = [checkStatus valueForKey:#"isSelected"];
return person.contactSelected;}
Is there an easy way to throw this into a Queue? I have read and tried to figure out how to send a NSManagedObject to queues, but when I create a child of the Parent MoC, It can not find the Entity "Contact". I don't know if there is a simpler way to do it or not!?
Thanks for your time, and
WhatWasIThinking!?!?!
Yes, this is really inefficient code. The fetch has to be done multiple times, i.e. for each cell as it becomes visible.
You should instead use an NSFetchedResultsController which is especially designed to work with table views. It will decide the appropriate number of trips to the store for fetches and optimize for speed and memory footprint.
Also, you will most likely use significantly less code.
Besides, a predicate string like recordID like %# does not make much sense. If you are using IDs they should be unique so you can index them and look them up really fast. LIKE, just like string functions such as CONTAINS are very slow in comparison.