NSFetchRequest fetchBatchSize not working - ios

I have been having problems with fetchBatchSize in an NSFetchedResultsController so I decided to take a step back and use it on just a plain NSFetchRequest. I created a very simple project with only a tableViewController that has 100 items. The items have only two properties, itemID and itemName. I use the following method to populate the data array:
func initializeItems() {
let request = NSFetchRequest<Item>(entityName: "Item")
let messageSort = NSSortDescriptor(key: "itemName", ascending: true)
request.sortDescriptors = [messageSort]
request.fetchBatchSize = 20
do {
let fetchedResults = try context.fetch(request)
if fetchedResults.count > 0 {
self.items = fetchedResults
//self.tableView.reloadData()
}
} catch {
print("Could not fetch \(error)")
}
}
This is called in viewDidLoad of the tableViewController. The context is defined as follows:
let appDelegate = UIApplication.shared.delegate as! AppDelegate
self.context = appDelegate.persistentContainer.viewContext
When I run the app, the table shows fully populated (as indicated by the scroll bar) with all rows.
I don't understand why the fetchBatchSize is being ignored.
Documentation is as follows:
The batch size of the objects specified in the fetch request.
The default value is 0. A batch size of 0 is treated as infinite, which disables the batch faulting behavior.
If you set a nonzero batch size, the collection of objects returned when an instance of NSFetchRequest is executed is broken into batches. When the fetch is executed, the entire request is evaluated and the identities of all matching objects recorded, but only data for objects up to the batchSize will be fetched from the persistent store at a time. The array returned from executing the request is a proxy object that transparently faults batches on demand. (In database terms, this is an in-memory cursor.)
You can use this feature to restrict the working set of data in your application. In combination with fetchLimit, you can create a subrange of an arbitrary result set.
I can see the 100 items in the fault array being loaded. But then it loops through 5 loads of 20 items full data prior to scrolling the tableView. The tableView shows about 15 rows at a time.
The behaviour I want is for the data to be loaded in 20 item batches as the table is scrolled

As shown in the documentation:
The fetch limit specifies the maximum number of objects that a request
should return when executed.
fetchLimit is what you're looking for.

I suspect something in your code actually iterates through that array and causes it to fire the fault. I'm not telling you do it deliberately yourself - it is quite likely that setup od your UITableView does that (pre-calculating cell height based on content, sectioning data, etc).
What I suggest is to open Core Data profiling tool in Instruments and check what is in the stack trace when faults are fired.

fetchBatchSize works best with NSFetchedResultsController. Put ur fetchRequest in NSFetchedResultsController and then use NSFetchedResultsController to access fetched result. Then definitely fetchBatchSize gonna work. Or u can google how NSFetchedResultsController is implemented in fetching result.

Related

How to correctly add additional section with FetchResultController?

For now I'm sorting all my sections with the next sort descriptors:
let sortDescriptors = [NSSortDescriptor(key: "category", ascending: true), NSSortDescriptor(key: "name", ascending: true)]
let request = DBOShopCardBox.createFetchRequest(predicate: predicate, sortDescriptors: sortDescriptors)
fetchedResultsController = NSFetchedResultsController(fetchRequest: request, managedObjectContext: DBContext.defaultContext, sectionNameKeyPath: "categoryOrderIndex", cacheName: nil) as? NSFetchedResultsController<NSManagedObject>
But now I want to the next:
I have an additional column, type. It has 3 values box, folder, free. I want to add one more(+1) section with ONLY records where type ==folder` in it and sort them alphabetically.
I was trying to add one more sort descriptor into my array like:
NSSortDescriptor(key: "type", ascending: true)
but it will sort ALL of them by type. But I need to get the records with type == folder, put them into the separate section and just later sort them. I'm confused now. Maybe someone can give me some hints or help me?
Thanks in advance!
That's not what NSFetchedResultsController does. It uses the result of a fetch (hence the name) to control (hence the name) the table view by serving as its model object. If you want the table view to express something the fetched results controller doesn't express (a different number of sections, etc.), you'll have to implement that yourself, probably as part of the data source methods (or you might subclass NSFetchedResultsController; I'm not sure about that part though). You can consult the fetched results controller's fetchedObjects to see what it fetched, but you will have to dictate explicitly that there is an extra section and what goes in it.
NSFetchedResultsController is a really cool class. It fetches from core data and updates the viewController when stuff is added, deleted, changed, or moved. It assigns indexPaths for every object it is monitoring so it can interface easily with a collectionView or a tableView. But you don't need to have a 1-to-1 relationship between a fetchedResultsController and a collectionView. You can have two fetchedResultsControllers - one for sections 0-3 and a different one for section 4.
The big challenge with this setup is to keep track of what kinds of indexPaths you are dealing with; the fetchedResultsControllers' indexPaths are no longer a one-to-one relation to the collectionView's. Make a few methods to convert from different indexPath spaces and back and make sure to call them correctly whenever you see an indexPath.

Encrypted core data take hell lot of time in deleting 400k+ records

I am using Encrypted CoreData Sqlite Store to encrypt my CoreData stack. I have a table in database which stores more than 400k records. I want to delete most of those records based on some activity. As batch delete is not available for this encrypted store what I need to do is, execute a fetch request to read data.
let fetchRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "MyEntity")
fetchRequest.includesPropertyValues = false
// Fetch data till minimum threshold limit
let newPredicate = NSPredicate(format: "%K <= %#", "recordTime", NSNumber(value: maxAllowedTimeStamp as Int64))
fetchRequest.predicate = newPredicate
let objectsToDelete = myDataController.executeFetchRequest(fetchRequest) as? [NSManagedObject]
Than I perform delete operation one-by-one the objects:
for object in objectsToDelete {
myDataController.deleteManagedObject(object)
}
This operation is taking more than 7-8 mins. I believe that core data is fast enough to provide fault object and it should not take this much time as you can see my predicate in which I am not taking any property value in order to reduce fetch time.
Can someone help me to understand what is taking so long to delete objects?
What's taking so long is that you're deleting 400k objects one at a time. It's not about being encrypted, or at least not mainly about that. This kind of operation has always been slow, and is the reason that batch deletes were added.
Deleting this many records without batching is fundamentally a very slow process. If you can't use a batch delete, you need to think seriously about why you need to delete this many objects and whether you could avoid that. Maybe don't create so many if you're going to delete them all. Or maybe don't delete them-- since if you created them once, maybe you create them again later? Or if it's an offline cache of user account data, maybe don't fetch all of that data. Or maybe handle the delete by removing the persistent store files instead of by deleting the objects one by one.

Sort Order vs Display Sort Order

I have an app that displays a section title followed by the detail items followed by the next section and its details. Everything works fine, but I would like to rearrange the order in which the sections are shown. The problem is that I need to order the core data by the report-id then status then the date to get the correct detail items to show under the proper section.
let sortDescriptor1 = NSSortDescriptor(key: #keyPath(Item.report.id), ascending:true)
let sortDescriptor2 = NSSortDescriptor(key: #keyPath(Item.report.status), ascending:true)
let sortDescriptor3 = NSSortDescriptor(key: #keyPath(Item.report.dateStarted), ascending:false)
let sortDescriptor4 = NSSortDescriptor(key: #keyPath(Item.date), ascending:true)
How can I change the sort/display order while still maintaining the proper relationship between the section (report) and the detail items associated with it? The report.id is a UUID so currently the reports end up in random order.
A fetchedResultsController has a property sectionNameKeyPath which can be used to group items togethers. This only works if the sectionNameKeyPath groups the items in the same order that they are sorted. For example: you can sort by date and then group by hour or week or any other time based grouping, but not by name. In your case you want the sections to be sorted in a way that does not not match how the items are grouped. There may be some clever solution for your particular situation, but since you didn't give any details I can only give a general solution.
The indexPath that is returned from a fetchedResultsController is really useful for interoperability with a tableView or collectionView. But it does not have to be a one-to-one relationship. For example you could have a setup where one section points to one fetchedResultsController and another section points to a different one. The key in doing this setup is to make sure to not confuse the fetchedResultsController indexPath with your collectionView indexPath. Generally I find having a separate object to manage converting the indexPath the easily way to keep it straight.
Create a separate object that sorts the sections after the fetchedResultsController does a fetch (and after a section is inserted or deleted). Inside the indexPathsManager have a dictionary the maps between the "local" indexPath and the fetchedResultsController indexPath. Make sure to sure use this object EVERY TIME you use indexPaths in the viewController. Also make sure to convert the indexPaths when you update the view after the fetchedResultsController delegates that there is a change. And updating the indexPathsManager when any sections are inserted or deleted.
TL;DR Sort the sections of the fetchedResultsController after the fetch and convert your tableView's indexPath to fetchedResultsController indexPaths.

Core Data sectionNameKeyPath with Relationship Attribute Performance Issue

I have a Core Data Model with three entities:
Person, Group, Photo with relationships between them as follows:
Person <<-----------> Group (one to many relationship)
Person <-------------> Photo (one to one)
When I perform a fetch using the NSFetchedResultsController in a UITableView, I want to group in sections the Person objects using the Group's entity name attribute.
For that, I use sectionNameKeyPath:#"group.name".
The problem is that when I'm using the attribute from the Group relationship, the NSFetchedResultsController fetches everything upfront in small batches of 20 (I have setFetchBatchSize: 20) instead of fetching batches while I'm scrolling the tableView.
If I use an attribute from the Person entity (like sectionNameKeyPath:#"name") to create sections everything works OK: the NSFetchResultsController loads small batches of 20 objects as I scroll.
The code I use to instantiate the NSFetchedResultsController:
- (NSFetchedResultsController *)fetchedResultsController {
if (_fetchedResultsController) {
return _fetchedResultsController;
}
NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] init];
NSEntityDescription *entity = [NSEntityDescription entityForName:[Person description]
inManagedObjectContext:self.managedObjectContext];
[fetchRequest setEntity:entity];
// Specify how the fetched objects should be sorted
NSSortDescriptor *groupSortDescriptor = [[NSSortDescriptor alloc] initWithKey:#"group.name"
ascending:YES];
NSSortDescriptor *personSortDescriptor = [[NSSortDescriptor alloc] initWithKey:#"birthName"
ascending:YES
selector:#selector(localizedStandardCompare:)];
[fetchRequest setSortDescriptors:[NSArray arrayWithObjects:groupSortDescriptor, personSortDescriptor, nil]];
[fetchRequest setRelationshipKeyPathsForPrefetching:#[#"group", #"photo"]];
[fetchRequest setFetchBatchSize:20];
NSError *error = nil;
NSArray *fetchedObjects = [self.managedObjectContext executeFetchRequest:fetchRequest error:&error];
if (fetchedObjects == nil) {
NSLog(#"Error Fetching: %#", error);
}
_fetchedResultsController = [[NSFetchedResultsController alloc] initWithFetchRequest:fetchRequest
managedObjectContext:self.managedObjectContext sectionNameKeyPath:#"group.name" cacheName:#"masterCache"];
_fetchedResultsController.delegate = self;
return _fetchedResultsController;
}
This is what I get in Instruments if I create sections based on "group.name" without any interaction with the App's UI:
And this is what I get (with a bit of scrolling on UITableView) if sectionNameKeyPath is nil:
Please, can anyone help me out on this issue?
EDIT 1:
It seems that I get inconsistent results from the simulator and Instruments: when I've asked this question, the app was starting in the simulator in about 10 seconds (by Time Profiler) using the above code.
But today, using the same code as above, the app starts in the simulator in 900ms even if it makes a temporary upfront fetch for all the objects and it's not blocking the UI.
I've attached some fresh screenshots:
EDIT 2:
I reset the simulator and the results are intriguing: after performing an import operation and quitting the app the first run looked like this:
After a bit of scrolling:
Now this is what happens on a second run:
After the fifth run:
EDIT 3:
Running the app the seventh time and eight time, I get this:
This is your stated objective: "I need the Person objects to be grouped in sections by the relationship entity Group, name attribute and the NSFetchResultsController to perform fetches in small batches as I scroll and not upfront as it is doing now."
The answer is a little complicated, primarily because of how an NSFetchedResultsController builds sections, and how that affects the fetching behavior.
TL;DR; To change this behavior, you would need to change how NSFetchedResultsController builds sections.
What is happening?
When an NSFetchedResultsController is given a fetch request with pagination (fetchLimit and/or fetchBatchSize), several things happen.
If no sectionNameKeyPath is specified, it does exactly what you expect. The fetch returns an proxy array of results, with "real" objects for the first fetchBathSize number of items. So for example if you have setFetchBatchSize to 2, and your predicate matches 10 items in the store, the results contain the first two objects. The other objects will be fetched separately as they are accessed. This provides a smooth paginated response experience.
However, when a sectionNameKeyPath is specified, the fetched results controller has to do a bit more. To compute the sections it needs to access that key path on all the objects in the results. It enumerates the 10 items in the results in our example. The first two have already been fetched. The other 8 will be fetched during enumeration to get the key path value needed to build the section information. If you have a lot of results for your fetch request, this can be very inefficient. There are a number of public bugs concerning this functionality:
NSFetchedResultsController initially takes too long to set up sections
NSFetchedResultsController ignores fetchLimit property
NSFetchedResultsController, Table Index, and Batched Fetch Performance Issue
... And several others. When you think about it, this makes sense. To build the NSFetchedResultsSectionInfo objects requires the fetched results controller to see every value in the results for the sectionNameKeyPath, aggregate them to the unique union of values, and use that information to create the correct number of NSFetchedResultsSectionInfo objects, set the name and index title, know how many objects in the results a section contains, etc. To handle the general use case there is no way around this. With that in mind, your Instruments traces may make a lot more sense.
How can you change this?
You can attempt to build your own NSFetchedResultsController that provides an alternative strategy for building NSFetchedResultsSectionInfo objects, but you may run into some of the same problems. For example, if you are using the existing fetchedObjects functionality to access members of the fetch results, you will encounter the same behavior when accessing objects that are faults. Your implementation would need a strategy for dealing with this (it's doable, but very dependant on your needs and requirements).
Oh god no. What about some kind of temporary hack that just makes it perform a little better but doesn't fix the problem?
Altering your data model will not change the above behavior, but can change the performance impact slightly. Batch updates will not have any significant effect on this behavior, and in fact will not play nicely with a fetched results controller. It may be much more useful to you, however, to instead set the relationshipKeyPathsForPrefetching to include your "group" relationship, which may improve the fetching and faulting behavior significantly. Another strategy may be to perform another fetch to batch fault these objects before you attempt to use the fetched results controller, which will populate the various levels of Core Data in-memory caches in a more efficient manner.
The NSFetchedResultsController cache is primarily for section information. This prevents the sections from having to be completely recalculated on each change (in the best case), but can actually make the initial fetch to build the sections take much longer. You will have to experiment to see if the cache is worthwhile for your use case.
If your primary concern is that these Core Data operations are blocking user interaction, you can offload them from the main thread. NSFetchedResultsController can be used on a private queue (background) context, which will prevent Core Data operations from blocking the UI.
Based on my experience a way to achieve your goal is to denormalize your model. In particular, you could add a group attribute in your Person entity and use that attribute as sectionNameKeyPath. So, when you create a Person you should also pass the group it belongs to.
This denormalization process is correct since it allows you to avoid fetching of related Group objects since not necessary. A cons could be that if you change the name of a group, all the persons associated with that name must change, on the contrary you can have incorrect values.
The key aspect here is the following. You need to have in mind that Core Data is not a
relational database. The model should not designed as a database schema, where normalization could take place, but it should be designed from the perspective of how the data are presented and used in the user interface.
Edit 1
I cannot understand your comment, could you explain better?
What I've found very intriguing though is that even if the app is
performing a full upfront fetch in the simulator, the app loads in
900ms (with 5000 objects) on the device despite the simulator where it
loads much slower.
Anyway, I would be interested in knowing details about your Photo entity. If you pre-fetch photo the overall execution could be influenced.
Do you need to pre-fetch a Photo within your table view? Are they thumbs (small photos)? Or normal images? Do you take advantage of External Storage Flag?
Adding an additional attribute (say group) to the Person entity could not be a problem. Updating the value of that attribute when the name of a Group object changes it's not a problem if you perform it in background. In addition, starting from iOS 8 you have available a batch update as described in Core Data Batch Updates.
After almost a year since I've posted this question, I've finally found the culprits that enable this behaviour (which slightly changed in Xcode 6):
Regarding the inconsistent fetch times: I was using a cache and at the time I was back and forth with opening, closing and resetting the simulator.
Regarding the fact that everything was fetched upfront in small batches without scrolling (in Xcode 6's Core Data Instruments that's not the case anymore - now it's one, big fetch which takes entire seconds):
It seems that setFetchBatchSize does not work correctly with parent/child contexts. The issue was reported back in 2012 and it seems that it's still there http://openradar.appspot.com/11235622.
To overcome this issue, I created another independent context with an NSMainQueueConcurrencyType and set its persistence coordinator to be the same that my other contexts are using.
More about issue #2 here: https://stackoverflow.com/a/11470560/1641848

Core Data working with subset of full data store...HOW?

I'm banging my head trying to figure out something in Core Data that I think should be simple to do, and I need some help.
I have a data store which contains data from the past two years, but in my app, I have certain criteria so that the user only works with a subset of that data (i.e. just the past month). I have created the predicates to generate the fetch request and all that works fine.
My issue is that I then want to run some additional predicates on this subset of data (i.e. I just want objects with name=Sally). I'd like to do so without having to re-run the original predicate with an additional predicate (in a NSCompoundPredicate); I'd rather just run it on the subset of data already created.
Can I just run a predicate on the fetch results?
Is the format of the predicate the same as for the initial calls into the core data store?
Thanks for any help.
You could filter the original array of results using a predicate. See the NSArray filteredArrayUsingPredicate method.
The only reason I would consider doing what you're talking about is if you find that modifying the predicate at the FetchedResultsController level caused significant performance degradation.
My recommendation is to get a reference to your fetchedResultsController in your view controller and update your predicate with a compound predicate matching your additional search parameters.
If you were to tie your viewControllers data source to a predicate of a predicate you wouldn't be able to properly utilize the NSFetchedResultsControllers Delegate methods that allow for easy, dynamic updating of Views such as table view and collection view.
/* assuming you're abstracting your datastore with a singleton */
self.fetchedResultsController = [DataStore sharedStore].fetchedResultsController;
self.fetchedResultsController.delegate = self;
[self.fetchedResultsController.fetchRequest setPredicate:yourPredicate];
Make sure to configure your fetch requests' batch size to a reasonable value to improve performance.

Resources