I'm reading this link for concurrency in core-data
Coredata - Concurrency
My app flow is:
One thread is contentiously inserting information in the background. And the user is playing in an app and update something.
-> Due to background thread all updates will be saved in database automatically as it will save my main context. So one can not discard what the changes he made as it will save automatically.
Before I go to update my core logic of inserting information, I need your advice.
Is there any way to handle my scenario easily?
Will concurrency work with the background and main thread?
Also, let me know if any demo or tutorial you have/finds. So I may refer it.
Thanks
Related
I have a multithreaded app using coredata.
when the app launches, there are many cloudkit NSoperations running simultaneously.
In each operation's completion block I am doing some core data processing.
Since the core data processing is being done in the background simultaneously by different operations, duplicate Managed Objects are being created.
I am using Coredata's background context and using performAndWait() method to do core data processing.
How can I solve this issue?
I thought using performAndWait() on background Context will ensure my background Core Data updates are all performed on one single thread. But still the duplicate Managed Objects are getting created.
As a workaround I'm using Core data's Unique attribute Constraint feature. But is this the right approach?
I was able to solve this.
The right way is to use Core data background context. Use performAndWait() to do all core data processing tasks inside it's block.
I would like to get some suggestion for making core data operation concurrent in my project. My project is running since two years, so that it has many implementations which can be optimized based on the availability of new features in objectiveC. Mainly, I am looking for optimizing CoreData operation.
Currently most of the data operations are done using main managed object context. Recently, I have implemented a new feature to download a big set of data and inserting in to database using core data after login. This was supposed to be execute in parallel with other operations in the application. Now I realized that the code written for core data is executing in the main thread, because the UI of application is blocking during the coredata operation. So I referred many blogs and came to know that there are two strategies in which core data concurrency can be achieved, Notifications with the help of multiple contexts and parent/child managed object contexts.
I tried the parent/child strategy as Apple is not preferring the other strategy. But I am getting random crashes with the exception “Collection was mutated while being enumerated” on executeFetchRequest. This exception starts happening after implementing the parent/child strategy. Can anyone help me to solve this issue?
Yeah , i know there are not so many blogs that describe efficient use of core data in project but luckily i found one... which points to your problem properly... check here-> https://medium.com/soundwave-stories/core-data-cffe22efe716#.3wcpw1ijo
also your exception is occurring because you are updating your database while it is being used somewhere to remove this exception you can do this like:
if you are fetching data in array or dictionary then do change statement like this
NSDictionary *myDict = [[coreDataDectionary] mutableCopy];
Now perform any operation on this array or dictionary which you fetch from database, it wont show any exception.
Hope this helps you.
Try this :
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, NULL), ^{
// DATA PROCESSING
dispatch_async(dispatch_get_main_queue(), ^{
// UPDATE UI
});
}
You should use completionBlock into your code. here is tutorial and explanation.
It will allow you to not freeze your UI application even if your download is not finished.
the execution of the code will continue even if the code inside the block isn't finished yet. There will be a callback action inside the block when the download will be over.
Use this Core Data stack to minimize UI locks when importing large datasets:
One main thread MOC with its own PSC.
One background MOC with its own PSC.
Merge changes into main thread MOC on background MOC's save notifications.
Yes, you can – and should – use two independent PSCs (NSPersistentStoreCoordinator) pointing to the same .sqlite file. It will reduce overall locking time to just SQLite locks, avoiding PSC-level locks, so overall UI locking time will be [SQLite write lock] + [main thread MOC read].
You can use background MOC with NSConfinementConcurrencyType within a background thread or even better within an NSOperation – I found it very convenient to process data and feed it to Core Data on the same thread.
Import in batches. Choose batch size empirically. Reset the background MOC after each save.
When processing really large datasets, with hundreds of thousands of objects, do not use refreshObject:mergeChanges: with main thread MOC on every save. It is slow and eventually will consume all of the available memory. Reload your FRCs instead.
And about "Collection was mutated while being enumerated". To-many relationships in Core Data are mutable sets, so you have to make a copy, or better sort them into NSArrays before iterating.
I am searching for the best possible way to update a fairly large core-data based dataset in the background, with as little effect on the application UI (main thread) as possible.
There's some good material available on this topic including:
Session 211 from WWDC 2013 (Core Data Performance Optimization and Debugging, from around 25:30 onwards)
Importing Large Data Sets from objc.io
Common Background Practices from objc.io (Core Data in the Background)
Backstage with Nested Managed Object Contexts
Based on my research and personal experience, the best option available is to effectively use two separate core-data stacks that only share data at the database (SQLite) level. This means that we need two separate NSPersistentStoreCoordinators, each of them having it's own NSManagedObjectContext. With write-ahead logging enabled on the database (default from iOS 7 onwards), the need for locking could be avoided in almost all cases (except when we have two or more simultaneous writes, which is not likely in my scenario).
In order to do efficient background updates and conserve memory, one also needs to process data in batches and periodically save the background context, so the dirty objects get stored to the database and flushed from memory. One can use the NSManagedObjectContextDidSaveNotification that gets generated at this point to merge the background changes into the main context, but in general you don't want to update your UI immediately after a batch has been saved. You want to wait until the background job is completely done and than refresh the UI (recommended in both the WWDC session and objc.io articles). This effectively means that the application main context remains out of sync with the database for a certain time period.
All this leads me to my main question, which is, what can go wrong, if I changed the database in this manner, without immediately telling the main context to merge changes? I'm assuming it's not all sunshine an roses.
One specific scenario that I have in my head is, what happens if a fault needs to be fulfilled for an object loaded in the main context, if the background operation has in between deleted that object from the database? Can this for instance happen on a NSFetchedResultsController based table view that uses a batchSize to fetch objects incrementally into memory? I.e., an object that has not yet been fully fetched gets deleted but than we scroll up to a point where the object needs to get loaded. Is this a potential problem? Can other things go wrong? I'd appreciate any input on this matter.
Great question!
I.e., an object that has not yet been fully fetched gets deleted but
than we scroll up to a point where the object needs to get loaded. Is
this a potential problem?
Unfortunately it'll cause problems. A following exception will be thrown:
Terminating app due to uncaught exception 'NSObjectInaccessibleException', reason: 'CoreData could not fulfill a fault for '0xc544570 <x-coredata://(...)>'
This blog post (section titled "How to do concurrency with Core Data?") might be somewhat helpful, but it doesn't exhaust this topic. I'm struggling with the same problems in an app I'm working on right now and would love to read a write-up about it.
Based on your question, comments, and my own experience, it seems the larger problem you are trying to solve is:
1. Using an NSFetchedResultsController on the main thread with thread confinement
2. Importing a large data set, which will insert, update, or delete managed objects in a context.
3. The import causes large merge notifications to be processed by the main thread to update the UI.
4. The large merge has several possible effects:
- The UI gets slow, or too busy to be usable. This may be because you are using beginUpdates/endUpdates to update a tableview in your NSFetchedResultsControllerDelegate, and you have a LOT of animations queing up because of the large merge.
- Users can run into "Could not fulfill fault" as they try to access a faulted object which has been removed from the store. The managed object context thinks it exists, but when it goes to the store to fulfill the fault the fault it's already been deleted. If you are using reloadData to update a tableview in your NSFetchedResultsControllerDelegate, you are more likely to see this happen than when using beginUpdates/endUpdates.
The approach you are trying to use to solve the above issues is:
- Create two NSPersistentStoreCoordinators, each attached to the same NSPersistentStore or at least the same NSPersistentStore SQLite store file URL.
- Your import occurs on NSManagedObjectContext 1, attached to NSPersistentStoreCoordinator 1, and executing on some other thread(s). Your NSFetchedResultsController is using NSManagedObjectContext 2, attached to NSPersistentStoreCoordinator 2, running on the main thread.
- You are moving the changes from NSManagedObjectContext 1 to 2
You will run into a few problems with this approach.
- An NSPersistentStoreCoordinator's job is to mediate between it's attached NSManagedObjectContexts and it's attached stores. In the multiple-coordinator-context scenario you are describing, changes to the underlying store by NSManagedObjectContext 1 which cause a change in the SQLite file will not be seen by NSPersistentStoreCoordinator 2 and it's context. 2 does not know that 1 changed the file, and you will have "Could not fulfill fault" and other exciting exceptions.
- You will still, at some point, have to put the changed NSManagedObjects from the import into NSManagedObjectContext 2. If these changes are large, you will still have UI problems and the UI will be out of sync with the store, potentially leading to "Could not fulfill fault".
- In general, because NSManagedObjectContext 2 is not using the same NSPersistentStoreCoordinator as NSManagedObjectContext 1, you are going to have problems with things being out of sync. This isn't how these things are intended to be used together. If you import and save in NSManagedObjectContext 1, NSManagedObjectContext 2 is immediately in a state not consistent with the store.
Those are SOME of the things that could go wrong with this approach. Most of these problems will become visible when firing a fault, because that accesses the store. You can read more about how this process works in the Core Data Programming Guide, while the Incremental Store Programming Guide describes the process in more detail. The SQLite store follows the same process that an incremental store implementation does.
Again, the use case you are describing - getting a ton of new data, executing find-Or-Create on the data to create or update managed objects, and deleting "stale" objects that may in fact be the majority of the store - is something I have dealt with every day for several years, seeing all of the same problems you are. There are solutions - even for imports that change 60,000 complex objects at a time, and even using thread confinement! - but that is outside the scope of your question.
(Hint: Parent-Child contexts don't need merge notifications).
Two Persistent Store Coordinators (pscs) is certainly the way to go with large datasets. File locking is faster than the locking within core data.
There's no reason you couldn't use the background psc to create thread confined NSManagedObjectContexts in which each is created for each operation you do in the background. However, instead of letting core data manage the queueing you now need to create NSOperationQueues and/or threads to manage the operations based on what you're doing in the background. NSManagedObjectContexts are free and not expensive. Once you do this you can hang onto your NSManagedObjectContext and only use it during that one operation and/or threads life time and build as many changes as you want and wait until the end to commit them and merge them to the main thread how ever you decide. Even if you have some main thread writes you can still at crucial points in your operations life time refetch/merge back into your threads context.
Also it's important to know that if you're working on large sets of data don't worry about merging contexts so as long as you aren't touching something else. For example if you have class A and class B and you have two seperate opertions/threads to work on them and they have no direct relationship you do not have to merge the contexts if one changes you can keep on rolling with the changes. The only major need for merging background contexts in this fashion is if there are direct relationships faulting. It would be better to prevent this though through some sort of serialization whether it be NSOperationQueue or what ever else. So feel free to work away on different objects in the background just be careful about their relationships.
I've worked on a large scale core data projects and had this pattern work very well for me.
Indeed, this is the best core data scenario you can work with. Almost no Main UI staleness, and easy background management of your data. When you want to tell the Main Context (and maybe a currently running NSFetchedResultsController) you listen for save notifications of the backgroundContext like this:
[[NSNotificationCenter defaultCenter]
addObserver:self selector:#selector(reloadFetchedResults:)
name:NSManagedObjectContextDidSaveNotification
object:backgroundObjectContext];
Then, you can merge changes, but waiting for the Main Thread context to catch them before saving. When you receive the mergeChangesFromContextDidSaveNotification notification the changes are not yet saved. Hence the performBlockAndWait is mandatory, so the Main context gets the changes and then the NSFetchedResultsController updates its values correctly.
-(void)reloadFetchedResults:(NSNotification*)notification
{
NSManagedObjectContext*moc=[notification object];
if ([moc isEqual:backgroundObjectContext])
{
// Delete caches of fethcedResults if you have a deletion
if ([[theNotification.userInfo objectForKey:NSDeletedObjectsKey] count]) {
[NSFetchedResultsController deleteCacheWithName:nil];
}
// Block the background execution of the save, and merge changes before
[managedObjectContext performBlockandWait:^{
[managedObjectContext
mergeChangesFromContextDidSaveNotification:notification];
}];
}
}
There is a pitfall no one has noticed. You can get the save notification before the background context has actually saved the object you want to merge. If you want to avoid problems by a faster Main Context asking for an object that has not been saved yet by the background context, you should (you really should) call obtainPermanentIDsForObjects before any background save. Then you are safe to call the mergeChangesFromContextDidSaveNotification. This will ensure that the merge receives a valid permanent Id for merging.
Here's the scenario:
I'm writing a DownloadManager, that allows the user to download, pause, cancel, download all, and pause all. The DownloadManager is a singleton, and uses AFNetworking to download files. It has it's own private managed object context, so that user can freely use other parts of the application (by adding, editing, deleting) core-data objects. I have a core-data entity DownloadInfo that stores the download information i.e. fileURL, fileSize, bytesRead, etc. The DownloadManager updates the download progress in DownloadInfo (one for each file).
I have a DownloadManagerViewController which uses NSFetchedResultsController to show the download status to the user. This download view controller is using the main managed object context.
Now let's say that I have 20 files in the download queue. And let's say that only 3 concurrent downloads are allowed. The download manager should download the file, and show the download progress.
Problem:
The DownloadInfo objects are being updated by the DownloadManager at a very high rate. The DownloadManagerViewController (responsible for showing the download progress) is updating the list using NSFetchedResultsControllerDelegate methods. The result is that a lot is happening in the main queue and application has very poor responsiveness.
How can I fix this? How can I make the application responsive, while showing the download progress?
I don't know how else to communicate that the download status between DownloadManager and DownloadManagerViewController. Is there another/ a better way to do this?
I don't want to use main managed object context in my DownloadManager, for reasons mentioned above. Note, that the DownloadManager is using AFNetworking which is handling the requests asynchronously, but eventually the DownloadInfo objects are updated in the main thread (as a result of the callback methods). Maybe there's a way to handle the downloads and status update operations in a background thread? but how? How will I communicate between the main thread and the background thread i.e. how will I tell the background thread to queue another file for download?
Thanks.
Rather than observing every change to the managed object context, consider implementing one or more notifications for the events that you actually want to update the screen. If the notification are being posted from a background thread, be sure to switch back to the main thread before triggering any UI update.
Alternatively, when the FRC delegate methods are called you receive information about what actually changed. You can analyse this and filter out the most frequent and least meaningful changes and just prevent them from resulting in a UI update.
Have you tried to use Instruments to see what's really going on in your ViewController? Time Profiler is going to tell you where the CPU is spending more time and will help you to identify the root of the problem. Without knowing this, we can't know exactly what performance tweaks you should do.
After using Instruments, if the NSFetchedResultsController is forcing a lot of updates of your VC, you should consider not to read the progress from CoreData. I've seen that a lot of my tables or views spend more time reading from CoreData than performing draws. Guessing that your problem is with CoreData reads, I would try to read progress from an NSMutableDictionary and just update the UI when the downloads pass a certan threshold.
The Apple "Concurrency with Core Data" documentation states the following when discussing using core data with background threads.
Saving in a Background Thread is Error-prone
Asynchronous queues and threads do not prevent an application from
quitting. (Specifically, all NSThread-based threads are “detached”—see
the documentation for pthread for complete details—and a process runs
only until all not-detached threads have exited.)
and in particular:
If you need to save on a background thread, you must write additional code such that the main thread prevents the application from quitting until all the save operation is complete.
What is the recommended approach for achieving this inside an IOS application?
In the app delegates applicationWillTerminate and related methods, you need to check if any background threads have unsaved changes and save them before allowing the app to terminate or go into the background.
I recommend taking a look at using Magical Record (https://github.com/magicalpanda/MagicalRecord/). It greatly simplifies dealing with core data on background threads. I recently found this and used it for a project. We've now undertaken a maintenance effort to update a variety of existing apps to use the new Magical Record Core Data wrapper. It has saved us tons of time and frustration in the few weeks we have been using it.