Core Data managed object context design recommendation - ios

Question

We are working on an Enterprise-level application, which will store tens of thousands of objects with Core Data, and we are having issues on several fronts.
Our application has several independent systems which operate on the data when needed. These systems include discovery of items, loading of items, synchronization and UI display. If we design our software correctly, there should be little to none merge conflicts due to the different systems modifying same objects. Each system has its own operation queues, all performing in the background. We wish to keep all object creation and modification in the background to minimize UI performance issues, especially during initial ramp up, where thousands of objects might be created from data on the server. Here we have hit several problems with our various design attempts. Huge memory consumption during these ramp ups, and incorrect orchestration of all the contexts and child contexts, causing deadlocks and crashes.
We have attempted the following designs:
One root NSPrivateQueueConcurrencyType managed object context which has one child NSMainQueueConcurrencyType context. The UI fetched results controllers use this child context to fetch results from. From the NSMainQueueConcurrencyType child context, we created one NSPrivateQueueConcurrencyType child context, which we called "savingContext" and each background operation created a child context of that "savingContext", did its changes, and finally did what we called a "deep save", recursively saving to the top. We initially chose this design to not have to deal with NSManagedObjectContextDidSaveNotification notifications from many different child contexts. We wrapped every call to the NSPrivateQueueConcurrencyType contexts and access to objects with performBlockAndWait:. Functionally, this design performed. All changes and inserts were saved to the persistent store, and UI was updated with the changes. This, introduced two issues. One was laggy UI during ramp up because of merged changes going through the NSMainQueueConcurrencyType child context, and more importantly, very high memory usage during ramp up. We would hit prohibitive RAM usages due to inability to call reset recursively on contexts (as the main UI child context is there too) and/or lack of knowledge when to call refreshObject:mergeChanges:. So we went a different road.
Have two top-level contexts linked with the persistent store coordinator, one NSPrivateQueueConcurrencyType for save child contexts, and a NSMainQueueConcurrencyType for UI display. The NSMainQueueConcurrencyType listens to NSManagedObjectContextDidSaveNotification notifications from the main NSPrivateQueueConcurrencyType context and merges them in the main thread. Each background operation creates a child context of the main NSPrivateQueueConcurrencyType context, also with private queue concurrency type, does what it does, performs a "deep save" recursively, which performs a save on the current context, a recursive call of deep save to its parent, calls reset on the current context and saves again. This way we avoid the memory issues, as created objects are released quickly after save. However, with this design, we have hit a plethora of issues such as dead locks, NSInternalInconsistencyException exceptions and fetched results controllers not updating the UI despite there being save notifications for the NSMainQueueConcurrencyType context. This also cause initial load times in the UI to slow a lot. In the previous design, the fetched results controller returned results very fast, while this has the UI blocked for several seconds until the view loads (we initialize the fetched results controller in viewDidLoad).
We have tried many intermediate designs, but they all revolve around the same issues, either very high memory usage, fetched results controller not updating the UI or deadlocks and NSInternalInconsistencyException exceptions.
I am really getting frustrated. I can't but feel as if our designs are overtly complicated for something that should be rather simple, and it is just our lack of understanding some fundamental that is killing us.
So what would you guys suggest? What arrangement would you recommend for our contexts? How should we manage different contexts in different threads? Best practices for freeing up inserted objects and resetting contexts? Avoiding dead locks? All help would be appreciated at this point.
I have also seen recommendations for the MagicalRecords category. Is it recommended? We have are already invested in using Core Data types, how difficult would it be to migrate using MR?

Answer 1

First, to manage your memory, your second architecture gives you much more flexibility.
Second, there are two kinds of memory to manage: malloc-ed memory and resident VM memory. You can have a low malloc-ed memory footprint and still have a large VM resident region. This is due, in my experience, to Core Data aggressively holding on to newly inserted items. I solve this problem with a post-save trimming notification.
Third, MOCs are cheap. Use'em and throw'em away. In other words, release memory early and often.
Fourth, try to do almost nothing data base wise on the main MOC. Yes, this sounds counter-productive. What I mean is that all of your complex queries really should be done on background threads and then have the results passed to the main thread or have the queries redone from the main thread while exploiting the now populated row-cache. By doing this, you keep the UI live.
Fifth, in my heavily multi-queued app, I try to have all of my saves really occur in the background. This keeps my main MOC fast and consistent with data coming in from the net.
Sixth, the NSFetchedResultsController is a quite useful but specialized controller. If your application pushes it outside of its area of competence, it starts locking up your interface. When that happens, I roll my own controller by listening for the -didSave notifications myself.