we are adopting (Swift)Realm as a data store in our iOS app and we are really pleased with it so far. We have a question around the design for the retrieval and storage of objects with Realm and multi-threading:
Is it acceptable to load objects in the main ui thread?
We know about the constraints that objects loaded with realm cannot be shared between threads.
We are also not seeing any performance issues yet, but our approach so far is to load all kinds of resources in background threads.
In the case where we load and filter some data and register a notification block, we don't see problems with using the main ui thread, but how would we handle a situation, where we for example want to display all data in a table view?
Is it acceptable to load objects in the main ui thread?
Yes, it is in most cases* acceptable and fast enough. It wouldn't be acceptable if reading from the database would block the user, but as there is no concept like faults, read access is always predictable fast.
Only if you have a really complex object graph, where you need to do heavy pre-processing to be able to display the objects on the UI, it would make sense to employ a background thread and/or caching to warrant a good user experience.
In the case where we load and filter some data and register a notification block, we don't see problems with using the main ui thread, but how would we handle a situation, where we for example want to display all data in a table view?
A UITableView only request those cells which are currently visible on the screen and reuses the view containers. A Realm collection is similar in this lazy nature, when you don't filter it, it doesn't enlarge the memory pressure, because you get only object accessors for those objects which you pull out of it. There is no need for pagination as long as you rely on the builtin Realm Results or List collections. Only if you need to apply a custom and complex filter in a way which isn't supported by Realm, it might be necessary to process that on a background thread.
Related
I am developing iOS an app which will gather big amounts of data from several sources (up to tens of thousands of objects, but simple objects, no images) and save it to my own database using core data. I then analyse this data and display the results to the user.
I want to know if there is any benefit to using a Main Queue Nsmanagedobjectcontext or if it is enough that I use a private one.
I also want to know what the benefit is of having several NSManagedObjectContext or if one is enough?
The concurrency model i am using currently only has one private queue nsmanagedobjectcontext connected to a persistant store coordinator. All the data analysis is performed on the private queue and then I simply pass the analyzed data to the main queue to display it. On older devices (iPhone 4) my application can sometimes crash when too much data is being loaded (i.e. downloaded from the external databases) at the same time, is this related to my choice of concurrency model?
Your current approach sounds fine. You only need a main thread context if you want the main thread to interact with the data, and in your case you don't so that's fine.
Your memory management is effectively unrelated and is more tied to how many things you have going on at once (it sounds like one) and how many objects you try to keep in main memory at any one time (it sounds like many) instead of faulting them out to the data store. This is what you need to look at / work on. Instruments can help you see how many objects you're keeping in memory.
At least call refreshObject:mergeChanges: with NO for merge changes to fault out any objects that you aren't using.
Also, remember that you're working on a mobile device and that processing up to tens of thousands of objects is a job better handled by a server...
I am developing an application that uses Core Data for internal storage. This application has the following functionalities :
Synchronize data with a server by downloading and parsing a large XML file then save the entries with core data.
Allow user to make fetches (large data fetches) and CRUD operations.
I have read through a lot and a lot of documentation that there are several patterns to follow in order to apply multithreading with Core Data :
Nested contexts : this patterns seems to have many performance issues (children contexts block ancestor when making fetches).
Use one main thread context and background worker contexts.
Use a single context (main thread context) and apply multithreading with GCD.
I tried the 3 mentioned approaches and i realized that 2 last ones work fine. However i am not sure if these approaches are correct when talking about performance.
Is there please a well known performant pattern to apply in order to make a robust application that implements the mentioned functionalities ?
rokridi,
In my Twitter iOS apps, Retweever and #chat, I use a simple two MOC model. All database insertions and deletions take place on a private concurrent insertionMOC. The main MOC merges through -save: notifications from the insertionMOC and during merge processing emits a custom UI update notification. This lets me work in a staged fashion. All tweets come in to the app are processed on the background and are presented to the UI when everything is done.
If you download the apps, #chat's engine has been modernized and is more efficient and more isolated from the main thread than Retweever's engine.
Anon,
Andrew
Apple recommends using separate context for each thread.
The pattern recommended for concurrent programming with Core Data is
thread confinement: each thread must have its own entirely private
managed object context. There are two possible ways to adopt the
pattern: Create a separate managed object context for each thread
and share a single persistent store coordinator. This is the
typically-recommended approach. Create a separate managed object
context and persistent store coordinator for each thread. This
approach provides for greater concurrency at the expense of greater
complexity (particularly if you need to communicate changes between
different contexts) and increased memory usage.
See the apple Documentation
As per apple documentation use Thread Confinement to Support Concurrency
Creating one managed object context per thread. It will make your life easy. This is for when you are parsing large data in background and also fetching data on main thread to display in UI.
About the merging issue there are some best ways to do them.
Never pass objects between thread, but pass object ids to other thread if necessary and access them from that thread, for example when you are saving data by parsing xml you should save them on current thread moc and get the ids of them, and pass to UI thread, In UI thread re fetch them.
You can also register for notification and when one moc will change you will get notified by the user info dictionary which will have updated objects, you can pass them to merge context method call.
My app connects to a Parse server asynchronously and downloads the necessary data into the app's Core Data store . I would then like to display this data in a tableview. But in most cases -- as the connection is asynchronous -- the table view can access the data store much faster than the downloaded does . In this situation, I get an empty table view cell, and just after that the data is ready in the data store.
What is the best way to deal with the delays caused by asynchronous downloads? Is there a concept that I'm missing? Is it NSFetchedResultsController?
What do you think is the best way to deal with the delays caused by
asynchronous downloads?
It depends on requirements you have. In particular, if the user can interact with UI during async downloads, you can do nothing on it, otherwise you could use just a spinner to alert him something is downloading and stop the interaction until the sync as finished.
Anyway, in both cases, you should say something about the download. In particular, are you saving data in a different thread (different from the main one)? If so you should merge the changes from the context you use in background to the context associated with the NSFetchedResultsController (always the main one since NSFetchedResultsController manages UI elements).
Is there a "concept" that I miss and is it NSFetchedResultsController?
Did you setup correctly the delegate NSFetchedResultsControllerDelegate? If so, the NSFetchedResultsController tracks changes on the entity you registered on your fetch request. Not changes will happen for other entities.
Asynchronous is a design problem that you will need to work with. Look to some of the other popular applications in your field and see how they solve it. Do they show a spinner (I personally hate that) or do they show some unobtrusive indicator that data is being downloaded (better)?
If you use a NSFetchedResultsController (which I am guessing by your question you are not currently) you will get the data displayed once it is saved in Core Data with no additional effort on your part. So you can at least show the data as soon as possible.
In the meantime, I recommend let the cells/table be empty and let the user know that your app is working. Display the data as soon as possible. Perhaps consider downloading the data in pieces so that they can start to see it asap.
We are working on an Enterprise-level application, which will store tens of thousands of objects with Core Data, and we are having issues on several fronts.
Our application has several independent systems which operate on the data when needed. These systems include discovery of items, loading of items, synchronization and UI display. If we design our software correctly, there should be little to none merge conflicts due to the different systems modifying same objects. Each system has its own operation queues, all performing in the background. We wish to keep all object creation and modification in the background to minimize UI performance issues, especially during initial ramp up, where thousands of objects might be created from data on the server. Here we have hit several problems with our various design attempts. Huge memory consumption during these ramp ups, and incorrect orchestration of all the contexts and child contexts, causing deadlocks and crashes.
We have attempted the following designs:
One root NSPrivateQueueConcurrencyType managed object context which has one child NSMainQueueConcurrencyType context. The UI fetched results controllers use this child context to fetch results from. From the NSMainQueueConcurrencyType child context, we created one NSPrivateQueueConcurrencyType child context, which we called "savingContext" and each background operation created a child context of that "savingContext", did its changes, and finally did what we called a "deep save", recursively saving to the top. We initially chose this design to not have to deal with NSManagedObjectContextDidSaveNotification notifications from many different child contexts. We wrapped every call to the NSPrivateQueueConcurrencyType contexts and access to objects with performBlockAndWait:. Functionally, this design performed. All changes and inserts were saved to the persistent store, and UI was updated with the changes. This, introduced two issues. One was laggy UI during ramp up because of merged changes going through the NSMainQueueConcurrencyType child context, and more importantly, very high memory usage during ramp up. We would hit prohibitive RAM usages due to inability to call reset recursively on contexts (as the main UI child context is there too) and/or lack of knowledge when to call refreshObject:mergeChanges:. So we went a different road.
Have two top-level contexts linked with the persistent store coordinator, one NSPrivateQueueConcurrencyType for save child contexts, and a NSMainQueueConcurrencyType for UI display. The NSMainQueueConcurrencyType listens to NSManagedObjectContextDidSaveNotification notifications from the main NSPrivateQueueConcurrencyType context and merges them in the main thread. Each background operation creates a child context of the main NSPrivateQueueConcurrencyType context, also with private queue concurrency type, does what it does, performs a "deep save" recursively, which performs a save on the current context, a recursive call of deep save to its parent, calls reset on the current context and saves again. This way we avoid the memory issues, as created objects are released quickly after save. However, with this design, we have hit a plethora of issues such as dead locks, NSInternalInconsistencyException exceptions and fetched results controllers not updating the UI despite there being save notifications for the NSMainQueueConcurrencyType context. This also cause initial load times in the UI to slow a lot. In the previous design, the fetched results controller returned results very fast, while this has the UI blocked for several seconds until the view loads (we initialize the fetched results controller in viewDidLoad).
We have tried many intermediate designs, but they all revolve around the same issues, either very high memory usage, fetched results controller not updating the UI or deadlocks and NSInternalInconsistencyException exceptions.
I am really getting frustrated. I can't but feel as if our designs are overtly complicated for something that should be rather simple, and it is just our lack of understanding some fundamental that is killing us.
So what would you guys suggest? What arrangement would you recommend for our contexts? How should we manage different contexts in different threads? Best practices for freeing up inserted objects and resetting contexts? Avoiding dead locks? All help would be appreciated at this point.
I have also seen recommendations for the MagicalRecords category. Is it recommended? We have are already invested in using Core Data types, how difficult would it be to migrate using MR?
First, to manage your memory, your second architecture gives you much more flexibility.
Second, there are two kinds of memory to manage: malloc-ed memory and resident VM memory. You can have a low malloc-ed memory footprint and still have a large VM resident region. This is due, in my experience, to Core Data aggressively holding on to newly inserted items. I solve this problem with a post-save trimming notification.
Third, MOCs are cheap. Use'em and throw'em away. In other words, release memory early and often.
Fourth, try to do almost nothing data base wise on the main MOC. Yes, this sounds counter-productive. What I mean is that all of your complex queries really should be done on background threads and then have the results passed to the main thread or have the queries redone from the main thread while exploiting the now populated row-cache. By doing this, you keep the UI live.
Fifth, in my heavily multi-queued app, I try to have all of my saves really occur in the background. This keeps my main MOC fast and consistent with data coming in from the net.
Sixth, the NSFetchedResultsController is a quite useful but specialized controller. If your application pushes it outside of its area of competence, it starts locking up your interface. When that happens, I roll my own controller by listening for the -didSave notifications myself.
I am currently using a method where I run a fetch request in the background to obtain object IDs, and then instantiating them with -existingObjectWithID:error:.
The problem is that these objects have to-many relation to a large number of objects. And the UI freezes for a while when these objects are accessed. (They are accessed all at once.)
I am guessing that the related objects are faults. I am trying to figure out a way to preload them in the background. Is there a solution to this problem?
Do you know for sure that it is your main thread that is causing the slowdown (sure sounds like it) - I'd use Instruments and "Time Profiler" to be sure, and there is also a way to turn on SQL debugging/timing too.
If it is your main thread, there are fantastic WWDC videos (take a look at 2010 too, not just 2011) on how to optimize Core Data.
Try the setRelationshipKeyPathsForPrefetching: method on NSFetchRequest. Pass in an array of keys that represent relationships that should be fetched rather than faulted.
Core data is not thread safe. So for background thread you should have separate managed context.
Typically core data don't take lots time to load. But if you are storing blobs (like image data) it can hit the performance. You should you NSFetchRequestController with page size you want to set. It much faster So you probably wont need to worry about about background fetching