On sync I overwrite all my local data with the server's data. For this I first call realm.delete(realm.objects(MyObj)) for all my objects. Then I save the response's objects with realm.add(obj, update: false). Everything is in a single transaction. The payload can take a while to process but it's not big enough to justify implementing pagination.
Can the user use the app normally during this process? Can they store new items that are deleted in the clearing part of the transaction, or that would trigger an error or be overwritten during the adding part? If yes how can I avoid this?
Realm uses a Multi-Version-Concurrency-Control algorithm. This uses locks to ensure exclusive write, while other threads can keep reading previous versions of the data. We have an article on our blog, which explains how that works in more depth.
Be aware that what you attempt to solve here is a non-trivial challenge.
Can they store new items that are deleted in the clearing part of the transaction, or that would trigger an error or be overwritten during the adding part?
While the background transaction is in progress, other write transactions would be blocked. If you do these writes from the main thread, you would block the main thread. If you do them from background threads, they would queue up and be executed after your sync transaction is completed.
The objects, which are deleted in the beginning would become inaccessible (which you can check via invalidated), because write transactions always operate on the latest version. If your objects have consistent primary keys across your sync operations, you can utilize those to re-fetch them and redo all modifications to the fresh instances. But note, that you need to store the primary keys (and all other object data) into memory, before beginning the write transaction, which implies an implicit refresh.
Related
I am currently implementing a web application in .net core(C#) using entity framework. While working on the project, I actually encountered quite a few challenges but I will start with the one which I think are most important. My questions are as follows:
Instead of frequent loading data from the database, I am having a set of static objects which is a mirror of the data in the database. However, it is tedious and error prone when I want to ensure any changes, i.e., adding/deleting/modifying of objects are being saved to the database at real time. Is there any good example or advice that I can refer to improve my approach to do this?
Another thing is that value of some objects' properties will be changed on the fly according to the value of some other objects' properties. Something like a spreadsheet where a cell's value will be changed automatically if the value in the cell that the formula is referring to changes. I do not have a solution to do this yet. Appreciate if anyone has any example that I can refer to. But this will add another layer of complexity to sync the changes of the objects in memory to database.
At the moment, I am unsure if there is any better approach. Appreciate if anyone can help. Thanks!
Basically, you're facing a problem that's called eventual consistency. Something changes and two or more systems need to be aware at the same time. The problem here is that both changes need to be applied in order to consider the operation successful. If either one fails, you need to know.
In your case, I would use the Azure Service Bus. You can create queues and put messages on a queue. An Azure Function would handle these queue messages. You would create two queues, one for database updates, and one for the in-memory update (I think changing this to a cache service may be something to think off). Now the advantage of these queues is that you can easily drop messages on these queues from anywhere. Because you mentioned the object is going to evolve, you may need to update these objects either in the database or in memory (cache).
Once you've done that, I'd create a topic, with two subscriptions. One forwarding messages to Queue 1, and the other to Queue 2. This will solve your primary problem. In case an object changes, just send it to the topic. Both changes (database and memory) will be executed automagically.
The only problem you have now, it that you mentioned you wanted to update the database in real-time. With this scenario, you're going to have to leave that.
Also, you need to make sure you have proper alerts in place for the queues so in case you did miss a message, or your functions didn't handle it well enough, you'll receive an alert to check & correct errors.
I'm totally agree with #nineedm's and answer, but there are also other solutions.
If you introduce cache, you will always face cache revalidation problem - you have to mark cache as invalid when data were changed. Sometimes it is easy, depending on nature of cached data and how often data are changed.
If you have just single application, MemoryCache can be enough with proper specified expiration options.
If there is a cluster - you have to look at Distributed Cache solutions, for example Redis. There is MS article about that Distributed caching in ASP.NET Core
I have an app that uses Realm as a staging database. It receives information from a bluetooth device, processes it, and sends the processed result to a server.
The incoming data from bluetooth gets stored in a Realm table (table1). A separate thread reads data from the Realm database, processes it, and stores it into a second table (table2) for uploading to a server. When it pulls this data and successfully processed it, it deletes it from table1.
The third thread pulls data from table2, and when it successfully sends, removes it from table2.
I'm using a database here in case, for whatever reason, the app is killed - data won't be lost... it will just resume where it left off when the app is restarted. But as you can see, the database is not something that hangs around (it's not like an address book or something... it is just temporary staging)
What I notice is that no matter what the heck I do, the realm database file just increases in size over time. I'll end up with a database that if I open it, will have one record in it, but the database file on disk could be 10s of MB in size if the app is running long enough.
Data is being processed on different background queues so as to not block any UX (one of the reasons I'm using Realm instead of CoreData). But I'm using things like autoreleasepools and the invalidate command to avoid objects that are read from having copies made (as suggested by many realm questions/answers)
What gives? I know I don't have a code sample here, but this just seems like a basic garbage collection problem in Realm. I've seen other questions related to this where people are like "why is my database so huge", and the answers suggest doing things like "writeCopyToPath", but that feels like an incredible hack, and regardless, it would be very difficult - this app is meant to be constantly connected and monitoring a bluetooth device, so to do this, it would mean stopping, making sure all threads that might alter the database are quiesced, doing the copy to compact the db, and then starting everything back up again. That just seems nonsensical to me. I might interrupt user operations for example. I don't want a user to not be able to do something because I decided it was time to do database maintenance.
I feel like I'm either missing some incredibly fundamental point in how to make Realm not keep junk around, or Realm is just the completely wrong solution for my problem. I've never seen this problem with databases - adding and deleting lots of records... quickly... seems like something a database should just be able to do without exploding in size.
Are you making sure that the background thread is not holding on to old versions of the Realm, preventing the space from being reused?
Quote from the docs (https://realm.io/docs/swift/latest/#seeing-changes-from-other-threads):
If a thread has no runloop (which is generally the case in a background thread), then Realm.refresh() must be called manually in order to advance the transaction to the most recent state.
Failing to refresh Realms on a regular basis could lead to some transaction versions becoming “pinned”, preventing Realm from reusing the disk space used by that version, leading to larger file sizes.
I'm in the middle of development of an iOS application after working quite some time with Core Data and Magical Record and having an error:
error: NULL _cd_rawData but the object is not being turned into a fault
I didn't know Core Data before this project and as it turns out I was very naive to think I can just work with Magical Record without worrying about concurrency, as I haven't dedicated any thoughts/work about managed contexts for the main thread and background threads.
After A LOT of reading about Core Data Managed Object Contexts and Magical Record, I understand that:
NSManagedObjects are not thread safe.
NSManagedObjectId IS thread safe.
I can use: Entity *localEntity = [entity MR_inContext:localContext] of Magical Record to work with an entity in a background thread's context.
I should use Magical Record's saveWithBlock:completion: and saveWithBlockAndWait: methods to get a managed context to use for background threads.
A little information regarding my application:
I'm using the latest version of Magical Record which is 2.2.
I have a backend server which my application talks to a lot.
Their communication is similar to Whatsapp, as it uses background threads for communicating with the server and updating managed objects upon successful responses.
I'm wrapping the model with DataModel objects that hold the managed objects in arrays for quick referencing of UI/background use.
Now - my questions are:
Should I fetch only from the UI thread? Is it okay that I'm holding the managed objects in DataModel objects?
How can I create a new entity from a background thread and use the newly created entity in the DataModel objects?
Is there a best design scenario that I should use? specifically when sending a request to the server and getting a response, should I create a new managed context and use it throughout the thread's activity?
Let me know if everything is clear. If not, I'll try and add clarity.
Any help or guidelines would be appreciated.
Thanks!
I'm not working with MagicalRecord, but these questions are more related to CoreData than to MagicalRecord, so I'll try to answer them :).
1) Fetching from main(UI) thread
There are many ways how to design app model, so two important things I've learned using CoreData for few years:
when dealing with UI, always fetch objects on main thread. As You correctly stated NSManagedObjects are not thread safe so you can't (well, can, but shouldn't) access their data from different thread. NSFetchedResultsController is Your best friend when you need to display long lists (eg. for messages – but watch out for request's batchSize).
you should design your storage and fetches to be fast and responsive. Use indexes, fetch only needed properties, prefetch relationships etc.
on the other hand if you need to fetch from large amount of data, you can use context on different thread and transfer NSManagedObjectIDs only. Let's say your user has huge amount of messages and you want to show him latest 10 from specific contact. You can create background context (private concurrency), fetch these 10 message IDs (NSManagedObjectIDResultType), store them in array (or any other suitable format for you), return them to your main thread and fetch those IDs only. Note that this approach speed things up if fetch takes long because of predicate/sortDescriptor not if the "problem" is in the process of turning faults into objects (eg. large UIImage stored in transformable attribute :) )
2) Creating entity in background
You can create the object in background context, store it's NSManagedObjectID after you save context (object has only temporary ID before save) and send it back to your main thread, where you can perform fetch by ID and get the object in your main context.
3) Working with background context
I don't know if it's exactly the best, but I'm pretty much satisfied with NSManagedObjectContext observation and merging from notifications. Checkout:
mergeChangesFromContextDidSaveNotification:
So, you create background context, add main context as observer for changes (NSManagedObjectContextObjectsDidChangeNotification) and background context automatically sends you notifications (every time you perform save) about all of it's changes – inserted/updated/deleted objects (no worries, you can just merge it by calling mergeChangesFromContextDidSaveNotification:). This has many advantages, such as:
everything is updated automatically (every object you have fetched in "observing context" gets updated/deleted)
every merge runs in memory (no fetches, no persisting on main thread)
if you implement NSFetchedResultsController's delegate method, everything updates automatically (not exactly everything – see below)
On the other side:
take care about merging policy (NSMangedObjectContext mergePolicy)
watchout for referencing managed objects that got deleted from background (or just another context)
NSFetchedResultsController updates only on changes to "direct" attributes (checkout this SO question)
Well, I hope it answers your questions. If everything comes up, don't hesitate to ask :)
Side note about child contexts
Also take a peek to child contexts. They can be powerful too. Basically every child context sends it's changes to parent context on save (in case of "base" context (no parent context), it sends it's changes to persistent coordinator).
For example when you are creating edit/add controller you can create child context from your main context and perform all changes in it. When user decides to cancel the operation, you just destroy (remove reference) the child context and no changes will be stored. If user decides to accept changes he/she made, save the child context and destroy it. By saving child context all changes are propagated to it's parent store (in this example your main context). Just be sure to also save the parent context (at some point) to persist these changes (save: method doesn't do the bubbling). Checkout documentation of managing parent store.
Happy coding!
I am building an app with synchronization support. Since the sync will require more than one request there is a risk of persistent errors during the sync (e.g. if some but not all of the requests fails).
If anything goes wrong during the synchronization I want to rollback all changes so the synchronization is performed fully or not at all.
It seems RestKit saves the managedObjectContext when it fetches data. This means I have no way of using an NSUndoManager to handle undo/rollbacks. What is the preferred way to do this? To backup the object store file (sqlite) and replace it if the synchronization fails would be one way but it does not seem very "pure".
I'm syncing with a MySQL database.
Initially, I was going to loop through all my new/modified objects and set all the foreign keys for that object and then do the next object, and so on... But that's a lot of fetch requests.
So instead I wanted to loop through all my new/modified objects and set the foreign keys one at a time. So the first pass over my objects sets fk1, my next sets fk2, so on...
Cool, fetch requests drastically reduced. Now I'm curious if I could thread these fk setters. They aren't dependent on each other, but they are modifying the same object, even though they're only setting one relationship, and it's a different relationship. Speaking in git terms, these changes could be 'merged' together without any conflict, but is it possible to push changes in one child managedObjectContext(childContext:save) up to the parentManagedObjectContext(parent:performBlock^{parent:save}) and pull it down in another, different child managedObjectContext(???)? Or will the merge policy only take one childContext's version of the object and leave the other fks effectively unchanged.
I know this exists: NSManagedObjectContext/refreshObject:mergeChanges:
But that's on an object by object level. Will that cause a bunch of fetches? Or will that update my whole context at once/in batches?
Following Apple's suggestion from here:
https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreData/Articles/cdImporting.html
I've created/updated my values before I start setting any relationships, so all entities already exist before I try to point any relationships at them.
Aside: We have a couple apps that could benefit from the concurrency, because they throw a considerable amount of data around, and with the quad core iPad apps, this would really help out with the time the initial sync takes.
I'n not sure what you are trying to do and why (you could write less lines in your question and be more clear), but here are some guidelines for working with Core Data:
-- NSManagedObjectContext is not thread safe. Therefore, you need to limit your access to this managed object context to happen inside 1 thread. Otherwise, you may end up having many errors you can't understand.
-- NSManagedObjectContexts apart from doing other things, serve like "snapshots" of your persistent store. That means that when you change an object, you save it to the persistent store, you post a NSManagedObjectContextDidSaveNotification and then call mergeChangesFromContextDidSaveNotification: in another place inside your program in order to load the latest data from the persistent store. Be careful of thread safety.
NSManagedObjectContext/refreshObject:mergeChanges: according to apple is not about just refreshing a managed object. If you pass YES as the second argument, it will write any pending changes from this managed object context to the persistent store, and will load any other changes to other properties of this object from the persistent store, thus "synchronizing" your object with the persistent store. If you pass NO as the second argument, the object loses any pending changes, and it is being turned to a fault. That means that when you attempt to access it, the Managed Object Context will reload the object as it was last saved to the database. It will NOT reload the entire managed object context. It will only operate on the object.
Aside: I have written a blog post that scratches the surface of asynchronous loading from a core data database. In my case, since I'm doing heavy lifting with the database, I ended up using an NSOperation that operates with its own NSManagedObjectContext, and using serial GCD queues to save large chunks of data, since it was faster than having multiple threads accessing the same persistent store, even if they operate on different managed object contexts.
I hope I helped.