I've read in another answer that Simperium should be able to handle Core Data lightweight migrations fine. However, I'm currently struggling with the simple case of adding an (optional) property to an existing entity.
To make the issue a bit easier to follow, let's go through an example. Let's say that my previous app version is 1.0 and a new version 1.1 introduces a new property foo on a Core Data entity.
Now, let's consider this scenario:
Device A and B both run version 1.0 and are attached to the same Simperium sync account, both up-to-date.
Device A upgrades to version 1.1, the property is created in the data base, and the user adds some data to the new foo property. This data is correctly synced to the Simperium backend and foo is visible in the web data browser.
Device B (still on version 1.0, i.e. without foo) syncs with Simperium. At this point I see Simperium warning: applyDiff for a member that doesn't exist, which is understandable because foo does not exist. Everything still expected and fine here.
Now device B upgrades to 1.1. When starting the app for the first time, Core Data creates the new foo column. However, now that the foo property is there, Simperium still does not pull its data from the backend, so devices A and B don't see the same data for foo!
I understand why this happens (Simperium has discarded the change the first time, because the property wasn't there yet, and then doesn't apply the same change again later when it could). However, I think this is quite problematic and effectively makes even the simplest changes to the data model very risky. Am I missing anything here? What would be a safe way to add a new property to an entity?
Sorry it took us a while to reply (please, by all means, feel free to open an issue on Github, anytime!).
For the time being, there are few caveats when it comes to Migrations.
I've filed this issue, to properly handle the New Attributes scenario, and this issue to deal with attribute renames.
Right now, although the protocol is resilient, and sync'ing should recover on its own (in the eventuality in which something happens, and a client goes off sync, ie. migrations), an attribute rename will cause glitches.
For now, i'm afraid that the easiest // safest way to deal migrations, would be to create another bucket, that would have the desired set of attributes.
Sorry about this inconvenience, we'll be iterating over this!
Related
I've spent the better part of a workday trying to solve this.
Background
I have a simple core data model, with books and reading sessions. The books have covers (images) that are stored as binary data with "Allows External Storage".
On iOS 11.4 and below, everything works fine all the time. When I save a new session everything gets updated properly.
Problem
Since iOS 12, when I create a new reading session and link it to the book, about every second time, core data generates a SQL statement that also updates the book cover field, sometimes resulting in a bad reference (to file on disk) which often results in the cover being nil when restarting the app, and almost always creates duplicate copy of the cover on disk (as can be seen in Simulator's _EXTERNAL_DATA folder).
In-memory context and objects remain correct though (and everything in the UI is therefore OK), until the app is restarted, then the cover is often nil.
iOS 12 specific
On iOS 12, I can deterministically reproduce the error in the simulator, on physical devices, and users have reported the error as well. I cannot reproduce the error on iOS 11.4, and no users reported the error previous to iOS 12.
Steps taken
I've enabled "-com.apple.CoreData.ConcurrencyDebug 1", so it shouldn't be that I'm accessing anything from the wrong queue. I've also enabled "-com.apple.CoreData.SQLDebug 3" so that I can see exactly what gets written.
I've made sure the Book instance (and therefore the cover) is not modified by my code before the association with the new Session by checking hasChanges, just before I do newSession.book = book and context.save().
To be 100% sure I'm not touching the cover property on any thread I've short-circuited my getters and setters for that property. No improvement.
I've tried using objectID to request an instance of the book just before the association and save. No improvement.
I've even tried the option where the context keeps strong references to all objects, just to make sure it was not some kind of memory management issue. No improvement.
Question
Any ideas for next steps?
Status update
This is a defect in iOS 12. See accepted answer below for a detailed description of a resonable workaround.
Update: The underlying Core Data issue appears to be resolved in iOS 12.1 (verified in beta 4). We will keep the workaround described below in our app, and won't be recommending using the External Storage option any time soon.
After talking to Apple engineers and filing the Radar mentioned above, we couldn’t wait around for a fix, so we took the hit and switched to storing files on the filesystem and managing it directly ourselves.
Another alternative that we considered was migrating our model not to allow External Storage for BLOBs, but I don't know what impact that would have had on performance and I was also worried about a model migration at a time when this part of iOS seems to be unstable, especially after reading stories like this in the past: Core Data: don’t store large files as binary data – Alexander Edge – Medium
It wasn't too much of a pain to implement local storage ourselves. You just need to have a unique identifier for each record that you can use to create a filename so you can map files to records. We added an extension to our Managed Object subclass with methods for reading, writing and deleting the files. Now, instead of calling e.g. article.photo = image.pngData(), we now need to call something like article.savePhoto(image.pngData()) and then we do similar when we want to retrieve the image. You can also add some code to these methods to support backwards compatibility with any images that are currently stored in Core Data.
Deletion was a little more tricky because our objects are deleted from multiple places in the code, including cascading deletes. In the end I opted to do it in the managed object's prepareForDeletion method but it is not ideal. There is plenty of discussion of how best to implement this here: cocoa - How to handle cleanup of external data when deleting unsaved Core Data objects? - Stack Overflow
Finally, to prevent our app crashing when a non-Optional binary attribute has disappeared because of this bug, I override awakeFromFetch in my Managed Object subclass to ensure that any required attributes are not nil, and if they are, I set them to a placeholder image so that they can be saved without the validation failing.
I'm in the middle of adding an "offline mode" feature to an app I'm currently working on. Basically the idea is that users should able to make changes to the data, for example, edit the description of an item, without being connected to the internet, and the changes should survive between app launches.
Each change would normally result in an API request when working online but situation is different in offline mode.
Right now this is implemented by storing all data coming from the API in a Core Data database that acts as a cache. Entities that can be edited by user in addition to normal attributes have the following ones:
locallyCreated - whether the object was created offline
locallyDeleted - object was deleted offline
locallyUpdated - updated
This makes it possible to look for new/deleted/updated objects and send corresponding API requests when doing sync.
This worked well for creating and deleting objects, however, one disadvantage I found with this approach is when new data is retrieved from the API all local changes (i.e. attributes of objects marked as locally updated) are lost, which means that they have to be stored separately somehow.
What would be the best way to approach this problem?
Since you have your locallyUpdated key, the obvious answer is to modify your code that imports server changes, so that it doesn't overwrite changes to any object marked as changed. One way or another you need to avoid overwriting those changes, and you're already keeping a record of which objects have changes, so you already have the tools for a basic solution.
But you'll soon run into the complexity of syncing data. What if the local object has changes on one key, but the incoming data from the server has changes on a different key? You can't resolve that just by knowing that the local copy has changed somehow. Maybe you decide that the server always wins, or that the local copy always wins. Those are easy, if they make sense for your app. If you need to merge changes though, you have some work ahead of you. You would need to record not only a Boolean value indicating that changes were made, but also a list of which keys had changed. This can get complicated, but it's the nature of data syncing.
I've done a lot of changes to my core data model. In the past we used the simple automatic migration. However this will fail this time. Since I really don't care about the data being migrated I just want to delete the persistent store if auto migration fails and set it up again. Is this a valid way to go? Any thing I have to be careful ? Could this get my app rejected ?
There are some definite problems with doing that, and you need to be careful with it. This answer had some good advice from the NSManagedObjectContext's documentation
A context always has a “parent” persistent store coordinator which
provides the model and dispatches requests to the various persistent
stores containing the data. Without a coordinator, a context is not
fully functional. The context’s coordinator provides the managed
object model and handles persistency. All objects fetched from an
external store are registered in a context together with a global
identifier (an instance of NSManagedObjectID) that’s used to uniquely
identify each object to the external store.
When faced with a similar situation in one of our apps, I opted to make a new persistent store, and deprecate the old one because our old store had been messed up on many of our devices by a previous bad migration. It ended up being a messier transition than I had hoped, but it did work.
The problems with your plan are not insurmountable, I'm just recommending caution. I liked Giao's advice of using NSManagedObjectContext's reset. When deleting and rebuilding, the persistent store coordinator could get confused. I worry because Apple seems to be doing so many things behind the scenes. I also worry because It seems like core data behaves differently on released apps than it does on our debug versions, especially in the upgrade process.
I think you are smart in recognizing that your automigrate is going to have trouble, and that you are looking for another path. In the recent past I've seen a group that really had to scramble for a month to deal with a failed data migration in their app.
when dealing with CoreData, I've run into a few problems I'm trying to nip in the bud for future proofing the system out of the gate. The simple fact of the matter is that I've never done anything like this before (work with CoreData that is). While I've managed to figure out how to work with it in the app, I need to know a decent practice to signal an app between versions that default data needs to be refresh on first app launch.
So right now, in my AppDelegate, I setup my managed object context, and I perform a fetch request to see if there are any records at all in a particular table/entity. I only want this to happen on first launch so im not constantly rewriting the contents of the DB every app launch. Anyways, so it goes ahead and uses Object Models to handle inserting of data amongst the entities in question (theres a few)
Now, for this version of the app, it's going into the store without an API (thats a far future thing), but between versions released to the app store, we may have to update specific information within the entities (for example: prices), again I only want this refresh to happen on app launch. Also, the schema MIGHT change, Im not sure if or when, but I'd like to make sure this can accomodate that just in case.
I figured, versioning the coredata "Add Model Version" would do the trick, set the new db version as the active version, but when I launch the app in the simulator, nothing happens which tells me that the data inside is being retained.
Any help towards what it is that I should do to accomodate this would be appreciated. Thank you!
You should find the Core Data Model Versioning and Data Migration guide useful:
https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreDataVersioning/Articles/Introduction.html
You'll also probably find Method for import initial data with coredata useful.
I have a weird problem with the automatic lightweight migration in iOS updating from an older version of the app I'm working on, to the next.
In iOS5 and 6, I don't have a problem.
In iOS4 specifically (I don't support older than that), after the migration, querying for one type of object returns object of an incorrect type. As in, if I did a fetch for Client objects, I got back an array of DataRecords instead. This doesn't happen for all objects but it's ... kind of a fatal problem.
This seems to be triggered by exceeding 50 entity types in the core data. I took my old data model and made a new version deriving from it (and set it to the new default) and all I did was add 4 more entity definitions to it. And in iOS4 the problem occurs.
Any ideas? I can't find anything about such a limit in the Apple docs.
Thanks!