I have a rather big CoreData Model which suddenly behaves weirdly. There are three relevant entities:
User
Story
Group
User has a to-many relationship named stories to Story objects and a to-many-relationship groups to Group objects. Both also have an appropriate inverse. I use mogenerator to generate convenience accessors for all the properties and relationships. Both are ordered to-many relationships.
Now it sometimes happens that when I ask the user object for its groups (like user.groups) that I actually get an NSOrdered set of XNGStory objects. In the few cases that I have observed this they were the same objects that are returned by user.stories. The two sets weren't the same (different pointers), but their contents were (I checked by calling array] valueForKeyPath:#"objectID.URIRepresentation"] on them and got the same URIs in the same order both cases).
It feels like there is some way in my app that stores stories in the groups relationship but I checked the code and there are only Groups-related classes touching groups relationship and only story-related classes touching the story-relationship. Also there is no interaction or relationship between the two they are used in completely separate parts of the app, so I have no idea how this kind of data corruption can happen.
Is there any reason why such a data corruption might happen? I would have expected CoreData to complain when I store objects in the wrong kind of relationship accidentally.
One more thing: So far we were only able to reproduce it on iOS 10.
Things we tried:
We suspected a memory issue first (because an object of the wrong type turns up where it's not supposed to be), but running with Address Sanitizer and NSZombies enabled didn't reveal any issues. Also, once the issue occurs it persists over app restarts and using the debugger to poke around in the objects also shows a consistent(ly wrong) data model so it's not just one pointer that's wrong.
Deactivated WAL (issue still exists)
Looked at sqlite file (both with and without WAL) after crash with Core Data Editor: It displays Group-objects for the groups relationship, so all seems to be good on the sqlite-database-level -> Somehow the relationships get mixed up when loaded from the file.
As best I can tell, this seems to be an iOS 10 bug.
We have a similar problem with an app that uses Core Data - a User object has a relationship Addresses (to Address objects) and Friends (to other User objects).
For some reason under iOS 10 Core Data occasionally chooses to return the Address objects in the Friends relationship.
We've regression tested back to iOS 9 and this issue doesn't occur.
There's an open bug on Open Radar for the same issue - 26826183. Interestingly we've only started seeing this issue under 10.0.2 but that Radar report was based on an early iOS 10 Beta.
We have tested this extensively and also conclude that it is an iOS 10 bug. It also does not seem to be related to any of our code, just the data model. We were able to recreate the issue in an example project using our data model, Xcode 8's code generation and iOS 10's new NSPersistentContainer (before it broke using mogenerator and the "old way of setting up a CoreData stack").
To reproduce the issue we do the following:
At startup, if there is no User-object, create a User-object and add 10 Group objects to it and save.
Then add a Story to it, still on the main-context, and save.
Kill the app (via iOS swipe-up or simply via pressing "stop" in Xcode)
Start the app again. On startup, fetch the User object, add a Story object to to the stories relationship and save.
Check the user.groups relationship on a background or main context. It contains XNGStory objects now, although it should still have XNGGroup objects.
I filed a Radar with the example project. OpenRadar is here (without the example project, as it contains our data model).
The issue seems to be highly dependent on the data-model setup. It occurs when all-but-on relationships on our User entity are ordered. If all are ordered, it doesn't occurs, neither does it occur if two or more are unordered.
Related
I'm using NSPersistentCloudKitContainer to synchronise data between different devices with CloudKit. It works perfectly well with a new project, however when I'm using it with old projects the old data which was added with NSPersistentContainer does not synchronise.
What I would like to achieve is to synchronise old data that was added with NSPersistentContainer after changing it to NSPersistentCloudKitContainer. Is it possible?
I've found a solution that works for my Core Data database - and mine is quite complex with multiple many-to-many relationships (A surgery/anaesthesia logbook app called Somnus)
I started by creating a new attribute for all my Core Data Entities called sentToCloud and setting it to FALSE by default in the Core Data model.
On the first load for an existing user:
Fetch request using the predicate "sentToCloud == FALSE" for each Entity type
Change sentToCloud to TRUE for each Object then save the MOC
This triggers NSPersistentCloudKitContainer to start syncing
I've done this in order of 'priority' that works for my database, assuming the iCloud sync sessions match the order in which Core Data is modified. In my testing this seems to be the case:
I first sync all child (or most child-like) Entities
Then sync their parents, and so on, up the tree
I sync the Object the user interacts with last, once everything else is in place so the relationships are intact and they don't think their data is borked while we wait for NSPersistentCloudKitContainer to reconnect all the relationships
I also leave any binary data (magically turned into a CKAsset behind-the-scenes) to last as it's not the most important part of my database
My database synced successfully from my iPad to iPhone and all relationships and binary data appear correct.
Now all I need is a way to tell the user when data is syncing (and/or some sort of progress) and for them to turn it off entirely.
ADDENDUM
So I tried this again, after resetting all data on the iCloud dashboard and deleting the apps on my iPhone & iPad.
Second time around it only synced some of the data. It seems like it still has a problem dealing with large sync requests (lots of .limitExceeded CKErrors in the console).
What's frustrating is that it's not clear if it's breaking up the requests to try again or not - I don't think it is. I've left it overnight and still no further syncing, only more .limitExceeded CKErrors.
Maybe this is why they don't want to sync existing data?
Personally, I think this is silly. Sometimes users will do a batch process on their data which would involve updating many thousands of Core Data objects in one action. If this is just going to get stuck with .limitExceeded CKErrors, NSPersistentCloudKitContainer isn't going to be a very good sync solution.
They need a better way of dealing with these errors (breaking up the requests into smaller requests), plus the ability to see what's going on (and perhaps present some UI to the user).
I really need this to work because as it stands, there is no way to synchronise many-to-many Core Data relationships using CloudKit.
I just hope that they're still working on this Class and improving it.
I've spent the better part of a workday trying to solve this.
Background
I have a simple core data model, with books and reading sessions. The books have covers (images) that are stored as binary data with "Allows External Storage".
On iOS 11.4 and below, everything works fine all the time. When I save a new session everything gets updated properly.
Problem
Since iOS 12, when I create a new reading session and link it to the book, about every second time, core data generates a SQL statement that also updates the book cover field, sometimes resulting in a bad reference (to file on disk) which often results in the cover being nil when restarting the app, and almost always creates duplicate copy of the cover on disk (as can be seen in Simulator's _EXTERNAL_DATA folder).
In-memory context and objects remain correct though (and everything in the UI is therefore OK), until the app is restarted, then the cover is often nil.
iOS 12 specific
On iOS 12, I can deterministically reproduce the error in the simulator, on physical devices, and users have reported the error as well. I cannot reproduce the error on iOS 11.4, and no users reported the error previous to iOS 12.
Steps taken
I've enabled "-com.apple.CoreData.ConcurrencyDebug 1", so it shouldn't be that I'm accessing anything from the wrong queue. I've also enabled "-com.apple.CoreData.SQLDebug 3" so that I can see exactly what gets written.
I've made sure the Book instance (and therefore the cover) is not modified by my code before the association with the new Session by checking hasChanges, just before I do newSession.book = book and context.save().
To be 100% sure I'm not touching the cover property on any thread I've short-circuited my getters and setters for that property. No improvement.
I've tried using objectID to request an instance of the book just before the association and save. No improvement.
I've even tried the option where the context keeps strong references to all objects, just to make sure it was not some kind of memory management issue. No improvement.
Question
Any ideas for next steps?
Status update
This is a defect in iOS 12. See accepted answer below for a detailed description of a resonable workaround.
Update: The underlying Core Data issue appears to be resolved in iOS 12.1 (verified in beta 4). We will keep the workaround described below in our app, and won't be recommending using the External Storage option any time soon.
After talking to Apple engineers and filing the Radar mentioned above, we couldn’t wait around for a fix, so we took the hit and switched to storing files on the filesystem and managing it directly ourselves.
Another alternative that we considered was migrating our model not to allow External Storage for BLOBs, but I don't know what impact that would have had on performance and I was also worried about a model migration at a time when this part of iOS seems to be unstable, especially after reading stories like this in the past: Core Data: don’t store large files as binary data – Alexander Edge – Medium
It wasn't too much of a pain to implement local storage ourselves. You just need to have a unique identifier for each record that you can use to create a filename so you can map files to records. We added an extension to our Managed Object subclass with methods for reading, writing and deleting the files. Now, instead of calling e.g. article.photo = image.pngData(), we now need to call something like article.savePhoto(image.pngData()) and then we do similar when we want to retrieve the image. You can also add some code to these methods to support backwards compatibility with any images that are currently stored in Core Data.
Deletion was a little more tricky because our objects are deleted from multiple places in the code, including cascading deletes. In the end I opted to do it in the managed object's prepareForDeletion method but it is not ideal. There is plenty of discussion of how best to implement this here: cocoa - How to handle cleanup of external data when deleting unsaved Core Data objects? - Stack Overflow
Finally, to prevent our app crashing when a non-Optional binary attribute has disappeared because of this bug, I override awakeFromFetch in my Managed Object subclass to ensure that any required attributes are not nil, and if they are, I set them to a placeholder image so that they can be saved without the validation failing.
I am currently having an issue with Simperium on iOS with CoreData. Upon launching the app for the first time, Simperium tries to sync with CoreData.
Sometimes it will work 100% correctly, and other times it will set some of the relationships to "nil" despite the Simperium data on the server NOT saying that. This is then NEVER fixed unless I re-install the entire application. And then I roll the dice again to see if the relationships are linked properly on startup.
I cannot find a pattern in this. The relationships that are nil are completely random. Sometimes this entity over here will have nil relationships, and then the next time I try it a different entity will have nil relationships.
All relationships are optional and there is nothing wrong with my CoreData file.
Has anybody had an issue like this? I found the exact same issue in a bug from 2014, but it's been forever since then.
Exact same issue I am having can be found here: https://github.com/Simperium/simperium-ios/issues/250
Side Note: If you read that issue, he also clarifies that he has a problem with editing the CD file WHILE Simperium is syncing, causing nil relationships. Has anybody confirmed this happening? If so this might be my problem.
It's highly likely that this glitch is caused by the scenario you've described (editing the CD file while Simperium is Sync'ing).
Core Data deals, internally, with locks to maintain data integrity. Accessing the sql storage directly might result in data corruption (i'm assuming you're editing the file via either a Firefox plugin, in the simulator, or accessing the file via a Filesystem API).
Please, try to reproduce the issue without accessing the Core Data's sqlite file directly (always go thru NSPersistentStoreCoordinator / NSManagedObjectContext).
If you do succeed, and there's effectively a bug, we'd love to get it fixed! (In which case, please, open an issue the main repository, including as many details as possible).
Thanks for your interest in Simperium!
When my iOS app is started for the first time, it initializes a few entities with default data. The same process is completed for every device of the same user when the app is first installed on the device. This leads to a problem with Simperium, because, even if I have a constraint on one of the entity's attributes, it create duplicates. How can avoid that from happening? Is there a way to make Simperium skip objects locally initialized ? Or to prevent it from inserting duplicates?
Would setting the simperiumKey of the entity to a custom constant value which is the same for all the devices of the user fix the issue? I mean, would that prevent Simperium from importing the very same entity from different devices that would produce duplicates and sync conflicts ?
Because I tried that option and it looks like the buckets on Simperium Server are ok, but when I dispose the view and I open it again the entities are not there anymore. When I restart the app they are back again... it looks like they desappear to reappear after resetting the app.. very strange.
UPDATE
The problem is that I get duplicated entities if I try to insert the same entity from different device. For example. When I setup my app for the first time, I have a function that initializes CURRENCY entities with codes and other things. The same operation would be done from a different device if the user decides to install the app on a new one. In this case, because the new device will initialize again the CURRENCY entities, those info will generate duplicates and conflict errors. I need a way to make Simperium understand that the entities locally inizialized in the devices must not be duplicated. I would remove the inheritance from SPManagedObject in order to stop Simperium from syncing the entities, but in their turn they have relations with other entities and that would definitively create problems with Simperium, because it will try to sync entities which have relations with objects not inheriting from SPManagedObject. Hope you now have a more clear idea.
After reading various posts on this subject I've understood that not signing out (that means: not calling signOutAndRemoveLocalData) and setting custom simperiumKey-s would prevent data duplication. I tested this solution and it apparently looks good. My app needs Simperium to sync data with other devices of the same user, but it mainly works with the local CoreData database. I hope this solution is good for this scenario.
I have an object (A) that has a To-Many relationship to another object (B).
Also, B holds an inverse relationship to A.
When I delete B, It still shown on A's relation ship count, unless i manually clean the inverse relationship of B before deletion.
I want it to happen synchronously, so i could update a UITableView and delete B's row,
instead of waiting for MOC's save action to complete.
Is there any way to handle that without manually cleaning B's inverse?
(I have tons of these relationships and it would be bad practice & hard to maintain)
Thanks!
That should work automatically if you set the "Delete Rule" for the inverse relationship from B to A to "Nullify" in the Core Data Model inspector in Xcode.
See Relationship Delete Rules in the "Core Data Programming Guide" for more information.
Almost 5 years later (at iOS 10 now) and I stumbled upon the same issue.
App started crashing after I decided to 'optimise' things by removing the saveContext() from almost everywhere, thinking that the in-memory representation is guaranteed to be correct (since includesPendingChanges is true by default).
However I was getting the issue described (and because later some UITableView needed to be updated, app was crashing).
Here are four separate approaches, any one of which solved the issue in my case:
(ordered from best to worst as far as I can judge)
calling processPendingChanges() (on the NSManagedObjectContext) after the deletion
calling refreshAllObjects() (on the NSManagedObjectContext) after the deletion
calling saveContext() after the deletion (as others have pointed out in the comments)
just waiting. for example, delaying the execution of the code that depends on the correct data by 0.1 seconds or so (using DispatchQueue.main.asyncAfter). (this of course is by far the worst approach and you should not implement it)
Things I am still not sure about:
I tried replicating it on a separate very small example project and I did not get the issue. So what is the underlying problem?
Is this expected behaviour or a bug? Is it documented anywhere?