I'm getting crash reports from a large percentage of my user base, and have tracked it down to a core data entity having every record deleted somehow.
This entity is one where the app downloads every row the first time you launch the app, and then the table is never written to again, it's just a read only lookup table.
It shouldn't be possible for anything to ever be removed from the entity, but that's what I'm seeing. All reports are the app functions fine for a while (sometimes days, other times minutes after a fresh install) and then suddenly starts crashing any time you try to do anything.
I've gone over the entire code base and nothing ever deletes data from the entity. The bug only occurs in a version of the app just released. Rolling all users back to the previous release - only two weeks old, has fixed the bug for for everyone. It's an enterprise app, I simply redeployed the old binary.
I went over every change in source control and found nothing related to the entity.
In fact all code related to this entity has not changed at all in the last few years. It's been perfectly stable all that time.
Other data is persisted to disk in the same -[NSManagedObjectContext save:] call, and that data does not vanish.
I'm stumped. How can data in an entity be erased? How can I debug this issue further?
I am unable to reproduce the bug on my own devices, although I have extensive logging in place (and am happy to add more — this is an enterprise app only ever used for work so user privacy isn't an issue).
Related
I have a custom database application using Access 2016 (MS365 account) with backend data on a file server and a local front end with many VBA-driven forms. All has been working sweetly for several years, until this morning when myself and the all other users started to receive "No Current Record" errors on certain forms (not all).
I assumed it was a rogue record or corrupt DB so I went back to the previous day's backup data but the same problem occurred. No front-end code changes have been done recently, and in any case different users are using slightly different versions of the FE system, yet all suffer the problem, so it's not specific to particular VBA.
I went back many weeks of full data backups - all throw the same error now, although had worked fine at the time of course.
I've repaired and compacted, uncompiled and so on, but it won't clear the error.
I have a strange hunch it might relate to an Office update, as I believe the problem was not present on a machine that happened to be disconnected from the internet, but this is only 80% certain.
Has anyone ever had problems arising after updates? I cannot think of anything that has changed in the data or the front end so I'm struggling to understand what else it can be.
There have been multiple reports now that this may be a bug caused by Patch Tuesday updates. Refer to https://www.devhut.net/latest-office-update-version-2204-source-of-new-bugs/
I've spent the better part of a workday trying to solve this.
Background
I have a simple core data model, with books and reading sessions. The books have covers (images) that are stored as binary data with "Allows External Storage".
On iOS 11.4 and below, everything works fine all the time. When I save a new session everything gets updated properly.
Problem
Since iOS 12, when I create a new reading session and link it to the book, about every second time, core data generates a SQL statement that also updates the book cover field, sometimes resulting in a bad reference (to file on disk) which often results in the cover being nil when restarting the app, and almost always creates duplicate copy of the cover on disk (as can be seen in Simulator's _EXTERNAL_DATA folder).
In-memory context and objects remain correct though (and everything in the UI is therefore OK), until the app is restarted, then the cover is often nil.
iOS 12 specific
On iOS 12, I can deterministically reproduce the error in the simulator, on physical devices, and users have reported the error as well. I cannot reproduce the error on iOS 11.4, and no users reported the error previous to iOS 12.
Steps taken
I've enabled "-com.apple.CoreData.ConcurrencyDebug 1", so it shouldn't be that I'm accessing anything from the wrong queue. I've also enabled "-com.apple.CoreData.SQLDebug 3" so that I can see exactly what gets written.
I've made sure the Book instance (and therefore the cover) is not modified by my code before the association with the new Session by checking hasChanges, just before I do newSession.book = book and context.save().
To be 100% sure I'm not touching the cover property on any thread I've short-circuited my getters and setters for that property. No improvement.
I've tried using objectID to request an instance of the book just before the association and save. No improvement.
I've even tried the option where the context keeps strong references to all objects, just to make sure it was not some kind of memory management issue. No improvement.
Question
Any ideas for next steps?
Status update
This is a defect in iOS 12. See accepted answer below for a detailed description of a resonable workaround.
Update: The underlying Core Data issue appears to be resolved in iOS 12.1 (verified in beta 4). We will keep the workaround described below in our app, and won't be recommending using the External Storage option any time soon.
After talking to Apple engineers and filing the Radar mentioned above, we couldn’t wait around for a fix, so we took the hit and switched to storing files on the filesystem and managing it directly ourselves.
Another alternative that we considered was migrating our model not to allow External Storage for BLOBs, but I don't know what impact that would have had on performance and I was also worried about a model migration at a time when this part of iOS seems to be unstable, especially after reading stories like this in the past: Core Data: don’t store large files as binary data – Alexander Edge – Medium
It wasn't too much of a pain to implement local storage ourselves. You just need to have a unique identifier for each record that you can use to create a filename so you can map files to records. We added an extension to our Managed Object subclass with methods for reading, writing and deleting the files. Now, instead of calling e.g. article.photo = image.pngData(), we now need to call something like article.savePhoto(image.pngData()) and then we do similar when we want to retrieve the image. You can also add some code to these methods to support backwards compatibility with any images that are currently stored in Core Data.
Deletion was a little more tricky because our objects are deleted from multiple places in the code, including cascading deletes. In the end I opted to do it in the managed object's prepareForDeletion method but it is not ideal. There is plenty of discussion of how best to implement this here: cocoa - How to handle cleanup of external data when deleting unsaved Core Data objects? - Stack Overflow
Finally, to prevent our app crashing when a non-Optional binary attribute has disappeared because of this bug, I override awakeFromFetch in my Managed Object subclass to ensure that any required attributes are not nil, and if they are, I set them to a placeholder image so that they can be saved without the validation failing.
I am currently having an issue with Simperium on iOS with CoreData. Upon launching the app for the first time, Simperium tries to sync with CoreData.
Sometimes it will work 100% correctly, and other times it will set some of the relationships to "nil" despite the Simperium data on the server NOT saying that. This is then NEVER fixed unless I re-install the entire application. And then I roll the dice again to see if the relationships are linked properly on startup.
I cannot find a pattern in this. The relationships that are nil are completely random. Sometimes this entity over here will have nil relationships, and then the next time I try it a different entity will have nil relationships.
All relationships are optional and there is nothing wrong with my CoreData file.
Has anybody had an issue like this? I found the exact same issue in a bug from 2014, but it's been forever since then.
Exact same issue I am having can be found here: https://github.com/Simperium/simperium-ios/issues/250
Side Note: If you read that issue, he also clarifies that he has a problem with editing the CD file WHILE Simperium is syncing, causing nil relationships. Has anybody confirmed this happening? If so this might be my problem.
It's highly likely that this glitch is caused by the scenario you've described (editing the CD file while Simperium is Sync'ing).
Core Data deals, internally, with locks to maintain data integrity. Accessing the sql storage directly might result in data corruption (i'm assuming you're editing the file via either a Firefox plugin, in the simulator, or accessing the file via a Filesystem API).
Please, try to reproduce the issue without accessing the Core Data's sqlite file directly (always go thru NSPersistentStoreCoordinator / NSManagedObjectContext).
If you do succeed, and there's effectively a bug, we'd love to get it fixed! (In which case, please, open an issue the main repository, including as many details as possible).
Thanks for your interest in Simperium!
I have seen successful core-data migrations of my 4Gb database on my iPad on application launch taking several minutes.
And now suddenly, some users report crashes after installing a new version and the app is kicked out with a: failed to launch in time error.
I just tested again by restoring an old database and I am sure that core data migration can take way more than 10 seconds.
But other people are concerned it should not and try to take it to the background, or at least out of the run loop at launch time:
iPhone app launch times and Core Data migration
Can this have anything to to with other conditions, e.g. being connected to a power source? Or have a battery level of more than 50 %?
Update: I reproduced a crash by just starting the app on the device (unplugged) instead of debugging.
Then I tried starting the app on the device with USB attached: Crash.
Then started the app via the debugger: No crash (and the migration took about 4 minutes.)
Extra info: I have only enterprise users (about 75 of them) and they all have a database of 4.5Gb. Some users have no problem upgrading and some have. The upgrades all take minutes if they succeed. The crashes always come after 20 seconds. (And they keep crashing if you try again on these devices).
I followed the advice to place the migration out of the run loop, but I am still wondering why the old method works on some devices and not on others. All users are on iOS 7.
This is a common launch problem. A Core Data migration can take any amount of time, 0 to N depending on the complexity of the model and the amount of data and the type of migration occurring.
Ideally you should not be creating your Core Data stack in the -applicationDidFinish... method and migration is one of the reasons.
My recommendation is to rework your launch so that you display something until the stack has initialized. This could be just your default image in a view. Then when the Core Data stack has initialized you can switch over to your full view controller stack.
I would also recommend taking this a bit further so that you can tell the user that a migration is in process and I would further put the migration on a background queue so that you can update the UI while the migration is happening.
Lastly, if you are doing a heavy migration, I would look into doing a lightweight migration instead. The lightweight migration is far faster as well as other benefits.
If you look at the crash log it will likely say that the app was killed because it took too long to startup. The watchdog process kills apps that take too long to startup - >20 seconds I think. This is because the core data migration process was run during app startup.
I'd recommend you manually run the migration in the background. The following new book on Core Data has code and explanation for how to do a background manual migration.
http://www.amazon.com/gp/aw/d/0321905768
It is not a rule that don't run migration on background thread, but its a suggestion because if you run on background thread and your app start running, it is not guaranteed that your core data stack will not touch.
You can take this migration out of the didFinishLaunching but make sure that stack is not touch. You can handle this by some check like place a viewController with message that app is updating which don't allow user to do anything, and at mean time you can perfrom background migration. When migration process finish you can simply dismiss that viewController take user to home viewController.
When your app is running on the iOS platform you can not gurantee every thing, like some time if native apps need more memory then memory will cut off from your app quota and can get some wired kills.
when dealing with CoreData, I've run into a few problems I'm trying to nip in the bud for future proofing the system out of the gate. The simple fact of the matter is that I've never done anything like this before (work with CoreData that is). While I've managed to figure out how to work with it in the app, I need to know a decent practice to signal an app between versions that default data needs to be refresh on first app launch.
So right now, in my AppDelegate, I setup my managed object context, and I perform a fetch request to see if there are any records at all in a particular table/entity. I only want this to happen on first launch so im not constantly rewriting the contents of the DB every app launch. Anyways, so it goes ahead and uses Object Models to handle inserting of data amongst the entities in question (theres a few)
Now, for this version of the app, it's going into the store without an API (thats a far future thing), but between versions released to the app store, we may have to update specific information within the entities (for example: prices), again I only want this refresh to happen on app launch. Also, the schema MIGHT change, Im not sure if or when, but I'd like to make sure this can accomodate that just in case.
I figured, versioning the coredata "Add Model Version" would do the trick, set the new db version as the active version, but when I launch the app in the simulator, nothing happens which tells me that the data inside is being retained.
Any help towards what it is that I should do to accomodate this would be appreciated. Thank you!
You should find the Core Data Model Versioning and Data Migration guide useful:
https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreDataVersioning/Articles/Introduction.html
You'll also probably find Method for import initial data with coredata useful.