Core Data sometimes loses data - ios

We are running an App through Citrix Secure Hub, it seems that sometimes there is a rollback with loosing some Data in CoreData.
As i understand, CoreData is having something like an working copy of all the objects, and sometimes its tries to persist that on the filesystem.
Well tried to simulate the behavior but without any success, we could not find out any data loss or rollbacked data in our test environment.
So is there a way to force iOS to write the current "working copy" on the disk to prevent any data loss when using too much memory (and maybe crash)? We call our save function after
As we already found out:
We were NOT using:
func applicationWillResignActive(_ application: UIApplication) {
print("applicationWillResignActive")
}
to save the context, could this be a problem (we are already saving the context after every created object) ?
At the Moment we dont really handle problems when the context could not be saved, are there any recommendations how to handle that in a productive environment? And is it a good thing to maybe crash to app to prevent the user from struggeling with data loss?
Edit: this is the used Core Data Handler:
import Foundation
import CoreData
let context = CoreDataManager.shared.managedObjectContext
func saveContext(_ completion: (() -> Void)? = nil) {
CoreDataManager.shared.save(completion)
}
func saveContextSync() {
CoreDataManager.shared.saveSync()
}
class CoreDataManager: NSObject {
static let shared = CoreDataManager()
lazy var managedObjectContext: NSManagedObjectContext = {
var managedObjectContext = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
managedObjectContext.persistentStoreCoordinator = persistentStoreCoordinator
return managedObjectContext
}()
And our save functionality:
#objc func save(_ completion: (() -> Void)?) {
saveAsync(completion)
}
func saveAsync(_ completion: (() -> Void)?) {
func save() {
context.perform {
do { try context.save() }
catch {
// HERE WE NEED TO HANDLE IT FOR A PRODUCTIVE ENVIRONMENT
}
completion?()
}
}
if Thread.isMainThread {
save()
} else {
DispatchQueue.main.async {
save()
}
}
}
func saveSync() {
func save() {
context.performAndWait {
do { try context.save() }
catch { print(error)
// TRY TO REPRODUCE MEMORY LOSS APP TO SEE WHAT HAPPENS
abort()
}
}
}
if Thread.isMainThread {
save()
} else {
DispatchQueue.main.sync {
save()
}
}
}
Edit 2: This question in Objective C should be very similar:
Core Data reverts to previous state without apparent reason
Edit 3: It seems that there is no crash, some users telling me that they are adding data, then just press the home button and after a couple of hours the data from the last "task" is lost.

There are three possible causes.
Write Conflicts
Core data generally wants writes to be done in a single synchronous way. If you write in multiple ways at the same time to the same object (even if they are touching different properties and don't strictly conflict), it will be a merge conflict. You can set a merge policy (by default the value is 'error' - meaning don't apply the changes) but that is really a bad solution because you are tell core-data to lose information silently. see NSPersistentContainer concurrency for saving to core data for a setup to prevent merge conflicts.
Closing the app with unsaved data
If you setup your core-data correctly this shouldn't happen. The correct way to setup core data to only read from the 'viewContext' and write in a single synchronous way. Each writing is done in a single atomic block and the UI is only updated after it is saved. If you are displaying information from a context that is not saved to disk this can be a problem. For example it appears that your app only uses a single main-thread context for both reading and writing. Making changes to that context and not calling save will leave the app in a state where there are major changes that are only in memory and not on disk.
There is an error saving to disk
This is by far the rarest event, but there are users that have really really full disks. If this happens there is NOTHING that you can do to save. There is physically no room left on the disk. Generally the correct thing to do is to tell the user and leave it at that.
Without knowing more about your particular setup it hard to say for certain what your problem is. I would recommend the following setup:
use NSPersistentContainer
only read from the viewContext and never write to it.
make an operation queue for writing to core data
for every operation, create a context, make changes to it, and then save. Do not pass any managed object into or out of these blocks.
This should deal with all but the third problem.

Related

Apple Core Data saving duplicates of entities when initializing entities

I am currently building an iOS app that utilizes Core Data to cache Firebase queries, so that my app won't be hitting the Firebase server multiple times for the same query.
My main issue right now is with fetching what will be labeled as IdeaEntity, where the initial fetch will result in the published array of IdeaEntity storing duplicates of the IdeaEntity elements. I will post images below, along with the code snippet for how I'm fetching IdeaEntity arrays from Core Data. However, when re-opening the app (when you close it down via app preview), the duplicates seem to disappear.
iOS Simulator 16.2; Showing proof of duplicate IdeaEntity
// Fetch either from Firebase or Core Data
func fetchIdeas() {
let request = NSFetchRequest<IdeaEntity>(entityName: "IdeaEntity")
do {
self.savedIdeas = try appContainer.viewContext.fetch(request)
if self.savedIdeas.isEmpty {
self.initialFetchIdeas()
}
} catch {
print("ERROR - \(error)")
}
}
// Initial Function to fetch IdeaEntity
private func initialFetchIdeas() {
do {
let uid = self.firebaseService.fetchUserID()
guard !uid.isEmpty else { throw AltyrErrors.cannotFetchUid }
self.firebaseService.fetchUser(withUid: uid) { [weak self] user in
guard let self = self else { return }
self.firebaseService.fetchIdeas(user: user) { ideas in
guard !(ideas.isEmpty) else { return }
ideas.forEach { idea in
let newIdea = IdeaEntity(context: self.appContainer.viewContext)
// ... Insert code for inserting parameters for `newIdea`
if let feedback = idea.feedback {
let newFeedback = FeedbackEntity(context: self.appContainer.viewContext)
// ... Insert code for inserting parameters for `newIdea` Feedback object
newIdea.feedback = newFeedback
}
}
self.save()
}
}
} catch {
// TODO: Error handling...
print("ERROR - \(error)")
}
}
Couple of assumptions:
FirebaseService functions are assumed to be 100% functional, as in they output the needed items in a proper manner.
The above functions are in a class named CoreDataService, which is a class that inherits from ObservableObject
savedProjects is a Published array of IdeaEntity
If there is a better manner of fetching with another library, a different method for fetching IdeaEntity, need any other code snippets / signatures, or anything else, please do let me know.
I had tried switching around fetchIdea with initialFetchIdeas within the private init (since CoreDataService is a singleton) to see if fetchIdea was missing anything that initialFetchIdeas had, though the result is the same (e.g., duplicate entities).
Edit: TL;DR of the solution, just use NSCache if you have really complex objects. It's not worth the effort to utilize Core Data.

CoreData multithreading fetch requires small delay before accessing attributes

I use coreData as persistent store.
To read data, I use (only essential parts are shown):
func fetchShoppingItems(completion: #escaping (Set<ShoppingItem>?, Error?) -> Void) {
persistentContainer.performBackgroundTask { (backgroundManagedContext) in
let fetchRequest: NSFetchRequest<CDShoppingItem> = CDShoppingItem.fetchRequest()
var shoppingItems: Set<ShoppingItem> = []
do {
let cdShoppingItems: [CDShoppingItem] = try backgroundManagedContext.fetch(fetchRequest)
for nextCdShoppingItem in cdShoppingItems {
nextCdShoppingItem.managedObjectContext!.performAndWait {
Thread.sleep(forTimeInterval: 0.1) // This seems to be required
let nextShoppingItem = ShoppingItem.init(name: nextCdShoppingItem.name!)
shoppingItems.insert(nextShoppingItem)
} // performAndWait
} // for all cdShoppingItems
completion(shoppingItems, nil)
return
} catch let error as NSError {
completion(nil, error)
return
} // fetch error
} // performBackgroundTask
} // fetchShoppingItems
To test the coreData implementation, I wrote a unit test that creates multiple threads that write to and read from coreData concurrently.
This test runs only successfully, if the instruction
Thread.sleep(forTimeInterval: 0.1) // This seems to be required
is inserted in the performAndWait closure.
If it is commented out, nextCdShoppingItem is often read back with nil attributes, and the function crashes due to the forced unwrap.
I am not sure, if nextCdShoppingItem.managedObjectContext!.performAndWait is correct or if I had to use backgroundManagedContext.performAndWait, but with backgroundManagedContext the effect is the same.
I do not understand why inserting a small delay before accessing an attribute of a managed object is necessary to avoid the problem.
Any hint is welcome!
EDIT:
I investigated the issue further, and found the following:
Every time nextCdShoppingItem is read back by the background thread (called read thread below) as nil, there is also another background thread that tries to save its own managedContext after all records in its managedContext have been deleted (called write thread below).
Apparently the read thread tries to fetch a record that has just been deleted by the write thread.
So the problem is definitively a multithreading issue, and I found a solution (see my answer below).
performAndWait will add the block to the queue and schedule it to run, just like perform, but performAndWait will not return until the block is complete. Since you are inside a loop of cdShoppingItems, the loop does not stop and wait for the block to return. By adding the thread sleep, you are essentially slowing down the loop and giving core data enough time to complete its fetch. The forced unwrap crash is probably an indication that it's lost its nextCdShoppingItem reference.
I would consider refactoring where you do not need to query core data inside a loop. If it's possible, add the name attribute to CDShoppingItem so you don't have to fetch it to build a ShoppingItem object.
Edit: took a stab at a refactor although I don't know your exact use case:
func fetchShoppingItems(completion: #escaping (Set<ShoppingItem>?, Error?) -> Void) {
persistentContainer.performBackgroundTask { (backgroundManagedContext) in
let fetchRequest: NSFetchRequest<CDShoppingItem> = CDShoppingItem.fetchRequest()
do {
var shoppingItems: Set<ShoppingItem> = []
let cdShoppingItems: [CDShoppingItem] = try backgroundManagedContext.fetch(fetchRequest)
for nextCdShoppingItem in cdShoppingItems {
if let name = nextCdShoppingItem.name {
let nextShoppingItem = ShoppingItem.init(name: name)
shoppingItems.insert(nextShoppingItem)
}
}
completion(shoppingItems, nil)
} catch let error as NSError {
print("Error fetching CDShoppingItem: \(error)")
completion(nil, error)
} // fetch error
return
} // performBackgroundTask
} // fetchShoppingItems
To prevent the multithreading issue, I tried 2 things:
1) Since iOS10, a persistentStore of SQL type maintains a connection pool for concurrent access to the pool, and it is possible to set a maximum pool size, see the WWDC video. I did so using
private lazy var persistentContainer: NSPersistentContainer = {
let container = NSPersistentContainer(name: modelName)
container.loadPersistentStores(completionHandler: { (storeDescription, error) in
if let error = error as NSError? {
// ...
} else {
storeDescription.setOption(NSNumber(1), forKey: NSPersistentStoreConnectionPoolMaxSizeKey)
}
})
return container
}()
to prevent concurrent access to the persistentStore. Unfortunately, this did not solve the problem for an unknown reason.
2) I then tried to serialize read and write operations by setting up a serial queue:
private let coreDataSerialQueue = DispatchQueue(label: "com.xxx.ShopEasy.coreDataManager") // serial by default
It is used for the read and write accesses in the following way:
coreDataSerialQueue.async {
let backgroundManagedContext = self.persistentContainer.newBackgroundContext()
backgroundManagedContext.performAndWait {
// …
} // performAndWait
} // coreDataSerialQueue.async
This did solve the problem.
Please note that it would be wrong to use
coreDataSerialQueue.async {
self.persistentContainer.performBackgroundTask { (backgroundManagedContext) in
// …
} // performBackgroundTask
} // coreDataSerialQueue.async
because performBackgroundTask would fork another asynchronous thread and thus break the serialization.

Am I stuck resolving many ThreadSafeReferences to use Realm with asynchronous tasks like network requests?

I have a set of NSOperations which make network requests. Imagine a case where we have a User object in Realm and I want to make some changes and then send a PATCH request to the server:
let operation = UpdateUserOperation(updatedUser)
Then the operation runs on a different thread so it has to resolve a thread safe reference:
class UpdateUserOperation : Operation {
var userReference : ThreadSafeReference<User>
init(_ user: User) {
userReference = ThreadSafeReference(to: user)
}
func main() {
// We're probably on a different thread so resolve the reference
let user = try! Realm().resolve(userReference)
// That invalidated `userReference` so we need to create a new
// one to use below...
userReference = ThreadSafeReference(to: user)
sendUserPatchRequest(user) { response in
// We might be in _another_ thread again :(
let realm = try! Realm()
let user = realm.resolve(userReference)
try! realm.write {
user.updateFromResponse(response)
}
}
}
}
This feels like a really unclean way to do this – re-fetching the user so many times to do a pretty simple task. It feels especially onerous because we need to re-up the thread-safe reference – they aren't re-usable. In Core Data, we'd be able to choose a single NSManagedObjectContext to do our work in and ensure thread safety by using managedObjectContext.perform { /* ... */ }, but that sort of functionality is unavailable in Realm.
Am I missing anything? Is there a better way to do this, or am I stuck re-fetching the object each time I need to use it?

CoreData performBackgroundTask conflict

I am using CoreData and am having issues when I try to saveContext in the performBackgroundTask function of the container.
I am calling this from multiple places at the same time. My understanding is that each time the thread will be different leading to issues when trying to save to the persistentStore.
I thought of using childContext's and other approaches until I came across the following article:
https://blog.five.agency/how-to-import-a-large-data-set-using-core-data-6c248a503148
In the article the coreData stack is as follows:
final class DataCoordinator {
//MARK: - singleton
static let sharedInstance = DataCoordinator()
//MARK: - init
public var container : NSPersistentContainer
private init() {
container = NSPersistentContainer(name: "Model")
container.loadPersistentStores(completionHandler: { (_, error) in
if let error = error {
fatalError("Unresolved error \(error)")
}
})
}
//MARK: - perform methods
static func performBackgroundTask(_ block: #escaping (NSManagedObjectContext) -> Void) {
DataCoordinator.sharedInstance.container.performBackgroundTask(block)
}
static func performViewTask(_ block: #escaping (NSManagedObjectContext) -> Void) {
block(DataCoordinator.sharedInstance.container.viewContext)
}
}
My question is using this approach, will the static func performBackgroundTask overcome this problem and always be giving me the same background context on the same queue or is this no different than what I was previously doing with container.performBackgroundTask everytime?
Wondering how I can save from multiple places at the same time on the same queue?
Hi first thing is its not about queue, NSManagedObjectContext run on different queues, like viewContext run on main queue. So in your case I think you want to use same NSManagedObjectContext to save from different places ? If this is the case you can get one newBackgroundContext and save the reference of it and use this context to save your task from all the places. Use the below method of NSPersistentContainer to get a background context.
func newBackgroundContext() -> NSManagedObjectContext
If my understanding about your problem is wrong please let me know more so I can give you other solution.
Again the below method of the tutorial u mentioned will not work for you
static func performBackgroundTask(_ block: #escaping (NSManagedObjectContext) -> Void) {
DataCoordinator.sharedInstance().container.performBackgroundTask(block)
}
because performBackgroundTask always create a new context. Apple doc says:
Each time this method is invoked, the persistent container creates a new NSManagedObjectContext with the concurrencyType set to NSManagedObjectContextConcurrencyType.privateQueueConcurrencyType. The persistent container then executes the passed in block against that newly created context on the context’s private queue.

Core Data Entity Unique Constraint Does Not Work

I am trying to set a constraint in core data with the new Entity Constraints inspector (To make the name of the item unique). All that I've read says it's pretty simple - Set the constraint and handle the error. I don't get any errors and can add the same entry as many times as I want.
The app does require IOS 9.0, Xcode tools requirement is set to 7.0
The constraint, category1Name, is a String.
My addItem code is:
func addNewRecord() {
//check to be sure the entry is not empty
if (categoryTextField.text == "") {
//prompt requiring a name
let ac = UIAlertController(title: nil, message: "Name Required", preferredStyle: .Alert)
ac.addAction(UIAlertAction(title: "Ok", style: .Default, handler: nil))
self.presentViewController(ac, animated: true, completion: nil)
} else {
let newManagedObject = NSEntityDescription.insertNewObjectForEntityForName("Category1", inManagedObjectContext: kAppDelegate.managedObjectContext) as! Category1
newManagedObject.category1Name = categoryTextField.text
newManagedObject.category1Description = categoryTextView.text
//bunch more items...
//save it
kAppDelegate.saveContext()
makeEntryFieldsEnabledNO()
performSegueWithIdentifier("unwindToCategoriesTableViewController", sender: self)
}//if else
}//addNewRecord
The AppDelegate save is standard:
func saveContext () {
if managedObjectContext.hasChanges {
do {
try managedObjectContext.save()
} catch {
//insert your standard error alert stuff here
let nserror = error as NSError
print("From the print line: Unresolved error \(nserror), \(nserror.userInfo)")
abort()
}//do catch
}//if moc
}//saveContext
Here's the Core Data constraint:
This app is iCloud enabled.
The managedObjectContext merge policy is set to NSMergeByPropertyObjectTrumpMergePolicy
lazy var managedObjectContext: NSManagedObjectContext = {
// Returns the managed object context for the application (which is already bound to the persistent store coordinator for the application.) This property is optional since there are legitimate error conditions that could cause the creation of the context to fail.
let coordinator = self.persistentStoreCoordinator
var managedObjectContext = NSManagedObjectContext(concurrencyType: .MainQueueConcurrencyType)
managedObjectContext.persistentStoreCoordinator = coordinator
managedObjectContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
return managedObjectContext
}()//var managedObjectContext
Any guidance would be appreciated.
It would appear Apple have finally fixed the crazy Xcode problem where changes you make in a data model file don't actually change.
Putting that aside, the current formula seems to be:
in your core data singleton ...
container = NSPersistentContainer(name: _nom)
// during development, right HERE likely delete the sql database file
// and start fresh, as described here stackoverflow.com/a/60040554/294884
container.loadPersistentStores { storeDescription, error in
if let error = error {
print("\n ERROR LOADING STORES! \(error) \n")
}
else {
print("\n STORES LOADED! \(storeDescription) \n")
}
self.container.viewContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
self.container.viewContext.automaticallyMergesChangesFromParent = true
}
You must use merge policy and automatically merges.
Then in your data model file
Don't bother unless every relationship has an inverse, with
"to one or many" correctly set
and (almost certainly, except in very unusual source data) your unique id for each entity is indicated as a constraint
Then when you add new data, you must
use the new background context supplied by the handy core data function which does that
so, never try to make your own separate thread
double-check you have done (1) and (2) !
when you do add a few entities, you must do that inside a perform
and when you have finished adding entities (ie on the new thread) you must while still in the perform ...
do a performAndWait which does two things
save the new items (on the new child thread), and then
save the new items (on the main view thread)
naturally for both 7 and 8, you have to check .hasChanges before saving
Easy right?
So something like
let pm = core.container.newBackgroundContext()
pm.perform {
for onePerson in someNewData {
... create your new CDPerson entity ...
}
pm.bake()
}
Note that the bake routine is within the perform block,
and it looks like this:
func bake() {
self.performAndWait {
if self.hasChanges {
do {
try self.save()
}
catch {
print("bake disaster type 1 \(error)")
}
}
// OPTIONALLY, SEE BELOW
if core.container.viewContext.hasChanges {
do {
try core.container.viewContext.save()
}
catch {
print("bake disaster type 2 \(error)")
}
}
// OPTIONALLY, SEE BELOW
}
}
To be clear, notice the pm.bake ... in the function bake(), the self in the first half is indeed that newBackgroundContext which is created for the loop inside the perform.
Note that these days you don't even need to save to the main context
Nowadays automaticallyMergesChangesFromParent seems to work perfectly, if you "do everything in the long list above".
• In the bake above, add a couple print lines to see what is saved to the viewContext. You'll see that nothing, at all, is ever saved. It's all done properly by the child/whatever relationships in the engine
• So in fact, in reality you can just omit that passage of the code. All you have to do is
func bake() {
self.performAndWait {
if self.hasChanges {
do {
try self.save()
}
catch {
print("bake disaster type 1 \(error)")
}
}
}
If you want to get an error when there are merge conflicts and handle them manually then you need to change your policy to NSErrorMergePolicy and you will get an error and in the user info the object IDs that you need to solve merge conflict , otherwise it will merge and save according to the specified merge policy.
The policy that you set will overwrite the object attributes but not relationships, if you want to overwrite the attributes and relationships then specify the NSMergeByPropertyObjectTrumpMergePolicy.
The comment from pbasdf above seems to be correct. Constraints in the inspector don't save. I used both methods suggested in the link provided - I added the constraint, I changed another attribute, did a file save, then changed that attribute back and did a file save again. The constraint now acts as I expect. I'll mark this as answered. pbasdf should get the credit.

Resources