Crashes during CoreData fetching on serial queue - ios

I went through many discutions and subjects about CoreData, but I keep getting the same problem.
Here's the context : I have an application which have to do several access to CoreData. I decided, for simplifying, to declare a serial thread specifically for access (queue.sync for fetching, queue.async for saving). I have a structure that is nested three times, and for recreating the entire structure, I fetch subSubObject, then SubObject and finally Object
But sometimes (like 1/5000 recreation of "Object") CoreData crash on fetching results, with no stack trace, with no crash log, only a
EXC_BAD_ACCESS (code 1)
Objects are not in cause, and the crash is weird because all access are done in the same thread which is a serial thread
If anyone can help me, I will be very grateful !
Here's the structure of the code :
private let delegate:AppDelegate
private let context:NSManagedObjectContext
private let queue:DispatchQueue
override init() {
self.delegate = (UIApplication.shared.delegate as! AppDelegate)
self.context = self.delegate.persistentContainer.viewContext
self.queue = DispatchQueue(label: "aLabel", qos: DispatchQoS.utility)
super.init()
}
(...)
public func loadObject(withID ID: Int)->Object? {
var object:Object? = nil
self.queue.sync {
let fetchRequest = NSFetchRequest<NSManagedObject>(entityName: "Name")
fetchRequest.predicate = NSPredicate(format: "id == %#", NSNumber(value: ID))
do {
var data:[NSManagedObject]
// CRASH HERE ########################
try data = context.fetch(fetchRequest)
// ###################################
if (data.first != nil) {
let subObjects:[Object] = loadSubObjects(forID: ID)
// Task creating "object"
}
} catch let error as NSError {
print("CoreData : \(error), \(error.userInfo)")
}
}
return object
}
private func loadSubObjects(forID ID: Int)->[Object] {
var objects:[Object] = nil
self.queue.sync {
let fetchRequest = NSFetchRequest<NSManagedObject>(entityName: "Name")
fetchRequest.predicate = NSPredicate(format: "id == %#", NSNumber(value: ID))
do {
var data:[NSManagedObject]
// OR HERE ###########################
try data = context.fetch(fetchRequest)
// ###################################
if (data.first != nil) {
let subSubObjects:[Object] = loadSubObjects(forID: ID)
// Task creating "objects"
}
} catch let error as NSError {
print("CoreData : \(error), \(error.userInfo)")
}
}
return objects
}
(etc...)

TL;DR: get rid of your queue, replace it with an operation queue. Run fetches on the main thread with viewContext do writing in one synchronous way.
There are two issues. First is that managedObjectContexts are not thread safe. You cannot access a context (neither for reading or for writing) except from the single thread that it is setup to work with. The second issue is that you shouldn't be doing multiple writing to core-data at the same time. Simultaneous writes can lead to conflicts and loss of data.
The crash is cause by accessing viewContext from a thread that is not the main thread. The fact that there is a queue ensuring that nothing else is accessing core data at the same time doesn't fix it. When core-data thread safety is violated core-data can fail at any time and in any way. That means that it may crash with hard to diagnose crash reports even at points in the code where you are on the correct thread.
You have the right idea that core-data needs a queue to work well when saving data, but your implementation is flawed. A queue for core-data will prevent write conflicts caused by writing conflicting properties to an entity at the same time from different context. Using NSPersistentContainer this is easy to set up.
In your core-data manager create a NSOperationQueue
let persistentContainerQueue : OperationQueue = {
let queue = OperationQueue.init();
queue.maxConcurrentOperationCount = 1;
return queue;
}()
And do all writing using this queue:
func enqueueCoreDataBlock(_ block: #escaping (NSManagedObjectContext) -> Swift.Void){
persistentContainerQueue.addOperation {
let context = self.persistentContainer.newBackgroundContext();
context.performAndWait {
block(context)
do{
try context.save();
} catch{
//log error
}
}
}
}
For writing use enqueueCoreDataBlock: which will give you a context to use and will execute every block inside the queue so you don't get write conflicts. Make sure that no managedObject leave this block - they are attached to the context which will be destroyed at the end of the block. Also you can't pass managedObjects into this block - if you want to change a viewContext object you have to use the objectID and fetch in the background context. In order for the changes to be seen on the viewContext you have to add to your core-data setup persistentContainer.viewContext.automaticallyMergesChangesFromParent = true
For reading you should use the viewContext from the main thread. As you are generally reading in order to display information to the user you aren't gaining anything by having a different thread. The main thread would have to wait for the information in any event so it is faster just the run the fetch on the main thread. Never write on the viewContext. The viewContext does not use the operation queue so writing on it can create write conflicts. Likewise you should treat any other contexts that you create (with newBackgroundContext or with performBackgroundTask) as readonly because they will also be outside of the writing queue.
At first I thought that NSPersistentContainer's performBackgroundTask had an internal queue, and initial testing supported that. After more testing I saw that it could also lead to merge conflicts.

Related

How can I make multiple calls of NSBatchUpdateRequest within DB transaction so that either all rows is updated or none is updated?

Is there a way, to make multiple NSBatchUpdateRequest calls executed within a DB transaction, so that either all DB rows is updated or none is updated (When exception thrown)?
The following code illustrate the problem.
func debug() {
let coreDataStack = CoreDataStack.INSTANCE
let backgroundContext = coreDataStack.backgroundContext
backgroundContext.perform {
let fetchRequest = NSTabInfo.fetchSortedRequest()
do {
var objectIDs: [NSManagedObjectID] = []
let nsTabInfos = try fetchRequest.execute()
//
// QUESTION: We are updating multiple rows of data directly in a persistent store.
// How can we ensure either all rows is updated, or none row is updated is exception
// happens in between?
//
for nsTabInfo in nsTabInfos {
let batchUpdateRequest = NSBatchUpdateRequest(entityName: "NSTabInfo")
batchUpdateRequest.predicate = NSPredicate(format: "self == %#", nsTabInfo.objectID)
batchUpdateRequest.propertiesToUpdate = ["name": nsTabInfo.name! + "XXX"]
batchUpdateRequest.resultType = .updatedObjectIDsResultType
let batchUpdateResult = try backgroundContext.execute(batchUpdateRequest) as? NSBatchUpdateResult
guard let batchUpdateResultX = batchUpdateResult else { return }
guard let managedObjectIDs = batchUpdateResultX.result else { return }
if let nsManagedObjectIDs = managedObjectIDs as? [NSManagedObjectID] {
objectIDs.append(contentsOf: nsManagedObjectIDs)
}
//
// Simulate some exception
// We notice the first row is updated & rest of the rows are unchanged.
// This leaves our data in inconsistent state.
//
throw "Custom error!!!"
}
if !objectIDs.isEmpty {
let changes = [NSUpdatedObjectsKey : objectIDs]
coreDataStack.mergeChanges(changes)
}
} catch {
backgroundContext.rollback()
error_log(error)
}
}
}
class CoreDataStack {
static let INSTANCE = CoreDataStack()
private init() {
}
private(set) lazy var persistentContainer: NSPersistentContainer = {
let container = NSPersistentContainer(name: "wenote")
container.loadPersistentStores(completionHandler: { (storeDescription, error) in
if let error = error as NSError? {
fatalError("Unresolved error \(error), \(error.userInfo)")
}
})
// So that when backgroundContext write to persistent store, container.viewContext will retrieve update from
// persistent store.
container.viewContext.automaticallyMergesChangesFromParent = true
return container
}()
private(set) lazy var backgroundContext: NSManagedObjectContext = {
let backgroundContext = persistentContainer.newBackgroundContext()
backgroundContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
return backgroundContext
}()
func mergeChanges(_ changes: [AnyHashable : Any]) {
NSManagedObjectContext.mergeChanges(
fromRemoteContextSave: changes,
into: [persistentContainer.viewContext, backgroundContext]
)
}
}
We write a demo code to illustrate the following
Performing NSBatchUpdateRequest multiple times within a loop.
An exception happens in between.
We wishes none of the row in persistent store is updated. However, a row is already updated before the exception thrown.
May I know what technique I can use, which is similar to SQLite transaction feature, so that either all rows is updated, or none of the row is updated when exception happens?
CoreData.framework doesn't open up SQLite level controls to the user, it provides you NSManagedObjectContext.
How does it work in a similar manner?
You pull as many objects in many as you need and do your changes on them.
When you are done with your changes, you do context.save().
In that way, you save all of your changes in one shot.
In all cases, pulling all objects in memory might not be possible or a good idea, so then you need to implement your own solution around how to send all of these changes to disk.
From the NSBatchUpdateRequest docs -
A request to Core Data to do a batch update of data in a persistent store without loading any data into memory.
When you execute this, you are doing the changes in store that you can't roll back. For a large data-set, you can do following -
Say you have to perform a series of updates (5 different steps) on 100k records as an operation.
Start in a background thread, pull objects in memory in batches of 1k at a time.
You can load 1k objects easily in memory, mutate them - go through all of your changes/steps one by one and save these changes on this batch. If this is successful, you move on to the next batch.
In case one intermediate step fails on a batch, you can then use either NSManagedObjectContext.rollback() or NSManagedObjectContext.reset() depending on your implementation.
Here's a popular SO post on the differences between the two in case official docs don't provide enough clarity.

Core Data Creates A New Object When Updating An Old One

To start with, I don't believe this is a duplicate of: Updating object value in core data is also creating a new object (It's in Obj-C and they were calling insertNewObject every time they segued.)
Background Info: I learned how to use CoreData from the Ray Wenderlich book and referred to it when writing this code. I rolled my own custom stack as outlined in Chapter 3 if you have the book. I can show the code for this if needed.
Queue is the Entity I'm trying to update.
It has 1 property: name - String
And 1 to-many relationship: tasks: Task
My CoreData logic is in a Struct which contains the managedContext.
I have a basic find/create function to create a Queue object. This works. It creates 1 and only 1 object.
func findOrCreateMainQueue() -> Queue? {
let queue = Queue(context: managedContext)
queue.name = "Queue32"
let queueFetch: NSFetchRequest<Queue> = Queue.fetchRequest()
queueFetch.predicate = NSPredicate(format: "%K == %#", #keyPath(Queue.name), "Queue32" as CVarArg)
do {
let results = try managedContext.fetch(queueFetch)
print(results.count)
if results.count > 0 {
return results.first!
} else {
try managedContext.save()
}
} catch let error as NSError {
print("Fetch error: \(error) description: \(error.userInfo)")
}
return nil
}
(As you can see by the queue.name suffix number I have tried a lot of different things.)
I have tried just about everything I can think of:
This code is basically copy/pasted from: How do you update a CoreData entry that has already been saved in Swift?
func addTaskToMainQueue2(task: Task) {
let queueFetch: NSFetchRequest<Queue> = Queue.fetchRequest()
queueFetch.predicate = NSPredicate(format: "%K == %#", #keyPath(Queue.name), "Queue32" as CVarArg)
do {
let results = try managedContext.fetch(queueFetch)
print(results.count)
if results.count > 0 {
var tasks = results[0].tasks?.mutableCopy() as? NSMutableOrderedSet
tasks?.add(task)
results[0].setValue(tasks, forKey: "tasks")
}
} catch let error as NSError {
print("Fetch error: \(error) description: \(error.userInfo)")
}
do {
try managedContext.save()
} catch let error as NSError {
print("Save error: \(error),description: \(error.localizedDescription)")
}
}
Which causes a second Queue object to be created with the "Queue32" name.
Here is another thing I tried:
func addTaskToMainQueue(task: Task) {
if var queue = findOrCreateMainQueue() {
var tasks = queue.tasks?.mutableCopy() as? NSMutableOrderedSet
tasks?.add(task)
queue.tasks = tasks
do {
try managedContext.save()
} catch let error as NSError {
print("Save error: \(error),description: \(error.localizedDescription)")
}
}
}
For the sake of space I won't add code for other things I've tried.
I've tried using the find/create function and updating in that method.
I've tried saving the queue as a local object and passing it to the addTask function which causes duplication as well.
It also doesn't matter if I pass in the Task or create one in the addTask function.
I am starting to believe my issue is something in my dataModel file causing this as I've tried a number of 'How to update a Core Data object' tutorials online and I get the same result each time.
awakeFromInsert() is called whenever I try to update an object. Not sure if this should be happening.
In other places in my app updating works. For example, if I add a Subtask to a Task. It works fine. However, if I want to change the name of another entity called Project the object duplicates. (Project has an id attribute which is fetched, then the name attribute is changed.)
Thank you in advance. I've been pulling my hair out for hours.
I admit not having read all of your code but if you create a new managed object like this
let queue = Queue(context: managedContext)
then it will be added to the managedContext and will be saved to disk at some point. So this code
if results.count > 0 {
return results.first!
} else {
try managedContext.save()
}
is irrelevant in regard to the queue object created earlier because it will be saved even if results.count is > 0, although at a later point. So this means you will have to delete queue when the fetch is successful which feels unnecessary, better to wait with creating it
if results.count > 0 {
return results.first!
} else {
let queue = Queue(context: managedContext)
queue.name = "Queue32"
try managedContext.save()
}
Off topic but I see you return nil if a new object was created rather than fetched, is this intended?

Core Data and Concurrency

I am using the performBackgroundTask function to pull data from firebase, compare it with data already stored in Core Data, save new data to Core Data, and call a completion handler when done.
I understand that Core Data is not thread safe but I am trying to do this concurrently.
static func cache(completion: #escaping (Void) -> Void) {
CoreDataHelper.persistentContainer.performBackgroundTask { (context) in
let dispatchGroup = DispatchGroup()
// fetch previously saved Core Data from main thread (1) and filter them (2)
let newsSourceIDs = NewsSourceService.getSaved().filter{$0.isEnabled}.map{$0.id!}
let oldArticleURLs = ArticleService.getSaved().map{$0.url!}
// create firebase database reference
let ref = Database.database().reference()
Constants.Settings.timeOptions.forEach { time in
let timeRef = ref.child("time\(time)minutes")
newsSourceIDs.forEach { newsSourceID in
dispatchGroup.enter()
// pull from Firebase Database
timeRef.child(newsSourceID).observeSingleEvent(of: .value, with: { (snapshot) in
guard let newsSourceDict = snapshot.value as? [String: [String:String]] else {
return
}
newsSourceDict.values.forEach { articleDict in
dispatchGroup.enter()
if oldArticleURLs.contains(articleDict["url"]!) {
dispatchGroup.leave()
return
}
// create article entity with firebase data
let article = Article(context: context)
article.date = articleDict["date"]
article.source = newsSourceID
article.time = Int16(time)
article.title = articleDict["title"]
article.url = articleDict["url"]
article.urlToImage = articleDict["urlToImage"]
dispatchGroup.leave()
}
dispatchGroup.leave()
})
}
}
// when done, save and call completion handler (3)
dispatchGroup.notify(queue: .main) {
do {
try context.save()
completion()
} catch {
fatalError("Failure to save context: \(error)")
}
}
}
}
Fetch from Core Data function:
static func getSaved() -> [Article] {
let fetchRequest: NSFetchRequest<Article> = Article.fetchRequest()
do {
let results = try CoreDataHelper.managedContext.fetch(fetchRequest)
return results
} catch let error as NSError {
print("Could not fetch \(error)")
}
return []
}
Can I fetch Core Data from the main thread during performBackgroundTask?
Should I filter with the high level filter function or using a special batch request (can I do that concurrently?)
How can I use dispatchGroup.notify(queue:) to determine when the creation and saving of Core Data is complete?
Can I fetch Core Data from the main thread during performBackgroundTask?
You can fetch from any thread if you use that method. You can't use the results on the main thread though. NSPersistentContainer provides the viewContext property for use on the main thread.
Should I filter with the high level filter function or using a special batch request (can I do that concurrently?)
I'd do it with a predicate on a normal non-batch request. Either of the ways you mention are possible. It depends on what kind of fetching and filtering you need. A batch request might be good if the fetch and filter takes a long time to run. Filtering the results after the fetch might be good if your filtering rules can't be expressed in a predicate.
How can I use dispatchGroup.notify(queue:) to determine when the creation and saving of Core Data is complete?
Add the notify call after your forEach closure. If you never enter, it'll execute immediately. If you do enter, it will execute when you match each enter with a leave.
One other detail: Your getSaved method should take a managed object context as an argument, and fetch with that context. Otherwise you're mixing contexts here. The performBackgroundTask creates one context, but you're using a different one in getSaved.
Another way to handle concurrency with Core Data(one of the easiest, but not optimized) would be using a "child" managedObjectContext with a concurrencyType of private, setting that new MOC's parent to be the MOC on your main thread.
let privateMOC = NSManagedObjectContext(concurrencyType: .privateQueueConcurrencyType)
privateMOC.parent = persistentManager.managedObjectContext
privateMOC.perform {
do {
try privateMOC.save()
} catch let error as NSError {
}
}
You would perform all of your needed core data actions inside of the .perform closure. When you run privateMOC.save() the changes are pushed up to the parent managedObjectContext on the main thread.

NSPersistentContainer concurrency for saving to core data

I've read some blogs on this but I'm still confused on how to use NSPersistentContainer performBackgroundTask to create an entity and save it. After creating an instance by calling convenience method init(context moc: NSManagedObjectContext) in performBackgroundTask() { (moc) in } block if I check container.viewContext.hasChanges this returns false and says there's nothing to save, if I call save on moc (background MOC created for this block) I get errors like this:
fatal error: Failure to save context: Error Domain=NSCocoaErrorDomain Code=133020 "Could not merge changes." UserInfo={conflictList=(
"NSMergeConflict (0x17466c500) for NSManagedObject (0x1702cd3c0) with objectID '0xd000000000100000 <x-coredata://3EE6E11B-1901-47B5-9931-3C95D6513974/Currency/p4>' with oldVersion = 1 and newVersion = 2 and old cached row = {id = 2; ... }fatal error: Failure to save context: Error Domain=NSCocoaErrorDomain Code=133020 "Could not merge changes." UserInfo={conflictList=(
"NSMergeConflict (0x170664b80) for NSManagedObject (0x1742cb980) with objectID '0xd000000000100000 <x-coredata://3EE6E11B-1901-47B5-9931-3C95D6513974/Currency/p4>' with oldVersion = 1 and newVersion = 2 and old cached row = {id = 2; ...} and new database row = {id = 2; ...}"
)}
So I've failed to get the concurrency working and would really appreciate if someone could explain to me the correct way of using this feature on core data in iOS 10
TL:DR: Your problem is that you are writing using both the viewContext and with background contexts. You should only write to core-data in one synchronous way.
Full explanation: If an object is changed at the same time from two different contexts core-data doesn't know what to do. You can set a mergePolicy to set which change should win, but that really isn't a good solution, because you will lose data that way. The way that a lot of pros have been dealing with the problem for a long time was to have an operation queue to queue the writes so there is only one write going on at a time, and have another context on the main thread only for reads. This way you never get any merge conflicts. (see https://vimeo.com/89370886#t=4223s for a great explanation on this setup).
Making this setup with NSPersistentContainer is very easy. In your core-data manager create a NSOperationQueue
//obj-c
_persistentContainerQueue = [[NSOperationQueue alloc] init];
_persistentContainerQueue.maxConcurrentOperationCount = 1;
//swift
let persistentContainerQueue = OperationQueue()
persistentContainerQueue.maxConcurrentOperationCount = 1
And do all writing using this queue:
// obj c
- (void)enqueueCoreDataBlock:(void (^)(NSManagedObjectContext* context))block{
void (^blockCopy)(NSManagedObjectContext*) = [block copy];
[self.persistentContainerQueue addOperation:[NSBlockOperation blockOperationWithBlock:^{
NSManagedObjectContext* context = self.persistentContainer.newBackgroundContext;
[context performBlockAndWait:^{
blockCopy(context);
[context save:NULL]; //Don't just pass NULL here, look at the error and log it to your analytics service
}];
}]];
}
//swift
func enqueue(block: #escaping (_ context: NSManagedObjectContext) -> Void) {
persistentContainerQueue.addOperation(){
let context: NSManagedObjectContext = self.persistentContainer.newBackgroundContext()
context.performAndWait{
block(context)
try? context.save() //Don't just use '?' here look at the error and log it to your analytics service
}
}
}
When you call enqueueCoreDataBlock the block is enqueued to ensures that there are no merge conflicts. But if you write to the viewContext that would defeat this setup. Likewise you should treat any other contexts that you create (with newBackgroundContext or with performBackgroundTask) as readonly because they will also be outside of the writing queue.
At first I thought that NSPersistentContainer's performBackgroundTask had an internal queue, and initial testing supported that. After more testing I saw that it could also lead to merge conflicts.

Managed Object Context as Singleton?

I am facing issues with my app where if i create or delete a new object, then save the object within a different entity object, then go back and try to make a new object of the first entity type, my app will crash.
I can then reopen then app and make the object that crashed the app with no issue.
This is all being done via core data, there is an exercise, exercises are saved as a routine, then creating a new exercise after having created a routine will crash the app. Furthermore, deleting an exercise and a routine then trying to create a new one straight after will also crash the app
I have spend a long time reading around this and believe the likely cause is managed object context and wondered if creating it as a singleton was the solution? I set up the MoC by running the below in each VC's viewdidload:
func getMainContext() -> NSManagedObjectContext {
let appDelegate = UIApplication.shared.delegate as! AppDelegate
return appDelegate.persistentContainer.viewContext
}
I then reference this VC level variable via .self wherever i need to reference the MoC to avoid clashes with creating further MoC within a VC.
I believed this should prevent issues as all core data work is linked to the shared MoC. However as documented above, there are still crashes occurring.
Below is a console print of the crash which hopefully will narrow down the source.
fatal error: Failure to save context: Error Domain=NSCocoaErrorDomain Code=134020 "(null)" UserInfo={NSAffectedObjectsErrorKey= (entity: UserExercise; id: 0x600000025060
The code block this is triggering off as an example as 1 location in the app it occurs is included below, to clarify this only occurs when i just deleted other objects, if i reloaded the app now this code would work and save just fine:
func createExercise() {
print("SAVE EXERCISE PRESSED")
if userExercise == nil {
print("SAVING THE NEW EXERCISE")
let newUserExercise = UserExercise(context: self.managedObjectContext!)
newUserExercise.name = userExerciseName.text
newUserExercise.sets = Int64(userSetsCount)
newUserExercise.reps = Int64(userRepsCount)
newUserExercise.dateCreated = NSDate()
newUserExercise.hasBeenTickedDone = false
} if self.associatedRoutineToAddTo != nil {
let fetchRequest: NSFetchRequest<UserRoutine> = UserRoutine.fetchRequest()
fetchRequest.predicate = NSPredicate(format: "name == %#", self.associatedRoutineToAddTo!)
do {
let existingUserRoutine = try self.managedObjectContext!.fetch(fetchRequest).first
print("RETRIVED ROUTINES ARRAY CONTAINING \(existingUserRoutine)")
existingUserRoutine?.addToUserexercises(newUserExercise)
print("EXERCISE SUCESSFULLY ADDED TO ROUTINE")
} catch {
print("Fetching Routine Failed")
}
} else if self.associatedRoutineToAddTo == nil {
print("THIS IS A FRESH EXERCISE WITHOUT A PARENT ROUTINE")
}
} else if let userExercise = userExercise {
print("UPDATING THE EXISTING EXERCISE")
userExercise.name = userExerciseName.text
userExercise.sets = Int64(userSetsCount)
userExercise.reps = Int64(userRepsCount)
}
do {
try self.managedObjectContext?.save()
print("THE EXERCISE HAS BEEN SAVED")
} catch {
fatalError("Failure to save context: \(error)")
}
The variable declarations are:
var managedObjectContext: NSManagedObjectContext!
var userExercise: UserExercise?
var associatedRoutineToAddTo : String?
var editingUserExerciseID: NSManagedObjectID?
var editingUserExercise: UserExercise?
I was receiving the "NSCocoaErrorDomain Code=134020 (null)" error because my new entity was not added to the proper CoreData Configuration.

Resources