NSBatchDeleteRequest causes Merge Conflict - ios

I have an application that will sync with a server with data that can change daily. During the sync, I remove all the data for some entities and reload it with new data. I am using the following code:
func SyncronizeUserComments(theData : [[AnyHashable : Any]])
{
// Delete User Comments for this User and Connection
let commentRequest : NSFetchRequest<NSFetchRequestResult> = PT_UserComments.fetchRequest()
commentRequest.predicate = NSPredicate(format: "connection = %# AND user == %#", Global_CurrentConnection!, Global_CurrentUser!)
coreData.processDeleteRequest(request: commentRequest)
// ADD the Comments to CoreData
for index in 0..<theData.count {
let result : [AnyHashable : Any] = theData[index]
if let commentID = result["Comment_ID"] as? String, let commentText = result["Comment_Text"] as? String, let commentTitle = result["Comment_Title"] as? String
{
let newUserComment = PT_UserComments(context: coreData.persistentContainer.viewContext)
newUserComment.connection = Global_CurrentConnection
newUserComment.user = Global_CurrentUser
newUserComment.comment_ID = commentID
newUserComment.comment_Text = commentText
newUserComment.comment_Title = commentTitle
}
}
// Add the User Comments
print("Added New User Comments: \(theData.count)")
coreData.saveContext()
}
func processDeleteRequest(request : NSFetchRequest<NSFetchRequestResult>)
{
let deleteRequest = NSBatchDeleteRequest(fetchRequest: request)
deleteRequest.resultType = .resultTypeObjectIDs
do {
let result = try coreData.persistentContainer.viewContext.execute(deleteRequest) as? NSBatchDeleteResult
let objectIDArray = result?.result as? [NSManagedObjectID]
let changes = [NSDeletedObjectsKey : objectIDArray]
NSManagedObjectContext.mergeChanges(fromRemoteContextSave: changes as Any as! [AnyHashable : Any], into: [coreData.persistentContainer.viewContext])
} catch {
fatalError("Fatal Error Deleting Data: \(error)")
}
coreData.saveContext()
}
When I call coreData.saveContext() I will get a Merge Conflict against the deleted data.
In reading about CoreData and the NSBatchDeleteRequest, this deletes at the SQL LITE level and bypasses the in memory cache.
The only way I have been able to get this to work is by setting:
context.mergePolicy = NSMergeByPropertyStoreTrumpMergePolicy
Is this correct, or am I doing something wrong? I am also setting this merge policy in my saveContext() in the Core Data Stack.

I just spent hours debugging the same issue, hopefully this can help someone.
The problem is that NSManagedObjectContext.mergeChanges(fromRemoteContextSave:, into:) updates the managed object context but does not update the row cache version number of the deleted objects relationships to match the updated version number (Z_OPT) in the database file, causing a mismatch at time of the save.
If you're using NSErrorMergePolicyType this will cause the next save to fail, (or even a later one when the relationships become flagged for save), even though everything but the version numbers match. I've not seen this mentioned in the related docs or WWDC video, but I guess Apple assumed people would always pick a non-default merge policy.
So picking NSMergeByPropertyStoreTrumpMergePolicy solves it, as mentioned in the question, but you might not want this policy for all your save operations. To avoid that I ended up writing a custom merge policy that only resolves version mismatches. The code is below (this is untested Swift as I originally wrote in Obj-C, but should be equivalent):
//Configure the merge as below before saving
context.mergePolicy = AllowVersionMismatchMergePolicy(merge: .errorMergePolicyType)
//...
//The custom merge policy
class AllowVersionMismatchMergePolicy: NSMergePolicy {
override func resolve(optimisticLockingConflicts list: [NSMergeConflict]) throws {
do {
//if the default resolve worked leave it alone
return try super.resolve(optimisticLockingConflicts: list)
} catch {
//if any of the conflict is not a simple version mismatch (all other keys being equal), fail
let hasValueConflict = list.contains { conflict -> Bool in
//compare object and row cache
if let objectSnapshot = conflict.objectSnapshot as NSObject?,
let cachedSnapshot = conflict.cachedSnapshot as NSObject? {
return !objectSnapshot.isEqual(cachedSnapshot)
}
//compare row cache and database
if let cachedSnapshot = conflict.cachedSnapshot as NSObject?,
let persistedSnapshot = conflict.persistedSnapshot as NSObject? {
return !cachedSnapshot.isEqual(persistedSnapshot)
}
//never happens, see NSMergePolicy.h
return true
}
if hasValueConflict {
throw error
}
//Use store rollback merge policy to resolve all the version mismatches
return try NSMergePolicy.rollback.resolve(optimisticLockingConflicts: list)
}
}
}

Related

How can I make multiple calls of NSBatchUpdateRequest within DB transaction so that either all rows is updated or none is updated?

Is there a way, to make multiple NSBatchUpdateRequest calls executed within a DB transaction, so that either all DB rows is updated or none is updated (When exception thrown)?
The following code illustrate the problem.
func debug() {
let coreDataStack = CoreDataStack.INSTANCE
let backgroundContext = coreDataStack.backgroundContext
backgroundContext.perform {
let fetchRequest = NSTabInfo.fetchSortedRequest()
do {
var objectIDs: [NSManagedObjectID] = []
let nsTabInfos = try fetchRequest.execute()
//
// QUESTION: We are updating multiple rows of data directly in a persistent store.
// How can we ensure either all rows is updated, or none row is updated is exception
// happens in between?
//
for nsTabInfo in nsTabInfos {
let batchUpdateRequest = NSBatchUpdateRequest(entityName: "NSTabInfo")
batchUpdateRequest.predicate = NSPredicate(format: "self == %#", nsTabInfo.objectID)
batchUpdateRequest.propertiesToUpdate = ["name": nsTabInfo.name! + "XXX"]
batchUpdateRequest.resultType = .updatedObjectIDsResultType
let batchUpdateResult = try backgroundContext.execute(batchUpdateRequest) as? NSBatchUpdateResult
guard let batchUpdateResultX = batchUpdateResult else { return }
guard let managedObjectIDs = batchUpdateResultX.result else { return }
if let nsManagedObjectIDs = managedObjectIDs as? [NSManagedObjectID] {
objectIDs.append(contentsOf: nsManagedObjectIDs)
}
//
// Simulate some exception
// We notice the first row is updated & rest of the rows are unchanged.
// This leaves our data in inconsistent state.
//
throw "Custom error!!!"
}
if !objectIDs.isEmpty {
let changes = [NSUpdatedObjectsKey : objectIDs]
coreDataStack.mergeChanges(changes)
}
} catch {
backgroundContext.rollback()
error_log(error)
}
}
}
class CoreDataStack {
static let INSTANCE = CoreDataStack()
private init() {
}
private(set) lazy var persistentContainer: NSPersistentContainer = {
let container = NSPersistentContainer(name: "wenote")
container.loadPersistentStores(completionHandler: { (storeDescription, error) in
if let error = error as NSError? {
fatalError("Unresolved error \(error), \(error.userInfo)")
}
})
// So that when backgroundContext write to persistent store, container.viewContext will retrieve update from
// persistent store.
container.viewContext.automaticallyMergesChangesFromParent = true
return container
}()
private(set) lazy var backgroundContext: NSManagedObjectContext = {
let backgroundContext = persistentContainer.newBackgroundContext()
backgroundContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
return backgroundContext
}()
func mergeChanges(_ changes: [AnyHashable : Any]) {
NSManagedObjectContext.mergeChanges(
fromRemoteContextSave: changes,
into: [persistentContainer.viewContext, backgroundContext]
)
}
}
We write a demo code to illustrate the following
Performing NSBatchUpdateRequest multiple times within a loop.
An exception happens in between.
We wishes none of the row in persistent store is updated. However, a row is already updated before the exception thrown.
May I know what technique I can use, which is similar to SQLite transaction feature, so that either all rows is updated, or none of the row is updated when exception happens?
CoreData.framework doesn't open up SQLite level controls to the user, it provides you NSManagedObjectContext.
How does it work in a similar manner?
You pull as many objects in many as you need and do your changes on them.
When you are done with your changes, you do context.save().
In that way, you save all of your changes in one shot.
In all cases, pulling all objects in memory might not be possible or a good idea, so then you need to implement your own solution around how to send all of these changes to disk.
From the NSBatchUpdateRequest docs -
A request to Core Data to do a batch update of data in a persistent store without loading any data into memory.
When you execute this, you are doing the changes in store that you can't roll back. For a large data-set, you can do following -
Say you have to perform a series of updates (5 different steps) on 100k records as an operation.
Start in a background thread, pull objects in memory in batches of 1k at a time.
You can load 1k objects easily in memory, mutate them - go through all of your changes/steps one by one and save these changes on this batch. If this is successful, you move on to the next batch.
In case one intermediate step fails on a batch, you can then use either NSManagedObjectContext.rollback() or NSManagedObjectContext.reset() depending on your implementation.
Here's a popular SO post on the differences between the two in case official docs don't provide enough clarity.

iOS: CoreData and Remote API synchronization pattern

I have such a problem.
/movies?page=0&size=20
api endpoint that returns paginated movies
Then I want to display this movies in UITableView. So it is easy to make pagination and load next pages of movies.
But now I want to add CoreData caching functionality and Repository pattern in between.
Something like this
MoviesRepository (to integrate both below dependencies)
MoviesDao (to access Core Data for movies)
MoviesService (to make remote api calls)
I consider to treat Core Data as single source of truth.
So MoviesRepository -> fetchMovies(page: 2, size: 10) should make something like this:
call MoviesDao to get [20,30) movies from CoreData
make request to MoviesService -> getMovies(page: 2, size: 10)
after getting response it should call something like MoviesDao -> syncMovies([remoteMovies])
then Core Data change should be detected and movies in table view should be updated (here i think about some RxSwift observable approach or other like Combine framework in feature, maybe som CoreData nofitications, or FetchResultsController?
But the most crucial thing here is this syncing CoreData with remote data, as meantime remote data could change and page 2 is not equal to the page 2 in core data and how to resolve this problem without refetching all the data.
Here is the code that i found in some tutorial but it replaces all the data not pages.
private func syncFilms(jsonDictionary: [[String: Any]], taskContext: NSManagedObjectContext) -> Bool {
var successfull = false
taskContext.performAndWait {
let matchingEpisodeRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "Film")
let episodeIds = jsonDictionary.map { $0["episode_id"] as? Int }.compactMap { $0 }
matchingEpisodeRequest.predicate = NSPredicate(format: "episodeId in %#", argumentArray: [episodeIds])
let batchDeleteRequest = NSBatchDeleteRequest(fetchRequest: matchingEpisodeRequest)
batchDeleteRequest.resultType = .resultTypeObjectIDs
// Execute the request to de batch delete and merge the changes to viewContext, which triggers the UI update
do {
let batchDeleteResult = try taskContext.execute(batchDeleteRequest) as? NSBatchDeleteResult
if let deletedObjectIDs = batchDeleteResult?.result as? [NSManagedObjectID] {
NSManagedObjectContext.mergeChanges(fromRemoteContextSave: [NSDeletedObjectsKey: deletedObjectIDs],
into: [self.persistentContainer.viewContext])
}
} catch {
print("Error: \(error)\nCould not batch delete existing records.")
return
}
// Create new records.
for filmDictionary in jsonDictionary {
guard let film = NSEntityDescription.insertNewObject(forEntityName: "Film", into: taskContext) as? Film else {
print("Error: Failed to create a new Film object!")
return
}
do {
try film.update(with: filmDictionary)
} catch {
print("Error: \(error)\nThe quake object will be deleted.")
taskContext.delete(film)
}
}
// Save all the changes just made and reset the taskContext to free the cache.
if taskContext.hasChanges {
do {
try taskContext.save()
} catch {
print("Error: \(error)\nCould not save Core Data context.")
}
taskContext.reset() // Reset the context to clean up the cache and low the memory footprint.
}
successfull = true
}
return successfull
}
}

Core Data Creates A New Object When Updating An Old One

To start with, I don't believe this is a duplicate of: Updating object value in core data is also creating a new object (It's in Obj-C and they were calling insertNewObject every time they segued.)
Background Info: I learned how to use CoreData from the Ray Wenderlich book and referred to it when writing this code. I rolled my own custom stack as outlined in Chapter 3 if you have the book. I can show the code for this if needed.
Queue is the Entity I'm trying to update.
It has 1 property: name - String
And 1 to-many relationship: tasks: Task
My CoreData logic is in a Struct which contains the managedContext.
I have a basic find/create function to create a Queue object. This works. It creates 1 and only 1 object.
func findOrCreateMainQueue() -> Queue? {
let queue = Queue(context: managedContext)
queue.name = "Queue32"
let queueFetch: NSFetchRequest<Queue> = Queue.fetchRequest()
queueFetch.predicate = NSPredicate(format: "%K == %#", #keyPath(Queue.name), "Queue32" as CVarArg)
do {
let results = try managedContext.fetch(queueFetch)
print(results.count)
if results.count > 0 {
return results.first!
} else {
try managedContext.save()
}
} catch let error as NSError {
print("Fetch error: \(error) description: \(error.userInfo)")
}
return nil
}
(As you can see by the queue.name suffix number I have tried a lot of different things.)
I have tried just about everything I can think of:
This code is basically copy/pasted from: How do you update a CoreData entry that has already been saved in Swift?
func addTaskToMainQueue2(task: Task) {
let queueFetch: NSFetchRequest<Queue> = Queue.fetchRequest()
queueFetch.predicate = NSPredicate(format: "%K == %#", #keyPath(Queue.name), "Queue32" as CVarArg)
do {
let results = try managedContext.fetch(queueFetch)
print(results.count)
if results.count > 0 {
var tasks = results[0].tasks?.mutableCopy() as? NSMutableOrderedSet
tasks?.add(task)
results[0].setValue(tasks, forKey: "tasks")
}
} catch let error as NSError {
print("Fetch error: \(error) description: \(error.userInfo)")
}
do {
try managedContext.save()
} catch let error as NSError {
print("Save error: \(error),description: \(error.localizedDescription)")
}
}
Which causes a second Queue object to be created with the "Queue32" name.
Here is another thing I tried:
func addTaskToMainQueue(task: Task) {
if var queue = findOrCreateMainQueue() {
var tasks = queue.tasks?.mutableCopy() as? NSMutableOrderedSet
tasks?.add(task)
queue.tasks = tasks
do {
try managedContext.save()
} catch let error as NSError {
print("Save error: \(error),description: \(error.localizedDescription)")
}
}
}
For the sake of space I won't add code for other things I've tried.
I've tried using the find/create function and updating in that method.
I've tried saving the queue as a local object and passing it to the addTask function which causes duplication as well.
It also doesn't matter if I pass in the Task or create one in the addTask function.
I am starting to believe my issue is something in my dataModel file causing this as I've tried a number of 'How to update a Core Data object' tutorials online and I get the same result each time.
awakeFromInsert() is called whenever I try to update an object. Not sure if this should be happening.
In other places in my app updating works. For example, if I add a Subtask to a Task. It works fine. However, if I want to change the name of another entity called Project the object duplicates. (Project has an id attribute which is fetched, then the name attribute is changed.)
Thank you in advance. I've been pulling my hair out for hours.
I admit not having read all of your code but if you create a new managed object like this
let queue = Queue(context: managedContext)
then it will be added to the managedContext and will be saved to disk at some point. So this code
if results.count > 0 {
return results.first!
} else {
try managedContext.save()
}
is irrelevant in regard to the queue object created earlier because it will be saved even if results.count is > 0, although at a later point. So this means you will have to delete queue when the fetch is successful which feels unnecessary, better to wait with creating it
if results.count > 0 {
return results.first!
} else {
let queue = Queue(context: managedContext)
queue.name = "Queue32"
try managedContext.save()
}
Off topic but I see you return nil if a new object was created rather than fetched, is this intended?

How to set the type of NSBatchDeleteResult of a NSBatchDeleteRequest?

I have some core that performs a Batch delete request:
extension NSManagedObject: Clearable {
/// Clears all objects of this type in coreData
static func clearAll() {
let context = AppDelegate.sharedInstance()?.coreDataHelper.objectContext()
let fetchRequest = NSFetchRequest<NSFetchRequestResult>(entityName: String(describing:self))
let batchDeleteRequest = NSBatchDeleteRequest(fetchRequest: fetchRequest)
do {
if let unwrappedContext = context {
unwrappedContext.shouldDeleteInaccessibleFaults = true
let result = try unwrappedContext.execute(batchDeleteRequest) as? NSBatchDeleteResult
DLog("result \(result.debugDescription)")
switch result!.resultType {
case .resultTypeCount:
DLog("resultTypeCount")
case .resultTypeObjectIDs:
DLog("resultTypeObjectIDs")
case .resultTypeStatusOnly:
DLog("resultTypeStatusOnly")
}
if let objectIDArray = result?.result as? [NSManagedObjectID] {
let changes = [NSDeletedObjectsKey : objectIDArray]
NSManagedObjectContext.mergeChanges(fromRemoteContextSave: changes, into: [unwrappedContext])
}
try context?.save()
}
} catch let error as NSError {
DLog("Error removing : \(error), \(error.localizedDescription)")
}
}
}
The code works fine, but the result of the batch delete is always .resultTypeStatusOnly
The documentation here https://developer.apple.com/library/content/featuredarticles/CoreData_Batch_Guide/BatchDeletes/BatchDeletes.html#//apple_ref/doc/uid/TP40016086-CH3-SW2 says under the title Updating Your Application After Execution that
it is important that you notify the application that the objects in
memory are stale and need to be refreshed.
And to do that the result type has to be .resultTypeObjectIDs, to be able to trigger the NSManagedObjectContext.mergeChanges(fromRemoteContextSave: changes, into: [unwrappedContext]) part. It's just not clear how you get that.
How does one set the type of the result?
You can set the resultType property on the batchDeleteRequest. So, also from https://developer.apple.com/library/content/featuredarticles/CoreData_Batch_Guide/BatchDeletes/BatchDeletes.html :
When the executeRequest completes successfully, a response is received. That response can take one of two forms. The form of the response is deteremined by setting the resultType property on the NSBatchDeleteRequest. The default value is NSStatusOnlyResultType, which returns nothing.
The other option is NSBatchDeleteObjectIDsResultType, which returns an array of NSManagedObjectID instances indicating which entities were deleted during the execution.
Hope this helps. Good luck!

aws dynamodb how to use object mapper with batch get in ios swift

Thanks in advance for any help. I am trying to get Batch items (Load multiple) items from one DynamoDb table using the AWS iOS SDK (Swift). I can load one item using the Block syntax, but I need to load 10 or more than that. I don't want to use 10 Block calls to load them individually. I tried to follow the attach stackoverflow Link (where the similar solution is given) but I am getting the following compiler error message. I come from Java background, hence could also be a syntax issue. Is it the right way to load multiple items? I don't want to use low level API. Any help, where I am going wrong. Thanks.
aws dynamodb how to use object mapper with batch get in ios
let dynamoDBObjectMapper = AWSDynamoDBObjectMapper.default()
var tasksList = Array<AWSTask<AnyObject>>()
for i in 1...10 {
tasksList.append(dynamoDBObjectMapper.load(AWSCards.self, hashKey: "SH_"+String(i), rangeKey: nil))
}
AWSTask.init(forCompletionOfAllTasksWithResults: tasksList).continueWithBlock { (task) -> AnyObject? in
if let cards = task.result as? [AWSCards] {
print(cards.count)
}
else if let error = task.error {
print(error.localizedDescription)
}
return nil
}
Have a try with the following codes (Swift 4.1, Feb 9th, 2018):
let dynamoDBObjectMapper = AWSDynamoDBObjectMapper.default()
var tasksList = Array<AWSTask<AnyObject>>()
for i in 1...10 {
tasksList.append(dynamoDBObjectMapper.load(AWSCards.self, hashKey: "SH_"+String(i), rangeKey: nil))
}
AWSTask<AnyObject>.init(forCompletionOfAllTasksWithResults: tasksList).continueWith { (task) -> Any? in
if let cards = task.result as? [AWSCards] {
print(cards.count)
}
else if let error = task.error {
print(error.localizedDescription)
}
return nil
}
Your question is "how to use the object mapper" but it might be more efficient for you to not use it.
However, there is a way to use it. See Niklas's answer here and here (he copy & pasted), but something about it strikes me as fishy. I want to make the assertion that it is not as fast as the built-in batch-get function, but I am unsure. I suspect that this does not complete the items in parallel, or at least not as efficiently as in BatchGetItem.
See the docs: "In order to minimize response latency, BatchGetItem retrieves items in parallel."
According to Yosuke, "Currently, AWSDynamoDBObjectMapper does not support the batch get item. You need to load one item at a time if you want to use the object mapper" as of 2016. This still seems to be the case. I am using a version a couple versions behind, but not too far behind. Someone check.
In conclusion, if you are loading one item at a time, you are likely missing out on the whole purpose of BatchGetItem (low latency).
Pulling from various sources, including John Davis's question here, I have tested and ran this BatchGetItem result. Here ya go.
import AWSDynamoDB
let primaryKeyToSortKeyDict : [String : String] = .... // Your stuff
var keys = [Any]()
for key in primaryKeyToSortKeyDict.keys {
let partitionKeyValue = AWSDynamoDBAttributeValue()
partitionKeyValue?.s = String(key)
let sortValue = AWSDynamoDBAttributeValue()
sortValue?.s = String(primaryKeyToSortKeyDict[key]!)
keys.append(["partitionKeyAttributeName": partitionKeyValue, "sortKeyAttributeName": sortValue])
}
let keysAndAttributesMap = AWSDynamoDBKeysAndAttributes()
keysAndAttributesMap?.keys = keys as? [[String : AWSDynamoDBAttributeValue]]
keysAndAttributesMap?.consistentRead = true
let tableMap = [table : keysAndAttributesMap]
let request = AWSDynamoDBBatchGetItemInput()
request?.requestItems = tableMap as? [String : AWSDynamoDBKeysAndAttributes]
request?.returnConsumedCapacity = AWSDynamoDBReturnConsumedCapacity.total
guard request != nil else {
print("Handle some error")
return
}
AWSDynamoDB.default().batchGetItem(request!) { (output, error) in
print("Here is the batchgetitem output")
if error == nil {
// do output stuff
} else {
// handle error
}
}

Resources