Suppose if i have thousand record coming from the web service API and if i want to add those record in core data in privateContext. How can i do it simultaneous and not linearly.
Can i apply something like dispatch_apply and run
let entity:NSEntityDescription = NSEntityDescription.entityForName(entityName,
inManagedObjectContext:managedContext)!
let managedObject:NSManagedObject = NSManagedObject(entity: entity,
insertIntoManagedObjectContext: managedContext)
simultaneously ?
Try this :
func persistAsyncInContext<T:AnyObject>(parentContext:NSManagedObjectContext , objetcsFromJSON:[T] , completion:()->Void){
let workQ = dispatch_queue_create(nil, DISPATCH_QUEUE_CONCURRENT)
let workGroup = dispatch_group_create()
for objectFromJSON in objetcsFromJSON{
dispatch_group_enter(workGroup)
let workerChildMoc = NSManagedObjectContext(concurrencyType: NSManagedObjectContextConcurrencyType.PrivateQueueConcurrencyType)
workerChildMoc.parentContext = parentContext
workerChildMoc.performBlock{
let managedObject = NSEntityDescription.insertNewObjectForEntityForName("entityName", inManagedObjectContext: workerChildMoc)
//if objectFromJSON is also an array then loop it and move the line above that creates new object into the loop.
do
{
try workerChildMoc.save()
}
catch
{
//Handle error
}
defer{
dispatch_group_leave(workGroup)
}
}
}
dispatch_group_notify(workGroup, dispatch_get_main_queue()) { () -> Void in
do{
try parentContext.save()
}
catch
{
//Handle error
}
//End of persist
completion()
}
}
This creates each object in a child context and saves them , BUT take into consider that doing this for a lot of objects is not efficient , You may want to create a 2D array of JSON objects (ex. if you have 1000 objects then chunk them to 10 arrays of 100 objects and parse the 10 arrays async on 10 child contexts).
Hope it helps.
Related
Is there a way, to make multiple NSBatchUpdateRequest calls executed within a DB transaction, so that either all DB rows is updated or none is updated (When exception thrown)?
The following code illustrate the problem.
func debug() {
let coreDataStack = CoreDataStack.INSTANCE
let backgroundContext = coreDataStack.backgroundContext
backgroundContext.perform {
let fetchRequest = NSTabInfo.fetchSortedRequest()
do {
var objectIDs: [NSManagedObjectID] = []
let nsTabInfos = try fetchRequest.execute()
//
// QUESTION: We are updating multiple rows of data directly in a persistent store.
// How can we ensure either all rows is updated, or none row is updated is exception
// happens in between?
//
for nsTabInfo in nsTabInfos {
let batchUpdateRequest = NSBatchUpdateRequest(entityName: "NSTabInfo")
batchUpdateRequest.predicate = NSPredicate(format: "self == %#", nsTabInfo.objectID)
batchUpdateRequest.propertiesToUpdate = ["name": nsTabInfo.name! + "XXX"]
batchUpdateRequest.resultType = .updatedObjectIDsResultType
let batchUpdateResult = try backgroundContext.execute(batchUpdateRequest) as? NSBatchUpdateResult
guard let batchUpdateResultX = batchUpdateResult else { return }
guard let managedObjectIDs = batchUpdateResultX.result else { return }
if let nsManagedObjectIDs = managedObjectIDs as? [NSManagedObjectID] {
objectIDs.append(contentsOf: nsManagedObjectIDs)
}
//
// Simulate some exception
// We notice the first row is updated & rest of the rows are unchanged.
// This leaves our data in inconsistent state.
//
throw "Custom error!!!"
}
if !objectIDs.isEmpty {
let changes = [NSUpdatedObjectsKey : objectIDs]
coreDataStack.mergeChanges(changes)
}
} catch {
backgroundContext.rollback()
error_log(error)
}
}
}
class CoreDataStack {
static let INSTANCE = CoreDataStack()
private init() {
}
private(set) lazy var persistentContainer: NSPersistentContainer = {
let container = NSPersistentContainer(name: "wenote")
container.loadPersistentStores(completionHandler: { (storeDescription, error) in
if let error = error as NSError? {
fatalError("Unresolved error \(error), \(error.userInfo)")
}
})
// So that when backgroundContext write to persistent store, container.viewContext will retrieve update from
// persistent store.
container.viewContext.automaticallyMergesChangesFromParent = true
return container
}()
private(set) lazy var backgroundContext: NSManagedObjectContext = {
let backgroundContext = persistentContainer.newBackgroundContext()
backgroundContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
return backgroundContext
}()
func mergeChanges(_ changes: [AnyHashable : Any]) {
NSManagedObjectContext.mergeChanges(
fromRemoteContextSave: changes,
into: [persistentContainer.viewContext, backgroundContext]
)
}
}
We write a demo code to illustrate the following
Performing NSBatchUpdateRequest multiple times within a loop.
An exception happens in between.
We wishes none of the row in persistent store is updated. However, a row is already updated before the exception thrown.
May I know what technique I can use, which is similar to SQLite transaction feature, so that either all rows is updated, or none of the row is updated when exception happens?
CoreData.framework doesn't open up SQLite level controls to the user, it provides you NSManagedObjectContext.
How does it work in a similar manner?
You pull as many objects in many as you need and do your changes on them.
When you are done with your changes, you do context.save().
In that way, you save all of your changes in one shot.
In all cases, pulling all objects in memory might not be possible or a good idea, so then you need to implement your own solution around how to send all of these changes to disk.
From the NSBatchUpdateRequest docs -
A request to Core Data to do a batch update of data in a persistent store without loading any data into memory.
When you execute this, you are doing the changes in store that you can't roll back. For a large data-set, you can do following -
Say you have to perform a series of updates (5 different steps) on 100k records as an operation.
Start in a background thread, pull objects in memory in batches of 1k at a time.
You can load 1k objects easily in memory, mutate them - go through all of your changes/steps one by one and save these changes on this batch. If this is successful, you move on to the next batch.
In case one intermediate step fails on a batch, you can then use either NSManagedObjectContext.rollback() or NSManagedObjectContext.reset() depending on your implementation.
Here's a popular SO post on the differences between the two in case official docs don't provide enough clarity.
I have such a problem.
/movies?page=0&size=20
api endpoint that returns paginated movies
Then I want to display this movies in UITableView. So it is easy to make pagination and load next pages of movies.
But now I want to add CoreData caching functionality and Repository pattern in between.
Something like this
MoviesRepository (to integrate both below dependencies)
MoviesDao (to access Core Data for movies)
MoviesService (to make remote api calls)
I consider to treat Core Data as single source of truth.
So MoviesRepository -> fetchMovies(page: 2, size: 10) should make something like this:
call MoviesDao to get [20,30) movies from CoreData
make request to MoviesService -> getMovies(page: 2, size: 10)
after getting response it should call something like MoviesDao -> syncMovies([remoteMovies])
then Core Data change should be detected and movies in table view should be updated (here i think about some RxSwift observable approach or other like Combine framework in feature, maybe som CoreData nofitications, or FetchResultsController?
But the most crucial thing here is this syncing CoreData with remote data, as meantime remote data could change and page 2 is not equal to the page 2 in core data and how to resolve this problem without refetching all the data.
Here is the code that i found in some tutorial but it replaces all the data not pages.
private func syncFilms(jsonDictionary: [[String: Any]], taskContext: NSManagedObjectContext) -> Bool {
var successfull = false
taskContext.performAndWait {
let matchingEpisodeRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "Film")
let episodeIds = jsonDictionary.map { $0["episode_id"] as? Int }.compactMap { $0 }
matchingEpisodeRequest.predicate = NSPredicate(format: "episodeId in %#", argumentArray: [episodeIds])
let batchDeleteRequest = NSBatchDeleteRequest(fetchRequest: matchingEpisodeRequest)
batchDeleteRequest.resultType = .resultTypeObjectIDs
// Execute the request to de batch delete and merge the changes to viewContext, which triggers the UI update
do {
let batchDeleteResult = try taskContext.execute(batchDeleteRequest) as? NSBatchDeleteResult
if let deletedObjectIDs = batchDeleteResult?.result as? [NSManagedObjectID] {
NSManagedObjectContext.mergeChanges(fromRemoteContextSave: [NSDeletedObjectsKey: deletedObjectIDs],
into: [self.persistentContainer.viewContext])
}
} catch {
print("Error: \(error)\nCould not batch delete existing records.")
return
}
// Create new records.
for filmDictionary in jsonDictionary {
guard let film = NSEntityDescription.insertNewObject(forEntityName: "Film", into: taskContext) as? Film else {
print("Error: Failed to create a new Film object!")
return
}
do {
try film.update(with: filmDictionary)
} catch {
print("Error: \(error)\nThe quake object will be deleted.")
taskContext.delete(film)
}
}
// Save all the changes just made and reset the taskContext to free the cache.
if taskContext.hasChanges {
do {
try taskContext.save()
} catch {
print("Error: \(error)\nCould not save Core Data context.")
}
taskContext.reset() // Reset the context to clean up the cache and low the memory footprint.
}
successfull = true
}
return successfull
}
}
To start with, I don't believe this is a duplicate of: Updating object value in core data is also creating a new object (It's in Obj-C and they were calling insertNewObject every time they segued.)
Background Info: I learned how to use CoreData from the Ray Wenderlich book and referred to it when writing this code. I rolled my own custom stack as outlined in Chapter 3 if you have the book. I can show the code for this if needed.
Queue is the Entity I'm trying to update.
It has 1 property: name - String
And 1 to-many relationship: tasks: Task
My CoreData logic is in a Struct which contains the managedContext.
I have a basic find/create function to create a Queue object. This works. It creates 1 and only 1 object.
func findOrCreateMainQueue() -> Queue? {
let queue = Queue(context: managedContext)
queue.name = "Queue32"
let queueFetch: NSFetchRequest<Queue> = Queue.fetchRequest()
queueFetch.predicate = NSPredicate(format: "%K == %#", #keyPath(Queue.name), "Queue32" as CVarArg)
do {
let results = try managedContext.fetch(queueFetch)
print(results.count)
if results.count > 0 {
return results.first!
} else {
try managedContext.save()
}
} catch let error as NSError {
print("Fetch error: \(error) description: \(error.userInfo)")
}
return nil
}
(As you can see by the queue.name suffix number I have tried a lot of different things.)
I have tried just about everything I can think of:
This code is basically copy/pasted from: How do you update a CoreData entry that has already been saved in Swift?
func addTaskToMainQueue2(task: Task) {
let queueFetch: NSFetchRequest<Queue> = Queue.fetchRequest()
queueFetch.predicate = NSPredicate(format: "%K == %#", #keyPath(Queue.name), "Queue32" as CVarArg)
do {
let results = try managedContext.fetch(queueFetch)
print(results.count)
if results.count > 0 {
var tasks = results[0].tasks?.mutableCopy() as? NSMutableOrderedSet
tasks?.add(task)
results[0].setValue(tasks, forKey: "tasks")
}
} catch let error as NSError {
print("Fetch error: \(error) description: \(error.userInfo)")
}
do {
try managedContext.save()
} catch let error as NSError {
print("Save error: \(error),description: \(error.localizedDescription)")
}
}
Which causes a second Queue object to be created with the "Queue32" name.
Here is another thing I tried:
func addTaskToMainQueue(task: Task) {
if var queue = findOrCreateMainQueue() {
var tasks = queue.tasks?.mutableCopy() as? NSMutableOrderedSet
tasks?.add(task)
queue.tasks = tasks
do {
try managedContext.save()
} catch let error as NSError {
print("Save error: \(error),description: \(error.localizedDescription)")
}
}
}
For the sake of space I won't add code for other things I've tried.
I've tried using the find/create function and updating in that method.
I've tried saving the queue as a local object and passing it to the addTask function which causes duplication as well.
It also doesn't matter if I pass in the Task or create one in the addTask function.
I am starting to believe my issue is something in my dataModel file causing this as I've tried a number of 'How to update a Core Data object' tutorials online and I get the same result each time.
awakeFromInsert() is called whenever I try to update an object. Not sure if this should be happening.
In other places in my app updating works. For example, if I add a Subtask to a Task. It works fine. However, if I want to change the name of another entity called Project the object duplicates. (Project has an id attribute which is fetched, then the name attribute is changed.)
Thank you in advance. I've been pulling my hair out for hours.
I admit not having read all of your code but if you create a new managed object like this
let queue = Queue(context: managedContext)
then it will be added to the managedContext and will be saved to disk at some point. So this code
if results.count > 0 {
return results.first!
} else {
try managedContext.save()
}
is irrelevant in regard to the queue object created earlier because it will be saved even if results.count is > 0, although at a later point. So this means you will have to delete queue when the fetch is successful which feels unnecessary, better to wait with creating it
if results.count > 0 {
return results.first!
} else {
let queue = Queue(context: managedContext)
queue.name = "Queue32"
try managedContext.save()
}
Off topic but I see you return nil if a new object was created rather than fetched, is this intended?
I am using the performBackgroundTask function to pull data from firebase, compare it with data already stored in Core Data, save new data to Core Data, and call a completion handler when done.
I understand that Core Data is not thread safe but I am trying to do this concurrently.
static func cache(completion: #escaping (Void) -> Void) {
CoreDataHelper.persistentContainer.performBackgroundTask { (context) in
let dispatchGroup = DispatchGroup()
// fetch previously saved Core Data from main thread (1) and filter them (2)
let newsSourceIDs = NewsSourceService.getSaved().filter{$0.isEnabled}.map{$0.id!}
let oldArticleURLs = ArticleService.getSaved().map{$0.url!}
// create firebase database reference
let ref = Database.database().reference()
Constants.Settings.timeOptions.forEach { time in
let timeRef = ref.child("time\(time)minutes")
newsSourceIDs.forEach { newsSourceID in
dispatchGroup.enter()
// pull from Firebase Database
timeRef.child(newsSourceID).observeSingleEvent(of: .value, with: { (snapshot) in
guard let newsSourceDict = snapshot.value as? [String: [String:String]] else {
return
}
newsSourceDict.values.forEach { articleDict in
dispatchGroup.enter()
if oldArticleURLs.contains(articleDict["url"]!) {
dispatchGroup.leave()
return
}
// create article entity with firebase data
let article = Article(context: context)
article.date = articleDict["date"]
article.source = newsSourceID
article.time = Int16(time)
article.title = articleDict["title"]
article.url = articleDict["url"]
article.urlToImage = articleDict["urlToImage"]
dispatchGroup.leave()
}
dispatchGroup.leave()
})
}
}
// when done, save and call completion handler (3)
dispatchGroup.notify(queue: .main) {
do {
try context.save()
completion()
} catch {
fatalError("Failure to save context: \(error)")
}
}
}
}
Fetch from Core Data function:
static func getSaved() -> [Article] {
let fetchRequest: NSFetchRequest<Article> = Article.fetchRequest()
do {
let results = try CoreDataHelper.managedContext.fetch(fetchRequest)
return results
} catch let error as NSError {
print("Could not fetch \(error)")
}
return []
}
Can I fetch Core Data from the main thread during performBackgroundTask?
Should I filter with the high level filter function or using a special batch request (can I do that concurrently?)
How can I use dispatchGroup.notify(queue:) to determine when the creation and saving of Core Data is complete?
Can I fetch Core Data from the main thread during performBackgroundTask?
You can fetch from any thread if you use that method. You can't use the results on the main thread though. NSPersistentContainer provides the viewContext property for use on the main thread.
Should I filter with the high level filter function or using a special batch request (can I do that concurrently?)
I'd do it with a predicate on a normal non-batch request. Either of the ways you mention are possible. It depends on what kind of fetching and filtering you need. A batch request might be good if the fetch and filter takes a long time to run. Filtering the results after the fetch might be good if your filtering rules can't be expressed in a predicate.
How can I use dispatchGroup.notify(queue:) to determine when the creation and saving of Core Data is complete?
Add the notify call after your forEach closure. If you never enter, it'll execute immediately. If you do enter, it will execute when you match each enter with a leave.
One other detail: Your getSaved method should take a managed object context as an argument, and fetch with that context. Otherwise you're mixing contexts here. The performBackgroundTask creates one context, but you're using a different one in getSaved.
Another way to handle concurrency with Core Data(one of the easiest, but not optimized) would be using a "child" managedObjectContext with a concurrencyType of private, setting that new MOC's parent to be the MOC on your main thread.
let privateMOC = NSManagedObjectContext(concurrencyType: .privateQueueConcurrencyType)
privateMOC.parent = persistentManager.managedObjectContext
privateMOC.perform {
do {
try privateMOC.save()
} catch let error as NSError {
}
}
You would perform all of your needed core data actions inside of the .perform closure. When you run privateMOC.save() the changes are pushed up to the parent managedObjectContext on the main thread.
I am currently running into a Core Data issue in which not all data saved in my child context is saved to its parent when called. The number of records saved vary, which leads me to believe there is some race condition involved, but after hours of looking at this, I am out of ideas.
Setup of Child Context
init(managedObjectContext: NSManagedObjectContext) {
_parentManagedObjectContext = managedObjectContext
// initialize managed object context
privateManagedObjectContext = NSManagedObjectContext(concurrencyType: .PrivateQueueConcurrencyType)
// configure
privateManagedObjectContext.parentContext = _parentManagedObjectContext
super.init()
}
Creating Records
private func insertPerson(name:String, hometown:String) {
let newPerson = NSEntityDescription.insertNewObjectForEntityForName("Person", inManagedObjectContext: privateManagedObjectContext) as! Person
newPerson.name = name
newPerson.hometown = hometown
}
Saving Records
private func saveChanges(to moc:NSManagedObjectContext) {
guard privateManagedObjectContext.hasChanges else { return }
moc.performBlockAndWait {
do {
print("Saving \(moc.insertedObjects.count) objects")
moc.mergePolicy = NSMergeByPropertyStoreTrumpMergePolicy
try moc.save()
if let parentContext = moc.parentContext {
self.saveChanges(to: parentContext)
}
} catch {
print("[ERROR] Could not save: \((error as NSError).localizedDescription)")
}
}
}
When I run the code, I get something like the following (let's say 50 records should be inserted):
Saving 50 records // privateManagedObjectContext
Saving 12 records // _parentManagedObjectContext
Again, the number of records inserted by the parent varies, but generally no more than 10-12 records. I've been unable to find a clear reason for this issue but it feels like a race condition with the objects not being saved to the child before the parent, but that was why I used performBlockAndWait. Any suggestions are MUCH appreciated.