I have a NSOperation subclass that has its private context, and a singleton data manager class that has a context on main queue. All the UI and crud operation are done by this singleton class and a background fetch from cloud kit is done by the NSOperation subclass. Had few doubts as below.
Following is the code i have in NSoperation subclass.Can below code create deadlock?
self.localStoreMOC?.performBlockAndWait({ () -> Void in
//Long process of fetching data from cloud and pushing changes to cloud happens here.
var error:NSErrorPointer = nil
if self.localStoreMOC!.hasChanges
{
do
{
try self.localStoreMOC!.save()
}
catch let error1 as NSError
{
error.memory = error1
}
if error == nil
{
self.localStoreMOC!.parentContext!.performBlockAndWait({
do
{
try self.localStoreMOC!.parentContext!.save()
}
catch let error1 as NSError
{
print("wasSuccessful error1 \(error1)")
}
})
}
}
}
If i have a another singleton class using this class NSManagedOBject do i need to pass them though ID ?
First, you need to turn on -com.apple.CoreData.ConcurrencyDebug 1 in your run time arguments. That will help insure you are calling everything on the proper thread/queue.
Second, you are doing a lot of forced unwrapping of optionals, that is a very bad habit to be in. Best to unwrap them properly or use the optional unwrapping.
Third, what happens when you pause the debugger? Where is the line of code that it is pausing on and what queues are you on?
Just turning on the concurrency debug will most likely show you your issue.
Item 2
If you are wanting to pass a reference to a NSManagedObject from one context to another then yes, you need to use the NSManagedObjectID as the NSManagedObject is not safe to pass between contexts.
Code Formatting
Was playing with the formatting a bit, the results may be of interest to you:
guard let local = localStoreMOC else { fatalError("Local store is nil") }
guard let parent = local.parentContext else { fatalError("Parent store is nil") }
local.performBlockAndWait {
//Long process of fetching data from cloud and pushing changes to cloud happens here.
if !local.hasChanges { return }
do {
try local.save()
parent.performBlockAndWait {
do {
try parent.save()
} catch {
print("wasSuccessful error1 \(error)")
}
}
} catch {
print("Failed to save local: \(error)")
}
}
This removes the forced unwrapping of optionals and prints both errors out if you get one in either case.
Update
Also, some developers, say that nested performblockandwait like above will cause deadlock.
performBlockAndWait will never cause a deadlock. It is more intelligent than that.
If you are going from one queue to another, then you want each queue to block and wait like the code describes.
If you were on the right queue and called performBlockAndWait then the call would effectively be a no-op and will NOT deadlock
Related
I use coreData as persistent store.
To read data, I use (only essential parts are shown):
func fetchShoppingItems(completion: #escaping (Set<ShoppingItem>?, Error?) -> Void) {
persistentContainer.performBackgroundTask { (backgroundManagedContext) in
let fetchRequest: NSFetchRequest<CDShoppingItem> = CDShoppingItem.fetchRequest()
var shoppingItems: Set<ShoppingItem> = []
do {
let cdShoppingItems: [CDShoppingItem] = try backgroundManagedContext.fetch(fetchRequest)
for nextCdShoppingItem in cdShoppingItems {
nextCdShoppingItem.managedObjectContext!.performAndWait {
Thread.sleep(forTimeInterval: 0.1) // This seems to be required
let nextShoppingItem = ShoppingItem.init(name: nextCdShoppingItem.name!)
shoppingItems.insert(nextShoppingItem)
} // performAndWait
} // for all cdShoppingItems
completion(shoppingItems, nil)
return
} catch let error as NSError {
completion(nil, error)
return
} // fetch error
} // performBackgroundTask
} // fetchShoppingItems
To test the coreData implementation, I wrote a unit test that creates multiple threads that write to and read from coreData concurrently.
This test runs only successfully, if the instruction
Thread.sleep(forTimeInterval: 0.1) // This seems to be required
is inserted in the performAndWait closure.
If it is commented out, nextCdShoppingItem is often read back with nil attributes, and the function crashes due to the forced unwrap.
I am not sure, if nextCdShoppingItem.managedObjectContext!.performAndWait is correct or if I had to use backgroundManagedContext.performAndWait, but with backgroundManagedContext the effect is the same.
I do not understand why inserting a small delay before accessing an attribute of a managed object is necessary to avoid the problem.
Any hint is welcome!
EDIT:
I investigated the issue further, and found the following:
Every time nextCdShoppingItem is read back by the background thread (called read thread below) as nil, there is also another background thread that tries to save its own managedContext after all records in its managedContext have been deleted (called write thread below).
Apparently the read thread tries to fetch a record that has just been deleted by the write thread.
So the problem is definitively a multithreading issue, and I found a solution (see my answer below).
performAndWait will add the block to the queue and schedule it to run, just like perform, but performAndWait will not return until the block is complete. Since you are inside a loop of cdShoppingItems, the loop does not stop and wait for the block to return. By adding the thread sleep, you are essentially slowing down the loop and giving core data enough time to complete its fetch. The forced unwrap crash is probably an indication that it's lost its nextCdShoppingItem reference.
I would consider refactoring where you do not need to query core data inside a loop. If it's possible, add the name attribute to CDShoppingItem so you don't have to fetch it to build a ShoppingItem object.
Edit: took a stab at a refactor although I don't know your exact use case:
func fetchShoppingItems(completion: #escaping (Set<ShoppingItem>?, Error?) -> Void) {
persistentContainer.performBackgroundTask { (backgroundManagedContext) in
let fetchRequest: NSFetchRequest<CDShoppingItem> = CDShoppingItem.fetchRequest()
do {
var shoppingItems: Set<ShoppingItem> = []
let cdShoppingItems: [CDShoppingItem] = try backgroundManagedContext.fetch(fetchRequest)
for nextCdShoppingItem in cdShoppingItems {
if let name = nextCdShoppingItem.name {
let nextShoppingItem = ShoppingItem.init(name: name)
shoppingItems.insert(nextShoppingItem)
}
}
completion(shoppingItems, nil)
} catch let error as NSError {
print("Error fetching CDShoppingItem: \(error)")
completion(nil, error)
} // fetch error
return
} // performBackgroundTask
} // fetchShoppingItems
To prevent the multithreading issue, I tried 2 things:
1) Since iOS10, a persistentStore of SQL type maintains a connection pool for concurrent access to the pool, and it is possible to set a maximum pool size, see the WWDC video. I did so using
private lazy var persistentContainer: NSPersistentContainer = {
let container = NSPersistentContainer(name: modelName)
container.loadPersistentStores(completionHandler: { (storeDescription, error) in
if let error = error as NSError? {
// ...
} else {
storeDescription.setOption(NSNumber(1), forKey: NSPersistentStoreConnectionPoolMaxSizeKey)
}
})
return container
}()
to prevent concurrent access to the persistentStore. Unfortunately, this did not solve the problem for an unknown reason.
2) I then tried to serialize read and write operations by setting up a serial queue:
private let coreDataSerialQueue = DispatchQueue(label: "com.xxx.ShopEasy.coreDataManager") // serial by default
It is used for the read and write accesses in the following way:
coreDataSerialQueue.async {
let backgroundManagedContext = self.persistentContainer.newBackgroundContext()
backgroundManagedContext.performAndWait {
// …
} // performAndWait
} // coreDataSerialQueue.async
This did solve the problem.
Please note that it would be wrong to use
coreDataSerialQueue.async {
self.persistentContainer.performBackgroundTask { (backgroundManagedContext) in
// …
} // performBackgroundTask
} // coreDataSerialQueue.async
because performBackgroundTask would fork another asynchronous thread and thus break the serialization.
We are running an App through Citrix Secure Hub, it seems that sometimes there is a rollback with loosing some Data in CoreData.
As i understand, CoreData is having something like an working copy of all the objects, and sometimes its tries to persist that on the filesystem.
Well tried to simulate the behavior but without any success, we could not find out any data loss or rollbacked data in our test environment.
So is there a way to force iOS to write the current "working copy" on the disk to prevent any data loss when using too much memory (and maybe crash)? We call our save function after
As we already found out:
We were NOT using:
func applicationWillResignActive(_ application: UIApplication) {
print("applicationWillResignActive")
}
to save the context, could this be a problem (we are already saving the context after every created object) ?
At the Moment we dont really handle problems when the context could not be saved, are there any recommendations how to handle that in a productive environment? And is it a good thing to maybe crash to app to prevent the user from struggeling with data loss?
Edit: this is the used Core Data Handler:
import Foundation
import CoreData
let context = CoreDataManager.shared.managedObjectContext
func saveContext(_ completion: (() -> Void)? = nil) {
CoreDataManager.shared.save(completion)
}
func saveContextSync() {
CoreDataManager.shared.saveSync()
}
class CoreDataManager: NSObject {
static let shared = CoreDataManager()
lazy var managedObjectContext: NSManagedObjectContext = {
var managedObjectContext = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
managedObjectContext.persistentStoreCoordinator = persistentStoreCoordinator
return managedObjectContext
}()
And our save functionality:
#objc func save(_ completion: (() -> Void)?) {
saveAsync(completion)
}
func saveAsync(_ completion: (() -> Void)?) {
func save() {
context.perform {
do { try context.save() }
catch {
// HERE WE NEED TO HANDLE IT FOR A PRODUCTIVE ENVIRONMENT
}
completion?()
}
}
if Thread.isMainThread {
save()
} else {
DispatchQueue.main.async {
save()
}
}
}
func saveSync() {
func save() {
context.performAndWait {
do { try context.save() }
catch { print(error)
// TRY TO REPRODUCE MEMORY LOSS APP TO SEE WHAT HAPPENS
abort()
}
}
}
if Thread.isMainThread {
save()
} else {
DispatchQueue.main.sync {
save()
}
}
}
Edit 2: This question in Objective C should be very similar:
Core Data reverts to previous state without apparent reason
Edit 3: It seems that there is no crash, some users telling me that they are adding data, then just press the home button and after a couple of hours the data from the last "task" is lost.
There are three possible causes.
Write Conflicts
Core data generally wants writes to be done in a single synchronous way. If you write in multiple ways at the same time to the same object (even if they are touching different properties and don't strictly conflict), it will be a merge conflict. You can set a merge policy (by default the value is 'error' - meaning don't apply the changes) but that is really a bad solution because you are tell core-data to lose information silently. see NSPersistentContainer concurrency for saving to core data for a setup to prevent merge conflicts.
Closing the app with unsaved data
If you setup your core-data correctly this shouldn't happen. The correct way to setup core data to only read from the 'viewContext' and write in a single synchronous way. Each writing is done in a single atomic block and the UI is only updated after it is saved. If you are displaying information from a context that is not saved to disk this can be a problem. For example it appears that your app only uses a single main-thread context for both reading and writing. Making changes to that context and not calling save will leave the app in a state where there are major changes that are only in memory and not on disk.
There is an error saving to disk
This is by far the rarest event, but there are users that have really really full disks. If this happens there is NOTHING that you can do to save. There is physically no room left on the disk. Generally the correct thing to do is to tell the user and leave it at that.
Without knowing more about your particular setup it hard to say for certain what your problem is. I would recommend the following setup:
use NSPersistentContainer
only read from the viewContext and never write to it.
make an operation queue for writing to core data
for every operation, create a context, make changes to it, and then save. Do not pass any managed object into or out of these blocks.
This should deal with all but the third problem.
I need to sync web database in my coredata, for which I perform service api calls. I am using Alamofire with Swift 3. There are 23 api calls, giving nearly 24k rows in different coredata entities.
My problem: These api calls blocks UI for a minute, which is a long time for a user to wait.
I tried using DispatchQueue and performing the task in background thread, though nothing worked. This is how I tried :
let dataQueue = DispatchQueue.init(label: "com.app.dataSyncQueue")
dataQueue.async {
DataSyncController().performStateSyncAPICall()
DataSyncController().performRegionSyncAPICall()
DataSyncController().performStateRegionSyncAPICall()
DataSyncController().performBuildingRegionSyncAPICall()
PriceSyncController().performBasicPriceSyncAPICall()
PriceSyncController().performHeightCostSyncAPICall()
// Apis which will be used in later screens are called in background
self.performSelector(inBackground: #selector(self.performBackgroundTask), with: nil)
}
An API call from DataSyncController:
func performStateSyncAPICall() -> Void {
DataSyncRequestManager.fetchStatesDataWithCompletionBlock {
success, response, error in
self.apiManager.didStatesApiComplete = true
}
}
DataSyncRequestManager Code:
static func fetchStatesDataWithCompletionBlock(block:#escaping requestCompletionBlock) {
if appDelegate.isNetworkAvailable {
Util.setAPIStatus(key: kStateApiStatus, with: kInProgress)
DataSyncingInterface().performStateSyncingWith(request:DataSyncRequest().createStateSyncingRequest() , withCompletionBlock: block)
} else {
//TODO: show network failure error
}
}
DataSyncingInterface Code:
func performStateSyncingWith(request:Request, withCompletionBlock block:#escaping requestCompletionBlock)
{
self.interfaceBlock = block
let apiurl = NetworkHttpClient.getBaseUrl() + request.urlPath!
Alamofire.request(apiurl, parameters: request.getParams(), encoding: URLEncoding.default).responseJSON { response in
guard response.result.isSuccess else {
block(false, "error", nil )
return
}
guard let responseValue = response.result.value else {
block (false, "error", nil)
return
}
block(true, responseValue, nil)
}
}
I know many similar questions have been already posted on Stackoverflow and mostly it is suggested to use GCD or Operation Queue, though trying DispatchQueues didn't work for me.
Am I doing something wrong?
How can I not block UI and perform the api calls simultaneously?
You can do this to run on a background thread:
DispatchQueue.global(qos: .background).async {
// Do any processing you want.
DispatchQueue.main.async {
// Go back to the main thread to update the UI.
}
}
DispatchQueue manages the execution of work items. Each work item submitted to a queue is processed on a pool of threads managed by the system.
I usually use NSOperationQueue with Alamofire, but the concepts are similar. When you set up an async queue, you allow work to be performed independently of the main (UI) thread, so that your app doesn't freeze (refuse user input). The work will still take however long it takes, but your program doesn't block while waiting to finish.
You really have only put one item into the queue.
You are adding to the queue only once, so all those "perform" calls wait for the previous one to finish. If it is safe to run them concurrently, you need to add each of them to the queue separately. There's more than one way to do this, but the bottom line is each time you call .async {} you are adding one item to the queue.
dataQueue.async {
DataSyncController().performStateSyncAPICall()
}
dataQueue.async {
DataSyncController(). performRegionSyncAPICall l()
}
I'm having troubles with Alamofire using Operation and OperationQueue.
I have an OperationQueue named NetworkingQueue and I push some operation (wrapping AlamofireRequest) into it, everything works fine, but during application living, at one moment all Alamofire request are not sent. My queue is getting bigger and bigger and no request go to the end.
I do not have a scheme to reproduce it anytime.
Does anybody have a clue for helping me?
Here is a sample of code
The BackgroundAlamoSession
let configuration = URLSessionConfiguration.background(withIdentifier: "[...].background")
self.networkingSessionManager = Alamofire.SessionManager(configuration: configuration)
AbstractOperation.swift
import UIKit
import XCGLogger
class AbstractOperation:Operation {
private let _LOGGER:XCGLogger = XCGLogger.default
enum State:String {
case Ready = "ready"
case Executing = "executing"
case Finished = "finished"
var keyPath: String {
get{
return "is" + self.rawValue.capitalized
}
}
}
override var isAsynchronous:Bool {
get{
return true
}
}
var state = State.Ready {
willSet {
willChangeValue(forKey: self.state.rawValue)
willChangeValue(forKey: self.state.keyPath)
willChangeValue(forKey: newValue.rawValue)
willChangeValue(forKey: newValue.keyPath)
}
didSet {
didChangeValue(forKey: oldValue.rawValue)
didChangeValue(forKey: oldValue.keyPath)
didChangeValue(forKey: self.state.rawValue)
didChangeValue(forKey: self.state.keyPath)
}
}
override var isExecuting: Bool {
return state == .Executing
}
override var isFinished:Bool {
return state == .Finished
}
}
A concrete Operation implementation
import UIKit
import XCGLogger
import SwiftyJSON
class FetchObject: AbstractOperation {
public let _LOGGER:XCGLogger = XCGLogger.default
private let _objectId:Int
private let _force:Bool
public var object:ObjectModel?
init(_ objectId:Int, force:Bool) {
self._objectId = objectId
self._force = force
}
convenience init(_ objectId:Int) {
self.init(objectId, force:false)
}
override var desc:String {
get{
return "FetchObject(\(self._objectId))"
}
}
public override func start(){
self.state = .Executing
_LOGGER.verbose("Fetch object operation start")
if !self._force {
let objectInCache:objectModel? = Application.main.collections.availableObjectModels[self._objectId]
if let objectInCache = objectInCache {
_LOGGER.verbose("object with id \(self._objectId) founded on cache")
self.object = objectInCache
self._LOGGER.verbose("Fetch object operation end : success")
self.state = .Finished
return
}
}
if !self.isCancelled {
let url = "[...]\(self._objectId)"
_LOGGER.verbose("Requesting object with id \(self._objectId) on server")
Application.main.networkingSessionManager.request(url, method : .get)
.validate()
.responseJSON(
completionHandler: { response in
switch response.result {
case .success:
guard let raw:Any = response.result.value else {
self._LOGGER.error("Error while fetching json programm : Empty response")
self._LOGGER.verbose("Fetch object operation end : error")
self.state = .Finished
return
}
let data:JSON = JSON(raw)
self._LOGGER.verbose("Received object from server \(data["bId"])")
self.object = ObjectModel(objectId:data["oId"].intValue,data:data)
Application.main.collections.availableobjectModels[self.object!.objectId] = self.object
self._LOGGER.verbose("Fetch object operation end : success")
self.state = .Finished
break
case .failure(let error):
self._LOGGER.error("Error while fetching json program \(error)")
self._LOGGER.verbose("Fetch object operation end : error")
self.state = .Finished
break
}
})
} else {
self._LOGGER.verbose("Fetch object operation end : cancel")
self.state = .Finished
}
}
}
The NetworkQueue
class MyQueue {
public static let networkQueue:SaootiQueue = SaootiQueue(name:"NetworkQueue", concurent:true)
}
How I use it in another operation and wait for for result
let getObjectOperation:FetchObject = FetchObject(30)
SaootiQueue.networkQueue.addOperations([getObjectOperation], waitUntilFinished: true)
How I use it the main operation using KVO
let getObjectOperation:FetchObject = FetchObject(30)
operation.addObserver(self, forKeyPath: #keyPath(Operation.isFinished), options: [.new], context: nil)
operation.addObserver(self, forKeyPath: #keyPath(Operation.isCancelled), options: [.new], context: nil)
queue.addOperation(operation)
//[...]
override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
if let operation = object as? FetchObject {
operation.removeObserver(self, forKeyPath: #keyPath(Operation.isFinished))
operation.removeObserver(self, forKeyPath: #keyPath(Operation.isCancelled))
if keyPath == #keyPath(Operation.isFinished) {
//Do something
}
}
A few clarifications:
My application is a radio player and I need, while playing music and the background, to fetch the currently playing program. This is why I need background Session.
In fact I also use the background session for all the networking I do when the app is foreground. Should I avoid that ?
The wait I'm using is from another queue and is never used in the main queue (I know it is a threading antipattern and I take care of it).
In fact it is used when I do two networking operation and the second one depends of the result of the second. I put a wait after the first operation to avoid KVO observing. Should I avoid that ?
Additional edit:
When I say "My queue is getting bigger and bigger and no request go to the end", it means that at one moment during application livecycle, random for the moment (I can not find a way to reproduce it at every time), Alamofire request don't reach the response method.
Because of that the Operation wrapper don't end and the queue is growing.
By the way I'm working on converting Alamofire request into URLRequest for having clues and I founded some problem on using the main queue. I have to sort what is due to the fact that Alamofire use the main queue for reponse method and I'll see if I find a potential deadlock
I'll keep you informed. Thanks
There are minor issues, but this operation implementation looks largely correct. Sure, you should make your state management thread-safe, and there are other stylistic improvements you could make, but I don't think this is critical to your question.
What looks worrisome is addOperations(_:waitUntilFinished:). From which queue are you waiting? If you do that from the main queue, you will deadlock (i.e. it will look like the Alamofire requests never finish). Alamofire uses the main queue for its completion handlers (unless you override the queue parameter of responseJSON), but if you're waiting on the main thread, this can never take place. (As an aside, if you can refactor so you never explicitly "wait" for operations, that not only avoids the deadlock risk, but is a better pattern in general.)
I also notice that you're using Alamofire requests wrapped in operations in conjunction with a background session. Background sessions are antithetical to operations and completion handler closure patterns. Background sessions continue after your app has been jettisoned and you have to rely solely upon the SessionDelegate closures that you set when you first configure your SessionManager when the app starts. When the app restarts, your operations and completion handler closures are long gone.
Bottom line, do you really need background session (i.e. uploads and downloads that continue after your app terminates)? If so, you may want to lose this completion handler and operation based approach. If you don't need this to continue after the app terminates, don't use background sessions. Configuring Alamofire to properly handle background sessions is a non-trivial exercise, so only do so if you absolutely need to. Remember to not conflate background sessions and the simple asynchronous processing that Alamofire (and URLSession) do automatically for you.
You asked:
My application is a radio player and I need, while playing music and the background, to fetch the currently playing program. This is why I need background Session.
You need background sessions if you want downloads to proceed while the app is not running. If your app is running in the background, though, playing music, you probably don't need background sessions. But, if the user chooses to download a particular media asset, you may well want background session so that the download proceeds when the user leaves the app, whether the app is playing music or not.
In fact I also use the background session for all the networking I do when the app is foreground. Should I avoid that ?
It's fine. It's a little slower, IIRC, but it's fine.
The problem isn't that you're using background session, but that you're doing it wrong. The operation-based wrapping of Alamofire doesn't make sense with a background session. For sessions to proceed in the background, you are constrained as to how you use URLSession, namely:
You cannot use data tasks while the app is not running; only upload and download tasks.
You cannot rely upon completion handler closures (because the entire purpose of background sessions is to keep them running when your app terminates and then fire up your app again when they're done; but if the app was terminated, your closures are all gone).
You have to use delegate based API only for background sessions, not completion handlers.
You have to implement the app delegate method to capture the system provided completion handler that you call when you're done processing background session delegate calls. You have to call that when your URLSession tells you that it's done processing all the background delegate methods.
All of this is a significant burden, IMHO. Given that the system is keeping you app alive for background music, you might contemplate using a standard URLSessionConfiguration. If you're going to use background session, you might need to refactor all of this completion handler-based code.
The wait I'm using is from another queue and is never used in the main queue (I know it is a threading antipattern and I take care of it).
Good. There's still serious code smell from ever using "wait", but if you are 100% confident that it's not deadlocking here, you can get away with it. But it's something you really should check (e.g. put some logging statement after the "wait" and make sure you're getting past that line, if you haven't already confirmed this).
In fact it is used when I do two networking operation and the second one depends of the result of the second. I put a wait after the first operation to avoid KVO observing. Should I avoid that ?
Personally, I'd lose that KVO observing and just establish addDependency between the operations. Also, if you get rid of that KVO observing, you can get rid of your double KVO notification process. But I don't think this KVO stuff is the root of the problem, so maybe you defer that.
If I create a NSManagedObjectContext on the main thread with NSMainQueueConcurrencyType must I use performBlock() method for all the save and performFetch methods. IE is it possible to do the following:
do {
managedObjectContext.save()
} catch let error as NSError {
print(error)
}
or should I always do this:
managedObjectContext.performBlock({
do {
managedObjectContext.save()
} catch let error as NSError {
print(error)
}
})
If I understand the documentation correctly I always have to use performBlock() or performBlockAndWait() but in the template code from XCode 7 they are not using blocks. Any help is appreciated!
If you are already on the main thread and have a NSMainQueueConcurrencyType context, you do not need to use performBlock.