Updated: I have prepared the sample which is reproduce the issue without magical record.Please download the test project using following URL:
https://www.dsr-company.com/fm.php?Download=1&FileToDL=DeadLockTest_CoreDataWithoutMR.zip
The provided project has following problem: deadlock on fetch
in performBlockAndWait called from main thread.
The issue is reproduced if code is compiled using XCode version > 6.4.
The issue is not reproduced if code is compiled using xCode == 6.4.
Old question was:
I am working on the support of IOS mobile application.
After the recent update of Xcode IDE from version 6.4 to version 7.0 ( with IOS 9 support ) I have faced with critical issue - application hangup.
The same build of the application ( produced from the same sources ) with xCode 6.4 works OK.
So, if the application is built using xCode > 6.4 - application hangs up on some cases.
if the application is built using xCode 6.4 - application works OK.
I have spent some time to research the issue and as the result I have prepared the test application with similar case like in my application which reproduces the problem.
The test application hangup on the Xcode >= 7.0 but works correctly on the Xcode 6.4
Download link of test sources:
https://www.sendspace.com/file/r07cln
The requirements for the test application is:
1. cocoa pods manager must be installed in the system
2. MagicalRecord framework of version 2.2.
Test application works in the following way:
1. At the start of the application it creates test database with 10000 records of simple entities and saves them to persistent store.
2. At the first screen of the application in the method viewWillAppear: it runs the test which causes deadlock.
Following algorithm is used:
-(NSArray *) entityWithId: (int) entityId inContext:(NSManagedObjectContext *)localContext
{
NSArray * results = [TestEntity MR_findByAttribute:#"id" withValue:[ NSNumber numberWithInt: entityId ] inContext:localContext];
return results;
}
…..
int entityId = 88;
NSManagedObjectContext *childContext1 = [NSManagedObjectContext MR_context];
childContext1.name = #"childContext1";
NSManagedObjectContext *childContext2 = [NSManagedObjectContext MR_context];
childContext2.name = #"childContext2";
NSArray *results = [self entityWithId:entityId inContext: childContext2];
for(TestEntity *d in results)
{
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name); /// this line is the reason of the hangup
}
dispatch_async(dispatch_get_main_queue(), ^
{
int entityId2 = 11;
NSPredicate *predicate2 = [NSPredicate predicateWithFormat:#"id=%d", entityId2];
NSArray *a = [ TestEntity MR_findAllWithPredicate: predicate2 inContext: childContext2];
for(TestEntity *d in a)
{
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name);
}
});
Two managed object contexts are created with concurrency type == NSPrivateQueueConcurrencyType (please check the code of MR_context of magical record framework). Both contexts has parent context with
concurrency type = NSMainQueueConcurrencyType. From the main thread application performs fetch in sync manner ( MR_findByAttribute and MR_findAllWithPredicate
are used performBlockAndWait with fetch request inside ). After the first fetch the second fetch is schedule on the main thread using dispatch_async().
As a result the application hangs up. It seems that deadlock has happened, please check the screenshot of the stack:
 here is the link, my reputation is too low to post images. https://cdn.img42.com/34a8869bd8a5587222f9903e50b762f9.png)
If to comment the line
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name); /// this line is the reason of the hangup
(which is the line 39 in ViewController.m of the test project ) the application becomes working OK. I believe this is because there is no read of name field of the test entity.
So with the commented line
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name);
there is no hangup on binaries built both with Xcode 6.4 and Xcode 7.0.
With the uncommented line
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name);
there is hangup on binary built with Xcode 7.0 and there is no hangup on binary built with Xcode 6.4.
I believe the issue is happens because of lazy-loading of entity data.
Has anybody problem with the described case? I will be grateful for any help.
This is why I don't use frameworks that abstract (i.e., hide) too many details of core data. It has very complex use patterns, and sometimes you need to know the details of how they interoperate.
First, I know nothing about magical record except that lots of people use it so it must be pretty good at what it does.
However, I immediately saw several completely wrong uses of core data concurrency in your examples, so I went and looked at the header files to see why your code made the assumptions that it does.
I don't mean to bash you at all, though this may seem like it at first blush. I want to help educate you (and I used this as an opportunity to take a peek at MR).
From a very quick look at MR, I'd say you have some misunderstandings of what MR does, and also core data's general concurrency rules.
First, you say this...
Two managed object contexts are created with concurrency type ==
NSPrivateQueueConcurrencyType (please check the code of MR_context of
magical record framework). Both contexts has parent context with
concurrency type = NSMainQueueConcurrencyType.
which does not appear to be true. The two new contexts are, indeed, private-queue contexts, but their parent (according to the code I glanced at on github) is the magical MR_rootSavingContext, which itself is also a private-queue context.
Let's break down your code example.
NSManagedObjectContext *childContext1 = [NSManagedObjectContext MR_context];
childContext1.name = #"childContext1";
NSManagedObjectContext *childContext2 = [NSManagedObjectContext MR_context];
childContext2.name = #"childContext2";
So, you now have two private-queue MOCs (childContext1 and childContext2), both children of another anonymous private-queue MOC (we will call savingContext).
NSArray *results = [self entityWithId:entityId inContext: childContext2];
You then perform a fetch on childContext1. That code is actually...
-(NSArray *) entityWithId:(int)entityId
inContext:(NSManagedObjectContext *)localContext
{
NSArray * results = [TestEntity MR_findByAttribute:#"id"
withValue:[NSNumber numberWithInt:entityId]
inContext:localContext];
return results;
}
Now, we know that the localContext in this method is, in this case, another pointer to childContext2 which is a private-queue MOC. It is 100% against the concurrency rules to access a private-queue MOC outside of a call to performBlock. However, since you are using another API, and the method name offers no assistance to know how the MOC is being accessed, we need to go look at that API and see if it hides the performBlock to see if you are accessing it correctly.
Unfortunately, the documentation in the header file offers no indication, so we have to look at the implementation. That call ends up calling MR_executeFetchRequest... which does not indicate in the documentation how it handles the concurrency either. So, we go look at its implementation.
Now, we are getting somewhere. This function does try to safely access the MOC, but it uses performBlockAndWait which will block when it is called.
This is an extremely important piece of information, because calling this from the wrong place can indeed cause a deadlock. Thus, you must be keenly aware that performBlockAndWait is being called anytime you execute a fetch request. My own personal rule is to never use performBlockAndWait unless there is absolutely no other option.
However, this call here should be completely safe... assuming it is not being called from within the context of the parent MOC.
So, let's look at the next piece of code.
for(TestEntity *d in results)
{
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name); /// this line is the reason of the hangup
}
Now, this is not the fault of MagicalRecord, because MR isn't even being used directly here. However, you have been trained to use those MR_ methods, which require no knowledge of the concurrency model, so you either forget or never learn the concurrency rules.
The objects in the results array are all managed objects that live in the childContext2 private-queue context. Thus, you may not ever access them without paying homage to the concurrency rules. This is a clear violation of the concurrency rules. While developing your application, you should enable concurrency debugging with the argument -com.apple.CoreData.ConcurrencyDebug 1.
This code snippet must be wrapped in either performBlock or performBlockAndWait. I hardly ever use performBlockAndWait for anything because it has so many drawbacks - deadlocks being one of them. In fact, just seeing the use of performBlockAndWait is a very strong indication that your deadlock is happening in there and not on the line of code that you indicate. However, in this case, it is at least as safe as the previous fetch, so let's make it a bit safer...
[childContext2 performBlockAndWait:^{
for (TestEntity *d in results) {
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name);
}
}];
Next, you dispatch to the main thread. Is that because you just want something to occur on a subsequent event loop cycle, or is it because this code is already running on some other thread? Who knows. However, you have the same problem here (I reformatted your code for readability as a post).
dispatch_async(dispatch_get_main_queue(), ^{
int entityId2 = 11;
NSPredicate *predicate2 = [NSPredicate predicateWithFormat:#"id=%d", entityId2];
NSArray *a = [TestEntity MR_findAllWithPredicate:predicate2
inContext:childContext2];
for (TestEntity *d in a) {
NSLog(#"e from fetchRequest %# with name = '%#'", d, d.name);
}
});
Now, we know that code starts out running on the main thread, and the search will call performBlockAndWait but your subsequent access in the for-loop again violates the core data concurrency rules.
Based on that, the only real problems I see are...
MR seems to honor the core data concurrency rules within their API, but you must still follow the core data concurrency rules when accessing your managed objects.
I really don't like the use of performBlockAndWait as it's just a problem waiting to happen.
Now, let's take a look at the screenshot of your hang. Hmmm... it's a classic deadlock, but it makes no sense because the deadlock happens between the main thread and the MOC thread. That can only happen if the main-queue MOC is a parent of this private-queue MOC, but the code shows that is not the case.
Hmmm... it didn't make sense, so I downloaded your project, and looked at the source code in the pod you uploaded. Now, that version of the code uses the MR_defaultContext as the parent of all MOCs created with MR_context. So, the default MOC is, indeed, a main-queue MOC, and now it all makes perfect sense.
You have a MOC as a child of a main-queue MOC. When you dispatch that block to the main queue, it's is now running as a block on the main queue. The code then calls performBlockAndWait on a context that is a child of a MOC for that queue, which is a huge no-no, and your are almost guaranteed to get a deadlock.
So, it seems that MR has since changed their code from using a main-queue as the parent of new contexts to using a private-queue as the parent of new contexts (most likely due to this exact problem). So, if you upgrade to the latest version of MR you should be fine.
However, I would still warn you that if you want to use MR in multithreading ways, you must know exactly how they handle the concurrency rules, and you must also make sure you obey them anytime you are accessing any core-data objects that are not going through the MR API.
Finally, I'll just say that I've done tons and tons of core data stuff, and I've never used an API that tries to hide the concurrency issues from me. The reason is that there are too many little corner cases, and I'd rather just deal with them in a pragmatic way up front.
Finally, you should almost never use performBlockAndWait unless you know exactly why its the only option. Having it be used as part of an API underneath you is even more scary... to me at least.
I hope this little jaunt has enlightened and helped you (and possible some others). It certainly shed a little bit of light for me, and helped reestablish some of my previous unfounded skittishness.
Edit
This is in response to the "non-magical-record" example you provided.
The problem with this code is the exact same problem I described above, relative to what was happening with MR.
You have a private-queue context, as a child to a main-queue context.
You are running code on the main queue, and you call performBlockAndWait on the child context, which has to then lock its parent context as it tries to execute the fetch.
It is called a deadlock, but the more descriptive (and seductive) term is deadly embrace.
The original code is running on the main thread. It calls into a child context to do something, and it does nothing else until that child complete.
That child then, in order to complete, needs the main thread to do something. However, the main thread can't do anything until the child is done... but the child is waiting for the main thread to do something...
Neither one can make any headway.
The problem you are facing is very well documented, and in fact, has been mentioned a number of times in WWDC presentations and multiple pieces of documentation.
You should NEVER call performBlockAndWait on a child context.
The fact that you got away with it in the past is just a "happenstance" because it's not supposed to work that way at all.
In reality, you should hardly every call performBlockAndWait.
You should really get used to doing asynchronous programming. Here is how I would recommend you rewrite this test, and whatever it is like that prompted this issue.
First, rewrite the fetch so it works asynchronously...
- (void)executeFetchRequest:(NSFetchRequest *)request
inContext:(NSManagedObjectContext *)context
completion:(void(^)(NSArray *results, NSError *error))completion
{
[context performBlock:^{
NSError *error = nil;
NSArray *results = [context executeFetchRequest:request error:&error];
if (completion) {
completion(results, error);
}
}];
}
Then, you change you code that calls the fetch to do something like this...
NSFetchRequest *request = [[NSFetchRequest alloc] init];
[request setEntity: testEntityDescription ];
[request setPredicate: predicate2 ];
[self executeFetchRequest:request
inContext:childContext2
completion:^(NSArray *results, NSError *error) {
if (results) {
for (TestEntity *d in results) {
NSLog(#"++++++++++ e from fetchRequest %# with name = '%#'", d, d.name);
}
} else {
NSLog(#"Handle this error: %#", error);
}
}];
We switched over to XCode7 and I just ran into a similar deadlock issue with performBlockAndWait in code that works fine in XCode6.
The issue seems to be an upstream use of dispatch_async(mainQueue, ^{ ... to pass back the result from a network operation. That call was no longer needed after we added concurrency support for CoreData, but somehow it was left and never seemed to cause a problem until now.
It's possible that Apple changed something behind the scenes to make potential deadlocks more explicit.
I am trying to create a signal that will cancel a NSURLSessionDataTask upon disposal. The problem is that I am not able to wait for the task to finish until I can send next values (implementing Server-sent Events), but I have to use the NSURLSessions delegate methods.
What I am doing right now is creating a RACSubject and returning it for every new request. Upon new events arrive, I sendNext: on the subject. The problem I have is figuring out when to efficiently cancel the task, if there are no more subscribers on the subject.
A workaround I found so far is creating a dummy signal and merging it with the subject (see below).
return [[RACSignal createSignal:^RACDisposable *(id <RACSubscriber> subscriber) {
return [RACDisposable disposableWithBlock:^{
if ( dataTask.state != NSURLSessionTaskStateCanceling && dataTask.state != NSURLSessionTaskStateRunning ) {
[dataTask cancel];
}
}];
}]
merge:self.requests[#(dataTask.taskIdentifier)][kSubjectKey]];
But there has to be a more elegant way, or? Plus a downside is, that the signal will never complete. If I sendCompleted within the dummy signal, the dispose block will be called immediately.
I am using ReactiveCocoa 2.5.x
Have you checked out this library. Basically all you need is to turn the delegate methods into blocks and then you can use declare the blocks inside of creatSignal:call. Check out this post if you want to wrap the delegate methods into blocks yourself.
In my app, I am using ReactiveCocoa to return signals to notify me when async api calls are completed (successfully or not). On the POST for saving the data, it only takes one object at a time:
- (RACSignal *)postJSONData:(NSDictionary *)dict toRelativeURL:(NSString *)urlString;.
The function that returns a RACSignal sends the subscriber a next:
[subscriber sendNext:json]or an Error: [subscriber sendError:jsonError].
This works great when saving a single object but I also have a scenario where I have to save multiple objects. These objects can be saved in any order (i.e. they are not dependent on one another) or sequentially - it doesn't matter to me.
I need to update the UI indicating the overall progress (Saving 1 of 4, Saving 2 of 4....) as well as a final progress update (Completed 4 of 4) and specific action to take when all have been processed (successful or not).
There are a number of ways to do this, but I'd like to do this the proper way using ReactiveCocoa. I'm thinking I either can do this with a concat: or then: with a rac_sequence map:^, but I'm not sure. On their github page, they show an example of addressing parallel work streams, but they use 2 discretely defined signals. I won't have my signals until I loop through each object I need to save. Would love some guidance or an example (even better!). Thanks in advance.
I'm doing something similar in my app where I start 3 different async network calls and combine them all into one signal that I can listen to. Basically I loop through all my objects and store the network signal in an array. I then call merge: and pass it the array of network signals.
NSMutableArray *recievedNames = [NSMutableArray new];
NSMutableArray *signals = [NSMutableArray new];
//go though each database that has been added and grab a signal for the network request
for (GLBarcodeDatabase *database in self.databases) {
[signals addObject:[[[[self.manager rac_GET:[database getURLForDatabaseWithBarcode:barcode] parameters:nil] map:^id(RACTuple *value) {
return [((NSDictionary *)value.second) valueForKeyPath:database.path];
}] doError:^(NSError *error) {
NSLog(#"Error while fetching name from database %#", error);
}]
}
//forward all network signals into one signal
return [[[RACSignal merge:signals] doNext:^(NSString *x) {
[recievedNames addObject:x];
}] then:^RACSignal *{
return [RACSignal return:[self optimalNameForBarcodeProductWithNameCollection:recievedNames]];
}];
Feel free to ask me questions about any of the operators I have used and I will do my best to explain them.
I am just learning ReactiveCocoa myself but I wanted to point out something important which also agrees with lightice11's answer. concat combines signals sequentially. Meaning you won't get anything from #2 or #3 until #1 completes. merge on the other hand interleaves the responses returning whatever comes next regardless of order. So for your scenario, it sounds like you really do want merge.
To quote the man, Justin Spahr-Summers:
concat joins the streams sequentially, merge joins the signals on an as-soon-as-values-arrive basis, switch only passes through the events from the latest signal.
I have an NSOperationQueue that handles importing data from a web server on a loop. It accomplishes this with the following design.
NSURLConnect is wrapped in an NSOperation and added to the Queue
On successful completion of the download (using a block), the data from the request is wrapped in another NSOperation that adds the relevant data to Core Data. This operation is added to the queue.
On successful completion (using another block), (and after a specified delay) I call the method that started it all and return to step 1. Thus, i make another server call x seconds later.
This works great. I'm able to get data from the server and handle everything on the background. And because these are just NSOperations I'm able to put everything in the background, and perform multiple requests at a time. This works really well.
The ONLY problem that I currently have is that I'm unable to successfully cancel the operations once they are going.
I've tried something like the following :
- (void)flushQueue
{
self.isFlushingQueue = YES;
[self.operationQueue cancelAllOperations];
[self.operationQueue waitUntilAllOperationsAreFinished];
self.isFlushingQueue = NO;
NSLog(#"successfully flushed Queue");
}
where self.isFlushingQueue is a BOOL that I use to check before adding any new operations to the queue. This seems like it should work, but in fact it does not. Any ideas on stopping my Frankenstein creation?
Edit (Solved problem, but from a different perspective)
I'm still baffled about why exactly I was unable to cancel these operations (i'd be happy to keep trying possible solutions), but I had a moment of insight on how to solve this problem in a slightly different way. Instead of dealing at all with canceling operations, and waiting til queue is finished, I decided to just have a data structure (NSMutableDictionary) that had a list of all active connections. Something like this :
self.activeConnections = [NSMutableDictionary dictionaryWithDictionary:#{
#"UpdateContacts": #YES,
#"UpdateGroups" : #YES}];
And then before I add any operation to the queue, I simply ask if that particular call is On or Off. I've tested this, and I successfully have finite control over each individual server request that I want to be looping. To turn everything off I can just set all connections to #NO.
There are a couple downsides to this solution (Have to manually manage an additional data structure, and every operation has to start again to see if it's on or off before it terminates).
Edit -- In pursuit of a more accurate solution
I stripped out all code that isn't relevant (notice there is no error handling). I posted two methods. The first is an example of how the request NSOperation is created, and the second is the convenience method for generating the completion block.
Note the completion block generator is called by dozens of different requests similar to the first method.
- (void)updateContactsWithOptions:(NSDictionary*)options
{
//Hard coded for ease of understanding
NSString *contactsURL = #"api/url";
NSDictionary *params = #{#"sortBy" : #"LastName"};
NSMutableURLRequest *request = [self createRequestUsingURLString:contactsURL andParameters:params];
ConnectionCompleteBlock processBlock = [self blockForImportingDataToEntity:#"Contact"
usingSelector:#selector(updateContactsWithOptions:)
withOptions:options andParsingSelector:#selector(requestUsesRowsFromData:)];
BBYConnectionOperation *op = [[BBYConnectionOperation alloc] initWithURLRequest:request
andDelegate:self
andCompletionBlock:processBlock];
//This used to check using self.isFlushingQueue
if ([[self.activeConnections objectForKey:#"UpdateContacts"] isEqualToNumber:#YES]){
[self.operationQueue addOperation:op];
}
}
- (ConnectionCompleteBlock) blockForImportingDataToEntity:(NSString*)entityName usingSelector:(SEL)loopSelector withOptions:(NSDictionary*)options andParsingSelector:(SEL)parseSelector
{
return ^(BOOL success, NSData *connectionData, NSError *error){
//Pull out variables from options
BOOL doesLoop = [[options valueForKey:#"doesLoop"] boolValue];
NSTimeInterval timeInterval = [[options valueForKey:#"interval"] integerValue];
//Data processed before importing to core data
NSData *dataToImport = [self performSelector:parseSelector withObject:connectionData];
BBYImportToCoreDataOperation *importOperation = [[BBYImportToCoreDataOperation alloc] initWithData:dataToImport
andContext:self.managedObjectContext
andNameOfEntityToImport:entityName];
[importOperation setCompletionBlock:^ (BOOL success, NSError *error){
if(success){
NSLog(#"Import %#s was successful",entityName);
if(doesLoop == YES){
dispatch_async(dispatch_get_main_queue(), ^{
[self performSelector:loopSelector withObject:options afterDelay:timeInterval];
});
}
}
}];
[self.operationQueue addOperation:importOperation];
};
}
Cancellation of an NSOperation is just a request, a flag that is set in NSOperation. It's up to your NSOperation subclass to actually action that request and cancel it's work. You then need to ensure you have set the correct flags for isExecuting and isFinished etc. You will also need to do this in a KVO compliant manner. Only once these flags are set is the operation finished.
There is an example in the documentation Concurrency Programming Guide -> Configuring Operations for Concurrent Execution. Although I understand that this example may not correctly account for all multi-threaded edge cases. Another more complex example is provided in the sample code LinkedImageFetcher : QRunLoopOperation
If you think you are responding to the cancellation request correctly then you really need to post your NSOperation subclass code to examine the problem any further.
Instead of using your own flag for when it is ok to add more operations, you could try the
- (void)setSuspended:(BOOL)suspend
method on NSOperationQueue? And before adding a new operation, check if the queue is suspended with isSuspended?
I have researched a ton of posts regarding Core Data on background threads, and I feel like I understand (on paper) what needs to be going on. I guess we'll see. I am working on migrating an existing OS X app to Core Data, and am having issues making new instances of my NSManagedObject on an async thread.
Here is a sample of the code I am running right after I have moved onto a background thread:
NSLog(#"JSON 1");
NSManagedObjectContext * context = [[NSManagedObjectContext alloc] init];
[context setPersistentStoreCoordinator:[[NSApp delegate] persistentStoreCoordinator]];
asset = (MTAssetInfo*)[NSEntityDescription insertNewObjectForEntityForName:#"Info" inManagedObjectContext:context];
NSLog(#"JSON 2");
The result is that the first log message (#"JSON 1") gets called 31 times, and the second one (#"JSON 2") is never called. The object isn't being made and returned correctly.
The model for this Info entity is quite complex with a few transformable attributes that may or may not be setup correctly. The weird thing is that similar code run on the main thread and the main MOC works great. No issues.
EDIT - Some more context
The async call originates from here:
for (NSNumber *sectionID in sectionsToShow) {
dispatch_group_async(group, queue, ^{
MTAssetInfo *asset = [self assetWithRefID:[sectionID unsignedIntegerValue]];
if (asset != nil) {
[sectionsLock lock];
[sectionsTemp addObject:asset];
[sectionsLock unlock];
}
});
}
The assetWithRefID method never returns with an object because of the other code snippet. It never successfully pulls an NSManagedObject out of the context on the background thread.
You are going to have to provide more information to get real help, but I bet your problem is an error happening in the NSManagedDocument background thread.
I'd register a NSNotificationCenter for ALL messages (name:nil object:nil) and just print them out. I bet you see a status change or error message in there that is failing.
You might want to try a #try/#catch block around it just to see if exceptions are being thrown.
Maybe it will give you more to go on.
One other suggestion... Swizzling isn't necessarily the right tool for production stuff, but it's almost unbeatable for debugging. I have method-swizled several entire classes, so that it sends a detailed NSNotification before/after each invocation.
It has saved me tons of time, and helped me track down some wicked bugs. Now, when something is going on in CoreData, I take out my set of classes, link them in, and see all the detail I want.
I know that does not exactly answer you question, but hopefully it will put you on the track so you can provide some more information and get it all fixed.
If that's too much for you, create a subclass and instantiate that, with a similar method for calling super. You can get a real idea of the entire flow pretty easily.