iOS -- Calling contentsOfDirectoryAtPath leaks memory in loop - ios

I'm not well versed in objective-c, so forgive me if this is a stupid question. I've created a background thread to parse a list of files that exists in a directory, where the files in the directory can change at any time.
I call "contentsOfDirectoryAtPath" in every iteration of the loop, and my allocations suddenly go over 300mb. I can't figure out how to get ARC to release the returned array. Can someone maybe point me in the right direction here?
-(void) offlineModeThread
{
NSString *dataDir = [ViewController getDataDirectory];
NSFileManager *nfm = [NSFileManager defaultManager];
while(1)
{
NSArray *files = [nfm contentsOfDirectoryAtPath:dataDir error:nil];
/*
if(files == nil)
break;
if([files count] <= 0)
{
files = nil;
[NSThread sleepForTimeInterval: 5.0f];
continue;
}
if(![ViewController obtainLock])
{
files = nil;
continue;
}
*/
//[ViewController releaseLock];
files = nil;
}
}
As you can see, i've tried releasing the array by setting 'files' to nil, but it doesn't work.

how are you creating this thread? NSThread? NSOperation and NSOperationQueue? GCD?
For NSThread and NSOperation, you're supposed to create your own autorelease pool as part of setting up the thread. That way temporary objects get managed correctly within that thread.
I believe GCD takes care of this detail for you.
In newer versions of Objective C (Objective C 2.0, I believe) you should use the new syntax
#autoreleasepool
{
//Code to use a local autorelease pool
}
For loops that generate a ton of temporary objects, you can create a local autorelease pool for any block of code. You could refactor your code like this:
while(1)
{
#autoreleasepool
{
NSArray *files = [nfm contentsOfDirectoryAtPath:dataDir error:nil];
/*
if(files == nil)
break;
if([files count] <= 0)
{
files = nil;
[NSThread sleepForTimeInterval: 5.0f];
continue;
}
if(![ViewController obtainLock])
{
files = nil;
continue;
}
*/
//[ViewController releaseLock];
files = nil;
}
}
That would cause it to create a new autorelease pool for each iteration of the loop, and drain it at the end (thus releasing all temporary objects created in that iteration.)
By the way, if you're using NSThread, don't. Learn how to use GCD instead. GCD (Grand Central Dispatch) makes MUCH more efficient use of system resources than NSThread. NSThread is based on POSIX threads, which are very expensive to create, and tie up physical memory on the device for the life of your application.
NSOperationQueues have been refactored in recent OS versions to use GCD under the covers, so they are similarly efficient internally, although harder to use than GCD for most things.

Well, I think that the answer is obviously. Take a look in the documentation.
Many programs create temporary objects that are autoreleased. These
objects add to the program’s memory footprint until the end of the
block. In many situations, allowing temporary objects to accumulate
until the end of the current event-loop iteration does not result in
excessive overhead; in some situations, however, you may create a
large number of temporary objects that add substantially to memory
footprint and that you want to dispose of more quickly. In these
latter cases, you can create your own autorelease pool block. At the
end of the block, the temporary objects are released, which typically
results in their deallocation thereby reducing the program’s memory
footprint.
And you can see this question.
So the solution is using your #autoreleasepool{} because you are creating many temporary object.

Related

Why sometime variable become NULL in block

I have some event from C++ written library which works in background thread:
virtual void OnData(const char* data)
{
NSLog(#"Here 'data' string is present %s", data);
#autoreleasepool {
NSString* sData= [NSString stringWithCString:data encoding:NSUTF8StringEncoding];
dispatch_async(dispatch_get_main_queue(), ^{
NSLog(#"Here _sometimes_ 'data'(%s) is nil (\0). But sData is always present %#", data, sData);
[callback OnData:sData];
});
};
}
And sometimes I have NULL(I suspect its garbage actually) in dispatch_async block in argument variable. But local NSString variable is always here. Why?
P.S. Do I actually must use #autoreleasepool in this situation?
You have no assurances about the lifespan of the buffer that const char *data was pointing to by the time the async block is performed. The data could be dangling pointer by that point (and should be assumed to be so). It's very dangerous to use C-style pointers in any asynchronous references or outside the context they were originally created.
You should either use memory managed objects (e.g. NSData, NSString, etc.) or, if you insist on using C-style pointers and need to reference this pointer in the asynchronous block, copy the data to your own buffer, use that buffer, and then free it when you're done using that buffer in your asynchronous routine. In this case, you have your sData, so just don't refer to data after that point, and you'll be fine.
P.S. You later ask whether you must use #autoreleasepool in this situation.
In short, in most cases, no additional autorelease pool is needed. Notably, when using Grand Central Dispatch (e.g. dispatch_async), it has its own autorelease pools, so you don't have to create one. And, when your main thread yield back to its run loop, again, it's pool is drained. In short, you only need manually created autorelease pools when instantiating your own NSThread objects.
Having said that, sometimes you will introduce autorelease pools if doing significant memory intensive operations prior to yielding back to the run loop. In that case, you'll add autorelease pools in order to reduce the peak memory usage of the app. But this would not appear to be one of those cases.
If you had something like this:
void CallOnData()
{
char *test = malloc(5 * sizeof(char));
strcpy(test, "test");
OnData(test);
free(test);
}
You should expect data to be "NULL" in the block.
And autorelease is not needed, assuming you're using ARC, which you should be.

When to use the autorelease pool [duplicate]

For the most part with ARC (Automatic Reference Counting), we don't need to think about memory management at all with Objective-C objects. It is not permitted to create NSAutoreleasePools anymore, however there is a new syntax:
#autoreleasepool {
…
}
My question is, why would I ever need this when I'm not supposed to be manually releasing/autoreleasing ?
EDIT: To sum up what I got out of all the anwers and comments succinctly:
New Syntax:
#autoreleasepool { … } is new syntax for
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
…
[pool drain];
More importantly:
ARC uses autorelease as well as release.
It needs an auto release pool in place to do so.
ARC doesn't create the auto release pool for you. However:
The main thread of every Cocoa app already has an autorelease pool in it.
There are two occasions when you might want to make use of #autoreleasepool:
When you are in a secondary thread and there is no auto release pool, you must make your own to prevent leaks, such as myRunLoop(…) { #autoreleasepool { … } return success; }.
When you wish to create a more local pool, as #mattjgalloway has shown in his answer.
ARC doesn't get rid of retains, releases and autoreleases, it just adds in the required ones for you. So there are still calls to retain, there are still calls to release, there are still calls to autorelease and there are still auto release pools.
One of the other changes they made with the new Clang 3.0 compiler and ARC is that they replaced NSAutoReleasePool with the #autoreleasepool compiler directive. NSAutoReleasePool was always a bit of a special "object" anyway and they made it so that the syntax of using one is not confused with an object so that it's generally a bit more simple.
So basically, you need #autoreleasepool because there are still auto release pools to worry about. You just don't need to worry about adding in autorelease calls.
An example of using an auto release pool:
- (void)useALoadOfNumbers {
for (int j = 0; j < 10000; ++j) {
#autoreleasepool {
for (int i = 0; i < 10000; ++i) {
NSNumber *number = [NSNumber numberWithInt:(i+j)];
NSLog(#"number = %p", number);
}
}
}
}
A hugely contrived example, sure, but if you didn't have the #autoreleasepool inside the outer for-loop then you'd be releasing 100000000 objects later on rather than 10000 each time round the outer for-loop.
Update:
Also see this answer - https://stackoverflow.com/a/7950636/1068248 - for why #autoreleasepool is nothing to do with ARC.
Update:
I took a look into the internals of what's going on here and wrote it up on my blog. If you take a look there then you will see exactly what ARC is doing and how the new style #autoreleasepool and how it introduces a scope is used by the compiler to infer information about what retains, releases & autoreleases are required.
#autoreleasepool doesn't autorelease anything. It creates an autorelease pool, so that when the end of block is reached, any objects that were autoreleased by ARC while the block was active will be sent release messages. Apple's Advanced Memory Management Programming Guide explains it thus:
At the end of the autorelease pool block, objects that received an autorelease message within the block are sent a release message—an object receives a release message for each time it was sent an autorelease message within the block.
People often misunderstand ARC for some kind of garbage collection or the like. The truth is that, after some time people at Apple (thanks to llvm and clang projects) realized that Objective-C's memory administration (all the retains and releases, etc.) can be fully automatized at compile time. This is, just by reading the code, even before it is run! :)
In order to do so there is only one condition: We MUST follow the rules, otherwise the compiler would not be able to automate the process at compile time. So, to ensure that we never break the rules, we are not allowed to explicitly write release, retain, etc. Those calls are Automatically injected into our code by the compiler. Hence internally we still have autoreleases, retain, release, etc. It is just we don't need to write them anymore.
The A of ARC is automatic at compile time, which is much better than at run time like garbage collection.
We still have #autoreleasepool{...} because having it does not break any of the rules, we are free create/drain our pool anytime we need it :).
Autorelease pools are required for returning newly created objects from a method. E.g. consider this piece of code:
- (NSString *)messageOfTheDay {
return [[NSString alloc] initWithFormat:#"Hello %#!", self.username];
}
The string created in the method will have a retain count of one. Now who shall balance that retain count with a release?
The method itself? Not possible, it has to return the created object, so it must not release it prior to returning.
The caller of the method? The caller does not expect to retrieve an object that needs releasing, the method name does not imply that a new object is created, it only says that an object is returned and this returned object may be a new one requiring a release but it may as well be an existing one that doesn't. What the method does return may even depend on some internal state, so the the caller cannot know if it has to release that object and it shouldn't have to care.
If the caller had to always release all returned object by convention, then every object not newly created would always have to be retained prior to returning it from a method and it would have to be released by the caller once it goes out of scope, unless it is returned again. This would be highly inefficient in many cases as one can completely avoid altering retain counts in many cases if the caller will not always release the returned object.
That's why there are autorelease pools, so the first method will in fact become
- (NSString *)messageOfTheDay {
NSString * res = [[NSString alloc] initWithFormat:#"Hello %#!", self.username];
return [res autorelease];
}
Calling autorelease on an object adds it to the autorelease pool, but what does that really mean, adding an object to the autorelease pool? Well, it means telling your system "I want you to to release that object for me but at some later time, not now; it has a retain count that needs to be balanced by a release otherwise memory will leak but I cannot do that myself right now, as I need the object to stay alive beyond my current scope and my caller won't do it for me either, it has no knowledge that this needs to be done. So add it to your pool and once you clean up that pool, also clean up my object for me."
With ARC the compiler decides for you when to retain an object, when to release an object and when to add it to an autorelease pool but it still requires the presence of autorelease pools to be able to return newly created objects from methods without leaking memory. Apple has just made some nifty optimizations to the generated code which will sometimes eliminate autorelease pools during runtime. These optimizations require that both, the caller and the callee are using ARC (remember mixing ARC and non-ARC is legal and also officially supported) and if that is actually the case can only be known at runtime.
Consider this ARC Code:
// Callee
- (SomeObject *)getSomeObject {
return [[SomeObject alloc] init];
}
// Caller
SomeObject * obj = [self getSomeObject];
[obj doStuff];
The code that the system generates, can either behave like the following code (that is the safe version that allows you to freely mix ARC and non-ARC code):
// Callee
- (SomeObject *)getSomeObject {
return [[[SomeObject alloc] init] autorelease];
}
// Caller
SomeObject * obj = [[self getSomeObject] retain];
[obj doStuff];
[obj release];
(Note the retain/release in the caller is just a defensive safety retain, it's not strictly required, the code would be perfectly correct without it)
Or it can behave like this code, in case that both are detected to use ARC at runtime:
// Callee
- (SomeObject *)getSomeObject {
return [[SomeObject alloc] init];
}
// Caller
SomeObject * obj = [self getSomeObject];
[obj doStuff];
[obj release];
As you can see, Apple eliminates the atuorelease, thus also the delayed object release when the pool is destroyed, as well as the safety retain. To learn more about how that is possible and what's really going on behind the scenes, check out this blog post.
Now to the actual question: Why would one use #autoreleasepool?
For most developers, there's only one reason left today for using this construct in their code and that is to keep the memory footprint small where applicable. E.g. consider this loop:
for (int i = 0; i < 1000000; i++) {
// ... code ...
TempObject * to = [TempObject tempObjectForData:...];
// ... do something with to ...
}
Assume that every call to tempObjectForData may create a new TempObject that is returned autorelease. The for-loop will create one million of these temp objects which are all collected in the current autoreleasepool and only once that pool is destroyed, all the temp objects are destroyed as well. Until that happens, you have one million of these temp objects in memory.
If you write the code like this instead:
for (int i = 0; i < 1000000; i++) #autoreleasepool {
// ... code ...
TempObject * to = [TempObject tempObjectForData:...];
// ... do something with to ...
}
Then a new pool is created every time the for-loop runs and is destroyed at the end of each loop iteration. That way at most one temp object is hanging around in memory at any time despite the loop running one million times.
In the past you often had to also manage autoreleasepools yourself when managing threads (e.g. using NSThread) as only the main thread automatically has an autorelease pool for a Cocoa/UIKit app. Yet this is pretty much legacy today as today you probably wouldn't use threads to begin with. You'd use GCD DispatchQueue's or NSOperationQueue's and these two both do manage a top level autorelease pool for you, created before running a block/task and destroyed once done with it.
It's because you still need to provide the compiler with hints about when it is safe for autoreleased objects to go out of scope.
Quoted from https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmAutoreleasePools.html:
Autorelease Pool Blocks and Threads
Each thread in a Cocoa application maintains its own stack of
autorelease pool blocks. If you are writing a Foundation-only program
or if you detach a thread, you need to create your own autorelease
pool block.
If your application or thread is long-lived and potentially generates
a lot of autoreleased objects, you should use autorelease pool blocks
(like AppKit and UIKit do on the main thread); otherwise, autoreleased
objects accumulate and your memory footprint grows. If your detached
thread does not make Cocoa calls, you do not need to use an
autorelease pool block.
Note: If you create secondary threads using the POSIX thread APIs
instead of NSThread, you cannot use Cocoa unless Cocoa is in
multithreading mode. Cocoa enters multithreading mode only after
detaching its first NSThread object. To use Cocoa on secondary POSIX
threads, your application must first detach at least one NSThread
object, which can immediately exit. You can test whether Cocoa is in
multithreading mode with the NSThread class method isMultiThreaded.
...
In Automatic Reference Counting, or ARC, the system uses the same
reference counting system as MRR, but it insertsthe appropriate memory
management method callsfor you at compile-time. You are strongly
encouraged to use ARC for new projects. If you use ARC, there is
typically no need to understand the underlying implementation
described in this document, although it may in some situations be
helpful. For more about ARC, see Transitioning to ARC Release Notes.
TL;DR
Why is #autoreleasepool still needed with ARC?
#autoreleasepool is used by Objective-C and Swift to work with autorelese inside
When you work with pure Swift and allocate Swift objects - ARC handles it
But if you decide call/use Foundation/Legacy Objective-C code(NSData, Data) which uses autorelese inside then #autoreleasepool in a rescue
//Swift
let imageData = try! Data(contentsOf: url)
//Data init uses Objective-C code with [NSData dataWithContentsOfURL] which uses `autorelese`
Long answer
MRC, ARC, GC
Manual Reference Counting(MRC) or Manual Retain-Release(MRR) as a developer you are responsible for counting references on objects manually
Automatic Reference Counting(ARC) was introduced in iOS v5.0 and OS X Mountain Lion with xCode v4.2
Garbage Collection(GC) was available for Mac OS and was deprecated in OS X Mountain Lion. Must Move to ARC
Reference count in MRC and ARC
//MRC
NSLog(#"Retain Count: %d", [variable retainCount]);
//ARC
NSLog(#"Retain Count: %ld", CFGetRetainCount((__bridge CFTypeRef) variable));
Every object in heap has an integer value which indicates how many references are pointed out on it. When it equals to 0 object is deallocated by system
Allocating object
Working with Reference count
Deallocating object. deinit is called when retainCount == 0
MRC
A *a1 = [[A alloc] init]; //this A object retainCount = 1
A *a2 = a1;
[a2 retain]; //this A object retainCount = 2
// a1, a2 -> object in heap with retainCount
Correct way to release an object:
release If only this - dangling pointer. Because it still can point on the object in heap and it is possible to send a message
= nil If only this - memory leak. deinit will not be called
A *a = [[A alloc] init]; //++retainCount = 1
[a release]; //--retainCount = 0
a = nil; //guarantees that even somebody else has a reference to the object, and we try to send some message thought variable `a` this message will be just skipped
Working with Reference count(Object owner rules):
(0 -> 1) alloc, new, copy, mutableCopy
(+1) retain You are able to own an object as many times as you need(you can call retain several times)
(-1) release If you an owner you must release it. If you release more than retainCount it will be 0
(-1) autorelease Adds an object, which should be released, to autorelease pool. This pool will be processed at the end of RunLoop iteration cycle(it means when all tasks will be finished on the stack)[About] and after that release will be applied for all objects in the pool
(-1) #autoreleasepool Forces process an autorelease pool at the end of block. It is used when you deal with autorelease in a loop and want to clear resources ASAP. If you don't do it your memory footprint will be constantly increasing
autorelease is used in method calls when you allocate a new object there and return it
- (B *)foo {
B *b1 = [[B alloc] init]; //retainCount = 1
//fix - correct way - add it to fix wrong way
//[b1 autorelease];
//wrong way(without fix)
return b;
}
- (void)testFoo {
B *b2 = [a foo];
[b2 retain]; //retainCount = 2
//some logic
[b2 release]; //retainCount = 1
//Memory Leak
}
#autoreleasepool example
- (void)testFoo {
for(i=0; i<100; i++) {
B *b2 = [a foo];
//process b2
}
}
ARC
One of biggest advantage of ARC is that it automatically insert retain, release, autorelease under the hood in Compile Time and as developer you should not take care of it anymore
Enable/Disable ARC
//enable
-fobjc-arc
//disable
-fno-objc-arc
Variants from more to less priority
//1. local file - most priority
Build Phases -> Compile Sources -> Compiler Flags(Select files -> Enter)
//2. global
Build Settings -> Other C Flags(OTHER_CFLAGS)
//3. global
Build Settings -> Objective-C Automatic Reference Counting(CLANG_ENABLE_OBJC_ARC)
Check if ARC is enabled/disabled
Preprocessor __has_feature function is used
__has_feature(objc_arc)
Compile time
// error if ARC is Off. Force to enable ARC
#if ! __has_feature(objc_arc)
#error Please enable ARC for this file
#endif
//or
// error if ARC is On. Force to disable ARC
#if __has_feature(objc_arc)
#error Please disable ARC for this file
#endif
Runtime
#if __has_feature(objc_arc)
// ARC is On
NSLog(#"ARC on");
#else
// ARC is Off
NSLog(#"ARC off");
#endif
Reverse engineering(for Objective-C)
//ARC is enabled
otool -I -v <binary_path> | grep "<mrc_message>"
//e.g.
otool -I -v "/Users/alex/ARC_experiments.app/ARC_experiments" | grep "_objc_release"
//result
0x00000001000080e0 748 _objc_release
//<mrc_message>
_objc_retain
_objc_release
_objc_autoreleaseReturnValue
_objc_retainAutoreleaseReturnValue
_objc_retainAutoreleasedReturnValue
_objc_storeStrong
Tool to Migrate Objective-C MRC to ARC
ARC generates errors where you should manually remove retain, release, autorelease and others issues
Edit -> Convert -> To Objective-C ARC...
New Xcode with MRC
If you enable MRC you get next errors(warnings)(but the build will be successful)
//release/retain/autorelease/retainCount
'release' is unavailable: not available in automatic reference counting mode
ARC forbids explicit message send of 'release'
There seems to be a lot of confusion on this topic (and at least 80 people who probably are now confused about this and think they need to sprinkle #autoreleasepool around their code).
If a project (including its dependencies) exclusively uses ARC, then #autoreleasepool never needs to be used and will do nothing useful. ARC will handle releasing objects at the correct time. For example:
#interface Testing: NSObject
+ (void) test;
#end
#implementation Testing
- (void) dealloc { NSLog(#"dealloc"); }
+ (void) test
{
while(true) NSLog(#"p = %p", [Testing new]);
}
#end
displays:
p = 0x17696f80
dealloc
p = 0x17570a90
dealloc
Each Testing object is deallocated as soon as the value goes out of scope, without waiting for an autorelease pool to be exited. (The same thing happens with the NSNumber example; this just lets us observe the dealloc.) ARC does not use autorelease.
The reason #autoreleasepool is still allowed is for mixed ARC and non-ARC projects, which haven't yet completely transitioned to ARC.
If you call into non-ARC code, it may return an autoreleased object. In that case, the above loop would leak, since the current autorelease pool will never be exited. That's where you'd want to put an #autoreleasepool around the code block.
But if you've completely made the ARC transition, then forget about autoreleasepool.

How do I properly dealloc and release objects being referred to in a while loop

The following code creates a memory leak. An asynchronous background process downloads images in tmp_pack_folder and another background thread is checking if the image count matches the total count expected, and then makes the images available to users once the download is complete.
The issue is that if the background process that is downloading images to the tmp_pack_folder fails for some reason, the following code becomes an infinite loop. This is a rare case, but when it does there is a memory leak. getAllFileNamesinFolder method is actually calling contentsOfDirectoryAtPath:bundleRoot of NSFileManager and it is called repeatedly. How to do I properly deallocate memory in this case (apart from preventing the infinite loop to begin with)
NSString *tmp_pack_folder = [packid stringByAppendingString:#"_tmp"];
if([fileMgr folderExists: tmp_pack_folder]){
NSArray *packImages = [fileMgr getAllFileNamesInFolder:tmp_pack_folder];
while(packImages.count != arrImages.count ){
packImages = [fileMgr getAllFileNamesInFolder:tmp_pack_folder]; //get the contents of the folder again.
if(cancel==YES){
break;
}
}
}
You say that you will rework this to "prevent the infinite loop." You should take that a step further and eliminate the loop altogether. If you ever find yourself with code that loops, polling some status, there's invariably an alternate, more efficient design. Bottom line, your memory situation is not the real problem: It's merely a symptom of a broader design issue.
I'd advise you move to an event-driven approach. So, rather than having a method that repeatedly performs the "am I done yet" logic, you should only check this status when triggered by the appropriate event (i.e. only when a download finishes/fails, and not before). This loop is probably causing to your memory problem, so don't fix the memory problem, but rather eliminate the loop altogether.
In answer to your question, one possible source of the memory problem arises from autorelease objects. These are objects that are allocated in such a manner that they are not released immediately when you're done with them, but rather only when the autorelease pool is drained (which generally happens for you automatically when you yield back to the app's run loop). But if you have some large loop that you repeatedly call, you end up adding lots of objects to an autorelease pool that isn't drained in a timely manner.
In some special cases, if you truly needed some loop (and to be clear, that's not the case here; you neither need nor want a loop in this case), you could employ your own custom #autoreleasepool, through which you'd effectively control the frequency of the draining of the pool.
But, at the risk of belaboring the point, this is simply not one of those situations. Don't use your own autorelease pool. Get rid of the loop. Only trigger the "am I done yet" logic when a download finishes/fails, and your problem should go away.
It's too bad Objective-C doesn't give us javascript-like promises. The way I solve this problem is by giving my asynch task a caller's interface like this:
- (void)doAsynchThingWithParams:(id)params completion:(void (^)(id))completion;
The params parameterize whatever the task is, and the completion handler takes result of the task.
This let's me treat several concurrent tasks like a todo list, with a completion handler that gets called with all the results once they've arrived.
// array is an array of params for each task e.g. urls for making url requests
// completion is called when all are complete with an array of results
- (void)doManyThingsWithParams:(NSArray *)array completion:(void (^)(NSArray *))completion {
NSMutableArray *todoList = [array mutableCopy];
NSMutableArray *results = [NSMutableArray array];
// results will always have N elements, one for each task
// nulls can be replaced by either good results or NSErrors
for (int i=0; i<array.count; ++i) results[i] = [NSNull null];
for (id params in array) {
[self doAsynchThingWithParams:params completion:^(id result) {
if (result) {
NSInteger index = [array indexOfObject:params];
[results replaceObjectAtIndex:index withObject:result];
}
[todoList removeObject:params];
if (!todoList.count) completion(results);
}];
}
}

ios memory going up very fast

I have a pretty general question here.
What would you do in general to find who's taking your memory?
I have a video encoder, the setup is pretty complex, the images are into a controller and the encoder is in another and i'm asking for the images and get them through delegates which are sometimes going through many levels of controllers, and i'm also using some dispatch_async calls in the process. Images are snapshots of an UIView and processed with CoreGraphics, i'm retaining the final image and releasing it in the other controller after use. Everything works fine, the memory is around 25Mb constantly, but what happens is that after I finish the encoding the memory is going up very fast, in maximum a minute is going from 25Mb to 330Mb and is of course crashing. I tried to put logs and see if is still asking for images but doesn't seem to be any problem, the encoder stops as expected. The encoder is set to run in background.
One important thing is that if I try to find leaks (or allocations because leaks are not reporting anything with ARC) the app is crashing sooner, but not because of the memory. I suspect that I messed the dispatches somehow and because of some delays caused by instruments something is not available at a specified time. However I have troubles finding this too without logs. Can I see logs when i'm debugging with instruments?
Thanks for any info that will help.
Edit: I succeeded to run the instruments with the allocs without doing anything, seems the crash is not consistent. I saved the instruments report and you can see how's the memory going up, there's an alloc that is causing this and i think the question resumes to how to read it. The file is here http://ge.tt/1PF97Pj/v/0?c
The problem here is that you're "busy-waiting" on adaptor.assetWriterInput.readyForMoreMediaData, -- i.e. calling it over and over in a tight loop. This is, generally speaking, bad practice. The headers state that this property is Key-Value Observable, so you would be better off restructuring your code to listen for Key-Value change notifications in order to advance the overall process. Even worse, depending on how AVAssetInputWriter works (I'm not sure if it's run-loop based or not), the act of busy-waiting here may actually prevent the asset input writer from doing any real work, since the run loop may be effectively deadlocked waiting for work to be done that might not happen until you let the run loop continue.
Now you may be asking yourself: How is busy-waiting causing memory pressure? It's causing memory pressure because behind the scenes, readyForMoreMediaData is causing autoreleased objects to be allocated every time you call it. Because you busy-wait on this value, checking it over and over in a tight loop, it just allocates more and more objects, and they never get released, because the run loop never has a chance to pop the autorelease pool for you. (see below for more detail about what the allocations are really for) If you wanted to continue this (ill-advised) busy-waiting, you could mitigate your memory issue by doing something like this:
BOOL ready = NO;
do {
#autoreleasepool {
ready = adaptor.assetWriterInput.readyForMoreMediaData;
}
} while (!ready);
This will cause any autoreleased objects created by readyForMoreMediaData to be released after each check. But really, you would be much better served in the long run by restructuring this code to avoid busy-waiting. If you absolutely must busy-wait, at least do something like usleep(500); on each pass of the loop, so you're not thrashing the CPU as much. But don't busy-wait.
EDIT: I also see that you wanted to understand how to figure this out from Instruments. Let me try to explain. Starting from the file you posted, here's what I did:
I clicked on the Allocations row in the top pane
Then I selected the "Created & Still Living" option (because if the things were getting destroyed, we wouldn't be seeing heap growth.)
Next, I applied a time filter by Option-dragging a small range in the big "ramp" that you see.
At this point, the window looks like this:
Here I see that we have tons of very similar 4K malloc'ed objects in the list. This is the smoking gun.
Now I select one of those, and expand the right pane of the window to show me a stack trace.
At this point, the window looks like this:
In the right panel we see the stack trace where that object is being created, and we see that it's being alloced way down in AVAssetInputWriter, but the first function below (visually above) the last frame in your code is -[AVAssetWriterInput isReadForMoreMediaData]. The autorelease in the backtrace there is a hint that this is related to autoreleased objects, and sitting in a tight loop like that, the standard autorelease mechanism never gets a chance to run (i.e. pop the current pool).
My conclusion from this stack is that something in -[AVAssetWriterInput isReadForMoreMediaData], (probably the _helper function in the next stack frame) does a [[foo retain] autorelease] before returning its result. The autorelease mechanism needs to keep track of all the things that have been autoreleased until the autorelease pool is popped/drained. In order to keep track those, it needs to allocate space for its "list of things waiting to be autoreleased". That's my guess as to why these are malloc blocks and not autoreleased objects. (i.e. there aren't any objects being allocated, but rather just space to keep track of all the autorelease operations that have happened since the pool was pushed -- of which there are MANY because you're checking this property in a tight loop.)
That's how I diagnosed the issue. Hopefully that will help you in the future.
To answer my question, the memory issue is fixed if i remove the dispatch_async calls, however now my UI is blocked which is not good at all. It should be a way to combine all this so i do not block it. Here is my code
- (void) image:(CGImageRef)cgimage atFrameTime:(CMTime)frameTime {
//NSLog(#"> ExporterController image");
NSLog(#"ExporterController image atFrameTime %lli", frameTime.value);
if (!self.isInBackground && frameTime.value % 20 == 0) {
dispatch_async(dispatch_get_main_queue(),^{
//logo.imageView.image = [UIImage imageWithCGImage:cgimage];
statusLabel.text = [NSString stringWithFormat:#"%i%%", frameCount/**100/self.videoMaximumFrames*/];
});
}
if (cgimage == nil || prepareForCancel) {
NSLog(#"FINALIZE THE VIDEO PREMATURELY cgimage == nil or prepareForCancel is YES");
[self finalizeVideo];
[logo stop];
return;
}
// Add the image to the video file
//dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0),^{
NSLog(#"ExporterController buffer");
CVPixelBufferRef buffer = [self pixelBufferFromCGImage:cgimage andSize:videoSize];
NSLog(#"ExporterController buffer ok");
BOOL append_ok = NO;
int j = 0;
while (!append_ok && j < 30) {
if (adaptor.assetWriterInput.readyForMoreMediaData) {
//printf("appending framecount %d, %lld %d\n", frameCount, frameTime.value, frameTime.timescale);
append_ok = [adaptor appendPixelBuffer:buffer withPresentationTime:frameTime];
if (buffer) CVBufferRelease(buffer);
while (!adaptor.assetWriterInput.readyForMoreMediaData) {}
}
else {
printf("adaptor not ready %d, %d\n", frameCount, j);
//[NSThread sleepForTimeInterval:0.1];
while(!adaptor.assetWriterInput.readyForMoreMediaData) {}
}
j++;
}
if (!append_ok) {
printf("error appending image %d times %d\n", frameCount, j);
}
NSLog(#"ExporterController cgimage alive");
CGImageRelease(cgimage);
NSLog(#"ExporterController cgimage released");
//});
frameCount++;
if (frameCount > 100) {
NSLog(#"FINALIZING VIDEO");
//dispatch_async(dispatch_get_main_queue(),^{
[self finalizeVideo];
//});
}
else {
NSLog(#"ExporterController prepare for next one");
//dispatch_async(dispatch_get_main_queue(),^{
//dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0),^{
NSLog(#"ExporterController requesting next image");
[self.slideshowDelegate requestImageForFrameTime:CMTimeMake (frameCount, (int32_t)kRecordingFPS)];
//});
}
}

Does #synchronized guarantees for thread safety or not?

With reference to this answer, I am wondering is this correct?
#synchronized does not make any code "thread-safe"
As I tried to find any documentation or link to support this statement, for no success.
Any comments and/or answers will be appreciated on this.
For better thread safety we can go for other tools, this is known to me.
#synchronized does make code thread safe if it is used properly.
For example:
Lets say I have a class that accesses a non thread safe database. I don't want to read and write to the database at the same time as this will likely result in a crash.
So lets say I have two methods. storeData: and readData on a singleton class called LocalStore.
- (void)storeData:(NSData *)data
{
[self writeDataToDisk:data];
}
- (NSData *)readData
{
return [self readDataFromDisk];
}
Now If I were to dispatch each of these methods onto their own thread like so:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[[LocalStore sharedStore] storeData:data];
});
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[[LocalStore sharedStore] readData];
});
Chances are we would get a crash. However if we change our storeData and readData methods to use #synchronized
- (void)storeData:(NSData *)data
{
#synchronized(self) {
[self writeDataToDisk:data];
}
}
- (NSData *)readData
{
#synchronized(self) {
return [self readDataFromDisk];
}
}
Now this code would be thread safe. It is important to note that if I remove one of the #synchronized statements however the code would no longer be thread safe. Or if I were to synchronize different objects instead of self.
#synchronized creates a mutex lock on the object you are syncrhonizing. So in other words if any code wants to access code in a #synchronized(self) { } block it will have to get in line behind all previous code running within in that same block.
If we were to create different localStore objects, the #synchronized(self) would only lock down each object individually. Does that make sense?
Think of it like this. You have a whole bunch of people waiting in separate lines, each line is numbered 1-10. You can choose what line you want each person to wait in (by synchronizing on a per line basis), or if you don't use #synchronized you can jump straight to the front and skip all the lines. A person in line 1 doesn't have to wait for a person in line 2 to finish, but the person in line 1 does have to wait for everyone in front of them in their line to finish.
I think the essence of the question is:
is the proper use of synchronize able to solve any thread-safe
problem?
Technically yes, but in practice it's advisable to learn and use other tools.
I'll answer without assuming previous knowledge.
Correct code is code that conforms to its specification. A good specification defines
invariants constraining the state,
preconditions and postconditions describing the effects of the operations.
Thread-safe code is code that remains correct when executed by multiple threads. Thus,
No sequence of operations can violate the specification.1
Invariants and conditions will hold during multithread execution without requiring additional synchronization by the client2.
The high level takeaway point is: thread-safe requires that the specification holds true during multithread execution. To actually code this, we have to do just one thing: regulate the access to mutable shared state3. And there are three ways to do it:
Prevent the access.
Make the state immutable.
Synchronize the access.
The first two are simple. The third one requires preventing the following thread-safety problems:
liveness
deadlock: two threads block permanently waiting for each other to release a needed resource.
livelock: a thread is busy working but it's unable to make any progress.
starvation: a thread is perpetually denied access to resources it needs in order to make progress.
safe publication: both the reference and the state of the published object must be made visible to other threads at the same time.
race conditions A race condition is a defect where the output is dependent on the timing of uncontrollable events. In other words, a race condition happens when getting the right answer relies on lucky timing. Any compound operation can suffer a race condition, example: “check-then-act”, “put-if-absent”. An example problem would be if (counter) counter--;, and one of several solutions would be #synchronize(self){ if (counter) counter--;}.
To solve these problems we use tools like #synchronize, volatile, memory barriers, atomic operations, specific locks, queues, and synchronizers (semaphores, barriers).
And going back to the question:
is the proper use of #synchronize able to solve any thread-safe
problem?
Technically yes, because any tool mentioned above can be emulated with #synchronize. But it would result in poor performance and increase the chance of liveness related problems. Instead, you need to use the appropriate tool for each situation. Example:
counter++; // wrong, compound operation (fetch,++,set)
#synchronize(self){ counter++; } // correct but slow, thread contention
OSAtomicIncrement32(&count); // correct and fast, lockless atomic hw op
In the case of the linked question you could indeed use #synchronize, or a GCD read-write lock, or create a collection with lock stripping, or whatever the situation calls for. The right answer depend on the usage pattern. Any way you do it, you should document in your class what thread-safe guarantees are you offering.
1 That is, see the object on an invalid state or violate the pre/post conditions.
2 For example, if thread A iterates a collection X, and thread B removes an element, execution crashes. This is non thread-safe because the client will have to synchronize on the intrinsic lock of X (synchronize(X)) to have exclusive access. However, if the iterator returns a copy of the collection, the collection becomes thread-safe.
3 Immutable shared state, or mutable non shared objects are always thread-safe.
Generally, #synchronized guarantees thread safety, but only when used correctly. It is also safe to acquire the lock recursively, albeit with limitations I detail in my answer here.
There are several common ways to use #synchronized wrong. These are the most common:
Using #synchronized to ensure atomic object creation.
- (NSObject *)foo {
#synchronized(_foo) {
if (!_foo) {
_foo = [[NSObject alloc] init];
}
return _foo;
}
}
Because _foo will be nil when the lock is first acquired, no locking will occur and multiple threads can potentially create their own _foo before the first completes.
Using #synchronized to lock on a new object each time.
- (void)foo {
#synchronized([[NSObject alloc] init]) {
[self bar];
}
}
I've seen this code quite a bit, as well as the C# equivalent lock(new object()) {..}. Since it attempts to lock on a new object each time, it will always be allowed into the critical section of code. This is not some kind of code magic. It does absolutely nothing to ensure thread safety.
Lastly, locking on self.
- (void)foo {
#synchronized(self) {
[self bar];
}
}
While not by itself a problem, if your code uses any external code or is itself a library, it can be an issue. While internally the object is known as self, it externally has a variable name. If the external code calls #synchronized(_yourObject) {...} and you call #synchronized(self) {...}, you may find yourself in deadlock. It is best to create an internal object to lock upon that is not exposed outside of your object. Adding _lockObject = [[NSObject alloc] init]; inside your init function is cheap, easy, and safe.
EDIT:
I still get asked questions about this post, so here is an example of why it is a bad idea to use #synchronized(self) in practice.
#interface Foo : NSObject
- (void)doSomething;
#end
#implementation Foo
- (void)doSomething {
sleep(1);
#synchronized(self) {
NSLog(#"Critical Section.");
}
}
// Elsewhere in your code
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
Foo *foo = [[Foo alloc] init];
NSObject *lock = [[NSObject alloc] init];
dispatch_async(queue, ^{
for (int i=0; i<100; i++) {
#synchronized(lock) {
[foo doSomething];
}
NSLog(#"Background pass %d complete.", i);
}
});
for (int i=0; i<100; i++) {
#synchronized(foo) {
#synchronized(lock) {
[foo doSomething];
}
}
NSLog(#"Foreground pass %d complete.", i);
}
It should be obvious to see why this happens. Locking on foo and lock are called in different orders on the foreground VS background threads. It's easy to say that this is bad practice, but if Foo is a library, the user is unlikely to know that the code contains a lock.
#synchronized alone doesn't make code thread safe but it is one of the tools used in writing thread safe code.
With multi-threaded programs, it's often the case of a complex structure that you want to be maintained in a consistent state and you want only one thread to have access at a time. The common pattern is to use a mutex to protect a critical section of code where the structure is accessed and/or modified.
#synchronized is thread safe mechanism. Piece of code written inside this function becomes the part of critical section, to which only one thread can execute at a time.
#synchronize applies the lock implicitly whereas NSLock applies it explicitly.
It only assures the thread safety, not guarantees that. What I mean is you hire an expert driver for you car, still it doesn't guarantees car wont meet an accident. However probability remains the slightest.
It's companion in GCD(grand central dispatch) is dispatch_once. dispatch_once does the same work as to #synchronized.
The #synchronized directive is a convenient way to create mutex locks on the fly in Objective-C code.
side-effects of mutex locks:
deadlocks
starvation
Thread safety will depend on usage of #synchronized block.

Resources