Memory leak using dataWithContentsOfURL in a loop - ios

Question

I'm using Xcode 4.6.3 and iOS 5.5/6.1.6 .
I am using a background thread to load large quantities of jpg from a server to iOS devices.
dispatch_async(kBgQueue, ^
{
// get the array of filenames to download
NSURL* url = [NSURL URLWithString:webPath];
NSArray* theArray = [NSArray arrayWithContentsOfURL:url];
if( theArray )
{
dispatch_async(dispatch_get_main_queue(), ^{
// disable screen buttons
[self setButtons:false];
});
[self loadImagesFromList:theArray sourceBundle:bundlePath destBundle:localBundlePath manager:manager];
if (!stopFlag) {
// if no memory error has occurred
NSLog(#"calling refresh after load_images");
dispatch_async(dispatch_get_main_queue(), ^{
[self refresh];
});
}
theArray = nil;
}
else
{
NSLog(#"Error loading bundle");
}
});
The background method:
-(void)loadImagesFromList:(NSArray *)theArray
sourceBundle:(NSString *)bundlePath
destBundle:(NSString *)localBundlePath
manager:(NSFileManager *)manager {
// initialize the progress and activity indicator
dispatch_async(dispatch_get_main_queue(), ^{
[self.activityIndictor startAnimating];
[UIApplication sharedApplication].networkActivityIndicatorVisible = YES;
[self.progressIndictor setProgress:0 animated:NO];
});
NSURL *url;
NSString *srcFile;
NSString *destFile;
NSError *error = nil;
int counter = 0;
float prog = 0;
float increment = 1.0 / [theArray count];
float stepSize = [theArray count] / 10;
for (NSString *file in theArray)
{
if (stopFlag) {
NSLog(#"I see stopFlag = true, counter = %d, prog = %f", counter, prog);
return;
}
srcFile = [bundlePath stringByAppendingPathComponent:file];
destFile = [localBundlePath stringByAppendingPathComponent:file];
counter += 1;
prog += increment;
if (counter == stepSize) {
dispatch_async(dispatch_get_main_queue(), ^{
self.progressIndictor.progress = prog;
});
counter = 0;
}
// only download if file isn't already here
BOOL fileExists = [manager fileExistsAtPath:destFile]; // check if we already have it
if (!fileExists) {
// jpg or folder check
if ([[destFile pathExtension] isEqualToString:#"jpg"]) {
url = [NSURL URLWithString:srcFile];
data = [NSData dataWithContentsOfURL:url
options:0
error:&error];
[data writeToFile:destFile options:NSDataWritingAtomic error:&error];
data = nil;
} else {
[manager createDirectoryAtPath:destFile withIntermediateDirectories:YES attributes:nil error:&error];
}
}
}
}
If the files exist, the loop zips through the array and exits back to the main thread ok.
If any files are missing, the download/write part seems to chew up the RAM and cause low memory warning to trigger. It takes several thousand files to do it.
I've tried declaring the variables outside the loop, and even doing the whole thing in the main thread to test if that was causing the leak.
I tried using the alternate dataWithContentsOfURL:options:error call.
I tried Instruments, but it is really slow and crashes often. Before crashing, it does show allocation going up, up, up slowly.
After several days on this, I'm stumped.

Answer 1

The first thing I'd suggest is using an #autoreleasepool to control the peak amount of memory consumed. Right now, you're downloading the contents into the NSData as an autorelease object, and when done, you're nil-ing that variable, which simple flags it to be deallocated once the autorelease pool is drained (which will not happen until loadImagesFromList is done). By (a) moving the variable declarations inside the for loop; and (b) wrapping this in an #autoreleasepool, your memory will be deallocated as the individual downloads finish.
-(void)loadImagesFromList:(NSArray *)theArray
sourceBundle:(NSString *)bundlePath
destBundle:(NSString *)localBundlePath
manager:(NSFileManager *)manager {
// initialize the progress and activity indicator
dispatch_async(dispatch_get_main_queue(), ^{
// your UI update here
});
int counter = 0;
float prog = 0;
float increment = 1.0 / [theArray count];
float stepSize = [theArray count] / 10;
for (NSString *file in theArray)
{
#autoreleasepool {
if (stopFlag) {
NSLog(#"I see stopFlag = true, counter = %d, prog = %f", counter, prog);
return;
}
NSString *srcFile = [bundlePath stringByAppendingPathComponent:file];
NSString *destFile = [localBundlePath stringByAppendingPathComponent:file];
counter += 1;
prog += increment;
if (counter == stepSize) {
dispatch_async(dispatch_get_main_queue(), ^{
self.progressIndictor.progress = prog;
});
counter = 0;
}
// only download if file isn't already here
BOOL fileExists = [manager fileExistsAtPath:destFile]; // check if we already have it
if (!fileExists) {
NSError *error = nil;
// jpg or folder check
if ([[destFile pathExtension] isEqualToString:#"jpg"]) {
NSURL *url = [NSURL URLWithString:srcFile];
NSData *data = [NSData dataWithContentsOfURL:url
options:0
error:&error];
[data writeToFile:destFile options:NSDataWritingAtomic error:&error];
} else {
[manager createDirectoryAtPath:destFile withIntermediateDirectories:YES attributes:nil error:&error];
}
}
}
}
}

Answer 2

You might want to refactor this code to use NSOperationQueue. This addresses the peak memory issue, but also let's you enjoy a degree of concurrency. Because iOS only allows 4-5 concurrent requests anyway, you want to limit the maximum number of concurrent operations to a reasonable number, and this mitigates network timeout risks if trying to run too many concurrent requests. (This maxConcurrentOperationCount feature is the main reason I suggest using operation queues.)
Anyway, that might look like:
-(void)loadImagesFromList:(NSArray *)theArray
sourceBundle:(NSString *)bundlePath
destBundle:(NSString *)localBundlePath
manager:(NSFileManager *)manager {
// initialize the progress and activity indicator
[[NSOperationQueue mainQueue] addOperationWithBlock:^{
// your UI update here
}];
int __block counter = 0;
float __block prog = 0;
float increment = 1.0 / [theArray count];
float stepSize = [theArray count] / 10;
NSOperationQueue *queue = [[NSOperationQueue alloc] init];
queue.maxConcurrentOperationCount = 4;
for (NSString *file in theArray)
{
[queue addOperationWithBlock:^{
if (stopFlag) {
NSLog(#"I see stopFlag = true, counter = %d, prog = %f", counter, prog);
return;
}
NSString *srcFile = [bundlePath stringByAppendingPathComponent:file];
NSString *destFile = [localBundlePath stringByAppendingPathComponent:file];
[[NSOperationQueue mainQueue] addOperationWithBlock:^{
counter += 1;
prog += increment;
if (counter == stepSize) {
self.progressIndictor.progress = prog;
counter = 0;
}
}];
// only download if file isn't already here
BOOL fileExists = [manager fileExistsAtPath:destFile]; // check if we already have it
if (!fileExists) {
NSError *error = nil;
// jpg or folder check
if ([[destFile pathExtension] isEqualToString:#"jpg"]) {
NSURL *url = [NSURL URLWithString:srcFile];
NSData *data = [NSData dataWithContentsOfURL:url
options:0
error:&error];
[data writeToFile:destFile options:NSDataWritingAtomic error:&error];
} else {
[manager createDirectoryAtPath:destFile withIntermediateDirectories:YES attributes:nil error:&error];
}
}
}];
}
}
There are other refinements I might suggest (e.g. implementing cancelation logic rather than looking at stopFlag), but I was trying to minimize the code changes. I'm just taking advantage of the fact that one can easily replace dispatch_async:
dispatch_async(dispatchQueue, ^{ ... });
with NSOperationQueue method addOperationWithBlock:
[operationQueue addOperationWithBlock:^{ ... }];
But now we can use a concurrent NSOperationQueue with maxConcurrentOperationCount of 4 or 5, and you suddenly enjoy a nice, constrained degree of concurrency. You may find that this is observably faster than downloading files sequentially.