Using AWSS3TransferUtility to upload data on iOS - ios

I've managed to successfully upload data using the AWS 2.0 SDK, however I'm having some trouble understanding the "re-wiring" of completion handlers that is meant to take place on resuming of the app.
What I believe I should be doing, is at comment 6, saving the upload task. Then if it reaches the completionHandler block, deleting it. However, if the app is terminated before this, I can look at my saved upload tasks in the block at comment 4. Any saved tasks would be "re-wired" to the completion handler.
For reference, imagine this code happening in an "Upload" class, and in a block, which is the "uploadSuccess()" code you see in the completionHandler.
For reference here is the blog that introduced the AWSS3TransferUtility, and here is its documentation.
Hopefully someone can guide me on this.
// 1. Get/Setup credentials
AWSCognitoCredentialsProvider *credentialsProvider = [[AWSCognitoCredentialsProvider alloc] initWithRegionType:AWSRegionUSEast1 identityPoolId:#"IdentityPoolId"];
AWSServiceConfiguration *configuration = [[AWSServiceConfiguration alloc] initWithRegion:AWSRegionUSEast1 credentialsProvider:credentialsProvider];
AWSServiceManager.defaultServiceManager.defaultServiceConfiguration = configuration;
// 2. Create AWSS3 Transfer blocks
AWSS3TransferUtilityUploadExpression *expression = [AWSS3TransferUtilityUploadExpression new];
expression.uploadProgress = ^(AWSS3TransferUtilityTask *task, int64_t bytesSent, int64_t totalBytesSent, int64_t totalBytesExpectedToSend) {
dispatch_async(dispatch_get_main_queue(), ^{
// Do something e.g. Update a progress bar.
});
};
AWSS3TransferUtilityUploadCompletionHandlerBlock completionHandler = ^(AWSS3TransferUtilityUploadTask *task, NSError *error) {
dispatch_async(dispatch_get_main_queue(), ^{
// Do something e.g. Alert a user for transfer completion.
// On failed uploads, `error` contains the error object.
if (error) {
uploadSuccess(NO);
} else {
uploadSuccess(YES);
}
});
};
// 3. Create Transfer Utility
AWSS3TransferUtility *transferUtility = [AWSS3TransferUtility defaultS3TransferUtility];
// 4. Rewire Transfer Utility blocks
[transferUtility
enumerateToAssignBlocksForUploadTask:^(AWSS3TransferUtilityUploadTask *uploadTask, __autoreleasing AWSS3TransferUtilityUploadProgressBlock *uploadProgressBlockReference, __autoreleasing AWSS3TransferUtilityUploadCompletionHandlerBlock *completionHandlerReference) {
NSLog(#"Upload task identifier = %lu", (unsigned long)uploadTask.taskIdentifier);
// Use `uploadTask.taskIdentifier` to determine what blocks to assign.
//*uploadProgressBlockReference = // Reassign your progress feedback block.
*completionHandlerReference = completionHandler;// Reassign your completion handler.
}
downloadTask:^(AWSS3TransferUtilityDownloadTask *downloadTask, __autoreleasing AWSS3TransferUtilityDownloadProgressBlock *downloadProgressBlockReference, __autoreleasing AWSS3TransferUtilityDownloadCompletionHandlerBlock *completionHandlerReference) {
}];
// 5. Upload data using Transfer Utility
[[transferUtility uploadData:myNSDataObject
bucket:#"bucketName"
key:#"keyName"
contentType:#"text/plain"
expression:expression
completionHander:completionHandler] continueWithBlock:^id(AWSTask *task) {
if (task.error) {
NSLog(#"Error: %#", task.error);
}
if (task.exception) {
NSLog(#"Exception: %#", task.exception);
}
if (task.result) {
AWSS3TransferUtilityUploadTask *uploadTask = task.result;
// 6. Should i be saving uploadTasks here?
}
return nil;
}];

Related

OCMOCK testing block

I am using the bolt framework for an Async task. How do I test the code, which is in continueWithBlock section?
BOOL wasFetchedFromCache;
[[store fetchFileAsync:manifestURL allowfetchingFromCache:YES fetchedFromCache:&wasFetchedFromCache]
continueWithBlock:^id(BFTask *task) {
NSData *fileContents = task.result;
NSError *localError;
// code to test
return nil
}];
In order to test async task, you should use XCTestExpectation which allows us to create an expectation that will be fulfilled in the future. That mean the future results returned is considered as an expectation in the test case, and the test will wait until receiving the results for asserted. Please take a look at the code below, which I write a simple async testing.
- (void)testFetchFileAsync {
XCTestExpectation *expectation = [self expectationWithDescription:#"FetchFileAsync"];
BOOL wasFetchedFromCache;
[[store fetchFileAsync:manifestURL allowfetchingFromCache:YES fetchedFromCache:&wasFetchedFromCache]
continueWithBlock:^id(BFTask *task) {
NSData *data = task.result;
NSError *localError;
XCTAssertNotNil(data, #"data should not be nil");
[expectation fulfill];
// code to test
return nil
}];
[self waitForExpectationsWithTimeout:15.0 handler:^(NSError * _Nullable error) {
if (error) {
NSLog(#"Timeout error");
}
}];
XCTAssertTrue(wasFetchedFromCache, #"should be true");
}

NSOperationQueue currentQueue not working

I am trying to run AFURLConnectionOperation seen below on the currentQueue as I want to keep my main thread free for user interatoin, however nothing happens when I call mainQeue.
However if I call the same AFURLConnectionOperation on mainQueue it works perfectly.
Pleas see following code
// Send Batch
NSArray *operations = [AFURLConnectionOperation batchOfRequestOperations:mutableOperations progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations) {
NSLog(#"%lu of %lu complete", (unsigned long)numberOfFinishedOperations, (unsigned long)totalNumberOfOperations);
} completionBlock:^(NSArray *operations) {
// check batch response
NSError *error;
for (AFHTTPRequestOperation *op in operations) {
if (op.isCancelled){
return ;
}
if (op.responseObject){
// Current JSON Batch complete
NSMutableArray *jsonObject = [NSJSONSerialization JSONObjectWithData:op.request.HTTPBody options:kNilOptions error:&error];
// Update sent_flag using current chunk
[[SyncModel sharedInstance] updateSentFlag:jsonObject];
}
if (op.error){
error = op.error;
NSLog(#"Error == %#", error);
}
}
}];
Then finally I call one or the other of the following code
[[NSOperationQueue mainQueue] addOperations:operations waitUntilFinished:NO]; // this works
[[NSOperationQueue currentQueue] addOperations:operations waitUntilFinished:NO]; // this dose not work
The reason is
You can use this method from within a running operation object to get a reference to the operation queue that started it. Calling this method from outside the context of a running operation typically results in nil being returned.
So,I guess,if you log [NSOperationQueue currentQueue],it is nil
If you want a new queue,use
[[NSOperationQueue alloc] init];
After adding the operation on queue, if the operation doesn't start eventually then there are two ways to get them executed.
Using wait block, for example during unit test using XCTest framework, use
XCTestExpectation *expectation1 = [self expectationWithDescription:#"ExtractColorsInternal function call on NSOperationQueue"];
dispatch_async(dispatch_get_main_queue(), ^{
[expectation1 fulfill];
});
[self waitForExpectationsWithTimeout:1000 handler:^(NSError *error) {
if (error != nil) {
NSLog(#"Error: %#", error.localizedDescription);
}
}];
call CFRunLoopRun(), which would execute the present operation in current queue succesfully

Download Multiple Items from Amazon S3 Bucket iOS

I've recently implemented the new AWS 2.0 iOS SDK in my application (yay, cocoapods!), and using the sample code from Amazon managed to properly configure access and downloads. I can successfully download a single item without issue, but I need to be able to download multiple files dynamically generated based on the current tableview. There doesn't appear to be a way to set up a batch download, so I'm simply trying to loop through an array of objects and trigger a download with each one. It works, but if the list includes more than a few items, it starts randomly misfiring. For example, if my dynamically created list has 14 items in it, 12 will be downloaded, and the other 2 aren't even attempted. The request just vanishes. In my testing, I added a sleep(1) timer, and then all 14 are triggered and downloaded, so I'm guessing that I'm overwhelming the download requests and they are getting dropped unless I slow it down. Slowing it down is not ideal... perhaps there is another way? Here is the code:
- (IBAction)downloadAllPics:(UIBarButtonItem *)sender {
if (debug==1) {
NSLog(#"Running %# '%#'", self.class, NSStringFromSelector(_cmd));
}
CoreDataHelper *cdh =
[(AppDelegate *)[[UIApplication sharedApplication] delegate] cdh];
// for loop iterates through all of the items in the tableview
for (Item *item in self.frc.fetchedObjects) {
NSString *docDir = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
NSString *downloadingFilePath1 = [NSString stringWithFormat:#"%#/%##2x.jpg",docDir, item.imageName];
NSURL *downloadingFileURL1 = [NSURL fileURLWithPath:downloadingFilePath1];
NSFileManager *fileManager = [NSFileManager defaultManager];
NSError *error = nil;
if ([fileManager fileExistsAtPath:downloadingFilePath1]) {
fileAlreadyExists = TRUE;
if (![fileManager removeItemAtPath:downloadingFilePath1
error:&error]) {
NSLog(#"Error: %#", error);
}
}
__weak typeof(self) weakSelf = self;
self.downloadRequest1 = [AWSS3TransferManagerDownloadRequest new];
self.downloadRequest1.bucket = S3BucketName;
// self.downloadRequest1.key = S3KeyDownloadName1;
self.downloadRequest1.key = [NSString stringWithFormat:#"images/%##2x.jpg", item.imageName];
self.downloadRequest1.downloadingFileURL = downloadingFileURL1;
self.downloadRequest1.downloadProgress = ^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite){
// update progress
dispatch_sync(dispatch_get_main_queue(), ^{
weakSelf.file1AlreadyDownloaded = totalBytesWritten;
weakSelf.file1Size = totalBytesExpectedToWrite;
});
};
// this launches the actual S3 transfer manager - it is successfully launched with each pass of loop
[self downloadFiles];
}
[cdh backgroundSaveContext];
}
That launches the downloadFiles method:
- (void) downloadFiles {
//if I add this sleep, all 14 download. If I don't usually 11-13 download.
sleep(1);
AWSS3TransferManager *transferManager = [AWSS3TransferManager defaultS3TransferManager];
__block int downloadCount = 0;
[[transferManager download:self.downloadRequest1] continueWithExecutor:[BFExecutor mainThreadExecutor] withBlock:^id(BFTask *task) {
if (task.error != nil){
if(task.error.code != AWSS3TransferManagerErrorCancelled && task.error.code != AWSS3TransferManagerErrorPaused){
NSLog(#"%s Errorx: [%#]",__PRETTY_FUNCTION__, task.error);
}
} else {
self.downloadRequest1 = nil;
}
return nil;
}];
}
There has got to be a way to download a dynamic list of files from an Amazon S3 bucket, right? Maybe there is a transfer manager that allows an array of files instead of doing them individually?
Any and all help is appreciated.
Zack
Sounds like request timeout interval setting issue.
First, when you configure AWSServiceConfiguration *configuration = ... try to configure the timeoutIntervalForRequest property. Also, maxRetryCount as well. maxRetryCount will attempt to download if failure for downloading each operation.
AWSServiceConfiguration *configuration = [[AWSServiceConfiguration alloc] initWithRegion:DefaultServiceRegionType
credentialsProvider:credentialsProvider];
[configuration setMaxRetryCount:2]; // 10 is the max
[configuration setTimeoutIntervalForRequest:120]; // 120 seconds
Second, for the multiple items downloading try to collect each AWSTask into one array and get the result at the end of group operation. ex)
// task collector
NSMutableSet *uniqueTasks = [NSMutableSet new];
// Loop
for (0 -> numOfDownloads) {
AWSS3TransferManagerDownloadRequest *downloadRequest = [AWSS3TransferManagerDownloadRequest new];
[downloadRequest setBucket:S3BucketNameForProductImage];
[downloadRequest setKey:filename];
[downloadRequest setDownloadingFileURL:sourceURL];
[showroomGroupDownloadRequests addObject:downloadRequest];
AWSTask *task = [[AWSS3TransferManager defaultS3TransferManager] download:downloadRequest];
[task continueWithBlock:^id(AWSTask *task) {
// handle each individual operation
if (task.error == nil) {
}
else if (task.error) {
}
// add to the tasks
[uniqueTasks addObject:task];
return nil;
}
[[AWSTask taskForCompletionOfAllTasks:tasks] continueWithBlock:^id(AWSTask *task) {
if (task.error == nil) {
// all downloads succeess
}
else if (task.error != nil) {
// failure happen one of download
}
return nil;
}];
The reason some requests seem to vanish is that you define AWSS3TransferManagerDownloadRequest as a property. self.downloadRequest1 = nil; is executed on the background thread, and it is possible that when [transferManager download:self.downloadRequest1] is executed, self.downloadRequest1 is nil.
You should remove the property and simply pass an instance of AWSS3TransferManagerDownloadRequest as an argument for - downloadFiles:.

sendAsynchronousRequest makes UI freezes

downloadImages is a button and whenever I press on it, a spinner should start rolling, an async request should ping Google (to make sure there is a connection) and after a response is received, I start to synchronically downloading images.
Somehow the spinner won't go and it seems as if the request is sync and not async.
- (IBAction)downloadImages:(id)sender {
NSString *ping=#"http://www.google.com/";
GlobalVars *globals = [GlobalVars sharedInstance];
[self startSpinner:#"Please Wait."];
NSURL *url = [[NSURL alloc] initWithString:ping];
NSURLRequest *request = [NSURLRequest requestWithURL:url cachePolicy:NSURLRequestReloadIgnoringCacheData timeoutInterval:5.0];
[NSURLConnection sendAsynchronousRequest:request queue:[NSOperationQueue mainQueue] completionHandler:^(NSURLResponse *response, NSData *data, NSError *error) {
if (data) {
for(int i=globals.farmerList.count-1; i>=0;i--)
{
//Definitions
NSString * documentsDirectoryPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
//Get Image From URL
NSString *urlString = [NSString stringWithFormat:#"https://myurl.com/%#",[[globals.farmerList objectAtIndex:i] objectForKey:#"Image"]];
UIImage * imageFromURL = [self getImageFromURL:urlString];
//Save Image to Directory
[self saveImage:imageFromURL withFileName:[[globals.farmerList objectAtIndex:i] objectForKey:#"Image"] ofType:#"jpg" inDirectory:documentsDirectoryPath];
}
[self stopSpinner];
}
}];
}
The spinner code:
//show loading activity.
- (void)startSpinner:(NSString *)message {
// Purchasing Spinner.
if (!connectingAlerts) {
connectingAlerts = [[UIAlertView alloc] initWithTitle:NSLocalizedString(message,#"")
message:nil
delegate:self
cancelButtonTitle:nil
otherButtonTitles:nil];
connectingAlerts.tag = (NSUInteger)300;
[connectingAlerts show];
UIActivityIndicatorView *connectingIndicator = [[UIActivityIndicatorView alloc] initWithActivityIndicatorStyle:UIActivityIndicatorViewStyleWhiteLarge];
connectingIndicator.frame = CGRectMake(139.0f-18.0f,50.0f,37.0f,37.0f);
[connectingAlerts addSubview:connectingIndicator];
[connectingIndicator startAnimating];
}
}
//hide loading activity.
- (void)stopSpinner {
if (connectingAlerts) {
[connectingAlerts dismissWithClickedButtonIndex:0 animated:YES];
connectingAlerts = nil;
}
// [self performSelector:#selector(showBadNews:) withObject:error afterDelay:0.1];
}
As asked: the getImageFromURL code
-(UIImage *) getImageFromURL:(NSString *)fileURL {
UIImage * result;
NSData * data = [NSData dataWithContentsOfURL:[NSURL URLWithString:fileURL]];
result = [UIImage imageWithData:data];
return result;
}
-(void) saveImage:(UIImage *)image withFileName:(NSString *)imageName ofType:(NSString *)extension inDirectory:(NSString *)directoryPath {
if ([[extension lowercaseString] isEqualToString:#"png"]) {
[UIImagePNGRepresentation(image) writeToFile:[directoryPath stringByAppendingPathComponent:[NSString stringWithFormat:#"%#.%#", imageName, #"png"]] options:NSAtomicWrite error:nil];
} else if ([[extension lowercaseString] isEqualToString:#"jpg"] || [[extension lowercaseString] isEqualToString:#"jpeg"]) {
[UIImageJPEGRepresentation(image, 1.0) writeToFile:[directoryPath stringByAppendingPathComponent:[NSString stringWithFormat:#"%#.%#", imageName, #"jpg"]] options:NSAtomicWrite error:nil];
} else {
NSLog(#"Image Save Failed\nExtension: (%#) is not recognized, use (PNG/JPG)", extension);
}
}
That's because you're creating an asynchronous operation and then telling it to execute on the main thread by using [NSOperationQueue mainQueue];.
Instead, create a new instance of NSOpeartionQueue and pass that as the parameter.
NSOperationQueue *myQueue = [[NSOperationQueue alloc] init];
This is an asynchronous problem. Asynchronism is infectious. That means, if any small part of the problem is asynchronous, the whole problem becomes asynchronous.
That is, your button action would invoke an asynchronous method like this (and itself becomes "asynchronous" as well):
- (IBAction)downloadImages:(id)sender
{
self.downloadImagesButton.enabled = NO;
[self asyncLoadAndSaveImagesWithURLs:self.urls completion:^(id result, NSError* error){
if (error != nil) {
NSLog(#"Error: %#", error);
}
dispatch_async(dispatch_get_main_queue(), ^{
self.downloadImagesButton.enabled = YES;
};
}];
}
So, your asynchronous problem can be described as this:
Given a list of URLs, asynchronously load each URL and asynchronously save them to disk. When all URLs are loaded and saved, asynchronously notify the call-site by calling a completion handler passing it an array of results (for each download and save operation).
This is your asynchronous method:
typedef void (^completion_t)(id result, NSError* error);
- (void) asyncLoadAndSaveImagesWithURLs:(NSArray*)urls
completion:(completion_t) completionHandler;
Asynchronous problems can be solved properly only by finding a suitable asynchronous pattern. This involves to asynchronize every part of the problem.
Lets start with your getImageFromURL method. Loading a remote resource is inherently asynchronous, so your wrapper method ultimately will be asynchronous as well:
typedef void (^completion_t)(id result, NSError* error);
- (void) loadImageWithURL:(NSURL*)url completion:(completion_t)completionHandler;
I leave it undefined how that method will be eventually implemented. You may use NSURLConnection's asynchronous convenient class method, a third party helper tool or your own HTTPRequestOperation class. It doesn't matter but it MUST be asynchronous for achieving a sane approach.
Purposefully, you can and should make your saveImage method asynchronous as well. The reason for making this asynchronous is, that this method possibly will get invoked concurrently, and we should *serialize* disk bound (I/O bound) tasks. This improves utilization of system resources and also makes your approach a friendly system citizen.
Here is the asynchronized version:
typedef void (^completion_t)(id result, NSError* error);
-(void) saveImage:(UIImage *)image fileName:(NSString *)fileName ofType:(NSString *)extension
inDirectory:(NSString *)directoryPath
completion:(completion_t)completionHandler;
In order to serialize disk access, we can use a dedicated queue disk_queue where we assume it has been properly initialized as a serial queue by self:
-(void) saveImage:(UIImage *)image fileName:(NSString *)fileName ofType:(NSString *)extension
inDirectory:(NSString *)directoryPath
completion:(completion_t)completionHandler
{
dispatch_async(self.disk_queue, ^{
// save the image
...
if (completionHandler) {
completionHandler(result, nil);
}
});
}
Now, we can define an asynchronous wrapper which loads and saves the image:
typedef void (^completion_t)(id result, NSError* error);
- (void) loadAndSaveImageWithURL:(NSURL*)url completion:(completion_t)completionHandler
{
[self loadImageWithURL:url completion:^(id image, NSError*error) {
if (image) {
[self saveImage:image fileName:fileName ofType:type inDirectory:directory completion:^(id result, NSError* error){
if (result) {
if (completionHandler) {
completionHandler(result, nil);
}
}
else {
DebugLog(#"Error: %#", error);
if (completionHandler) {
completionHandler(nil, error);
}
}
}];
}
else {
if (completionHandler) {
completionHandler(nil, error);
}
}
}];
}
This loadAndSaveImageWithURL method actually performs a "continuation" of two asynchronous tasks:
First, asynchronously load the image.
THEN, if that was successful, asynchronously save the image.
It's important to notice that these two asynchronous tasks are sequentially processed.
Up until here, this all should be quite comprehensive and be straight forward. The tricky part follows now where we try to invoke a number of asynchronous tasks in an asynchronous manner.
Asynchronous Loop
Suppose, we have a list of URLs. Each URL shall be loaded asynchronously, and when all URLs are loaded we want the call-site to be notified.
The traditional for loop is not that appropriate for accomplishing this. But imagine we would have a Category for a NSArray with a method like this:
Category for NSArray
- (void) forEachApplyTask:(task_t)transform completion:(completion_t)completionHandler;
This basically reads: for each object in the array, apply the asynchronous task transform and when all objects have been "transformed" return a list of the transformed objects.
Note: this method is asynchronous!
With the appropriate "transform" function, we can "translate" this to your specific problem:
For each URL in the array, apply the asynchronous task loadAndSaveImageWithURL and when all URLS have been loaded and saved return a list of the results.
The actual implementation of the forEachApplyTask:completion: may appear a bit tricky and for brevity I don't want to post the complete source here. A viable approach requires about 40 lines of code.
I'll provide an example implementation later (on Gist), but lets explain how this method can be used:
The task_t is a "block" which takes one input parameter (the URL) and returns a result.
Since everything must be treated asynchronously, this block is asynchronous as well, and the eventual result will be provided via a completion block:
typedef void (^completion_t)(id result, NSError* error);
typedef void (^task_t)(id input, completion_t completionHandler);
The completion handler may be defined as follows:
If the tasks succeeds, parameter error equals nil. Otherwise, parameter error is an NSError object. That is, a valid result may also be nil.
We can quite easily wrap our method loadAndSaveImageWithURL:completion: and create a block:
task_t task = ^(id input, completion_t completionHandler) {
[self loadAndSaveImageWithURL:input completion:completionHandler];
};
Given an array of URLs:
self.urls = ...;
your button action can be implemented as follows:
- (IBAction)downloadImages:(id)sender
{
self.downloadImagesButton.enabled = NO;
task_t task = ^(id input, completion_t completionHandler) {
[self loadAndSaveImageWithURL:input completion:completionHandler];
};
[self.urls forEachApplyTask:task ^(id results, NSError*error){
self.downloadImagesButton.enabled = YES;
if (error == nil) {
... // do something
}
else {
// handle error
}
}];
}
Again, notice that method forEachApplyTask:completion: is an asynchronous method, which returns immediately. The call-site will be notified via the completion handler.
The downloadImages method is asynchronous as well, there is no completion handler though. This method disables the button when it starts and enables it again when the asynchronous operation has been completed.
The implementation of this forEachApplyTask method can be found here: (https://gist.github.com/couchdeveloper/6155227).
From your code what I can understand is its not due to assyncronous call to load url. but the following code may heavy.
For assynchronous image loading try https://github.com/rs/SDWebImage
//Get Image From URL
NSString *urlString = [NSString stringWithFormat:#"https://myurl.com/%#",[[globals.farmerList objectAtIndex:i] objectForKey:#"Image"]];
UIImage * imageFromURL = [self getImageFromURL:urlString];
//Save Image to Directory
[self saveImage:imageFromURL withFileName:[[globals.farmerList objectAtIndex:i] objectForKey:#"Image"] ofType:#"jpg" inDirectory:documentsDirectoryPath];
Happy coding :)

Using AFNetworking NSOperations to download a number of files serially.....runs out of memory

Note: I'm using ARC.
I have some code that makes 1 request to an http server for a list of files (via JSON). It then parses that list into model objects which it uses to add a download operation (for downloading that file) to a different nsoperationqueue and then once it's done adding all of those operations (queue starts out suspended) it kicks off the queue and waits for all the operations to finish before continuing. (Note: this is all done on background threads so as not to block the main thread).
Here's the basic code:
NSURLRequest* request = [NSURLRequest requestWithURL:parseServiceUrl];
AFHTTPRequestOperation *op = [[AFHTTPRequestOperation alloc] initWithRequest:request];
op.responseSerializer = [AFJSONResponseSerializer serializer];
[op setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
//NSLog(#"JSON: %#", responseObject);
// Parse JSON into model objects
NSNumber* results = [responseObject objectForKey:#"results"];
if ([results intValue] > 0)
{
dispatch_async(_processQueue, ^{
_totalFiles = [results intValue];
_timestamp = [responseObject objectForKey:#"timestamp"];
NSArray* files = [responseObject objectForKey:#"files"];
for (NSDictionary* fileDict in files)
{
DownloadableFile* file = [[DownloadableFile alloc] init];
file.file_id = [fileDict objectForKey:#"file_id"];
file.file_location = [fileDict objectForKey:#"file_location"];
file.timestamp = [fileDict objectForKey:#"timestamp"];
file.orderInQueue = [files indexOfObject:fileDict];
NSNumber* action = [fileDict objectForKey:#"action"];
if ([action intValue] >= 1)
{
if ([file.file_location.lastPathComponent.pathExtension isEqualToString:#""])
{
continue;
}
[self downloadSingleFile:file];
}
else // action == 0 so DELETE file if it exists
{
if ([[NSFileManager defaultManager] fileExistsAtPath:file.localPath])
{
NSError* error;
[[NSFileManager defaultManager] removeItemAtPath:file.localPath error:&error];
if (error)
{
NSLog(#"Error deleting file after given an Action of 0: %#: %#", file.file_location, error);
}
}
}
[self updateProgress:[files indexOfObject:fileDict] withTotal:[files count]];
}
dispatch_sync(dispatch_get_main_queue(), ^{
[_label setText:#"Syncing Files..."];
});
[_dlQueue setSuspended:NO];
[_dlQueue waitUntilAllOperationsAreFinished];
[SettingsManager sharedInstance].timestamp = _timestamp;
dispatch_async(dispatch_get_main_queue(), ^{
callback(nil);
});
});
}
else
{
dispatch_async(dispatch_get_main_queue(), ^{
callback(nil);
});
}
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
NSLog(#"Error: %#", error);
callback(error);
}];
[_parseQueue addOperation:op];
and then the downloadSingleFile method:
- (void)downloadSingleFile:(DownloadableFile*)dfile
{
NSURLRequest* req = [NSURLRequest requestWithURL:dfile.downloadUrl];
AFHTTPRequestOperation* reqOper = [[AFHTTPRequestOperation alloc] initWithRequest:req];
reqOper.responseSerializer = [AFHTTPResponseSerializer serializer];
[reqOper setCompletionBlockWithSuccess:^(AFHTTPRequestOperation* op, id response)
{
__weak NSData* fileData = response;
NSError* error;
__weak DownloadableFile* file = dfile;
NSString* fullPath = [file.localPath substringToIndex:[file.localPath rangeOfString:file.localPath.lastPathComponent options:NSBackwardsSearch].location];
[[NSFileManager defaultManager] createDirectoryAtPath:fullPath withIntermediateDirectories:YES attributes:Nil error:&error];
if (error)
{
NSLog(#"Error creating directory path: %#: %#", fullPath, error);
}
else
{
error = nil;
[fileData writeToFile:file.localPath options:NSDataWritingFileProtectionComplete error:&error];
if (error)
{
NSLog(#"Error writing fileData for file: %#: %#", file.file_location, error);
}
}
[self updateProgress:file.orderInQueue withTotal:_totalFiles];
}
failure:^(AFHTTPRequestOperation* op, NSError* error)
{
[self updateProgress:dfile.orderInQueue withTotal:_totalFiles];
NSLog(#"Error downloading %#: %#", dfile.downloadUrl, error.localizedDescription);
}];
[_dlQueue addOperation:reqOper];
}
What I'm seeing is a constant spike in memory as more files get downloaded. It's like the responseObject or maybe even the whole completionBlock is not being let go of.
I've tried making the responseObject __weak as well as fileData. I've tried adding an autoreleasepool and I've tried making the actual file domain object __weak too but still memory climbs and climbs.
I've run Instruments and not seen any leaks persay but it never gets to a point where all the files have been downloaded before it runs out of memory with a big fat "can't allocate region" error. Looking at allocations, I see a bunch of connection:didFinishLoading and connection:didReceiveData methods that never seem to be let go of, however. I can't seem to debug it further than that though.
My question: Why is it running out of memory? What is not getting deallocated and how can I get it to do such?
There is a few things going on here. The biggest is that you are downloading the entire file, storing it in memory, and then writing it out to disk when the download is complete. Even with just one file of 500 MB, you will run out of memory.
The correct way to do this is using an NSOutputStream with asynchronous downloads. The key is to write out the data as soon as it arrives. It should look like this:
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
[self.outputStream write:[data bytes] maxLength:[data length]];
}
Also of note, you are creating your weak references inside the block, not outside. Because of that, you are still creating a retain cycle and leaking memory. When you create weak references, it should look like this.
NSOperation *op = [[NSOperation alloc] init];
__weak NSOperation *weakOp = op;
op.completion = ^{
// Use only weakOp within this block
};
Lastly, your code is using #autoreleasepool. NSAutoreleasePool, and the ARC equivalent #autoreleasepool are only useful in very limited situations. As a general rule, if you aren't absolutely sure you need one, you don't.
With the help of a friend, I was able to figure out the problem.
The problem was actually in the first block of code:
[_dlQueue waitUntilAllOperationsAreFinished];
Apparently , waiting for all operations to finish meant none of those operations would be released either.
Instead of that, I ended up adding a final operation to the queue that would do the final processing and callback and memory is much more stable now.
[_dlQueue addOperationWithBlock:^{
[SettingsManager sharedInstance].timestamp = _timestamp;
dispatch_async(dispatch_get_main_queue(), ^{
callback(nil);
});
}];
What kind of file you are downloading? If you are working with Images or videos you nee to clear URLCache as when you doneload images it create CFDATA and some information in cache and it does not cleared out. You need to clear it explicitly when your single file download completed. It will never caught as a leak also.
NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:0 diskCapacity:0 diskPath:nil];
[NSURLCache setSharedURLCache:sharedCache];
[sharedCache release];
If you are using ARC replace
[sharedCache release];
with
sharedCache = nil;
Hope It may help you.

Resources