During downloading multiple files using following asynchronous method of AWS iOS, I want to keep track which image is downloaded.
Following code is used for asynchronous image download.
S3TransferManager *tm = [S3TransferManager new];
S3TransferOperation *downloadFileOperation = [tm downloadFile:storeFilePath bucket:Bucket key:downloadPath];
Above method is in for loop, so there could be n images to download.
Delegate method which notify image is downloaded
-(void)request:(AmazonServiceRequest *)request didCompleteWithResponse:(AmazonServiceResponse *)response { }
But I did not find anything, using which I can manage that which actual image is downloaded. whether it was the firstID image or second one.
Any idea on where I can put some extra data , which can be received on image download ?
You can set requestTag:
S3PutObjectRequest *putObjectRequest = [ [ S3PutObjectRequest alloc ] initWithKey:keyFile inBucket:self.s3BucketName ];
putObjectRequest.requestTag = urlStringFile;
putObjectRequest.filename = fileName;
and then analyse it:
- (void)request:(AmazonServiceRequest*) request didCompleteWithResponse:(AmazonServiceResponse*) response
{
NSLog(#"Upload finished. RequestTag = %#", request.requestTag);
}
As far as, I have to only download content from Amazon s3, I used ASIHTTPRequest.
(Though this is not maintained since 2011, but I found its very useful and easy to use for my app).
Code Example,
// Initialize network Queue.
ASINetworkQueue *networkQueue = [[ASINetworkQueue alloc] init];
[networkQueue reset];
[networkQueue setRequestDidFinishSelector:#selector(requestDone:)]; //This is where download completion will be notified.
//Initialize Request.
ASIS3ObjectRequest *request = [ASIS3ObjectRequest requestWithBucket:#"Bucket_Name" key:#"/Path/file"];
This is what I was looking, I need all the information about what I am downloading on download completion. This userInfo contains all the data, which is available on download completion.
NSMutableDictionary *userInfo = [[NSMutableDictionary alloc] initWithObjectsAndKeys:#"Data", #"Key", nil];
request.userInfo = userInfo;
userInfo = nil;
// End of userInfo set.
[networkQueue addOperation:request]; // add request in ASINetworkQueue object. We can also add multiple request here.
And last,
[networkQueue go]; // This will start downloading.
// Delegate method, where download completion will be notified
- (void)requestDone:(ASIS3Request *)request
{
NSLog(#"UserInfo : %#", request.userInfo); // Request data, to manage which request is complete.
}
Done.
Related
My app offers the option to download 3430 high resolution images from our server, each image of size 50k - 600k bytes.
The original approach was to just download all of them - but we realized that gave a lot of NSURLErrorTimedOut errors and crashed our program. We then implemented it such that we download all of the images, but in batches of 100 images at a time. Someone on SO suggested we actually implement our download like this:
Create a list of all file URLs that need to be downloaded.
Write your code so that it downloads these URLs sequentially. I.e. do
not let it start downloading a file until the previous one has
finished (or failed and you decided to skip it for now).
Use NSURLSession's support for downloading an individual file to a
folder, don't use the code to get an NSData and save the file
yourself. That way, your application doesn't need to be running while
the download finishes.
Ensure that you can tell whether a file has already been downloaded or
not, in case your download gets interrupted, or the phone is restarted
in mid-download. You can e.g. do this by comparing their names (if
they are unique enough), or saving a note to a plist that lets you
match a downloaded file to the URL where it came from, or whatever
constitutes an identifying characteristic in your case.
At startup, check whether all files are there. If not, put the missing
ones in above download list and download them sequentially, as in #2.
Before you start downloading anything (and that includes downloading
the next file after the previous download has finished or failed), do
a reachability check using the Reachability API from Apple's
SystemConfiguration.framework. That will tell you whether the user has
a connection at all, and whether you're on WiFi or cellular (in
general, you do not want to download a large number of files via
cellular, most cellular connections are metered).
We create a list of all images to download here:
- (void)generateImageURLList:(BOOL)batchDownloadImagesFromServer
{
NSError* error;
NSFetchRequest* leafletURLRequest = [[[NSFetchRequest alloc] init] autorelease];
NSEntityDescription* leafletURLDescription = [NSEntityDescription entityForName:#"LeafletURL" inManagedObjectContext:managedObjectContext];
[leafletURLRequest setEntity:leafletURLDescription];
numberOfImages = [managedObjectContext countForFetchRequest:leafletURLRequest error:&error];
NSPredicate* thumbnailPredicate = [NSPredicate predicateWithFormat:#"thumbnailLocation like %#", kLocationServer];
[leafletURLRequest setPredicate:thumbnailPredicate];
self.uncachedThumbnailArray = [managedObjectContext executeFetchRequest:leafletURLRequest error:&error];
NSPredicate* hiResPredicate = [NSPredicate predicateWithFormat:#"hiResImageLocation != %#", kLocationCache];
[leafletURLRequest setPredicate:hiResPredicate];
self.uncachedHiResImageArray = [managedObjectContext executeFetchRequest:leafletURLRequest error:&error];
}
We use NSURLSession to download an individual image to a folder by calling hitServerForUrl and implementing didFinishDownloadingToURL:
- (void)hitServerForUrl:(NSURL*)requestUrl {
NSURLSessionConfiguration *defaultConfigurationObject = [NSURLSessionConfiguration defaultSessionConfiguration];
NSURLSession *defaultSession = [NSURLSession sessionWithConfiguration:defaultConfigurationObject delegate:self delegateQueue: nil];
NSURLSessionDownloadTask *fileDownloadTask = [defaultSession downloadTaskWithURL:requestUrl];
[fileDownloadTask resume];
}
- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didFinishDownloadingToURL:(NSURL *)location {
if (isThumbnail)
{
leafletURL.thumbnailLocation = kLocationCache;
}
else
{
leafletURL.hiResImageLocation = kLocationCache;
}
// Filename to write to
NSString* filePath = [leafletURL pathForImageAtLocation:kLocationCache isThumbnail:isThumbnail isRetina:NO];
// If it's a retina image, append the "#2x"
if (isRetina_) {
filePath = [filePath stringByReplacingOccurrencesOfString:#".jpg" withString:#"#2x.jpg"];
}
NSString* dir = [filePath stringByDeletingLastPathComponent];
[managedObjectContext save:nil];
NSError* error;
[[NSFileManager defaultManager] createDirectoryAtPath:dir withIntermediateDirectories:YES attributes:nil error:&error];
NSURL *documentURL = [NSURL fileURLWithPath:filePath];
NSLog(#"file path : %#", filePath);
if ([[NSFileManager defaultManager] fileExistsAtPath:filePath]) {
//Remove the old file from directory
}
[[NSFileManager defaultManager] moveItemAtURL:location
toURL:documentURL
error:&error];
if (error){
//Handle error here
}
}
This code calls loadImage, which calls `hitServer:
-(void)downloadImagesFromServer{
[self generateImageURLList:NO];
[leafletImageLoaderQueue removeAllObjects];
numberOfHiResImageLeft = [uncachedHiResImageArray count];
for ( LeafletURL* aLeafletURL in uncachedHiResImageArray)
{
//// Do the same thing again, except set isThumb = NO. ////
LeafletImageLoader* hiResImageLoader = [[LeafletImageLoader alloc] initWithDelegate:self];
[leafletImageLoaderQueue addObject:hiResImageLoader]; // do this before making connection!! //
[hiResImageLoader loadImage:aLeafletURL isThumbnail:NO isBatchDownload:YES];
//// Adding object to array already retains it, so it's safe to release it here. ////
[hiResImageLoader release];
uncachedHiResIndex++;
NSLog(#"uncached hi res index: %ld, un cached hi res image array size: %lu", (long)uncachedHiResIndex, (unsigned long)[uncachedHiResImageArray count]);
}
}
- (void)loadImage:(LeafletURL*)leafletURLInput isThumbnail:(BOOL)isThumbnailInput isBatchDownload:(BOOL)isBatchDownload isRetina:(BOOL)isRetina
{
isRetina_ = isRetina;
if (mConnection)
{
[mConnection cancel];
[mConnection release];
mConnection = nil;
}
if (mImageData)
{
[mImageData release];
mImageData = nil;
}
self.leafletURL = leafletURLInput;
self.isThumbnail = isThumbnailInput;
NSString* location = (self.isThumbnail) ?leafletURL.thumbnailLocation :leafletURL.hiResImageLocation;
//// Check if the image needs to be downloaded from server. If it is a batch download, then override the local resources////
if ( ([location isEqualToString:kLocationServer] || (isBatchDownload && [location isEqualToString:kLocationResource])) && self.leafletURL.rawURL != nil )
{
//NSLog(#"final loadimage called server");
//// tell the delegate to get ride of the old image while waiting. ////
if([delegate respondsToSelector:#selector(leafletImageLoaderWillBeginLoadingImage:)])
{
[delegate leafletImageLoaderWillBeginLoadingImage:self];
}
mImageData = [[NSMutableData alloc] init];
NSURL* url = [NSURL URLWithString:[leafletURL pathForImageOnServerUsingThumbnail:self.isThumbnail isRetina:isRetina]];
[self hitServerForUrl:url];
}
//// if not, tell the delegate that the image is already cached. ////
else
{
if([delegate respondsToSelector:#selector(leafletImageLoaderDidFinishLoadingImage:)])
{
[delegate leafletImageLoaderDidFinishLoadingImage:self];
}
}
}
Currently, I'm trying to figure out how to download the images sequentially, such that we don't call hitServer until the last image is finished downloading. Do I need to be downloading in the background? Thank you for suggestions!
My app offers the option to download 3430 high resolution images from our server, each image of size 50k - 600k bytes.
This seems like a job for on-demand resources. Just turn these files into on-demand resources obtained from your own server, and let the system take care of downloading them in its own sweet time.
This sounds very much like an architectural issue. If you fire off downloads without limiting them of course you're going to start getting timeouts and other things. Think about other apps and what they do. Apps that give the user the ability to do multiple downloads often limit how may can occur at once. iTunes for example can queue up thousands of downloads, but only runs 3 at a time. Limiting to just one at a time will only slow things down for your users. You need a balance that consider your user's available bandwidth.
The other part of this is to again consider what your users want. Does every one of your uses want every single image? I don't know what you are offering them, but in most apps which access resources like images or music, it's up to the user what and when they download. Thus they only download what they are interested in. So I'd recommend only downloading what the users are viewing or have somehow requested they want to download.
I am using objective-C to write an app which needs to dispatch 100 web request and the response will be handled in the call back. My question is, how can I execute web req0, wait for call back, then execute web req1 and so on?
Thanks for any tips and help.
NSURL *imageURL = [[contact photoLink] URL];
GDataServiceGoogleContact *service = [self contactService];
// requestForURL:ETag:httpMethod: sets the user agent header of the
// request and, when using ClientLogin, adds the authorization header
NSMutableURLRequest *request = [service requestForURL:imageURL
ETag: nil
httpMethod:nil];
[request setValue:#"image/*" forHTTPHeaderField:#"Accept"];
GTMHTTPFetcher *fetcher = [GTMHTTPFetcher fetcherWithRequest:request];
fetcher.retryEnabled = YES;
fetcher.maxRetryInterval = 0.3;
fetcher.minRetryInterval = 0.3;
[fetcher setAuthorizer:[service authorizer]];
[fetcher beginFetchWithDelegate:self
didFinishSelector:#selector(imageFetcher:finishedWithData:error:)];
}
- (void)imageFetcher:(GTMHTTPFetcher *)fetcher finishedWithData:(NSData *)data error:(NSError *)error {
if (error == nil) {
// got the data; display it in the image view. Because this is sample
// code, we won't be rigorous about verifying that the selected contact hasn't
// changed between when the fetch began and now.
// NSImage *image = [[[NSImage alloc] initWithData:data] autorelease];
// [mContactImageView setImage:image];
NSLog(#"successfully fetched the data");
} else {
NSLog(#"imageFetcher:%# failedWithError:%#", fetcher, error);
}
}
You can't simply call this code in a loop as GTMHTTPFetcher works asynchronously so the loop, as you see, will iterate and start all instances without any delay.
A simple option is to put all of the contacts into a mutable array, take the first contact from the array (remove it from the array) and start the first fetcher. Then, in the finishedWithData callback, check if the array contains anything, if it does remove the first item and start a fetch with it. In this way the fetches will run serially one after the other.
A better but more complex solution would be to create an asynchronous NSOperation (there are various guides on the web) which starts a fetch and waits for the callback before completing. The benefit of this approach is that you can create all of your operations and add them to an operation queue, then you can set the max concurrent count and run the queue - so you can run multiple fetch instances at the same time. You can also suspend the queue or cancel the operations if you need to.
Basically my app will retrieve an array of Data from database and upload it to the server(one at a time/ one after the other). I want to stop all the request when one of the data failed to upload (please check on the comments).
Code:
// 0 means need to upload to server
NSString *condition = [NSString stringWithFormat:#"isUploaded=\"0\""];
// array of ID on my database to be uploaded
NSArray *arrayOfID = [Registered distinctValuesWithAttribute:#"registeredID" predicate:[NSPredicate predicateWithFormat:condition]];
// loop every index and upload it to server
for (int i=0; i<arrayOfID.count && !isBreak; i++) {
// get the entity using ID
NSString *condition = [NSString stringWithFormat:#"registeredID=\"%#\"",[arrayOfID objectAtIndex:i]];
Registered *entity = [Registered getWithPredicate:[NSPredicate predicateWithFormat:condition]];
if (entity) {
__weak ASIFormDataRequest *request = [ASIFormDataRequest requestWithURL:url];
[self setRequest:request withEntity:entity]; // set delegate,POST,etc.
[request setCompletionBlock:^{
// returns dictionary (success/failed)
NSDictionary *dict = [[request responseString] JSON];
if ([[dict valueForKey:#"status"] isEqualToString:#"success"]) {
// set IsUploaded to 1 after successful upload to server
[entity setIsUploaded:[NSNumber numberWithInt:1]];
[Registered commit];
// Any necessary ideas that would make my code better
// and continue the POST request and proceed to the next entity???
}
else {
// I want to cancel all the request here and get out to this loop
}
}];
[request setFailedBlock:^{
// I want to cancel all the request here and get out to this loop
}];
// start startSynchronous
[request startSynchronous];
} else {
[GlobalMethods ShowAlertView:#"Database Error" message:#"Please try again later"];
isBreak=YES;
}
}
You could tweak your code slightly and use the provided ASINetworkQueue class. From the documentation, if one request in the queue fails, by default the rest are cancelled automatically.
You can use a break statement to immediately exit a loop. However, because the callbacks are asynchronous, putting one in the failure block will not have the effect you're looking for...assuming the failure block is fired in reaction to a server response and not something in the API itself, that loop will complete and fire off all your requests--and the enclosing method will return--long before any of your requests has the time to come back from the server and call its failure block (this is networking code, after all). So you're not going to be able to use a failure block to interrupt the creation of additional requests in the loop; that's not how asynchronous calls work.
I'm working with an app that requests data from an OAuth2.0 protected server. When I use the GTM OAuth Library to retrieve data, the program continues to run while the data is being downloaded in the background. I need some sort of mechanism to either force my application to wait until the didFinishWithData selector is called,or I need a way to notify my ViewController of the download's completion, so I can then utilize the data immediately.
I've tried conditional blocks, but those aren't doing it for me. I've also tried polling the object whose data I'm interested in, but if I do that, the data never seems to download. I've heard I can somehow utilize the Notification Center to accomplish this task, so I'll look more into that while I'm waiting for replies here.
Here is basically what is going on:
-(void) getAlert{
// Define the URL of the API module we'd like to utilize.
NSURL *url = [[NSURL alloc] initWithString:#"https://access.active911.com/interface/open_api/api/alerts"];
// Constructs a an HTTP request object to send to the server in order to obtain data.
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:url];
[request setValue:#"1" forHTTPHeaderField:#"alert_days"];
// This fetcher sends the request along with the authentication header in a recognizable manner.
GTMHTTPFetcher *fetcher = [[GTMHTTPFetcher alloc] initWithRequest:request];
// Attach the OAuth credentials for the fetcher's use.
[fetcher setAuthorizer:auth];
// Execute the operation.
[fetcher waitForCompletionWithTimeout:10];
NSLog(#"About to get alert");
[fetcher beginFetchWithDelegate:self didFinishSelector:#selector(responseHandler:finishedWithData:finishedWithError:)];
NSLog(#"got alert");
}
-(void)responseHandler:(id)valueNotUsed finishedWithData:(NSData *)data finishedWithError:(NSError *)error{
// Retrieve the server data in a usable object
// All that's being done here is conversion to an NSDictionary
// followed by the creation of subdictionaries from that dictionary
// until our final value can be picked directly out of the resulting dict
NSData *jsonData = [[NSData alloc] initWithData:data];
NSError *dictError;
NSDictionary* json = [NSJSONSerialization
JSONObjectWithData:jsonData //1
options:kNilOptions
error:&dictError];
NSDictionary *token = [json objectForKeyedSubscript:#"message"];
NSArray *alerts = [token objectForKeyedSubscript:#"alerts"];
NSDictionary *alertData = alerts[0];
mapCode = [alertData objectForKeyedSubscript:#"map_code"];
NSString *city = [alertData objectForKeyedSubscript:#"city"];
NSLog(#"Map code: '%#' with city '%#' and access token %#", mapCode, city, accessToken);
}
And I need to pass the mapCode to my view controller.
Thanks for the help!
First off, please rethink about having the UI halt while you fetch results from the server. This can create an extremely bad UX for the app and only should be done if absolutely necessary.
Second, does your responseHandler method work? And do you only need mapCode in the VC that responseHandler is in?
If so, you don't even need to use Notifications. Simply do:
-(void)responseHandler:(id)valueNotUsed finishedWithData:(NSData *)data finishedWithError:(NSError *)error{
...
...
mapCode = [alertData objectForKeyedSubscript:#"map_code"];
[self updateVCWithMapCode:mapCode];
}
That will call the method after the response has been received. Passing it explicitly too so you don't need to have mapCode be a property as well.
I'm quite new to Amazon S3 and I'm having difficulty downloading large files from S3.
I have successfully downloaded a file that is 35MB every time, but when the size of the file is really big around 500 MB - 1.7GB the application crashes.
When trying on the simulator I would get can't allocate region error after about 1GB of the download.
So then I tried it on the device. Now it seems to just crash at a random time and
no crash report is put in the device, therefor I'm having an issue debugging this problem.
At first I thought it was the device or even the simulator. But i'm not really sure.
Someone mentioned that S3 framework times out the downloads randomly occasionally for large files. Could this be the case?
I'm building the file by opening a data file seeking to the end, adding the data, then closing the file until the download is complete.
I'm not sure how to debug this problem.
Any help would be appreciated.
Thank you.
I am a maintainer of the AWS SDK for iOS. We recently patched the S3GetObjectResponse to allow the streaming of the data directly to disk without keeping the response data in memory.
S3GetObjectResponse.m
To enable this, you simply need to set the stream when creating your request:
NSOutputStream *outputStream = [[[NSOutputStream alloc] initToFileAtPath:FILE_NAME append:NO] autorelease];
[outputStream open];
S3GetObjectRequest *getObjectRequest = [[[S3GetObjectRequest alloc] initWithKey:FILE_NAME withBucket:BUCKET_NAME] autorelease];
getObjectRequest.outputStream = outputStream;
[s3 getObject:getObjectRequest];
Update: We added a post to our AWS Mobile Developer Blog on downloading large files with the AWS SDK for iOS that includes this info as well as other tips.
S3GetObjectRequest has NSMutableData* body where it appends all the data it downloads.
For large files as download progresses data is appended constantly, and it goes over the VM limit of 90MB and then app gets killed by iOS.
Quick and dirty workaround is to create your own S3GetObjectRequest and S3GetObjectResponse classes. AWS framework instantiates Response based on Class Name of Request (Class name of Request without last 7 chars "Request" and appends it with "Response", and tries to instantiate new class of that name).
Then to override -(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data to release body all the time.
This is quick and dirty fix simply because you still have constant data allocation, appending and then release. But it works when you are in a pinch. For my usage of downloading files of 150-700mb, this simple hack kept memory usage of the app at 2.55mb average, +/- 0.2mb.
As stated by the author of ASIHTTP library, it is no longer maintained.
Request - LargeFileS3GetObjectRequest.h
#interface LargeFileS3GetObjectRequest : S3GetObjectRequest
#end
Request - LargeFileS3GetObjectRequest.m
#implementation LargeFileS3GetObjectRequest
#end
Response - LargeFileS3GetObjectResponse.h
#interface LargeFileS3GetObjectResponse : S3GetObjectResponse
#end
Response - LargeFileS3GetObjectResponse.m
#implementation LargeFileS3GetObjectResponse
-(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
// allow original implementation to send data to delegates
[super connection:connection didReceiveData:data];
// release body and set it to NULL so that underlying implementation doesn't
// append on released object, but instead allocates new one
[body release];
body = NULL;
}
#end
Hope it helps.
You may want to stream the data to your application via ASIHTTPRequest
http://allseeing-i.com/ASIHTTPRequest/S3
NSString *secretAccessKey = #"my-secret-access-key";
NSString *accessKey = #"my-access-key";
NSString *bucket = #"my-bucket";
NSString *path = #"path/to/the/object";
ASIS3ObjectRequest *request = [ASIS3ObjectRequest requestWithBucket:bucket key:path];
[request setSecretAccessKey:secretAccessKey];
[request setAccessKey:accessKey];
[request startSynchronous];
if (![request error]) {
NSData *data = [request responseData];
} else {
NSLog(#"%#",[[request error] localizedDescription]);
}
/* Set up the Amazon client */
_s3 = [[AmazonS3Client alloc] initWithAccessKey:k_Amazon_ACCESS_KEY_ID withSecretKey:k_Amazon_SECRET_KEY];
_s3.endpoint = [AmazonEndpoints s3Endpoint:SA_EAST_1];
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(queue, ^{
/* Open a file stream for the download */
NSOutputStream *outputStream = [[NSOutputStream alloc] initToFileAtPath:[DOCUMENTS_DIRECTORY stringByAppendingPathComponent:k_Amazon_Video_Local_File_Name] append:NO];
[outputStream open];
/* Set up the s3 get object */
S3GetObjectRequest *getVideoRequest = [[S3GetObjectRequest alloc] initWithKey:k_Amazon_Video_Path withBucket:#""];
/* Set the stream */
getVideoRequest.outputStream = outputStream;
/* Get the response from Amazon */
S3GetObjectResponse *getObjectResponse = [_s3 getObject:getVideoRequest];
dispatch_async(dispatch_get_main_queue(), ^{
if(getObjectResponse.error != nil)
{
NSLog(#"S3 Error: %#", getObjectResponse.error);
}
else
{
NSLog(#"S3 - Video download complete and successful");
[[NSUserDefaults standardUserDefaults] setBool:YES forKey:k_Amazon_Video_Downloaded];
}
[[UIApplication sharedApplication] setNetworkActivityIndicatorVisible:NO];
});
});