Large size image uploading issue with latest iOS dropbox sdk - ios

I am trying to upload 4 to 5 mb size image on dropbox using latest iOS dropbox sdk. But every time I get Error code =-1001 The operation couldn't be completed.
My internet connection speed is fair enough (2 mbps).
I have also tried by reducing image size to 512kb and it gets uploaded. But if I try to upload 5 or 6 images in sequence than again i get same error after successful upload of 2 or 3 images.
I have tried using below methods.
[self.restClient uploadFile:filename toPath:destDir withParentRev:nil fromPath:localPath]
[self.restClient uploadFileChunk:nil offset:0 fromPath:localPath]

According to the documentation a -1001 error means NSURLErrorTimedOut, meaning that the connection took too long. There are a number of factors that can contribute to this, including the device, the device's network stack and Internet connection, the particular route between your ISP and the destination servers, etc.
When it comes to your Internet connection, you only upload as fast as your Internet service provider (ISP) allows. Consumer ISP's normally provide less upload speed compared to download speed. In addition to this, your speeds may be slower than what your ISP rates it. Sometimes resetting or retrying your connection gets you a different route and better speeds, but that is outside of our control. Some ISPs also throttle sustained connections so if you see an initial high connection speed followed by lower speeds, that could be the reason.
When using uploadFile, you're attempting to upload the entirety of the file in one HTTPS request. If the file is big enough, along with the above considerations, this may make the request take too long and fail. For that reason, chunked uploading can be better.
When using uploadFileChunk, you're only attempting to upload a portion of the file per HTTPS request, meaning each request doesn't need to take as long and is less likely to fail.
In either case, one more thing to be aware of is how many simultaneous requests you're attempting to make. If you have too many running at the same time, they may all slow down, causing these failures. That being the case, you'll want to limit how many you run concurrently. Some developers choose a limit of 4 simultaneous connections, but the right number for your app will depend on the specifics of the scenario, so you may want to try a few different numbers.
Edit, here's some basic sample code for using chunked uploading:
- (void) doTest {
if ([[DBSession sharedSession] isLinked]) {
NSLog(#"Running test...");
// Write a file to the local documents directory
NSString *text = #"Hello world. Usually something much longer here.";
NSString *filename = #"working-draft.txt";
NSString *localDir = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];
self.localPath = [localDir stringByAppendingPathComponent:filename];
[text writeToFile:self.localPath atomically:YES encoding:NSUTF8StringEncoding error:nil];
self.dbClient = [[DBRestClient alloc] initWithSession:[DBSession sharedSession]];
self.dbClient.delegate = self;
// Start the upload. No upload ID known yet
[self.dbClient uploadFileChunk:nil offset:0 fromPath:self.localPath];
}
}
- (void)restClient:(DBRestClient *)client uploadedFileChunk:(NSString *)uploadId newOffset:(unsigned long long)offset
fromFile:(NSString *)localPath expires:(NSDate *)expiresDate {
NSLog(#"uploadedFileChunk: %#, newOffset: %llu, fromFile: %#, expires: %#", uploadId, offset, localPath, expiresDate);
NSFileManager *fileManager = [NSFileManager defaultManager];
NSDictionary *attrs = [fileManager attributesOfItemAtPath:self.localPath error: NULL];
// in real use, there's probably a dangerous race condition here in checking the size each time, but this is just for demonstration purposes
if (offset >= [attrs fileSize]) {
// all data has been uploaded
NSLog(#"Finishing upload...");
[self.dbClient uploadFile:#"chunked_uploaded_file.txt" toPath:#"/" withParentRev:nil fromUploadId:uploadId];
} else {
// more data to upload
NSLog(#"Continuing upload...");
[self.dbClient uploadFileChunk:uploadId offset:offset fromPath:localPath];
}
}
- (void)restClient:(DBRestClient *)client uploadFileChunkFailedWithError:(NSError *)error {
NSLog(#"uploadFileChunkFailedWithError: %#", error);
// in real use, the app should probably implement some retrying logic here
}
- (void)restClient:(DBRestClient *)client uploadFileChunkProgress:(CGFloat)progress {
NSLog(#"uploadFileChunkProgress: %f", progress);
}

Related

Getting byte Data from File

WHAT IM DOING I am trying to get an audio file (could be up to an hour long. eg. a Podcast) that I've recorded with AVAudioRecorder to be uploaded to our backend. In addition to being uploaded to the server it needs to be able to be "Paused" and "Resumed" if the user chooses. Because of this, I believe, I need to use dataWithBytesNoCopy:buffer on the NSData class to achieve this.
WHERE IM AT I know for a fact I can get the data with using the passed self.mediaURL property:
if (self.mediaURL) {
NSData *audioData = [NSData dataWithContentsOfURL:self.mediaURL];
if (audioData) {
[payloadDic setObject:audioData forKey:#"audioData"];
}
}
However, this will not give me the desired functionality. I am trying to keep track of the bytes uploaded so that I can resume if the user pauses.
QUESTION How can I use the provided self.mediaURL so that I can retrieve the file and be able to calculate the byte length like this example?
Byte *buffer = (Byte*)malloc((long)audioFile.size);
NSUInteger buffered =[rep getBytes:buffer fromOffset:0.0 length:(long)rep.size error:nil];
NSMutableData *body = [[NSMutableData alloc] init];
body = [NSMutableData dataWithBytesNoCopy:buffer length:buffered freeWhenDone:YES];
Instead of making things more complicated for yourself by trying to reinvent the wheel, use what the system gives you. NSURLSession lets you do a background upload. You hand the task to the session (created using the background session configuration) and just walk away. The upload takes place in pieces, when it can. No "pause" or "resume" needed; the system takes care of everything. Your app doesn't even have to be running. If authentication is needed, your app will be woken up in the background as required. This architecture is just made for the situation you describe.
If the problem is that you want random access to file data without having to read the whole thing into a massive NSData, use NSFileHandle.

Queue of AFHTTPRequestOperations creating Memory Buildup

I just updated to AFNetworking 2.0 and I am re-writing my code to download data & insert it into Core Data.
I download JSON data files (anywhere from 10-200mb files), write them to disk, then pass them off to background threads to process the data. Below is the code that downloads the JSON & write it to disk. If I just let this run (without even processing the data), the app uses up memory until it is killed.
I assume as the data is coming in, it is being stored in memory, but once I save to disk why would it stay in memory? Shouldn't the autorelease pool take care of this? I also set the responseData, and downloadData to nil. Is there something blatantly obvious that I am doing wrong here?
#autoreleasepool
{
for(int i = 1; i <= totalPages; i++)
{
NSString *path = ....
NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:path]];
AFHTTPRequestOperation *op = [[AFHTTPRequestOperation alloc] initWithRequest:request];
op.responseSerializer =[AFJSONResponseSerializer serializer];
[op setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject)
{
//convert dictionary to data
NSData *downloadData = [NSKeyedArchiver archivedDataWithRootObject:responseObject];
//save to disk
NSError *saveError = nil;
if (![fileManager fileExistsAtPath:targetPath isDirectory:false])
{
[downloadData writeToFile:targetPath options:NSDataWritingAtomic error:&saveError];
if (saveError != nil)
{
NSLog(#"Download save failed! Error: %#", [saveError description]);
}
}
responseObject = nil;
downloadData = nil;
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
DLog(#"Error: %#", error);
}];
}
[mutableOperations addObject:op];
}
NSArray *operations = [AFURLConnectionOperation batchOfRequestOperations:mutableOperations progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations) {
DLog(#"%lu of %lu complete", (unsigned long)numberOfFinishedOperations, (unsigned long)totalNumberOfOperations);
} completionBlock:^(NSArray *operations) {
DLog(#"All operations in batch complete");
}];
mutableOperations = nil;
[manager.operationQueue addOperations:operations waitUntilFinished:NO];
Thanks!
EDIT #1
Adding an #autoreleasepool within my complete block seemed to slow the memory usage a bit, but it still builds up and eventually crashes the app.
If your JSON files are really 10-200mb each, this would definitely cause memory problems, because this sort of request is going to load the responses in memory (rather than streaming them to persistent storage). Worse, because your using JSON, I think the problem is twice as bad, because you're going to be loading this into a dictionary/array, which also takes up memory. So, if you have four 100mb downloads going on, your peak memory usage could be of the order of magnitude of 800mb (100mb for the NSData plus ~100mb for the array/dictionary (possibly much larger), times four for the four concurrent requests). You could quickly run out of memory.
So, a couple of reactions:
When dealing with this volume of data, you'd want to pursue a streaming interface (a NSURLConnection or NSURLSessionDataTask where you write the data as it comes in, rather than holding it in memory; or use NSURLSessionDownloadTask which does this for you), one that writes the data directly to persistent storage (rather than trying to hold it in a NSData in RAM as it's being downloaded).
If you use NSURLSessionDownloadTask, this is really simple. If you need to support iOS versions prior to 7.0, I'm not sure if AFNetworking supports streaming of the responses directly to persistent storage. I'd wager you could write your own response serializer that does that, but I haven't tried it. I've always written my own NSURLConnectionDataDelegate methods that download directly to persistent storage (e.g. something like this).
You might not want to use JSON for this (because NSJSONSerialization will load the whole resource into memory, and then parse it to a NSArray/NSDictionary, also in memory), but rather use a format that lends itself to streamed parsing of the response (e.g. XML) and write a parser that stores the data to your data store (Core Data or SQLite) as it's being parsed, rather than trying to load the whole thing in RAM.
Note, even NSXMLParser is surprisingly memory inefficient (see this question). In the XMLPerformance sample, Apple demonstrates how you can use the more cumbersome LibXML2 to minimize the memory footprint of your XML parser.
By the way, I don't know if your JSON includes any binary data that you have encoded (e.g. base 64 or the like), but if so, you might want to consider a binary transfer format that doesn't have to do this conversion. Using base-64 or uuencode or whatever can increase your bandwidth and memory requirements. (If you're not dealing with binary data that has been encoded, then ignore this point.)
As an aside, you might want to use Reachability to confirm the user's connection type (Wifi vs cellular), because it is considered bad form to download that much data over cellular (at least not without the user's permission), not only because of speed issues, but also the risk of using up an excessive portion of their carrier's monthly data plan. I've even heard that Apple historically rejected apps that tried to download too much data over cellular.

Xcode - iOS - Simply upload a file to FTP Server

I'm trying to work with FTP Servers.
I have googled around for everything and everything is hard to understand for beginners like me. SimpleFTPSample is hard to understand because it is so much at a time. views, buttons, labels, textflelds, upload, download, request, list, get. Same with BlackRaccoon and everything else.
How to Simply and programily upload "test.txt" to FTP Server: "192.168.1.111" in Xcode (iPhone app) without views or button. Just code that can be in the ViewDidLoad for example.
Maybe something like this?:
NSURL* url = [NSURL URLWithString:#"ftp://username:pw#189.92.32.34"];
CFReadStreamRef stream = CFReadStreamCreateWithFTPURL(NULL, (__bridge CFURLRef) url);
stream.delegate= self;
[stream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
[stream open];
but which file?
expand this, or write a new code. i don't know, this is new for me.
Thanks Jonathan
As the writer of Black Raccoon perhaps I'm biased (well, I KNOW I'm biased), but I've attempted to make it as simple and powerful as possible. Let's look at what you want to do, upload a file:
There are four things we need to upload a file - start up code, then four delegate methods: overwrite check, data, success and the fail. Let's assume that you read the entire file into memory (okay for small files less than 2 megs).
First, you need this in your header:
BRRequestUpload *uploadData; // Black Raccoon's upload object
NSData *uploadData; // data we plan to upload
Now for the code part:
- (IBAction) uploadFile :(id)sender
{
//----- get the file path for the item we want to upload
NSString *applicationDocumentsDir = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
NSString *filepath = [NSString stringWithFormat: #"%#/%#", applicationDocumentsDir, #"file.text"];
//----- read the entire file into memory (small files only)
uploadData = [NSData dataWithContentsOfFile: filepath];
//----- create our upload object
uploadFile = [[BRRequestUpload alloc] initWithDelegate: self];
//----- for anonymous login just leave the username and password nil
uploadFile.path = #"/home/user/myfile.txt";
uploadFile.hostname = #"192.168.1.100";
uploadFile.username = #"yourusername";
uploadFile.password = #"yourpassword";
//----- we start the request
[uploadFile start];
}
The first will be asking your code if you want to overwrite an existing file.
-(BOOL) shouldOverwriteFileWithRequest: (BRRequest *) request
{
//----- set this as appropriate if you want the file to be overwritten
if (request == uploadFile)
{
//----- if uploading a file, we set it to YES (if set to NO, nothing happens)
return YES;
}
}
Next, Black Raccoon will ask you for chunks of data to send. If you have a very large file you NEVER want to try to send it all in one shot - Apple's API will choke and drop data. However, we only have one small chunk so we do this:
- (NSData *) requestDataToSend: (BRRequestUpload *) request
{
//----- returns data object or nil when complete
//----- basically, first time we return the pointer to the NSData.
//----- and BR will upload the data.
//----- Second time we return nil which means no more data to send
NSData *temp = uploadData; // this is a shallow copy of the pointer
uploadData = nil; // next time around, return nil...
return temp;
}
Remember we can ONLY do this for a small file.
Next we have our completion handler (if things worked according to plan):
-(void) requestCompleted: (BRRequest *) request
{
if (request == uploadFile)
{
NSLog(#"%# completed!", request);
uploadFile = nil;
}
}
Lastly we have our failure handler:
-(void) requestFailed:(BRRequest *) request
{
if (request == uploadFile)
{
NSLog(#"%#", request.error.message);
uploadFile = nil;
}
}
It would be WONDERFUL if it was as simple as saying [BRFtpUploadTo: dest srcfile: srcFile destfile: dstFile] but there are many reasons why you SHOULDN'T. Part of it has to do with how Apple has implemented their internal FTP. There are also the issues of blocking, errors, etc. In the end, FTP sounds like it should be trivial but ends up being a bit of a nightmare.
FTP is non-trivial which is why there are so many implementations. I'm not arguing that Black Raccoon is the best, but it is maintained with response to issues being between minutes to a couple of days.
It may look daunting at first, but Black Raccoon is, in my opinion, one of the better FTP libraries. I've spent a lot of time and effort to make it a quality product with excellent response to issues. How do I do this for free? Volume. ;)
Good luck with whatever FTP software you end up with!
Upload path is required when uploading. That is the way FTP works.
The port is the standard FTP port. I know of no way to change this without violating the API. If you figure it out, you stand a good chance of not passing Apple's check.
This code will upload/download any file.
I do not know how to make this work under secure conditions. This uses Apple's FTP protocol. There are other FTP packages that have built this from scratch and are far more intelligent. I would look into them.
BR was designed because I needed simple FTP communication. White Raccoon didn't do this for me because at the time (it has since been modernized).

iOS: sending .jpg image as base64 to tcp server

This one is going to kill me. I'm so close to getting this done except for this one stupid problem. And I am not sure I will be able to adequately describe the problem, but I'll try.
My enterprise app uses the iPhone camera to take pictures of receipts of purchases made by our field personnel. I used a real cool API for turning the jpeg data to base 64 (https://github.com/nicklockwood/Base64) to send via TCP connection to the VB 2010 server, which reads it as a text string and converts it back to a binary.
When the base64 file is created, it is first saved to disk on the phone, because there may be more images. Then when ready to send, the process will read each base64 file and send it one at a time.
The text string created by the base64 function is quite large, and at first it was only sending about 131,000 bytes, which would convert back to binary easily enough but would render about 1/4 to 1/3 of the image. I just figured that the data was being truncated because the app was trying to get ahead of itself.
So then I found a nice snippet that showed me how to use the NSStreamEventHasSpaceAvailable event to split the base64 string into several chunks and send them sequentially. (http://www.ios-developer.net/iphone-ipad-programmer/development/tcpip/tcp-client) That works great insofar as it sends the full file -- that is, the resulting file received by the server is the correct size, the same as the base64 file before it's sent.
The problem here is that at some point the file received by the server is corrupted because it seems to start all over at the beginning of the file... in other words, the data starts to repeat itself.
The odd part is that the repeating part starts at exactly the same spot in the received file every time: at position 131016. It doesn't start the repetition at the end of the file, it just interrupts the file at that point and jumps back to the beginning. And it happens that that was the size of the file that was sent before I started using the HasSpaceAvailable event. I can't figure out what the significance of the value 131,016 is. Maximum buffer size somewhere?
Using all kinds of NSLogs and breakpoints, I have pretty much determined that the data is leaving the phone that way, and not being scrambled by the server. I also wrote in an NSMailComposeViewer method that would email me the base64 file as an attachment, and it comes through perfectly.
Here is the code for when the file is read from disk and sent to the server:
int i;
for (i = 0; i < [imageList count];i++){
NSArray *documentsPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *documentsDirectory= [documentsPath objectAtIndex:0]; //Get the docs directory
NSString *imagePath = [documentsDirectory stringByAppendingPathComponent:imageFileName];
imageFileName = [imageList objectAtIndex:i] ;
NSLog(#"Image index: %d - image file name: %#",i,imageFileName);
BOOL fileExists = [[NSFileManager defaultManager] fileExistsAtPath:imagePath];
if(fileExists == YES){
NSString *imageReceipt = [NSString stringWithContentsOfFile:imagePath encoding:NSASCIIStringEncoding error:nil];
int32_t imageStringLen = [imageReceipt length];
NSString *imageSize = [NSString stringWithFormat: #"%d",imageStringLen];
NSString *currentImage = [NSString stringWithFormat:#"image,%#,%#,%#",imageFileName,imageSize,imageReceipt]; //creates a CSV string with a header string with the filename and file size, and then appends the image data as the final comma-separated string.
data = [[NSMutableData alloc] initWithData:[currentImage dataUsingEncoding:NSASCIIStringEncoding]];
[outputStream write:[data bytes] maxLength:[data length]];
And then here is the code that uses the HasSpaceAvailable event:
case NSStreamEventHasSpaceAvailable:
if (data != nil)
{
//Send rest of the packet
int ActualOutputBytes = [outputStream write:[data bytes] maxLength:[data length]];
int totalLength = [data length];
if (ActualOutputBytes >= totalLength)
{
//It was all sent
data = nil;
}
else
{
//Only partially sent
[data replaceBytesInRange:NSMakeRange(0, ActualOutputBytes) withBytes:NULL length:0]; //Remove sent bytes from the start
}
}
break;
(I especially like this code because it would allow placing a ProgressView control on the screen.)
The network stream event handler code is in the root view controller, but the image data in base64 is being sent from another view controller. My instinct tells me that this is not a problem because it has worked fine until now, but with much shorter strings.
Now, there's one other issue that may be related -- and probably is. I can't seem to complete the transfer of the data unless I close the app. The server doesn't see it until the connection is closed, I guess. I have tried placing [outputStream close] in various places in the code to no avail. I've also tried terminating the base64 string with a linefeed or carriage return or both.
The server is programmed to save the file when it has seen the correct number of bytes, but that never happens until the app is closed. I know by using WireShark on the server that some of the data is being received, but the remaining data, as I have said, doesn't arrive until the app is closed.
I suspect that this last part (completing the transfer) is the problem, but for the life of me, I can't find anything online that addresses this, unless I am just too ignorant to know what search terms to use.... which is highly likely.
I hope I have given enough information. Can anyone help me?
EDIT
The solution to the problem appeared to be different than what was suspected in the original answer. Through the discussion in the comments the QP has been led to a solution. Here is a summary:
The transmission of data through a raw TCP socket requires thorough handling of the enqueing logic, which was not properly taken into account by the QP. I recommended to use the socket library CocoaAsyncSocket which handles this part of the task and which in turn led the QP to a working solution.
Original Answer:
I'd guess NSString is not up to the task. It's made to hold, well, strings.
Try to read the file into an NSData directly instead and sending it as binary. (Its ascii in the end, isn't it?) Besides, this will be much more resource friendly than your current code.

IOS: Using stringWithContentsOfURL when network is unavailable

In this code poll from within my app for a reachable network
("http://soxxx9.cafe24.com/event.php")
NSString * szURL =[NSString stringWithFormat:#"http://soxxx9.cafe24.com/event.php"];
NSURL *url = [NSURL URLWithString:[szURL stringByAddingPercentEscapesUsingEncoding:NSUTF8StringEncoding ]];
NSString *strData;
while(1)
{
NSError *error = nil;
strData = [NSString stringWithContentsOfURL:url
encoding:NSUTF8StringEncoding
error:&error];
if(!error)
break;
//String data is not owned by me, no need to release
}
If you have a better way, please teach me.
This code seems to be heavily power consuming when network is out : you'll try million times to download something that is unreachable...
Have a look at the Reachability class, provided by Apple (http://developer.apple.com/library/ios/#samplecode/Reachability/Introduction/Intro.html). You'll find ARCified versions on gitHub (https://github.com/tonymillion/Reachability for example).
The idea is to register for notifications about the network reachability.
So, in your code :
Check network resource availability before retrieving the string you want.
If this is available, use your code WITHOUT the while(TRUE)
Check your string for any error while retrieving it client side = in your code
If the network is not available, you'll have to inform the user that network is unreachable, and register for reachability notifications to retrieve your string as soon as it is reachable again for example.
You should a class to handle the connection for you. This way you have more control of what's going on with it. MKNetworkKit is a solution, you can check it here.

Resources