decoding a HUGE NSString, running out of memory - ios

Question

I'm looking for ideas on how to improve a process of decoding a 40+MB NSString with base64 encoding and saving it to a file while being able to fit the process into iPad 1's 256 MB of RAM
I get the NSString from NSXMLParser:
id pointerToString;
- (void)parser:(NSXMLParser *)parser foundCharacters:(NSString *)string{
if ([currentElement isEqualToString:#"myElement"])
{
pointerToString = [string retain];
}
}
Then I use the pointerToString in a callback:
[handler performSelector: action withObject: pointerToString];
In the callback (id value is the pointerToString). I initialize NSData with the pointerToString while decoding it with base64 encoding.
^(id value)
{
if ( [[value class] isSubclassOfClass:[NSString class]] )
{
NSData *data = [NSData dataFromBase64String:value];
[data writeToFile:file.path atomically:YES];
}
}
the iPad 1 device runs out of memory and gets killed by the iOS when the memory allocation reaches around 130MB after or during the NSData call.
I have determined that in order to process the 40+MB NSString this way, I'd need about 180+MB of RAM (this is what the maximum memory allocation is on iPad 2 & 3, where the process works because of more RAM)
Any ideas/tips ?
Thank you

Answer 1

Edit:
When dealing with a file of this size, you probably do not want to load the entire multi-megabyte file in memory at one time, neither the huge input file nor the almost-as-huge output file. You should be parsing this in a streaming fashion, decoding the data in your foundCharacters as you go along, not holding any significant portions in memory.
The traditional techniques, though, may hold your entire XML file memory in three phases of the process:
As you download the XML file from the server;
As the XML parser parses that file; and
As you do the Base64-decode of the file.
The trick is to employ a streaming technique, that does these three processes at once, for small chunks of the single, large XML file. Bottom line, as you're downloading the entire 50mb file, grab a few kb, parse the XML, and if you're parsing the Base64-encoded field, perform the Base64-decode for that few kb, and the proceed to the next chunk of data.
For an example of this (at least the streaming XML downloading-and-parsing, not including the Base64-decoding), please see Apple's XMLPerformance sample project. You'll see that it will demonstrate two XML parsers, the NSXMLParser that we're all familiar with, as well as the less familiar LibXML parser. The issue with NSXMLParser is that, left to it's own devices, will load the entire XML file in memory before it starts parsing, even if you use initWithContentsOfURL.
In my previous answer, I mistakenly claimed that by using initWithContentsOfURL, the NSXMLParser would parse the URL's contents in nice little packets as they were being downloaded. The foundCharacters method of NSXMLParserDelegate protocol seems so analogous to the NSURLConnectionDelegate method, didReceiveData, that I was sure that NSXMLParser was going to handle the stream just like NSURLConnection does, namely returning information as the download was in progress. Sadly, it doesn't.
By using LibXML, though, like the Apple XMLPerformance sample project, you can actually use the NSURLConnection ability of streaming, and thus parse the XML on the fly.
I have created a little test project, but I might suggest that you go through Apple's XMLPerformance sample project in some detail. But in my experiment, a 56mb XML file consumed well over 100mb when parsing and converting via NSXMLParser but only consumed 2mb when using LibXML2.
In your comments, you describe the desire to download the Base64-encoded data to a file and then decode that. That approach seems a lot less efficient, but certainly could work. By the way, on that initial download, you have the same memory problem (that I solve above). I urge you to make sure that your initial download of the Base64-encoded data does not blithely load it into RAM like most routines do. You want to, assuming you're using NSURLConnection, write the data to the NSOutputStream as you receive the data in didReceiveData, not hold it in RAM.
See the didReceiveResponse in AdvancedGetController.m of Apple's AdvancedURLConnections example for an example of how to write a file as it's being received, rather than typical patterns of adding it to a NSMutableData (because most of these routines just assume you're dealing with a reasonably sized file). (Ignore all the stuff in that AdvancedURLConnections sample about authentication and the like, but focus on understanding how it's writing to the NSOutputStream as it goes.) This technique will address the first of the three problems listed at the top of this answer, but not the latter two. For that, you'll have to contemplate using LibXML2 as illustrated in Apple's XMLPerformance sample project, or other similar techniques.

Answer 2

The method
- (void)parser:(NSXMLParser *)parser foundCharacters:(NSString *)string
is probably not receiving all the data at once.
Doc is saying
"Sent by a parser object to provide its delegate with a string representing all or part of the characters of the current element."
So it is called multiple times.
It looks like you are trying to write the whole string at once (sorry if I am wrong).
So you could append the received data to the file by doing the following:
You can use a combination of
-writeData:
and
-seekToEndOfFile
methods from NSFileHandle class for writing NSData to the end of a file.
But be carefull with your base64 encoding on partial data receivment!