what is the expected memory usage of NSKeyedArchiver during the encoding process ?
in encoding ~4MB of an object graph into an NSData object, Xcode reveals the process uses almost 4-5x the memory just in the process of encoding (which is eventually causing a crash with larger object graphs), but the final, encoded NSData object remains around ~4MB. A picture is attached below, where the massive spike occurs during the encoding process. Is this expected behavior?
Related
I'm doing some research on how iPhone manage the heap and stack but it's very difficult to find a good source of information about this. I'm trying to trace how a password is kept in memory, even after the NSString is deallocated.
As far as I can tell, an iPhone will not clear the memory content (write zeros or garbage) once the release count in ARC go down to 0. So the string with the password will live in memory until that memory position is overridden.
There's a debug option in Xcode, Malloc Scribble, to debug memory problems that will fill deallocated memory with 0x55, by enabling/disabling this option (and disabling Zombies), and after a memory dump of the simulator (using gcore) I can check if the content has been replaced in memory with 0x55.
I wonder if this is something that can be done with the Apple Store builds, fill deallocated memory with garbage data, if my assumption that iPhone will not do that by default is correct or not, or if there's any other better option to handle sensitive data in memory, and how it should be cleared after using it (Mutable data maybe? write in that memory position?)
I don't think that there's something that can be done on the build settings level. You can, however, apply some sort of memory scrubbing yourself by zeroing the memory (use memset with the pointer to your string).
As #Artal was saying, memset can be used to write in a memory position. I found this framework "iMAS Secure Memory" that can be useful to handle this:
The "iMAS Secure Memory" framework provides a set of tools for
securing, clearing, and validating memory regions and individual
variables. It allows an object to have it's data sections overwritten
in memory either with an encrypted version or null bytes
They have a method that it should be useful to clear a memory position:
// Return NO if wipe failed
extern inline BOOL wipe(NSObject* obj) {
NSLog(#"Object pointer: %p", obj);
if(handleType(obj, #"", &wipeWrapper) == YES) {
if (getSize(obj) > 0){
NSLog(#"WIPE OBJ");
memset(getStart(obj), 0, getSize(obj));
}
else
NSLog(#"WIPE: Unsupported Object.");
}
return YES;
}
What is the difference between
+ (instancetype)dataWithBytes:(const void *)bytes length:(NSUInteger)length;
and
+ (instancetype)dataWithBytesNoCopy:(void *)bytes length:(NSUInteger)length;
Also,
+ (instancetype)dataWithBytesNoCopy:(void *)bytes length:(NSUInteger)length freeWhenDone:(BOOL)b;
if b == YES, will it free the bytes automatically after converted to data?
I am working on an app and almost finished it. But the last problem is it crashes with memory error when it runs on device. It only crashes when on device, but in simulator it is perfect.
"malloc: * error for object 0x17415d0c0: Invalid pointer dequeued from free list * set a breakpoint in malloc_error_break to debug";
I have been working on this issue for several days:
iOS - My app crashes with memory Error if only runs on Device
But finally I found the problem, inside my Encryption and Decryption function, I have this:
Byte *buffer = (Byte*)malloc(asciiDataLength);
After I process with buffer, I convert it to NSData:
NSData *plainData = [NSData dataWithBytesNoCopy:buffer length:asciiDataLength freeWhenDone:YES];
This code caused my app to crash continuously, I changed it to
NSData *plainData = [NSData dataWithBytes:buffer length:asciiDataLength];
free(buffer);
Then my app never crash again.
So, I have to free the Byte by myself, ARC will not free it for me.
+ dataWithBytes:length::
Creates and returns a data object containing a given number of bytes copied from a given buffer.
+ dataWithBytesNoCopy:length::
Creates and returns a data object that holds length bytes from the buffer bytes.
dataWithBytes makes a copy of the buffer for the data, while the NoCopy version does not.
Important note: in the discussion section of dataWithBytesNoCopy:length::
The returned object takes ownership of the bytes pointer and frees it on deallocation. Therefore, bytes must point to a memory block allocated with malloc.
This means that initialising with this method essentially hands ownership of the memory to the NSData object, which will release it with free once it is done. If you try to initialise it with memory that you didn't allocate with malloc, your app will crash when the data object is deallocated.
dataWithBytesNoCopy is useful for when you get the bytes in a buffer from somewhere else, and are ready to hand them over to the NSData object, and won't use them yourself again outside of that.
If you want to initialise the data with memory you manage yourself, use + dataWithBytesNoCopy:length:freeWhenDone:. This is useful if the buffer will be stored somewhere persistently, and not changed or released.
However, if you are not sure how to correctly manage this memory manually, it is better to use dataWithBytes. The other methods are present for performance reasons, as avoiding copying large chunks of data can save a lot of time, but if you aren't sure how to use them, it's probably best not to — an app that doesn't crash is preferable to an app that crashes quickly.
[[NSData alloc] initWithBytes:buffer length:buflength] create a data object containing buflength bytes copied from the buffer bytes.
[NSData dataWithBytesNoCopy:buffer length:buflength] creates a data object that holds buflength bytes from the buffer bytes. The returned object takes ownership of the buffer pointer and frees it on deallocation. Therefore, buffer must point to a memory block allocated with malloc.
I'm parsing a JSON file on an iPad which has about 53 MB. The parsing is working fine, I'm using Yajlparser which is a SAX parser and have set it up like this:
NSData *data = [NSData dataWithContentsOfFile:path options:NSDataReadingMappedAlways|NSDataReadingUncached error:&parseError];
YAJLParser *parser = [[YAJLParser alloc] init];
parser.delegate = self;
[parser parse:data];
Everything worked fine until now, but the JSON-file became bigger and now I'm suddenly experiencing memory warnings on the iPad 2. It receives 4 Memory Warnings and then just crashes. On the iPad 3 it works flawlessly without any mem warnings.
I have started profiling it with Instruments and found a lot of CFNumber allocations (I have stopped Instruments after a couple of minutes, I had it run before until the crash and the CFNumber thing was at about 60 mb or more).
After opening the CFNumber detail, it showed up a huge list of allocations. One of them showed me the following:
and another one here:
So what am I doing wrong? And what does that number (e.g. 72.8% in the last image) stand for? I'm using ARC so I'm not doing any Release or Retain or whatever.
Thanks for your help.
Cheers
EDIT: I have already asked the question about how to parse such huge files here: iPad - Parsing an extremely huge json - File (between 50 and 100 mb)
So the parsing itself seems to be fine.
See Apple's Core Data documentation on Efficiently Importing Data, particularly "Reducing Peak Memory Footprint".
You will need to make sure you don't have too many new entities in memory at once, which involves saving and resetting your context at regular intervals while you parse the data, as well as using autorelease pools well.
The general sudo code would be something like this:
while (there is new data) {
#autoreleasepool {
importAnItem();
if (we have imported more than 100 items) {
[context save:...];
[context reset];
}
}
}
So basically, put an autorelease pool around your main loop or parsing code. Count how many NSManagedObject instances you have created, and periodically save and reset the managed object context to flush these out of memory. This should keep your memory footprint down. The number 100 is arbitrary and you might want to experiment with different values.
Because you are saving the context for each batch, you may want to import into a temporary copy of your store in case something goes wrong and leaves you with a partial import. When everything is finished you can overwrite the original store.
Try to use [self.managedObjectContext refreshObject:obj refreshChanges:NO] after certain amount of insert operations. This will turn NSManagedObjects into faults and free up some memory.
Apple Docs on provided methods
I'm using the following openssl functions:
PKCS5_PBKDF2_HMAC_SHA1
EVP_BytesToKey
EVP_aes_256_cbc
EVP_sha1
EVP_CIPHER_CTX_init
EVP_EncryptInit_ex
EVP_DecryptInit_ex
EVP_CIPHER_CTX_cleanup
EVP_DecryptUpdate
EVP_DecryptFinal_ex
EVP_EncryptUpdate
EVP_EncryptFinal_ex
When decrypting data on a background thread, there are no issues if the data size is fairly small.
For data around 500 kilobytes the decrypt routine will always crash on EVP_DecryptUpdate but is fine on the main thread.
What gives?
I am using the ASIHTTPRequest Library in my project. It works very well with iOS5.1 in an app using ARC. (files in this library use compiler flag -fno-objc-arc) However, when I run analyzer it shows multiple potential memory leaks, especially in ASIHTTPRequest. I am a little reluctant to start making changes in this library, that is widely used, is quite complex and is working fine in my project.
Suggestions?
an example:
ASIHTTPRequest.m line 1515
// Find out how much data we've uploaded so far
[self setTotalBytesSent:[NSMakeCollectable([(NSNumber *)CFReadStreamCopyProperty((CFReadStreamRef)[self readStream], kCFStreamPropertyHTTPRequestBytesWrittenCount) autorelease]) unsignedLongLongValue]];
if (totalBytesSent > lastBytesSent) {
Analyzer message:
call to CFReadStreamCopyProperty returns a core foundation object with a +1 retain count of
object sent autorelease message
object sent autorelease message
object leaked: allocated object is not referenced later in this excecution path and has a retain count of +1