On iOS, our application is downloading a zip file that's approximately 400MB. We're getting intermittent crashing while the file is downloading.
Current I'm using [NSFileHandle writeData:] to write the data as it comes in, and is not being stored in memory. But I'm wondering if the operating system is somehow storing it in memory?
Would NSOutputStream be a better solution to downloading large file? Or possibly standard unix file descriptors?
Our file handle is setup like this:
NSFileManager * fileManager = [NSFileManager defaultManager];
[fileManager createFileAtPath:tmpFilePath.path contents:nil attributes:nil];
_zipFile = [NSFileHandle fileHandleForWritingAtPath:tmpFilePath.path];
Currently my NSURLConnection delegate method looks like this:
- (void) connection:(NSURLConnection *) connection didReceiveData:(NSData *) data {
[_zipFile writeData:data];
}
So the data that comes in from the request is not stored or appended to any other data objects. Shouldn't this just write to disk and not effect memory?
Thanks
I don't think there's anything wrong with your use of NSFileHandle. I confess that I've always used NSOutputStream, but I just tried it both ways (NSFileHandle and NSOutputStream), and both appear to be quite respectful in terms of the memory consumed. Downloading a 40mb file, the allocations spiked to 3mb at the start of the download but quickly leveled off at 1mb):
So, I'd run your app through "Allocations" and "Leaks" (if you choose "Leaks", you get both) and see what it looks like.
If you haven't already, run your code through the static analyzer ("Analyze" on the "Product" menu), to make sure you don't have any issues. (You should have a clean bill of health there; if there are any issues reported, you must fix those.) I'd also make sure that zombies are turned off (because in the process of keeping track of all of those released objects, it creates a zombie object for each ... small, but it will consume memory.
Related
I have an app that downloads a whole bunch of data from over 100+ APIs upon successful login. I successfully download the data, and then use iExplorer to extract the data container folders (Documents, Library and Tmp) from the fully loaded application.
I would like to take a blank version of the original app, in .ipa format, and insert those data container folders into that fully loaded .ipa file. Then I will be able to take this new fully loaded .ipa, and use a deployment software to deploy it to a bunch of local user's devices. So everyone will have this fully loaded app.
Please, has anyone done this? Please provide some feedback, and don't argue with my methodology, because this has be done this way due to requirements. Maybe there is a step I'm missing? I'm not sure.
With the source code in hand, you can run the app in the simulator (no need for iExplorer), wait for it to download all the files and browse to the folder on your computer where the app was installed.
From there you can put aside any files you want along with their respective folders. If you're using Coredata there should be a SQLITE database file there somewhere (typically in your Application Support folder) and this might be all you need but it is hard to tell without looking at your implementation details.
Once you have the files you need set aside, add them to the app bundle via Xcode and create code to check whether files already exist (in which case you don't want to replace them), and if not copy all files needed from the bundle into their respective folders.
Here's some semi pseudo-code for you:
NSDictionary *userPrefs = [[NSUserDefaults standardUserDefaults] objectForKey:self.email];
if (![userPrefs[kInitialSetupCompleted] boolValue])
NSFileManager *fileManager = [NSFileManager defaultManager];
NSURL *destinationFilePath = ...
NSURL *seedFilePath = [NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:kCoreDataStoreName ofType:nil]];
NSError* err = nil;
if (![fileManager copyItemAtURL:seedPath toURL:destinationFilePath error:&err]) {
NSLog(#"Could not copy seed data. error: %#", err);
// Deal with error
} else {
// Set user defaults kInitialSetupCompleted to YES
}
}
I have an app that is using NSFileWrapper to create a backup of the user's data. This backup file contains text and media files (compression is not relevant here). Sometimes these backup files get quite large, over 200 MB in size. When I call NSFileWrapper -writeToURL... it appears to load the entire contents into memory as part of the writing process. On older devices, this causes my app to be terminated by the system due to memory constraints.
Is there a simple way to avoid having NSFileWrapper load everything into memory? I've read through every NSFileWrapper question on here that I could find. Any suggestions on how to tackle this?
Here is the current file structure of the backup file:
BackupContents.backupxyz
user.txt
- folder1
- audio files
asdf.caf
asdf2.caf
- folder2
- audio files
asdf3.caf
Again, please don't tell me to compress my audio files. That would only be a band-aid to a flawed design.
It seems like I could just move/copy all of the files into a directory using NSFileManager and then make that directory a package. Should I go down that path?
When an NSFileWrapper tree gets written out to disk, it will attempt to perform a hard-link of the original file to the the new location, but only if you supply a parameter for the originalContentsURL.
It sounds like you're constructing the file wrapper programmatically (for the backup scenario), so your files are probably scattered all over the filesystem. This would mean that when you writeToURL, you don't have an originalContentsURL. This means the hard-link logic is going to get skipped, and the file will get loaded so it can get rewritten.
So, if you want the hard-linking behavior, you need to find a way to provide an originalContentsURL. This is most easily done by supplying an appropriate URL to the initial writeToURL call.
Alternatively, you could try subclassing NSFileWrapper for regular files, and giving them an NSURL that they internally hang on to. You'd need to override writeToURL to pass this new URL up to super, but that URL should be enough to trigger the hard-link code. You'd want to then use this subclass of NSFileWrapper for the large files you want hard-linked in to place.
I'm storing some files in the Library directory in an iOS app, using the following methods to construct it. In the end, I can call [MyClass dataDirectory] to do my file handling and all is well. I've recently discovered, however, that some files seem to be mysteriously disappearing out of this directory. According to the documentation, this should not be the case. Is this a safe place to store persistent files?
The console output of this directory is: ~/var/mobile/Containers/Data/Application/{id}/Library/Data
+ (NSString*)libraryDirectory
{
return [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory, NSUserDomainMask, YES) lastObject];
}
+ (NSString*)dataDirectory
{
NSString* dir = [[self libraryDirectory] stringByAppendingPathComponent:#"Data"];
BOOL isDir=NO;
NSError * error = nil;
NSFileManager *fileManager = [NSFileManager new];
if (![fileManager fileExistsAtPath:dir isDirectory:&isDir] && isDir)
{
[[NSFileManager defaultManager] createDirectoryAtPath:dir
withIntermediateDirectories:YES
attributes:nil
error:&error];
}
[self addSkipBackupAttributeToItemAtURL:[NSURL fileURLWithPath:dir isDirectory:YES]];
if (error != nil) {
DDLogError(#"Fatal error creating ~/Library/Data directory: %#", error);
}
return dir;
}
And the skip method:
+ (BOOL)addSkipBackupAttributeToItemAtURL:(NSURL *)URL
{
if ([[NSFileManager defaultManager] fileExistsAtPath:[URL path]])
{
assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);
NSError *error = nil;
BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
forKey: NSURLIsExcludedFromBackupKey error: &error];
if(!success){
DDLogError(#"Error excluding %# from backup %#", [URL lastPathComponent], error);
}
return success;
}
return YES;
}
In the code you posted, the first problem is here:
if (![fileManager fileExistsAtPath:dir isDirectory:&isDir] && isDir)
At the point where this is evaluated, isDir will default to NO, and will be set to NO if the file does not exist or is not a directory. This will prevent the directory from being created. Remove && isDir or change to || !isDir to get the logic you want.
Now on to your original question:
Is this (a subdirectory of NSLibraryDirectory) a safe place to store persistent files?
Yes. NSLibraryDirectory is backed up by default. To comply with the iOS Data Storage Guidelines an application should not store user-created data in that location, but it is a safe place to store application data. NSApplicationSupportDirectory is a directory that is generally within the NSLibraryDirectory, and is the preferred place to store this kind of data. Data within that location will be backed up, and will be migrated during application and OS updates.
The iOS Data Storage Guidelines, File System Programming Guide, and App Programming Guide for iOS all provide guidance on where to put files, and how they will be backed up from standard file system locations.
Unless those files have had their NSURLIsExcludedFromBackupKey/kCFURLIsExcludedFromBackupKey resource metadata value altered. Then it gets much more complicated.
Files 'Excluded From Backup'
Generally, if a file outside of a Documents directory can be backed up, the system assumes it can also purge it under low space or other conditions. This is why setting NSURLIsExcludedFromBackupKey to YES on a file allows the file to persist even in low storage conditions. If your application sets NSURLIsExcludedFromBackupKey to YES for a file, your application assumes responsibility for the life of that file.
The catch here is that the backup process and the purge process do not follow the same logic. Apple's documentation indicates that for the purposes of controlling the backup behavior, it is possible to set NSURLIsExcludedFromBackupKey on a directory. The children of that directory will effectively inherit that resource value (in practice, this may not be accurate). The purge process, however, does not seem to have the same behavior. It may not check the backup exclusions of the parent directories and apply it to children, and as a result if a file does not have NSURLIsExcludedFromBackupKey explictly set it may be purged.
This gets even more complicated. If you were to read the documentation for the constant NSURLIsExcludedFromBackupKey you would see:
Some operations commonly made to user documents cause this property to be reset to false; consequently, do not use this property on user documents.
This actually applies to much more than user documents. For example, if you were to perform an atomic write on a file such as:
[thing writeToURL:URL atomically:YES encoding:NSUTF8StringEncoding error:&error]
If the file at URL had NSURLIsExcludedFromBackupKey set to YES before the write, it would now appear to be set to NO. An atomic write like this will first create a temporary file, write to that, and replace the original with the new file. In doing so, file and URL resource flags are not preserved. The original file had the NSURLIsExcludedFromBackupKey resource value set, the newly created file at the same location now does not. This is just one example; many Foundation APIs perform atomic writes like this implictly.
There are scenarios where this gets even more complex. When an application is updated it is installed into a new location with a new application container path. Data inside the old application container is migrated. There are few guarantees regarding what may or may not be migrated as part of the update process. It may be everything, it may be only some things. In particular there are is no guidance concerning how files or directories marked with the NSURLIsExcludedFromBackupKey resource attribute will be treated. In practice it seems that these are often the least likely files to be migrated, and when they are migrated the NSURLIsExcludedFromBackupKey attribute is rarely preserved.
OS updates are also an issue. Historically Over-The-Air updates have been problematic and have caused the NSURLIsExcludedFromBackupKey resource attribute to be effectively cleared or ignored. A "major" OS update will clear the device and restore from a backup - which is equivalent to migrating to new hardware. Files marked with the NSURLIsExcludedFromBackupKey resource attribute will not be migrated, and the application will have to re-create them.
Update scenarios are described in TechNote 2285: Testing iOS App Updates
Because of this, when using NSURLIsExcludedFromBackupKey it is generally best to set the value on every access, and as always should be done through the File Coordination APIs (unless you are writing to a shared group container, which is an entirely different set of issues). If the NSURLIsExcludedFromBackupKey resource attribute value is lost files can be purged at any time. Ideally an application should not depend on the NSURLIsExcludedFromBackupKey or how the OS may (or may not!) handle it, but instead be designed such that the data could be recreated on demand. That may not always be possible.
It's clear from your question and the code that you posted that you are somewhat dependant on NSURLIsExcludedFromBackupKey ensuring that your file(s) have an application-controlled lifetime. As you can see from the above, that may not always be the case: there are many, many common scenarios where that resource attribute value can disappear, and with it your files.
It is also worth noting that NSFileProtection attributes work the same way, and can disappear in the same scenarios (and a few more).
TL;DR; What should I do?
Based on your question, code, and the description of the behavior you are seeing:
Setting the NSURLIsExcludedFromBackupKey value on the directory containing the file(s) you are interested in preserving may not be enough to prevent them from being purged. It would be wise to set NSURLIsExcludedFromBackupKey on every access to the actual files, rather than just a parent directory. Also attempt to ensure this resource value is set after any write to the file, especially through a high level API that may be doing atomic writes, etc.
All NSFileManager and file reading/writing operations should use file coordination. Even in an application that is single threaded there will be other processes interacting with "your" files. Processes like the daemons that run backups or purge files during low space conditions. Between your -fileExistsAtPath: and the -setResourceValue:forKey:error: another process could alter, delete, or move your file and its attributes. -setResourceValue:forKey:error: will actually return YES and no error in many cases where it did nothing, like the file not existing.
Files and directories marked with NSURLIsExcludedFromBackupKey are the responsibility of the application to manage. The application should still purge those files or their contents at some appropriate time, or set limits on their growth. If you look at the per-application disk usage information on a device, you can probably guess the names of some applications that do not do this correctly.
Test update scenarios as described in TechNote 2285: Testing iOS App Updates. often. Ideally the iOS Simulator would have a "Simulate Low Disk Space" capability similar to simulating memory warnings, but at this time it does not.
If at all possible, alter application logic to recreate these files if they go missing.
In the documentation you linked it is stated that
Critical data should be stored in the /Documents directory. Critical data is any data that cannot be recreated by your app, such as user documents and other user-generated content.
It is also mentioned that
Cached data should be stored in the /Library/Caches directory. Examples of files you should put in the Caches directory include (but are not limited to) database cache files and downloadable content, such as that used by magazine, newspaper, and map apps. Your app should be able to gracefully handle situations where cached data is deleted by the system to free up disk space.
The directory you are using is not explicitly mentioned for storing user data, it is used by the system and is not save for your data. It's guarantied to be untouched by an update of your app, but that's it
To find the documents folder you could do something like
NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *documentsFolderPath = [paths firstObject];
I'm aware that there are similar questions posted, but this is a very specific issue that may or may not be related to code, it might be due to where I'm sourcing the file, and I need some advice.
I have an iPad app and am detecting whether there is an internet connection. If there is, then a .csv file is downloaded, saved, then split into an array. If not then a file held in the main bundle is used.
When using the file held in the main bundle, I can extract the data. The problem I have is when I try to download the .csv. The file is held on Document Manager, a Content Management System that is the only secure area for the file to be held for the company I work for and is therefore unavoidable. If I use this code:
NSString *urlString = #".../view-document.cgi?f=fundsspreadsheetc.csv";
(sorry, I need to keep the full link confidential)
NSURL *csvURL = [NSURL URLWithString:[urlString stringByAddingPercentageEscapesUsingEncoding:NSASCIIStringEncoding]];
NSData *urlData = [NSData dataWithContentsOfURL:csvURL options:NSDataReadingMappedAlways error:nil];
then urlData returns nil.
The original url opens the file on a windows laptop, but with a file name of view-document.cgi. I don't know if this is relevant.
If I change the url to:
"
http://download.finance.yahoo.com/d/quotes.csv?s=^GSPC+^IXIC+^dji+^GSPC+^BVSP+^GSPTSE+^FTSE+^GDAXI+^FCHI+^STOXX50E+^AEX+^IBEX+^SSMI+^N225+^AXJO+^HSI+^NSEI&f=sl1d1t1c1ohgv&e=.csv"
from a question set by shebi, then my code works.
What do I need to do to get my file to download?
much appreciated
Thanks for the help, it was problem with the link.
I'm looking for a better way to download a bunch of files from my iCloud container to my sandbox. This is what I currently use:
for (FileAttachments *fileLink in items) {
NSURL *cloudFileURL = [fileLink urlForCloudAttachmentWithName];
NSURL *fileURL = [backupCloudLocalDirectoryURL URLByAppendingPathComponent: fileLink.fileName];
NSFileCoordinator *fileCoordinator = [[NSFileCoordinator alloc] initWithFilePresenter:nil];
[fileCoordinator coordinateReadingItemAtURL:fileURL options:NSFileCoordinatorReadingWithoutChanges error:&error
byAccessor:^(NSURL *newURL) {
NSError *blockError = nil;
[fileManager copyItemAtURL:cloudFileURL toURL:fileURL error:&blockError];
}];
}
}
Is there any problem with making a copy of the iCloud item this way? In production, I have some users complaining that all their files weren't downloaded. Is it better to use NSFileManager's startDownloadingUbiquitousItemAtURL:error instead? If so, why?
Thanks.
It's still not completely clear to me from comments how you're discovering the URLs, however some important details that affect your situation are:
Using coordinateReadingItemAtURL:block: has nothing to do with downloading files from iCloud. The purpose of NSFileCoordinator is to coordinate among readers and writers of a file so that, for example, you don't get two threads trying to write to the same file at the same time. You use file coordinators with iCloud because the iCloud system needs to read and write files and so does your app. Using a coordinator avoids corrupting the file, but again, has nothing to do with downloading the file.
To download a file from iCloud you need to use startDownloadingUbiquitousItemAtURL:error: and then wait until the file downloads. The normal flow here is:
a. Use NSMetadataQuery to find files that exist in the iCloud account
b. Use startDownloadingUbiquitousItemAtURL:error: to make sure they're available on the local device.
The reason you need to use this call is simply because that's how iCloud works on iOS. Files don't download until you ask them to download, and this is how you ask. [On OS X it's different, everything automatically downloads.]
You cannot simply copy from an iCloud URL to another location unless you already know that the file has been downloaded. The file-copy operation does not download the file, so if the file isn't available locally, the copy will fail.
You must use a metadata query to identify the files and their download status then, if they have not been downloaded initiate the download, and using the metadata query determine when the download is complete and then copy the file from the ubiquity container to the apps sandbox directory using a file coordinator. If you try copying the file while it's partially downloaded you may get some strange results.
I had the same problem with you.
My case is that: When network disconnects, iCloud service copy file from iCloud container to sandbox. When this line executes, it can not go into the block to copy file. This is the reason why this file can not be copied.
fileCoordinator coordinateReadingItemAtURL:fileURL options:NSFileCoordinatorReadingWithoutChanges error:&error byAccessor:^(NSURL *newURL)
My solution is: Before copying file from iCLoud container to sandbox, you must check network. If it is not available, don't execute this code (return method). If network connects, execute fileCoordinator and copy.
More info: When copy file from ICloud container to sandbox, method fileManager copyItemAtURL:toURL:error: is OK because I implement this method and it's good.
Hope this works.