I have an app that lets you download "modules" that can expand your app usage.
When the user downloads a module, I fetch a ZIP file from a server and extract it to his Caches folder. (Each of these zips could be sized anywhere from 60k to 2MB).
Unfortunately, there are over 300 modules available, and many of the users download at least 50-60 of these to their device.
Lately, I got many complaints that modules just disappear off the user device, so I did some investigation and came across the following wording in Apple's documentation.
iOS will delete your files from the Caches directory when necessary,
so your app will need to degrade gracefully if it's data files are
deleted.
And also the following article explaining further about this situation:
http://iphoneincubator.com/blog/data-management/local-file-storage-in-ios-5
My problem is, I have no actual way of degrading gracefully, since I can't automatically let the user download so many modules. It could take hours depending on the internet connection and size of the modules.
So I have a few questions:
Did any of you ever had to deal with a similar situation, and if yes, how?
Does anyone know when exactly iOS purges the Cache? What is considered "low space" warning? This way I could at least give the user a warning that he doesnt have enough space to install a new module.
Is there a way to receive some sort of warning before the Cache folder is cleared?
This is a really frustrating move from Apple and I don't really see a way out. Would really love to hear some ideas from you.
Edit: After some testing, this seems to work just fine and doesn't get purged.
This isn't 100% confirmed yet, but it worked fine on our basic tests (I'll post more thorough results as they come). It seems saving the data to the app's "Application Support" folder resolves these issues, as this folder isn't purged.
The docs state:
Use the Application Support directory constant NSApplicationSupportDirectory, appending your <bundle_ID> for:
Resource and data files that your app creates and manages for the user. You might use this directory to store app state information, computed or downloaded data, or even user created data that you manage on behalf of the user / Autosave files.`
You could get to that folder as follows (Notice appending of the bundle ID as requested by the official apple docs):
[[NSSearchPathForDirectoriesInDomains(NSApplicationSupportDirectory, NSUserDomainMask, YES) lastObject] stringByAppendingPathComponent:[[NSBundle mainBundle] bundleIdentifier]]
Hope this helps anyone , and as I said, I will post more thorough test results during the weekend.
Edit 2:
Its very important to add this line to prevent syncing of temporary content from Application Support. I use it in the end of my applicationDidFinishLaunching:
[[NSURL URLWithString:NSApplicationSupportDir] setResourceValue:#(YES) forKey:NSURLIsExcludedFromBackupKey error:nil];
Related
Apple's data storage guidelines state the following:
2) Data that can be downloaded again or regenerated should be stored
in the /Library/Caches directory.
...and (emphasis mine):
4) Use the "do not back up" attribute for specifying files that should remain on device, even in low storage situations. Use this attribute
with data that can be recreated but needs to persist even in low
storage situations for proper functioning of your app or because
customers expect it to be available during offline use. This attribute
works on marked files regardless of what directory they are in,
including the Documents directory. These files will not be purged and
will not be included in the user's iCloud or iTunes backup. Because
these files do use on-device storage space, your app is responsible
for monitoring and purging these files periodically.
The page that Apple links to with a more detailed discussion of this topic does not mention anything about the attribute doing anything to prevent cached data from being purged.
So does anyone know if the "do not back up" attribute actually works like "do not backup and do not delete" for items placed in /Library/Caches, or if files still need to be stored inside of the application's Documents directory to ensure that they are not deleted when the device is running low on space?
I've made a quick test on iPhone 5 iOS 7.1.1:
I put some files to “/Library/Caches” (NSCachesDirectory) and mark them with NSURLIsExcludedFromBackupKey attribute. Then I put some more big files to the same directory in a normal way.
I then made a low disk space warning by taking a long video with the camera app.
After the warning, files marked with “do not backup” was not deleted from Cache, but the other files was! So, this attribute is really working and made two different things in spite of its name – excludes from backup and preserves from being purged on low space warning.
To me this is very clear, and the directory structure is the one I would follow first, just as Apple says. If you both want to save the file and not back it up, put it in Documents, preferably in a folder that you mark.
Even if you observe some behavior today using Caches, it may change in the future.
Currently researching this same question and also made the assumption that adding that attribute will prevent a file from being purged in a low disk space scenario (You know, because apple's documentation literally says that). I thought I would update this with my findings since it has been a few years since the accepted answer. I have a database that can be recreated and contains no user generated data, so I store it in caches, but contains some data that can affect the proper functioning of the application, so I don't want it to be purged when running out of disk space (currently happening and logged as a bug). I have experimented with adding the NSURLIsExcludedFromBackupKey to the file itself and file is still being removed after purge happens. So it does not seem to prevent purge as of iOS 11.3. Not sure what the next move is as I really don't want to move the database for all our users, but that's probably the next step so it will be safe in Documents. Please correct me if someone has a different experience.
When a file is passed into an iOS application by the document interaction system, a copy of the file is stored in the application bundle's Documents/Inbox folder. After the application has processed the file, it obviously needs to remove the file from Documents/Inbox, otherwise the folder will continue to grow and waste storage on the device.
I am uncomfortable with this simple solution (A), however, because my app needs to interact with the user before it can finish processing and removing the file. If the user suspends the app during this interaction period, and the app then gets killed while it is in the background, the stale file will not be removed when the app starts up the next time. Of course I can improve my app to cover this scenario, but I suspect that there will always be another border case that will leave me with an "unclean" Documents/Inbox folder.
A preferrable solution (B) therefore would be to remove the Documents/Inbox folder at an appropriate time (e.g. when the app launches normally, i.e. not via document interaction). I am still uncomfortable with this because I would be accessing a filesystem path whose location is not officially documented anywhere. For instance, my app would break if in a future version of iOS the document interaction system no longer places files in Document/Inbox.
So my questions are:
Would you recommend solution A or B?
Do you use a different approach and can you maybe give an outline of how your app manages Document/Inbox?
Last but not least: Do you know a piece of official Apple documentation that covers the topic and that I have overlooked?
Since I have asked this question, I have implemented the following solution:
I have redesigned my app so that it now immediately processes a file handed over to it via documentation interaction, without involving the user at all. Unless the app crashes, or is suspended and killed, in the middle of processing the file, this should always leave me with a clean Documents/Inbox.
To cover the (rare) case of a crash or suspend/kill, my app removes the Documents/Inbox folder when it is launched normallly (i.e. without the purpose of document interaction). To achieve this the Documents/Inbox folder path is necessarily hardcoded.
Here are the thoughts that went into the solution:
Redesigning the app
Initially it seemed like a good idea to offer the user a choice how she wanted a file to be processed - after all, offering a choice would make the app more flexible and provide the user with more freedom, right?
I then realized that I was trying to shift the responsibility for deciding how document interaction should be handled to the user. So I bit the bullet, made the hard decisions up-front, and then went happily on to implement a simple and straightforward document interaction system in my app.
As it turns out, no user interaction also means that the app is easier to use, so here's a win-win situation, both for me as a developer and for the users of my app.
Removing Documents/Inbox folder during app launch
Removing the Documents/Inbox folder during app launch is just an afterthought, not an essential part of how my app handles document interaction
Therefore I am quite willing to take the risk that Apple might change the filesystem path of the inbox folder at some point in the future. The worst thing that can happen is that my app will start to accumulate a few files that are leftovers from crash or suspend/kill scenarios.
And finally, a few thoughts for further development:
If it ever turns out that there really should be more than one way how the app can handle document interaction, I would add a user preference so that the user has to make a decision up-front, and the app does not need to stop its processing to ask the user for a choice.
If it ever turns out that user interaction is absolutely unavoidable in the middle of the document interaction handling process, I would look at this approach: 1) Before the user is allowed to interact, move the file from Documents/Inbox to some sort of "staging" folder; 2) Let user interaction take place; 3) Process the file in the "staging" folder, in whatever way the user chose. The important thing here is that the "staging" folder is in a known location and is completely managed by the app. Should the user suspend and then kill the app in the middle of the user interaction step, an appropriate action can simply be taken when the app is launched for the next time.
EDIT
In iOS 7 it is no longer possible to remove Documents/Inbox once it has been created. The NSFileManager method removeItemAtPath:error: returns Cocoa error 513 which resolves to NSFileWriteNoPermissionError (cf. this list of Foundation constants). The error does not seem to be related to POSIX permissions, however, it rather looks as if the system itself interferes with the attempt at deletion (possibly protection of the app bundle structure?).
Also noteworthy is that nowadays Apple explicitly names Documents/Inbox in the documentation of the UIApplicationDelegate method application:openURL:sourceApplication:annotation:. They also say that
[...] Your app has permission to read and delete files in this directory but does not have permission to write to them. If you want to modify a file, you must move it to a different directory first.
There is more stuff about possible encryption of the files in the docs, but you should read up on that yourself.
This problem has become much more complicated with the introduction of the Files app, and the "Open in place" feature.
If you do not have "Supports opening documents in place" switched on, in your info.plist, then things are pretty much the same, and files opened from any other app still appear in the Documents/Inbox directory. But files opened from the Files app appear in another inbox, currently at tmp/<bundle ID of app>-inbox. It is still recommended to delete the file once you are done with it, but there is less need to occasionally clean the directory, because the tmp directory gets cleaned by iOS once in a while anyways.
If you do have "Supports opening documents in place" switched on, then things change drastically. Files opened from the Files app and some other apps are no longer copied into an inbox, but they are passed to you at their original location. That is typically some location inside the Files app itself, inside another app referenced from the Files app, or even some general iCloud location. If you expose the files in your Documents folder, then it could even be one of your own app's files.
Needless to say, if you get such a file, you must not delete it. But if the file comes in an inbox, which will still happen a lot as well, then you must delete it. In order to determine this, the options of the application:openURL:options: call contains an UIApplicationOpenURLOptionsOpenInPlaceKey key. If that has value (NSNumber) NO, then the file is in an inbox, and it should be deleted. It it has value YES, then it is opened in-place, and must not be deleted.
Note that for files that are opened in place, you also need to gain permission to use them first. You do that but surrounding the access to the file by startAccessingSecurityScopedResource and stopAccessingSecurityScopedResource calls. See Apple documentation for details.
I am just now facing this same problem. Like you I don't have a good solution but in response to your questions I'm leaning towards option A over option B because I don't like the idea of potentially having an issue with future releases of the OS. Since, in my case, there is no user interaction once I display the preview of the document, I can go ahead and delete it when when I receive the –documentInteractionControllerDidEndPreview: delegate callback. This is theoretical in that I haven't coded this yet and won't get to it for a while as it is a low priority item. If it doesn't work or there are other issues, I will report back here. The Google search I entered in order to find documentation from Apple pointed to this StackOverflow post. I haven't seen any other useful information from Apple or anyone else on the subject.
I've been having trouble getting an app submitted to the App Store. This is due to the fact that that database, which is updatable, is too large for the iCloud backup limitations. Most of the data in the db is static, but one table records the user's schedule for reviewing words (this is a vocabulary quiz).
As far as I can tell, I have two or three realistic options. The first is to put the whole database into the Library/Cache directory. This should be accepted, because it's not backed up to iCloud. However, there's no guarantee that it will be maintained during app updates, per this entry in "Make App Backups More Efficient" at this url:
http://developer.apple.com/library/IOs/#documentation/iPhone/Conceptual/iPhoneOSProgrammingGuide/PerformanceTuning/PerformanceTuning.html
Files Saved During App Updates
When a user downloads an app update, iTunes installs the update in a new app directory. It then moves the user’s data files from the old installation over to the new app directory before deleting the old installation. Files in the following directories are guaranteed to be preserved during the update process:
<Application_Home>/Documents
<Application_Home>/Library
Although files in other user directories may also be moved over, you should not rely on them being present after an update.
The second option is to put the data into the NSDocuments or NSLibrary directory, as mark it with the skipBackupFlag. However, one problem is this flag doesn't work for iOS 5.0 and previous per this entry in "How do I prevent files from being backed up to iCloud and iTunes?" at
https://developer.apple.com/library/ios/#qa/qa1719/_index.html
Important The new "do not back up" attribute will only be used by iOS 5.0.1 or later. On iOS 5.0 and earlier, applications will need to store their data in <Application_Home>/Library/Caches to avoid having it backed up. Since this attribute is ignored on older systems, you will need to insure your app complies with the iOS Data Storage Guidelines on all versions of iOS that your application supports
This means that even if I use the "skipBackupFlag", I'll still have the problem that the database is getting backed up to the cloud, I think.
So, the third option, which is pretty much of an ugly hack, is to split the database into two. Put the updatable part into the NSLibrary or NSDocuments directory, and leave the rest in application resources. This would have the small, updatable part stored on the cloud, and leave the rest in the app resources directory. The problem is that this splits the db for no good reason, and introduces possible performance issues with having two databases open at once.
So, my question is, is my interpretation of the rules correct? Am I going to have to go with option 3?
p.s. I noticed in my last post cited urls were edited to links without the url showing. How do I do this?
Have you considered using external file references as described in https://developer.apple.com/library/IOS/#releasenotes/DataManagement/RN-CoreData/_index.html . Specifically, refer to "setAllowsExternalBinaryDataStorage:" https://developer.apple.com/library/IOS/documentation/Cocoa/Reference/CoreDataFramework/Classes/NSAttributeDescription_Class/reference.html#//apple_ref/occ/instm/NSAttributeDescription/setAllowsExternalBinaryDataStorage: . Pushing out large data into a separate file can help reduce database size .
My app was recently rejected from the app store for storing data in the Documents directory. I had moved it there because with the latest change, the db must now be writeable - it's no longer read-only.
In researching the solution, I've read that it's actually preferable to use NSLibraryDirectory. Is this so, and more importantly, will that address Apple's concerns? Their complaint is that the app lets the user download to much content (it doesn't let the user download any unless you count the db), and that it's storing "too much data" in the "incorrect location." The data is 8 mb, but could grow to about 10 or 12 mb max.
Actually its because of iCloud.
using iCloud, Application's document directory is synced to cloud and to other devices and hence Apple want developers to store only that data in document directory which they want to sync with iCloud.
I came to know this form one of my friends who work # Apple California and I'm not really sure if this material is on Apple's documentation.
I had this issue with an update to a suite of apps I develop the other week. The funny thing was that only five of the seven apps (exactly the same code base) were rejected.
I believe the issue in my case was duplicating assets from the .app bundle into ~/Documents.
My first attempt to comply with their new storage guidelines was to implement the do not backup switch on the files I was copying into ~/Documents. No deal with that so I had to change my implementation to not copy the data at all. The apps were promptly approved.
Your implementation is probably different but in my experience Apple no longer likes you copying things from the app bundle into ~/Documents, as it could be duplicating data unnecessarily (in their view).
They suggest copying into ~/Caches (or whatever it is), but this can be cleared in low storage situations and may not be right for your case either.
Hope that helps.
A product I wrote a year ago uses CoreData (with a SQLite data store). This database file is stored in ~/Library/Application Support//. This was approved by Apple without issue.
"Application Support" does not exist in ~/Library, so you will need to create it.
Documents is not a great place to store your database file for several reasons. Apple has their reasons, since they rejected your app. Another reason is that the Documents directory is accessible by the user (via iTunes), and unless the user deleting your database file is no big deal to the operation of your app, it is best to put it where they can not do anything with it directly and/or inadvertently.
You can still put those files in the Documents folder, you just have to give them an attribute that lets the file system know not to back them up to the iCloud
This is a great example of how to do so on different iOS versions
I store some data in my iOS app directly in a local .sqlite file. I chose to do this instead of CoreData because the data will need to be compatible with non-Apple platforms.
Now, I'm trying to come up with the best way to sync this file over iCloud. I know you can't sync it directly, for many reasons. I know CoreData is able to sync its DBs, but even ignoring that using CD would essentially lock this file into Apple platforms (I think? I've only looked into CD a bit), I need the iCloud syncing of this file to work across ALL of iCloud's supported platforms - which is supposed to include Windows. I have to assume that there won't be any compatibility for the CoreData files in the Windows API. Planning out the best way to accomplish this would be a lot easier if Apple would tell us any more than "There will be a Windows API [eventually?]"
In addition, I'll eventually need to implement at least one more sync service to support platforms that iCloud does not. It would be helpful, though not required, if the method I use for iCloud can be mostly reused for future services.
For these reasons, I don't think CoreData can help me with this. Am I correct in thinking this?
Moving on from there, I need to devise an algorithm for this, or find an existing one or an existing 3rd party solution. I haven't stumbled across anything yet. However, I have been mulling over a couple possible methods I could implement:
Method 1:
Do something similar to how CoreData syncs sqlite DBs: send "transaction logs" to iCloud instead and build each local sqlite file off of those.
I'm thinking each device would send a (uniquely named) text file listing all the sql commands that that device executed, with timestamps. The device would store how far along in each list of commands it has executed, and continue from that point each time the file is updated. If it received updates to multiple log files at once, it would execute each command in timestamp order.
Things could get 'interesting' efficiency-wise once these files get large, but it seems like a solvable problem.
Method 2:
Periodically sync a copy of the working database to iCloud. Have a modification timestamp field in every record. When an updated copy of the DB comes through, query all the records with newer timestamps than some reference time and update the record in the local DB from the new data.
I see many potential problems with this method:
-Have to implement something further to recognize record deletion.
-The DB file could get conflicts. It might be possible to deal with them by handling each conflict version in timestamp order.
-Determining the date to check each update from could be tricky, as it depends on which device the update is coming from.
There are a lot of potential problems with method 2, but method 1 seems doable to me...
Does anyone have any suggestions as to what might be the best course of action? Any better ideas than my "Method 1" (or reasons why it wouldn't work)?
Try those two solutions from Ray Wenderlich:
Exporting/Importing data through mail:
http://www.raywenderlich.com/1980/how-to-import-and-export-app-data-via-email-in-your-ios-app
File Sharing with iTunes:
http://www.raywenderlich.com/1948/how-integrate-itunes-file-sharing-with-your-ios-app
I found it quite complex but helped me a lot.
Both method 1 and method 2 seem doable. Perhaps a combination of the two in fact - use iCloud to send a separate database file that is a subset of data - i.e. just changed items. Or maybe another file format instead of sqlite db - XML/JSON/CSV etc.
Another alternative is to do it outside of iCloud - i.e. a simple custom web service for syncing. So each change gets submitted to a central server via JSON/XML over HTTP, and then other devices pull updates from that.
Obviously it depends how much data and how many devices you want to sync across, and whether you have access to an appropriate server and/or budget to cover running such a server. iCloud will do that for "free" but all it really does is transfer files. A custom solution allows you to define your syncing model as you wish, but you have to develop and manage it and pay for it.
I've considered the possibility of transferring a database file through iCloud but I think that I would run into classic problems of timing - slow start for the user - and corrupted databases if the app is run on multiple devices simultaneously. (iPad/iPhone for example).
Sooo. I've had to use the transaction logs method. It really is difficult to implement, but once in place, seems ok.
I am using Apple's SharedCoreData sample as the base for this work. This link requires an Apple Developer Account.
I did find a much much better solution from Tim Roadley however this only works for IOS and I needed both IOS and MacOS.
rant> iCloud development really has to get easier and more stable! /rant