Backing up .sqlite (Core Data) - ios

I have a core data based application that uses Dropbox to backup and restore data. The way I backup is pretty straight forward. I copy the .sqlite file on the user's dropbox.
Now my backup and restore functionality are working fine. The issue is with the .sqlite file itself. It appears that the .sqlite file is incomplete.
I entered about 125 entries in my application and took a backup. The backup appeared in my dropbox but when I use a .sqlite explorer tool to see the contents, I only see records upto 117th entry.
I tried updating the first entry and then again observing .sqlite file but no changes again.
What's even more strange is that the app appears to have recorded all changes. When I add a new entry or update an existing one and restart the app, the newly added data seems to persist.
But this newly added data does not appear in my .sqlite file.
I am backing up using this code:
AppDelegate *appDelegate = [[UIApplication sharedApplication] delegate];
NSString *filePath = [[[appDelegate applicationDocumentsDirectory] path] stringByAppendingPathComponent:#"MyApp.sqlite"];
if (account) {
if ([filesystem isShutDown]) {
filesystem = [[DBFilesystem alloc] initWithAccount:account];
[DBFilesystem setSharedFilesystem:filesystem];
}
DBPath *newPath = [[DBPath root] childPath:[NSString stringWithFormat:#"Backup - %#.sqlite", [NSDate date]]];
DBFile *file = [[DBFilesystem sharedFilesystem] createFile:newPath error:nil];
[file writeContentsOfFile:filePath shouldSteal:NO error:nil];
[filesystem shutDown];
}
I also copied the .sqlite file from the Simulator's folder and tried seeing it in the .sqlite browser. It still exhibits the same behaviour. Any reason why this must be happening?

Starting with iOS 7 / OS X 10.9, Core Data uses "Write-Ahead Logging" (WAL) as default
journaling mode for the underlying SQLite store file. This is explained in
Technical Q&A QA1809: New default journaling mode for Core Data SQLite stores in iOS 7 and OS X Mavericks
With the WAL mode, Core Data keeps the main store file untouched and
appends transactions to a -wal file in the same location. After the
Core Data context is saved, the -wal file is not deleted, and the data
in that file is not merged to the store file either. Therefore, simply
making copies of the store file will likely cause data loss and
inconsistency.
That should explain why your .sqlite file alone is incomplete.
As as solution you can (also explained in that Technical Note):
Disable WAL-mode (and use the "old" rollback journaling mode) for the SQLite store by setting the
#{NSSQLitePragmasOption:#{#"journal_mode":#"DELETE"}};
option when adding the persistent store, or
Use the
- (NSPersistentStore *)migratePersistentStore:(NSPersistentStore *)store toURL:(NSURL *)URL options:(NSDictionary *)options withType:(NSString *)storeType error:(NSError **)error
method to make a backup copy of the Core Data store.

Related

Cannot get the latest CoreData DB file in Swift

I have built a app using CoreData for persisting contents. Everything works fine. Now I would like to export the SQLite DB file of my stored data.
I checked the DB file's path by downloading container from my device. It seems to be "~\Library\Application Support\MainData.sqlite".
Screenshot of filepath:
I called shareDatabase() to share DB file using AirDrop to my Mac, and it works normally:
func shareDatabase() {
try? (UIApplication.shared.delegate as! AppDelegate).persistentContainer.viewContext.save()
let fileName = "MainData.sqlite"
let filePath = URL(fileURLWithPath: NSHomeDirectory()).appendingPathComponent("Library").appendingPathComponent("Application Support").appendingPathComponent(fileName)
let activityVC = UIActivityViewController(activityItems: [filePath], applicationActivities: nil)
present(activityVC, animated: true, completion: nil)
}
Strangely, the transferred MainData.sqlite is not the latest one, but the one with modified date of yesterday. I also used DB Browser opened the file. It does not contain the latest data.
Screenshot: info of AirDrop-shared file:
However, the MainData.sqlite from downloaded container is latest updated. The latest data are in this file, checking with DB Browser.
Screenshot: info of file in container:
Please help point out what is wrong with my codes.
Added contents:
Thank #user2782993 for referring the Apple answer on the topic: https://developer.apple.com/library/content/qa/qa1809/_index.html
I also read about this technical Q&A from apple library. But it is not very clear how to implement the mode changing option in XCode generated lazy var persistentContainer.
Also, this thread mentioned,
For sanity sake you should ensure you only have one connection to the
persistent store when you do this, i.e. only one persistent store
instance in a single persistent store coordinator.
I'm not sure how to close the existing connection. Any ideas? (better in Swift)
Try to use this:
https://stackoverflow.com/a/18870738/2782993
or this:
https://stackoverflow.com/a/20252663/2782993
Check this out:
http://pinkstone.co.uk/how-to-remove-wal-files-in-core-data/
Excerpt:
"Since iOS 7 and OS X 10.9 the default journalling mode in SQLite Stores is WAL. In addition to the main store file you’ll find a WAL file with the same (or larger) size as the store file, and a less important SHM file.
Prior to this implementation it was easy to save the context, extract the store file and ship it with an app as a pre-made data store. That’s no longer possible, because by default all data changes are written to the WAL file and do not sync with the main store file.
This is not a problem if you’re not shipping a pre-made store file with your app, but if you do, then this “improvement” has just ruined your way of delivering prewritten data stores.
Lucky for us we can switch this entire WAL business off by passing an option when creating our NSPersistentStoreCoordinator."
Here is the Apple answer on the topic:
https://developer.apple.com/library/content/qa/qa1809/_index.html
Please notice, synching the WAL data is called a checkpoint operation.
The important parts are in bold
Excerpt:
"A: The failure occurs because the default journaling mode for Core Data SQLite stores was changed to Write-Ahead Logging (WAL) in iOS 7 and OS X Mavericks. With the WAL mode, Core Data keeps the main store file untouched and appends transactions to a -wal file in the same location. After the Core Data context is saved, the -wal file is not deleted, and the data in that file is not merged to the store file either. Therefore, simply making copies of the store file will likely cause data loss and inconsistency.
...
To safely back up and restore a Core Data SQLite store, you can do the following:
Use the following method of NSPersistentStoreCoordinator class, rather than file system APIs, to back up and restore the Core Data store:
- (NSPersistentStore *)migratePersistentStore:(NSPersistentStore *)store toURL:(NSURL *)URL options:(NSDictionary *)options withType:(NSString *)storeType error:(NSError **)error
Note that this is the option we recommend.
Change to rollback journaling mode when adding the store to a persistent store coordinator if you have to copy the store file. Listing 1 is the code showing how to do this:
Listing 1 Use rollback journaling mode when adding a persistent store
NSDictionary *options = #{NSSQLitePragmasOption:#{#"journal_mode":#"DELETE"}};
if (! [persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType
configuration:nil
URL:storeURL
options:options
error:&error]) {
// error handling.
}
For a store that was loaded with the WAL mode, if both the main store file and the corresponding -wal file exist, using rollback journaling mode to add the store to a persistent store coordinator will force Core Data to perform a checkpoint operation, which merges the data in the -wal file to the store file. This is actually the Core Data way to perform a checkpoint operation. On the other hand, if the -wal file is not present, using this approach to add the store won't cause any exceptions, but the transactions recorded in the missing -wal file will be lost.
Bundle the main store file and the -wal file into a document package and manipulate them as a single item.
For more information about the default journaling mode change, please see WWDC 2013 session 207 What's New in Core Data and iCloud.
NOTE: In iOS 6.x and Mountain Lion, the default is rollback journaling mode, in which Core Data creates a -journal file to temporarily store transactions, updates the main store file in place and deletes the -journal file after saving the context. The store file therefore contains the up-to-date database."

How long do files stay in the documents directory (iOS)

If I save a file to the documents directory in a folder, how long does it stay on the disk? Forever (in that case I would have to delete it manually if I wish to do so) or it is deleted every time the app closes?
Files in the documents directory will persist until either the user deletes the app, resets their device or you remove the files in code. Also note however that the contents of the documents directory are backed up by iTunes so could persist longer than you would otherwise expect.
On the other hand, the tmp directory is not so persistent and its contents won't necessarily survive an app relaunch (they're also not backed up by iTunes) so you could use this if you don't want persistant storage. /Library/Caches/ is similar in that it isn't backed up but it is persistent. From the docs regarding tmp:
The system will periodically purge these files [in the tmp folder] when your app is not running; therefore, you cannot rely on these files persisting after your app terminates.
Basically, if you want to store something for a short period and it doesn't matter if it's deleted by the OS, use tmp. If you want something that will persist app launches and whatnot but still won't be backed up via iTunes, use the cache folder in Library. It you want something that persists and is backed up, use documents.
Extra info based on comment
If the images should disappear when the app is relaunched then you should use tmp. But take note that the tmp directory is not guaranteed to be cleared on an app relaunch. It's certainly possible that the images will still be there the next time you load the app. It's a bit of a gamble really. If it is vital to kill the images then wipe them manually somewhere suitable:
NSArray* tmpDirectory = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:NSTemporaryDirectory() error:NULL];
for (NSString *file in tmpDirectory) {
[[NSFileManager defaultManager] removeItemAtPath:[NSString stringWithFormat:#"%#%#", NSTemporaryDirectory(), file] error:NULL];
}

Copy iCloud file to sandbox

I'm looking for a better way to download a bunch of files from my iCloud container to my sandbox. This is what I currently use:
for (FileAttachments *fileLink in items) {
NSURL *cloudFileURL = [fileLink urlForCloudAttachmentWithName];
NSURL *fileURL = [backupCloudLocalDirectoryURL URLByAppendingPathComponent: fileLink.fileName];
NSFileCoordinator *fileCoordinator = [[NSFileCoordinator alloc] initWithFilePresenter:nil];
[fileCoordinator coordinateReadingItemAtURL:fileURL options:NSFileCoordinatorReadingWithoutChanges error:&error
byAccessor:^(NSURL *newURL) {
NSError *blockError = nil;
[fileManager copyItemAtURL:cloudFileURL toURL:fileURL error:&blockError];
}];
}
}
Is there any problem with making a copy of the iCloud item this way? In production, I have some users complaining that all their files weren't downloaded. Is it better to use NSFileManager's startDownloadingUbiquitousItemAtURL:error instead? If so, why?
Thanks.
It's still not completely clear to me from comments how you're discovering the URLs, however some important details that affect your situation are:
Using coordinateReadingItemAtURL:block: has nothing to do with downloading files from iCloud. The purpose of NSFileCoordinator is to coordinate among readers and writers of a file so that, for example, you don't get two threads trying to write to the same file at the same time. You use file coordinators with iCloud because the iCloud system needs to read and write files and so does your app. Using a coordinator avoids corrupting the file, but again, has nothing to do with downloading the file.
To download a file from iCloud you need to use startDownloadingUbiquitousItemAtURL:error: and then wait until the file downloads. The normal flow here is:
a. Use NSMetadataQuery to find files that exist in the iCloud account
b. Use startDownloadingUbiquitousItemAtURL:error: to make sure they're available on the local device.
The reason you need to use this call is simply because that's how iCloud works on iOS. Files don't download until you ask them to download, and this is how you ask. [On OS X it's different, everything automatically downloads.]
You cannot simply copy from an iCloud URL to another location unless you already know that the file has been downloaded. The file-copy operation does not download the file, so if the file isn't available locally, the copy will fail.
You must use a metadata query to identify the files and their download status then, if they have not been downloaded initiate the download, and using the metadata query determine when the download is complete and then copy the file from the ubiquity container to the apps sandbox directory using a file coordinator. If you try copying the file while it's partially downloaded you may get some strange results.
I had the same problem with you.
My case is that: When network disconnects, iCloud service copy file from iCloud container to sandbox. When this line executes, it can not go into the block to copy file. This is the reason why this file can not be copied.
fileCoordinator coordinateReadingItemAtURL:fileURL options:NSFileCoordinatorReadingWithoutChanges error:&error byAccessor:^(NSURL *newURL)
My solution is: Before copying file from iCLoud container to sandbox, you must check network. If it is not available, don't execute this code (return method). If network connects, execute fileCoordinator and copy.
More info: When copy file from ICloud container to sandbox, method fileManager copyItemAtURL:toURL:error: is OK because I implement this method and it's good.
Hope this works.

coredata - deliver / setup default data

I use coreData in my iOS App. It's possible, that the user Add, Delete Data into the Database.
I have to deliver default data ( some different data-sets ).
At the moment, I'm creating the database by first Application launch. I read data from a csv file an create the database with it.
The csv is in the Application sandbox; the coreData (managedDocument) is in ApplicationDocument (creation on runtime...).
It works perfect for me - but I ask me, will Apple allow that, if I push the App to the AppStore?
There is nothing wrong with this approach and it can't be a reason for rejection. There is also another way to do it. You can create the database the way you do it now, copy the .sqlite file and provide it as your default database. Then copy it on app first run. The following code will do it:
NSURL *storeURL = [[self applicationDocumentsDirectory] URLByAppendingPathComponent: #"YourDBName.sqlite"];
if (![fileManager fileExistsAtPath:storeURL.path]) {
NSString *defaultStorePath = [[NSBundle mainBundle] pathForResource:#"YourDBName" ofType:#"sqlite"];
if (defaultStorePath) {
[fileManager copyItemAtPath:defaultStorePath toPath:storeURL.path error:NULL];
}
}
With this approach you will not need to include your csv file in your bundle.
Yes, apple does allow shipping a database populated by default.
the standard way to do it is to ship a default database in your bundle, then at launch time check if there is a database in your application documents directory, and if it does not exist, then copy the database from your bundle to the documents directory.

iOS CoreData - prepopulate db with existing indexes

I am building a project where I need to prepopulate a coredata database with existing data.
I built a parser to create the sqlite file in the iOS Simulator, everything works fine.
I am using a single entity, and one of the attribute is indexed.
Performance after parsing my data file into core data is great, everything is good.
Now I am using the generated sqlite file (~200Mb) in project with same data model, same index, etc... and on first startup I copy over the db file to prepopulate the data
NSString *defaultStorePath = [[NSBundle mainBundle] pathForResource:#"myproject" ofType:#"sqlite"];
NSString *storePath = [[[self applicationDocumentsDirectory] path] stringByAppendingPathComponent: #"myproject.sqlite"];
NSError *error;
if (![[NSFileManager defaultManager] fileExistsAtPath:storePath])
{
if ([[NSFileManager defaultManager] copyItemAtPath:defaultStorePath toPath:storePath error:&error])
NSLog(#"Copied starting data to %#", storePath);
else
NSLog(#"Error copying default DB to %# (%#)", storePath, error);
}
The copying works fine, and the data can be accessed normally.
However the performance is terrible, and the index is clearly not being used.
A look at the size of the sqlite file after the copy operation, it went from 200Mb to 120Mb.
Everything looks alright in the model, what needs to be indexed is checked as indexed.
1) Is there a way for the index data not to be removed when copying the sqlite over?
2) Is it possible to programmatically rebuild the index?
3) Any other thoughts?
Check at the Apple documentation about this issue:
Although Core Data supports SQLite as one of its persistent store types, the database format is private. You cannot create a SQLite database using native SQLite API and use it directly with Core Data (nor should you manipulate an existing Core Data SQLite store using native SQLite API). If you have an existing SQLite database, you need to import it into a Core Data store (see “Efficiently Importing Data”).
To sum up, don't do it. The database schema is private and may change.
I use CSV files to pre-load all my initial data to CoreData on background when the application starts for the first time. Beware of multithreading CoreData access, by the way.
Hope it helps.
Looks like the problem was that the project didn't get cleaned well in between tests, it could have been a bug in Xcode 4.3 I was using at the time.
The same methodology is working fine now.

Resources