I use sqlite in an iOS app. Sometimes I replace a sqlite file at runtime and close and reopen my connections to it. But sqlite often seems to return query results from the old file when connected to the new file. My guess so far is that it's somehow keeping and using the page cache from the old file.
So my question is this: under what circumstances does SQLite's page cache get dropped?
Got it. I was walking through all my connections and closing and reopening each. When I simply closed them all (and let them get reopened later on demand as needed), the cache appeared to get cleared. My guess is that the cache is per-file and not per-connection, and it gets reset when all the connections to the file are closed. This can be tricky to achieve in a multithreaded environment.
Related
I have developed a hybrid mobile app that has been running on IOS, Android and Chrome reasonably well for over 5 years. My app uses an indexedDB database to maintain the state of the app and save it between restarts, however there are some cases where IOS can clear the indexedDB particularly if the client device is running low on memory.
I would like some help on how to maintain a parallel copy of the indexedDB using the native SQL database on the IOS device, in order to increase the resilience of my app, based on the hopeful expectation that IOS will not arbitrarily decide to clear an internal SQL database belonging to the app.
I understand how to communicate in both directions between Objective C and Javascript but I have not the foggiest idea in Objective C how to:
Create a SQL database in the app file system
Choose where to place the SQL database in the app file system
How to write a new key value pair to the database
Overwrite an existing key value pair in the database
Read back all key value pairs from the database (I expect there could be up to 1000 of these in practice) and pass these efficiently back to the javascript code.
Clear the database and start again.
Any pointers to useful resources on how to achieve this or better still coded examples would be most appreciated.
I would just like to add that I am not using Cordova or any other similar app development environment, so please do not suggest a SQLite plugin that is part of a bigger environment. I have got to where I am with a bespoke coding approach and I would like to keep it that way.
I have explored the idea of paralleling up each indexedDB save with a write to a file in the IOS App Documents Folder with the hope that this is less likely to be wiped than the Caches Folder.
Following each indexedDB setItem with a key-value-pair I pass a request to the objective-C IOS app code to create a text file in a sub folder created with the App Documents Folder, with the name 'key'.txt and the contents set to value.
Following each indexedDB removeItem, I pass a request to the objective-C IOS app code to delete the corresponding text file 'key'.txt.
Following each indexedDB clearAll, I delete the entire sub folder created above.
Now when the app starts up and discovers an empty localForage database, I pass a request to the objective-C IOS app code to test if the sub-folder with key-items is there or not.
If it is not, then this is a fresh install of the app and continue as normal for such a case.
if it is, then this is a case of the indexedDB database having been deleted.
In such case, I request the objective-C IOS app code to return the set of keys, by examining the contents of the folder created above and stripping off the .txt bit and in the case of the IOS Simulator ignoring the DS_Store file.
One by one I then request the contents of each key file and load them back into the previously empty localForage database and when this has been done I can continue as if it had not been deleted.
I found that it is necessary to use a zero duration timeout in the javascript before requesting each value to prevent call stack exceeded errors when restoring large databases.
This approach seems to work and I can test this any time by using the Safari Developer Resources tab actions to clear the database and then manually restarting the app. Using the same tab you can watch the indexedDB database being repopulated.
Due to the size of my database, I actually created a set of sub folders of different types of key, so that I could choose the order in which the database items were restored, especially as my app is often brought back to life in the background following a significant location change and in such cases there is an imposed maximum time limit on how long the app is given to do such a recovery. This refinement is of course optional and only needed for large databases.
The following notes are for anyone who wants to try this approach and assumes using Objective C in XCode 10.1
Use NSHomeDirectory() and stringByAppendingPathComponent #"Documents" to get the Documents folder.
Use stringByAppendingPathComponent to create a sub folder path for the keys sub folder.
Use fileExistsAtPath to check whether the keys sub folder exists already
Use createDirectoryAtPath if it does not.
When saving or changing items in indexedDB use the stringByAppendingPathComponent to create the key file name path e.g. Base.txt for a key of 'Base'.
Use fileHandleforWritingAtPath to get the fileHandle for a file
if fileHandle does not exist then need to create it using writeToFile to create the 'key' file
if fileHandle does exist then truncateFileAtOffseyt:0 (important) to clear it and then use writeData to create a new version of the 'key' file.
In both of the above specify UTF8 encoding.
When removing items from indexedDB do the same to get the key file path and then use removeItemAtPath.
The device can be cleared by removing the entire sub folder using removeItemAtPath.
The restore process uses contentsOfDirectoryAtPath to read the set of keys in the sub folder.
The restore process for each item uses stringWithContentsOfFile to read data files and return the contents enclosed in quotes using a call to stringByEvaluatingJavaScriptFromString
Hope this helps.
I am having problem reading a file. I'm using MagicalRecord as my CoreData wrapper. I successfully save or update object. With my NSLog I can see it, everything is fine and I can use it in my database. But every time I want to see SQLite file with my application (I'm using Datum LE), file is empty. I cannot access that file directly within my Library folder in my app. I copy that file to my desktop and it is empty. What am I doing wrong? So once more, everything inside my iOS application works fine, I can see records being saved and I can fetch them normally.
You are clearly looking at the wrong file. If your app is saving (I assume, across app restarts), the date is definitely saved.
One way to find out is to NSLog the persistent store URL and check the referenced file.
I believe the issue you are encountering is the new default journaling mode that Core Data uses.
See this article
https://developer.apple.com/library/ios/qa/qa1809/_index.html
Basically your changes are not written to the .sqlite file, but are found in .sqlite-wal. HOWEVER, most SQLite reading apps I have tried blow away the -wal when you open the .sqlite, so good luck.
You could try changing the journaling mode for debugging purposes
My app is using Core Data framework with storage type of sqlite . I received several feedback commenting that user data stored in Core Data is lost, it seems the sqlite file read fail. After i got the sqlite files from user, there are 3 files (.sqlite, .sqlite-wal, .sqlite-shm).
But only empty results can be queried in these sqlite files.
What I got from the users is they may power off their iPhone while the app is active.
I just noticed Core Data begin use WAL as the default journal_mode in ios7, while it's default as DELETE prior to ios7. I didn't set up the journal_mode manually in my code.
I'm not sure what is the root cause of this issue. But data lost is unacceptable to users anyway. Did anyone happen to meet this issue before?
Should I set up the journal_mode to DELETE manually for all iOS version?
Thank you!
Update：
User reported it works normally on previous day, but then everything is lost on next day.
I need a solution for a feature in an app my company is building. It is an IOS app for iPhone and iPad and we support ios5 and above only.
The end result we need is for the user to tap a button inside the settings area that dumps the apps internal database cleanly, attaches it to an email for the user to then address to whoever they wish.
Attaching to an email is not an issue. What I am concerned about is if it is reasonable to think that exporting a copy of an sqlite database to an email as an attachment would leave the user with a usable copy of the database. My customer base is not good about doing backups but the data is nevertheless very important to them.
Will a hot export of an sqlite database be usable, i.e. is this supported, or will the database be corrupt sometimes/always?
Keep in mind that all this function has to do is make it so that a usable copy of the database gets off the device and stored safely offline so that I can then walk them through restoring the database by copying it back into the device using iExplorer or PhoneView.
In the future we plan to add a robust export and import feature, but for now we are looking for a solution that we can implement in a day or less.
Thanks in advance for anybodys input and gasp some objective-c code that would dump a clean copy of an sqlite database in such a way that the database will be usable and not corrupted.
You don't need to make a hot copy of the database. When the user chooses the backup feature of your app, close the database, copy the file, and then reopen the database. You can now attach the copied file to the email.
I have a feature in my app for doing just this. I've never had an issue with a corrupt database file. As long as there are no active transactions running, the sqlite file should be OK to copy, especially if there are no open handles.
Another option would be to use the sqlite3_backup_init and related functions to copy the database.
I have a simple app that downloads some data from a webservice and displays it in tableviews. In order to allow offline access, the app caches this data in the cache directory on the device itself. I keep a list of all of the cached files for reference, and save that to the cache directory as well. So after all of the downloads are complete I have the following files:
XMLFileList
XML1
XML2
...
I launched this app a couple weeks ago, and have since had to release a few minor updates as necessary to fix some minor bugs. It seems, however, that when a user updates the app on their device, they no longer see the downloaded resources. I have managed to get this to happen on a development device, by updating through the app store, then loading a debug build, and determined the following problem
It would seem that it is not displaying the data in the downloaded files because they no longer exist. It is also not re-downloading the files because they show up in the saved list. This provides a rather interesting question: why did the cached XML files get deleted, but not the master list?
What exactly is going on behind the scenes during an app update that I could account for, or reproduce to ensure this does not continue happening?
If you're putting files into /Library/Caches (or whatever it is), then there is no guarantee that those files will be there the next time you run your app. Anything you put into /Documents, however, will be there. And in fact, anything in /Documents will also be backed up to iCloud (unless you set the bit newly available in iOS 5.0.1 that lets you opt-out files in /Documents from being backed up.)
Anyway, the point is that you should read Apple's docs on what files you should put where. (I searched around for the correct reference unsuccessfully; I've seen it before though.) It boils down to this: Anything that is user generated should go in /Documents, anything that can be re-fetched from a server should go in /Library/Caches and may be purged by the OS.
Cache directory contents are not guaranteed to be preserved. But I don't see any documented guarantee that the entire directory will be cleared all at once. (Maybe only the biggest and/or oldest contents or some such?)