We have been using Realm for a while and some of our users have been experiencing some data loss related to Realm. We think we have narrowed it down to our compaction method for when the file gets too big. We would like to ask for a little advice on if this is the proper way to recreate our Realm file. This method is called on applicationDidEnterBackground.
We wrote a sample of what we are doing below:
public static func compact() {
// Get the original file path
let configuration = RealmSampleClient.shared.config
guard let originalFileURL = configuration.fileURL else {
return
}
// check if the file size is bigger than 10mb, if not return
guard let attr = try? FileManager.default.attributesOfItem(atPath: originalFileURL.absoluteString),
let fileSize = attr[FileAttributeKey.size] as? UInt64,
fileSize > 500_000_000 else {
return
}
// create a filepath for a copy
let date = Date()
let dateFormatter = DateFormatter()
dateFormatter.dateFormat = "yyyyMMddHHmmss"
let dateString = "\(dateFormatter.string(from: date)).realm"
let copyFileURL = originalFileURL.deletingLastPathComponent().appendingPathComponent(dateString)
// copy the Realm file
do {
let realm = try Realm(configuration: configuration)
try realm.writeCopy(toFile: copyFileURL, encryptionKey: configuration.encryptionKey)
} catch {
return
}
// remove the old file and copy the new one
do {
removeRealmFile(at: originalFileURL)
try FileManager.default.copyItem(at: copyFileURL, to: originalFileURL)
} catch {
}
// remove a copy if it exists
guard FileManager.default.fileExists(atPath: copyFileURL.path) else { return }
do {
try FileManager.default.removeItem(at: copyFileURL)
} catch {
}
}
private static func removeRealmFile(at url: URL = databaseUrl) {
let realmURLs = [
url,
url.appendingPathExtension("lock"),
url.appendingPathExtension("note"),
url.appendingPathExtension("management"),
]
realmURLs.forEach { URL in
guard FileManager.default.fileExists(atPath: URL.path) else { return }
do {
try FileManager.default.removeItem(at: URL)
} catch {
}
}
}
Thanks your your help
I can see no kind of compacting code here, only a copy of the database file. So I assume you have left that out to keep the code compact here.
Anyway, you do this operation when the app enters background mode. Do you register a background task for that? If the compacting operation takes too much time the task gets killed by iOS, I think this is the problem.
You can explicitly ask the OS for more background execution time with UIApplication.shared.beginBackgroundTask but this is also a very time limited amount, usually 3 minutes.
But this is all digging in the dark, you should post more code to see how your background task is set up.
As per Realm doc, it's recommended to wrap your code with autoreleasepool, like this
do {
let realm = try Realm(configuration: configuration)
try realm.writeCopy(toFile: copyFileURL, encryptionKey: configuration.encryptionKey)
} catch {
return
}
and doing this in a backgroundtask will definitely help, a friendly advice is to always handle errors, you are just returning in the catch block, a log may help ..
Looking more into the doc I can see that RealmSwift integrates a compacting feature now, more details here :
https://realm.io/docs/swift/latest/#compacting-realms
We refactored the flow of our app a bit and it has seemed to solve our problems. It was related to accessing realm too soon during the startup process, possible on multiple threads.
Related
Is there any way I can completely nuke everything from my Realm Cloud, including existing schema definitions?
There is a way to delete the Realms from the Realm Object Server.
Here's information I collected on a Realm Forums Post
Here's a link to the official docs.
This is super important though. The docs I am linking are for Docs 3.0. Self-hosting appears to be going away so the 3.16 Docs no longer include this information.
There are two steps
Remove server files
Remove all local files
These both have to be done or else Realm will try to re-sync itself and your data will never go away.
The first function deletes a Realm Cloud instance and if successful, deletes the local realm files.
//
//MARK: - delete database
//
func handleDeleteEverything() {
let realm = RealmService //Singleton that returns my realm cloud
try! realm.write {
realm.deleteAll()
}
guard let currentUser = SyncUser.current else {return}
let adminToken = currentUser.refreshToken!
let urlString = "https://your_realm.cloud.realm.io" //from RealmStudio upper right corner
let endPoint = "\(urlString)/realms/files/realm_to_delete"
let url = URL(string: endPoint)
var request = URLRequest(url: url!)
request.httpMethod = "DELETE"
request.addValue(adminToken, forHTTPHeaderField: "Authorization")
let task = URLSession.shared.dataTask(with: request) { data, response, error in
if let err = error {
print("err = \(err.localizedDescription)")
return
}
print("Realm has been deleted")
self.deleteLocalRealmFiles() //remove local files
}
task.resume()
}
and then the function to remove the local files. This function is a bit different than what appears on the Realm Forums post with the addition of this function in Realm 4.2
try Realm.deleteFiles(for: config)
and the function that calls it
func deleteLocalRealmFiles() {
do {
let config = Realm.Configuration.defaultConfiguration
let isSuccess = try Realm.deleteFiles(for: config)
if isSuccess == true {
print("local files were located and deleted")
} else {
print("no local files were deleted, files were not found")
}
} catch let error as NSError {
print(error.localizedDescription)
}
}
I think, you can check this link.
https://forum.realm.io/t/is-it-possible-to-reset-the-default-realm-without-creating-a-new-instance/1466
This one solve my realm database issue and can reset all schemas
I hope it works well!
I have an app that has species and photos. I am adding cloudKit to the app. I have a working solution, but now I need to add a completion handler as if the user downloads new species that include images, this takes some time (of course depending on how many images). However, the app allows the user to work during most of this process as it runs in the background.
The issue is if an image is not yet fully downloaded and the user select that species the app crashes, naturally.
I need to input a completion handler (or if someone has a better idea) that will allow me to use an activity indicator until the full process is completed. I found a few examples, but they don't take into account multiple download processes, like my images and thumbnails.
Here is my code. Note that I have removed some of the irrelevant code to reduce the amount shown.
func moveSpeciesFromCloud() {
let predicate = NSPredicate(value: true)
let query = CKQuery(recordType: RemoteRecords.speciesRecord, predicate: predicate)
CKDbase.share.privateDB.perform(query, inZoneWith: nil) {
records, error in
if error != nil {
print(error!.localizedDescription)
} else {
guard let records = records else { return }
for record in records {
DispatchQueue.main.async {
self.remoteVersion = record[RemoteSpecies.remoteSpeciesVersion] as! Int
self.remoteSpeciesID = record[RemoteSpecies.remoteSpeciesID] as! Int
self.speciesDetail = AppDelegate.getUserDatabase().getSpeciesDetails(self.remoteSpeciesID)
self.localVersion = self.speciesDetail.version
// being sure that remote version is newer than local version
if self.localVersion >= self.remoteVersion {
print("Species version not newer")
} else {
self.commonNameLabel = record[RemoteSpecies.remoteCommonName] as! String
self.speciesLabel = record[RemoteSpecies.remoteSpeciesName] as! String
self.genusLabel = record[RemoteSpecies.remoteGenusName] as! String
self.groupLabel = record[RemoteSpecies.remoteGroupName] as! String
self.subGroupLabel = record[RemoteSpecies.remoteSubGroupName] as! String
self.speciesDetailsLabel = record[RemoteSpecies.remoteSpeciesDetails] as! String
// Here I sync records to SQLite, but removed code as not relevant.
// now syncing Photos, Thumbs, Groups, SubGroups and Favorties
self.syncPhotosFromCloud(self.remoteSpeciesID)
self.syncThumbsFromCloud(self.remoteSpeciesID)
}
}
}
}
}
}
Here is the code for the Thumbnails (Images are same process)
func syncThumbsFromCloud(_ id: Int) {
let predicate = NSPredicate(format: "thumbSpeciesID = \(id)")
let query = CKQuery(recordType: RemoteRecords.thumbsRecord, predicate: predicate)
CKDbase.share.privateDB!.perform(query, inZoneWith: nil)
{
records, error in
if error != nil {
print(error!.localizedDescription)
} else {
guard let records = records else { return }
for record in records {
DispatchQueue.main.async {
self.thumbName = (record.object(forKey: RemoteThumbs.remoteThumbName) as? String)!
self.thumbID = (record.object(forKey: RemoteThumbs.remoteThumbID) as? Int)!
if let asset = record[RemoteThumbs.remoteThumbFile] as? CKAsset,
let data = try? Data(contentsOf: (asset.fileURL)),
let image = UIImage(data: data)
{
let filemgr = FileManager.default
let dirPaths = filemgr.urls(for: .documentDirectory,
in: .userDomainMask)
let fileURL = dirPaths[0].appendingPathComponent(self.thumbName)
if let renderedJPEGData = image.jpegData(compressionQuality: 1.0) {
try! renderedJPEGData.write(to: fileURL)
}
}
// syncing records to SQLite
AppDelegate.getUserDatabase().syncThumbsFromCloudToSQLite(id: self.thumbID, name: self.thumbName, speciesID: id)
}
}
}
}
}
I call it here on SyncVC:
#IBAction func syncCloudToDevice(_ sender: Any) {
let cloudKit = CloudKit()
cloudKit.moveSpeciesFromCloud()
cloudKit.moveFavoritessFromCloud()
}
If I missed a detail, please let me know.
Any assistance would be greatly appreciated.
I'm kind of concerned that both the previous answers don't help answer your question.. One is asking you to restructure your database and the other is asking you to become dependent on a third-party library.
My suggestion would be to make your perform(_:inZoneWith:) into a synchronous operation so that you can easily perform one after another. For example:
func performSynchronously(query: CKQuery) throws -> [CKRecord] {
var errorResult: Error?
var recordsResult: [CKRecord]?
let semaphore = DispatchSemaphore(value: 0)
CKDbase.share.privateDB!.perform(query, inZoneWith: nil) { records, error in
recordsResult = records
errorResult = error
semaphore.signal()
}
// Block this thread until `semaphore.signal()` occurs
semaphore.wait()
if let error = errorResult {
throw error
} else {
return recordsResult ?? []
}
}
Ensure that you call this from a background thread so as to not block your UI thread! For example:
// ... start your activity indicator
DispatchQueue(label: "background").async {
do {
let records1 = try performSynchronously(query: CKQuery...)
// parse records1
let records2 = try performSynchronously(query: CKQuery...)
// parse records2
DispatchQueue.main.async {
// stop your activity indicator
}
} catch let e {
// The error e occurred, handle it and stop the activity indicator
}
}
Of course, please just use this code as inspiration on how to use a semaphore to convert your asynchronous operations into synchronous ones. Here's a good article that discusses semaphores in depth.
Well, in general that sort of things are easy to do with RxSwift. You set activity indicator to on/off in .onSubscribe() and .onTerminated(), respectively, and you get the end result in subscriber/observer when it is ready. Specifically for CloudKit, you can use RxCloudKit library.
Is there a reason why you made the pictures a separate record type? I would just add the thumbnail and the full photo to the Species record type:
thumbnail = Bytes data type (1MB max)
photo = Asset data type (virtually limitless)
That way when you do your initial Species query, you will instantly have your thumbnail available, and then you can access the CKAsset like you are currently doing and it will download in the background. No second query needed which will make your code simpler.
I'm new to RxSwift. I have a BackendProvider with handles the communication with my API. I want to have a config file sync so that I can retrieve some parameters dynamically. I have a fallback case with a local stored JSON file that I can to access in case my API is not reachable or my JSON parsing fails:
ConfigFileBackendService
open func getLatestConfig() -> Observable<ConfigFile?> {
let urlString = IoC.urlProviderService.getConfigFileUrl()?.absoluteString ?? ""
let configFileJSONData = IoC.backendCommunicationService.getJsonData(url: urlString)
return configFileJSONData.map { data in
if let configFile = try? JSONDecoder().decode(ConfigFile.self, from: data) {
return configFile
} else {
return nil
}
}
}
ConfigFileProcessService
This is the one that falls back to the local stored file:
func getConfigFile() -> Observable<ConfigFile> {
return IoC.configFileBackendService.getLatestConfig()
.map { configFile in
guard let configFile = configFile else { fatalError() }
return configFile
}
.catchError { error in
// Use default config
let localURL = IoC.urlProviderService.getLocalConfigFileUrl()
do {
let data = try Data(contentsOf: localURL)
let configFile = try JSONDecoder().decode(ConfigFile.self, from: data)
return Observable.just(configFile)
} catch {
fatalError("Error loading local config")
}
}
}
This approach works, but I'm having doubts with the .map / .catchError blocks. Is there a better way to handle the error case? Maybe should I go with, onNext and then onError? Thanks in advance!
What you have looks good except for the multiple approaches to handling errors. In one case you use a try? and another uses do... catch and presumably your getJsonData(url:) can emit an Observable error. You are all over the place. I suggest you pick one error handling system and stick to it. The most flexible one is Event.error. So something like this:
func getLatestConfig() -> Observable<ConfigFile> {
let urlString = IoC.urlProviderService.getConfigFileUrl()?.absoluteString ?? ""
let configFileJSONData = IoC.backendCommunicationService.getJsonData(url: urlString)
return configFileJSONData.map { try JSONDecoder().decode(ConfigFile.self, from: $0) }
}
Note that I'm just letting decoding errors route into an Observable error event. No need to deal with nil that way.
func getConfigFile() -> Observable<ConfigFile> {
return IoC.configFileBackendService.getLatestConfig()
.catchError { _ in
let localURL = IoC.urlProviderService.getLocalConfigFileUrl()
let data = try! Data(contentsOf: localURL)
let configFile = try! JSONDecoder().decode(ConfigFile.self, from: data)
return Observable.just(configFile)
}
}
Since you are crashing if either try fails anyway, just put a ! on them. It has the same effect. You should consider putting the error block into a separate, testable, function because there's no guarantee you are going to hit it during a regular run of the program and it could be broken without you ever realizing it.
Lastly, with the above there is no reason to provide an onError: handler in a subscribe because the getConfigFile() observable will never emit an error. You might want to have the function return a Driver instead to make the fact more explicit.
Following #Daniel T. approach, I got rid of nil and made the code crash only if the local config file loading process fails:
open class ConfigFileBackendService: ConfigFileBackendServiceProtocol {
open func getLatestConfig() -> Observable<ConfigFile> {
let urlString = IoC.urlProviderService.getConfigFileUrl()
let configFileJSONData = IoC.backendCommunicationService.getJsonData(url: urlString)
return configFileJSONData.map { try JSONDecoder().decode(ConfigFile.self, from: $0) }
}
}
class ConfigFileProcessService: ConfigFileProcessServiceProtocol {
func getConfigFile() -> Observable<ConfigFile> {
return IoC.configFileBackendService.getLatestConfig()
.catchError { [weak self] error in
// Load local config file
NSLog(error.localizedDescription)
guard let strongSelf = self else { fatalError() }
return Observable.just(strongSelf.getLocalFile())
}
}
private func getLocalFile() -> ConfigFile {
let localURL = IoC.urlProviderService.getLocalConfigFileUrl()
guard
let data = try? Data(contentsOf: localURL),
let configFile = try? JSONDecoder().decode(ConfigFile.self, from: data)
else { fatalError("Error loading local config") }
return configFile
}
}
I am using Realm mobile database. I update Realm objects using the following code. I want to get a callback with whether or not that Realm object was successfully updated.
do {
let realm = try Realm()
realm.beginWrite()
if let Settings = realm.objects(ModelClass).first {
Settings.settingsVal = settingsValue
realm.add(Settings, update: true)
}
try realm.commitWrite()
}
catch {
}
There are a number of different ways to achieve what you are looking to do, including using closures/delegates to provide a callback mechanism, however the simplest way to know that the update was successful would be to add code inside the try block:
do {
let realm = try Realm()
realm.beginWrite()
if let Settings = realm.objects(ModelClass).first {
Settings.settingsVal = settingsValue
realm.add(Settings, update: true)
}
try realm.commitWrite()
print("Success")
}
catch{
}
}
Below code is work. The images are saved successfully in document directory, but the problem is only first time the collectionViewController can load images with path successfully. I have to delete all images to store new images or it will show the error message
"fatal error: unexpectedly found nil while unwrapping an Optional
value".
Because the path is unavailable, readnsdata = NSData(contentsOfFile: filepath)! will cause error.
I have no idea why only the first time it can work.
path :
"/var/mobile/Containers/Data/Application/29306029-BDCF-4BEA-93A6-D5626CBAAA90/Documents/x.jpg"
func writeNSDataToDisk(imageData:NSData){
let myindex = imgPathArray.count
let fileName = "\(self.imgPathArray.count)"
let paths = NSSearchPathForDirectoriesInDomains(NSSearchPathDirectory.DocumentDirectory, NSSearchPathDomainMask.UserDomainMask, true)
let docs: String = paths[0] as String
let filepath: String = (docs as NSString).stringByAppendingPathComponent("\(fileName).jpg")
let test = imageData.writeToFile(filepath, atomically: true)
if test {
self.imgPathArray.insert(filepath, atIndex: myindex)
print("The picture \(fileName).jpg is been saved.")
self.readORwriteList(true)//write list to txt file
}
print(self.imgPathArray)
}
func readNSDataFromDisk(fileIndex:Int) -> NSData{
let checkValidation = NSFileManager.defaultManager()
var readnsdata = NSData()
if (fileIndex <= self.imgPathArray.count) {
let filepath = self.imgPathArray[fileIndex]
if (checkValidation.fileExistsAtPath(filepath)){
print("File is available")
print("load \(fileIndex).jpg,filepath is \(filepath)")
readnsdata = NSData(contentsOfFile: filepath)!
if readnsdata.length != 0 {
getImageProperties(readnsdata)
}
}
else{
print("File is not available!!!")
}
}
return readnsdata
}
The solution to my problem :
Instead of storing absolute file path, I name the files in a regular way and search them by their name. There is no need to store path.
The URLs for the files are now constructed relative to the Documents directory URL every time the app is run.
Thanks
First a side note. Apple's docs specifically recommend against using fileExistsAtPath the way you're doing it here.
NOTE
Attempting to predicate behavior based on the current state of
the file system or a particular file on the file system is not
recommended. Doing so can cause odd behavior or race conditions. It’s
far better to attempt an operation (such as loading a file or creating
a directory), check for errors, and handle those errors gracefully
than it is to try to figure out ahead of time whether the operation
will succeed.
Try replacing this…
if (checkValidation.fileExistsAtPath(filepath)){
print("File is available")
print("load \(fileIndex).jpg,filepath is \(filepath)")
readnsdata = NSData(contentsOfFile: filepath)!
if readnsdata.length != 0 {
getImageProperties(readnsdata)
}
}
else{
print("File is not available!!!")
}
…with this…
do {
readnsdata = try NSData(contentsOfFile: filepath, options: .DataReadingMappedIfSafe)
if readnsdata.length != 0 {
getImageProperties(readnsdata)
}
}
catch let e {
print("Couldn't read file at \(filepath) because \(e)")
}
This approach gives you the information you were looking for without having to speculate. Just run your code and see what happens when the NSData initializer throws! :)
[Update: Off-topic opinion]
While it's a good habit not to sprinkle a long method with returns, there's not a lot going on here. Personally, I think the code comes out more readable without the temporary readnsdata variable. This way, imo, both the happy path and the default return values are clear on first reading:
func readNSDataFromDisk2(fileIndex:Int) -> NSData{
if (fileIndex <= self.imgPathArray.count) {
let path = self.imgPathArray[fileIndex]
do {
let data = try NSData(contentsOfFile: path, options: .DataReadingMappedIfSafe)
if data.length != 0 {
getImageProperties(data)
}
return data
}
catch let e {
print("Couldn't read file at \(path) because \(e)")
}
}
return NSData()
}
replace readnsdata = NSData(contentsOfFile: filepath)! with readnsdata = NSData(contentsOfFile: filepath)?. hope this will help :)