Swift Realm. How to put 3 millions objects in database - ios

Question

I make offline dictionary app. And now I convert dictionary file to realm database.
Convert function:
if let path = Bundle.main.path(forResource: "dictionary", ofType: "dsl") {
do {
let data = try String(contentsOfFile: path, encoding: .utf8)
let myStrings = data.components(separatedBy: .newlines)
for (index, row) in myStrings.enumerated() {
if(row.containsChineseCharacters)
{
let firstWord = CNDict()
firstWord.word = row
firstWord.pinyin = myStrings[index+1]
firstWord.translate = myStrings[index+2]
try! realm.write {
realm.add(firstWord)
}
}
}
print("The task end.")
} catch {
print(error)
}
}
When I try to convert the dictionary immediately, the database file becomes a lot of gigabytes and crashes around the middle
Splitting the dictionary in parts is not an option, because there are about 3 million lines. It will take very much... (realm plugin crashes)
I need help on how to add as many values to the database with our crashes.

Answer 1

The issue is that your file is large and at this point, you load it into the memory:
let data = try String(contentsOfFile: path, encoding: .utf8)
And then you double the memory footprint here:
let myStrings = data.components(separatedBy: .newlines)
So my guess is that you received out of memory signal from the system.
Instead of loading into the memory all the data and than double it, you can use lazy collection. It will read and parse the line only when it needed for writing. It will not load all lines at once. One downside of lazy collections in Swift is that they could not provide all the functions that we get used to.
Here the full code for playground that solves your issue. You could and maybe should optimize some parts of it, but anyway it just showing how it could be done with lazy collection. (I changed some names, but it's still what you want).
import Foundation
extension String {
var containsOneSymbol: Bool {
return contains("1")
}
}
extension Character {
var isNewLine: Bool {
let string = String(self)
let set = CharacterSet(charactersIn: string)
return !set.isDisjoint(with: CharacterSet.newlines)
}
}
/// Add Object subclass for Realm
#objcMembers
final class CNDict {
dynamic var word = ""
dynamic var pinyin = ""
dynamic var translate = ""
}
final class ModelWriterWrapper {
private let bufferCapacity = 3
private var buffer: [String] = []
init() {
buffer.reserveCapacity(bufferCapacity)
}
func proccess(line: String) {
guard buffer.count == bufferCapacity else {
assert(buffer.count < bufferCapacity, "Buffer failer count: \(buffer.count)!")
buffer.append(line)
return
}
if let firstLine = buffer.first, firstLine.containsOneSymbol {
let dict = CNDict()
dict.word = firstLine
dict.pinyin = buffer[1]
dict.translate = buffer[2]
print("Ready for writing into DB \n word: \(dict.word) pinyin: \(dict.pinyin) translate: \(dict.translate)")
}
buffer.removeFirst()
buffer.append(line)
}
}
let data = stride(from: 0, to: 100_000, by: 1).map { "Line number \($0)\n"}.joined()
var line: String = ""
let myLines = data.lazy.map { char -> String in
line.append(char)
guard !char.isNewLine else {
defer { line = "" }
return line
}
return ""
}.filter { !$0.isEmpty }
let databaseWritter = ModelWriterWrapper()
myLines.forEach {
databaseWritter.proccess(line: $0)
}
If have any questions regarding the code, please ask.