Fast way to trim lines at the beginning of a log file on iOS - ios

Question

I'm looking for a fast, optimized way to trim log files on iOS. I want to specify that my log files have a maximum number of lines (e.g., 10,000). Appending new lines to the end of a text file seems relatively simple. However, I haven't yet found a fast way to trim lines at the beginning of the file. Here's the (slow) code I came up with.
guard let fileURL = self.fileURL else {
return
}
guard let path = fileURL.path else {
return
}
guard let fileHandle = NSFileHandle(forUpdatingAtPath: path) else {
return
}
fileHandle.seekToEndOfFile()
fileHandle.writeData(message.dataUsingEncoding(NSUTF8StringEncoding)!)
fileHandle.writeData("\n".dataUsingEncoding(NSUTF8StringEncoding)!)
currentLineCount += 1
// TODO: This could probably use some major optimization
if currentLineCount >= maxLineCount {
if let fileString = try? NSString(contentsOfURL: fileURL, encoding: NSUTF8StringEncoding) {
var lines = fileString.componentsSeparatedByCharactersInSet(NSCharacterSet.newlineCharacterSet())
lines.removeFirst()
let newData = lines.joinWithSeparator("\n")
fileHandle.seekToFileOffset(0)
fileHandle.writeData(newData.dataUsingEncoding(NSUTF8StringEncoding)!)
}
}
fileHandle.closeFile()

Answer 1

There are two aspects of your question. First, your code removes a single
line from the log file. Therefore, once the limit is reached, every new
log message causes the entire file to be read, shortened, and be re-written.
It would be more effective to use a "high-water mark" and a "low-water mark". For example, if you want the last 10.000 lines to be preserved,
let the log file grow until it has 15.000 lines, and only then truncate
it to 10.000 lines. This reduces the number of "trim actions"
considerably.
The second part is about the truncating itself. Your code loads the
file into an NSString, which requires the conversion of UTF-8 data
to Unicode characters
(and fails if there is a single invalid byte in the log file).
Then the string is split into an array, one array element removed,
the array concatenated to a string again, and then written back
to the file, which converts the Unicode characters to UTF-8.
I haven't done performance tests, but I can imagine that it could be
faster to operate on binary data only, without the conversions to
NSString, Array and back. Here is a possible implementation
which removes a given number of lines from the start of a file:
func removeLinesFromFile(fileURL: NSURL, numLines: Int) {
do {
let data = try NSData(contentsOfURL: fileURL, options: .DataReadingMappedIfSafe)
let nl = "\n".dataUsingEncoding(NSUTF8StringEncoding)!
var lineNo = 0
var pos = 0
while lineNo < numLines {
// Find next newline character:
let range = data.rangeOfData(nl, options: [], range: NSMakeRange(pos, data.length - pos))
if range.location == NSNotFound {
return // File has less than `numLines` lines.
}
lineNo++
pos = range.location + range.length
}
// Now `pos` is the position where line number `numLines` begins.
let trimmedData = data.subdataWithRange(NSMakeRange(pos, data.length - pos))
trimmedData.writeToURL(fileURL, atomically: true)
} catch let error as NSError {
print(error.localizedDescription)
}
}

Answer 2

I have updated Martin R answer to Swift 3, and I also changed it so that we can pass the number of lines to keep instead of the number of lines to remove:
func removeLinesFromFile(fileURL: URL, linesToKeep numLines: Int) {
do {
let data = try Data(contentsOf: fileURL, options: .dataReadingMapped)
let nl = "\n".data(using: String.Encoding.utf8)!
var lineNo = 0
var pos = data.count-1
while lineNo <= numLines {
// Find next newline character:
guard let range = data.range(of: nl, options: [ .backwards ], in: 0..<pos) else {
return // File has less than `numLines` lines.
}
lineNo += 1
pos = range.lowerBound
}
let trimmedData = data.subdata(in: pos..<data.count)
try trimmedData.write(to: fileURL)
} catch let error as NSError {
print(error.localizedDescription)
}
}

Answer 3

Instead of writing the new line to the log, and processing the appended content afterwards, do both write and trimming in one step:
let fileString = (try? NSString(contentsOfURL: fileURL, encoding: NSUTF8StringEncoding)) as NSString? ?? ""
var lines = fileString.characters.split("\n").map{String($0)}
lines.append(message)
// this also more generic as it will remove any number of extra lines
lines.removeFirst(max(currentLineCount - maxLineCount), 0))
let newLogContents = lines.joinWithSeparator("\n")
(newLogContents as NSString).writeToURL(fileURL, atomically: true, encoding: NSUTF8StringEncoding)