Solve memory issue with migrating a big bunch of data - ios

Question

In my previous version of my app I stored all user data into .dat files (with NSKeyedArchiver), but in my new version I want to upgrade to a real(m) database.
I'm trying to import all of this data (and that can be a LOT) into Realm. But it's taking so much memory that the debugger eventually kill my app before the migration has finished. The 'strange' thing is that the data on hard disk is only 1.5 mb big, but it's taking memory for more than 1gb so I'm doing something wrong.
I also tried to work with multiple threads, but that didn't help. Well it speeded up the migration process (which is good), but it also took the same amount of memory.
Who can help me out? See my code below for more information..
FYI Async can be found here https://github.com/duemunk/Async
import Async
let startDate = NSDate(timeIntervalSince1970: 1388534400).startOfDay // Start from 2014 jan 1st
let endDate = NSDate().dateByAddingTimeInterval(172800).startOfDay // 2 days = 3600 * 24 * 2 = 172.800
var pathDate = startDate
let calendar = NSCalendar.currentCalendar()
let group = AsyncGroup()
var allPaths = [(Int, Int)]()
while calendar.compareDate(pathDate, toDate: endDate, toUnitGranularity: .Month) != .OrderedDescending {
// Components
let currentMonth = calendar.component(.Month, fromDate: pathDate)
let currentYear = calendar.component(.Year, fromDate: pathDate)
allPaths.append((currentYear, currentMonth))
// Advance by one month
pathDate = calendar.dateByAddingUnit(.Month, value: 1, toDate: pathDate, options: [])!
}
for path in allPaths {
group.background {
// Prepare path
let currentYear = path.0
let currentMonth = path.1
let path = (Path.Documents as NSString).stringByAppendingPathComponent("Stats_\(currentMonth)_\(currentYear).dat")
print(path)
if NSFileManager.defaultManager().fileExistsAtPath(path) {
NSKeyedUnarchiver.setClass(_OldStatisticsDataModel.self, forClassName: "StatisticsDataModel")
if let statistics = NSKeyedUnarchiver.unarchiveObjectWithFile(path) as? [_OldStatisticsDataModel] {
// Loop through days
for i in 1...31 {
let dateComponents = NSDateComponents()
dateComponents.year = currentYear
dateComponents.month = currentMonth
dateComponents.day = i
dateComponents.hour = 0
dateComponents.minute = 0
// Create date from components
let userCalendar = NSCalendar.currentCalendar() // user calendar
guard let date = userCalendar.dateFromComponents(dateComponents) else {
continue
}
// Search for order items
let filtered = statistics.filter {
if let date = $0.date {
let dateSince1970 = date.timeIntervalSince1970
return date.startOfDay.timeIntervalSince1970 <= dateSince1970 && date.endOfDay.timeIntervalSince1970 >= dateSince1970
}
return false
}
if filtered.isEmpty == false {
// Create order
let transaction = Transaction()
transaction.employee = Account.API().administratorEmployee()
let order = Order()
order.status = PayableStatus.Paid
order.createdDate = date.timeIntervalSince1970
order.paidDate = date.timeIntervalSince1970
// Loop through all found items
for item in filtered {
// Values
let price = (item.price?.doubleValue ?? 0.0) * 100.0
let tax = (item.tax?.doubleValue ?? 0.0) * 100.0
// Update transaction
transaction.amount += Int(price)
// Prepare order item
let orderItem = OrderItemm()
orderItem.amount = item.amount
orderItem.price = Int(price)
orderItem.taxPercentage = Int(tax)
orderItem.name = item.name ?? ""
orderItem.product = Product.API().productForName(orderItem.name, price: orderItem.price, tax: orderItem.taxPercentage)
// Add order item to order
order.orderItems.append(orderItem)
}
if order.orderItems.isEmpty == false {
print("\(date): \(order.orderItems.count) order items")
// Set transaction for order
order.transactions.append(transaction)
// Save the order
Order.API().saveOrders([order])
}
}
}
}
}
}
}
group.wait()

Answer 1

As in the comments of my question was suggested by #bdash, autoreleasepool did the trick.
I use AsyncSwift as syntactic sugar for grand central dispatch, but when using block groups the group keeps a reference to the block which caused that the memory wasn't released. I still make use of a group now but I leave the group after it's finished.
I provided my code example below, to make things clearer. Using multiple threads gave me incredibly (like 10 times faster) more performance while memory won't go above 150MB. Previously app was crashing at 1,3GB due to memory pressure.
let group = AsyncGroup()
var allPaths = [(Int, Int)]()
// Some logic to fill the paths -> not interesting
for path in allPaths {
group.enter() // The following block will be added to the group
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) {
autoreleasepool { // Creating an autoreleasepool to free up memory for the loaded statistics
// Stripped unnecessary stuff
if NSFileManager.defaultManager().fileExistsAtPath(path) {
// Load the statistics from .dat files
NSKeyedUnarchiver.setClass(_OldStatisticsDataModel.self, forClassName: "StatisticsDataModel")
if let statistics = NSKeyedUnarchiver.unarchiveObjectWithFile(path) as? [_OldStatisticsDataModel] {
// Loop through days
for i in 1...31 {
autoreleasepool {
// Do the heavy stuff here
}
}
}
}
}
group.leave() // Block has finished, now leave the group
}
}
group.wait()