iOS Large SQLite data set slow performance - ios

I have an application that I am working on for iOS that is to display trails on a map based on the bounding box of the current view. So as the user moves around the map it will load the appropriate set of trails.
The SQLite table is LARGE, 260k rows, where each row is a point on a trail. The table has a primary key (int), latitude (float), longitude (float) and a name (varchar). The latitude and longitude columns are indexed. When I query I am looking for where the latitude is between the lower right and upper left latitude and the longitude is between the upper left and lower right longitude. The query works flawlessly on desktop as well as on the phones as it returns the expected results. The issue is that on my Mac the query returns instantaniously where as on the phone it may take as little as 4 seconds before returning anything. The bottle neck does appear to be the database query and I'm starting to think its a limitation of the hardware.
I have tried using CoreData which is where I noticed the issue first. Then I moved to using FMDB to access the database and am still having issues.
I have done no tweaking to the database or to the connection.
The guts of the queryForTrails method
if( ![db open] ) {
[db release];
NSLog(#"Error opening DB: %#", dbPath);
}
FMResultSet *trails = [db executeQueryWithFormat:#"SELECT zname, zlatitude, zlongitude FROM ztrail WHERE ztype = 1 and zlatitude BETWEEN %# and %# AND zlongitude BETWEEN %# and %# order by zname", lrLat, ulLat, ulLon,lrLon];
//Start to load the map with data
NSString *lastTrailName=#"";
int idx = 0;
CLLocationCoordinate2D *trailCoords = nil;
NSUInteger coordSize = 20;
trailCoords = malloc(sizeof(CLLocationCoordinate2D)*coordSize);
while( [trails next] ) {
NSString *trailName = [trails stringForColumnIndex:0];
NSString *lat = [trails stringForColumnIndex:1];
NSString *lon = [trails stringForColumnIndex:2];
if( [lastTrailName compare:trailName] != NSOrderedSame ) {
if(idx > 0) {
[trailLines addObject:[MKPolyline polylineWithCoordinates:trailCoords count:idx]];
free(trailCoords);
idx = 0;
coordSize = 20;
}
lastTrailName = trailName;
trailCoords = malloc(sizeof(CLLocationCoordinate2D)*coordSize);
}
if(idx == coordSize) {
coordSize *= 2;
trailCoords = realloc(trailCoords, sizeof(CLLocationCoordinate2D) * coordSize);
}
trailCoords[idx++] = CLLocationCoordinate2DMake([lat doubleValue], [lon doubleValue]);
}
//Build the new polyline
[trailLines addObject:[MKPolyline polylineWithCoordinates:trailCoords count:idx]];
//NSLog(#"Num Trails: %d", [trailLines count]);
free(trailCoords);
//NSLog(#"Num of Points %d for %#",idx, lastTrailName);
if( [trailLines count] > 0 ) {
dispatch_async(dispatch_get_main_queue(),^{
[mapView addOverlays:trailLines];
});
}
I can provide some NSLog data if needed. I will also be doing the same application for Android so I'd like to try and work out the performance issues now.

Related

Finding the nearest, lower number in a sorted NSArray

I have a sorted array of times like so
[0.0, 1.2, 4.3, 5.9, 7.2, 8.0]
While an audio file plays, I want to be able to take the current time and find what the nearest, lower number is in the array.
My approach would be to traverse the array, possible in reverse order as it feels like it should be faster. Is there a better way?
The playback SHOULD be linear, but might be fast-forwarded/rewound, so I would like to come up with a solution that takes that into account, but I'm not really sure how else to approach the problem.
The method you are looking for is -[NSArray indexOfObject:inSortedRange:options:usingComparator:]. It performs a binary search. With the options:NSBinarySearchingInsertionIndex option, if the value isn't found exactly, it returns the index where the object would be inserted, which is the index of the least larger element, or the count of items in the array.
NSTimeInterval currentTime = ...;
NSUInteger index = [times indexOfObject:#(currentTime)
inSortedRange:NSMakeRange(0, times.count)
options:NSBinarySearchingInsertionIndex
usingComparator:^(id object0, id object1) {
NSTimeInterval time0 = [object0 doubleValue];
NSTimeInterval time1 = [object1 doubleValue];
if (time0 < time1) return NSOrderedAscending;
else if (time0 > time1) return NSOrderedDescending;
else return NSOrderedSame;
}];
// If currentTime was not found exactly, then index is the next larger element
// or array count..
if (index == times.count || [times[index] doubleValue] > currentTime) {
--index;
}
The fastest* way to find something in a sorted array is binary search: if there are n items, check the element at index n/2. If that element is greater than what you're looking for, check the element at index n/4; otherwise, if it's less than what you're looking for, check the element at index 3n/4. Continue subdividing in this fashion until you've found what you want, i.e. the position where the current time should be. Then you can pick the preceding element, as that's the closest element that's less than the current time.
However, once you've done that once, you can keep track of where you are in the list. As the user plays through the file, keep checking to see if the time has passed the next element and so on. In other words, remember where you were, and use that when you check again. If the user rewinds, check the preceding elements.
*Arguably, this isn't strictly true -- there are surely faster ways if you can make a good guess as to the probable location of the element in question. But if you don't know anything other than that the element appears somewhere in the array, it's usually the right approach.
I'm not sure if it's the best approach, but I think it'll get the job done (assuming your array is always ascending order).
- (NSNumber *) incrementalClosestLowestNumberForNumber:(NSNumber *)aNumber inArray:(NSArray *)array {
for (int i = 0; i < array.count; i++) {
if ([array[i] floatValue] == [aNumber floatValue]) {
return aNumber;
}
else if ([array[i] floatValue] > [aNumber floatValue]) {
int index = (i > 0) ? i - 1 : 0;
return array[index];
}
}
return #0;
}
Then call it like this:
NSArray * numbArray = #[#0.0, #1.2, #4.3, #5.9, #7.2, #8.0];
NSNumber * closestNumber = [self closestLowestNumberForNumber:#2.4 inArray:numbArray];
NSLog(#"closest number: %#", closestNumber);
I'm not sure if someone else knows a special way that is much faster.
Based on some of the other answers / comments, I came up with this, perhaps one of them can point out if a whole is somewhere.
- (NSNumber *) quartalClosestLowestNumberForNumber:(NSNumber *)compareNumber inArray:(NSArray *)array {
int low = 0;
int high = array.count - 1;
NSNumber * lastNumber;
int currentIndex = 0;
for (currentIndex = low + (high - low) / 2; low <= high; currentIndex = low + (high - low) / 2) {
NSNumber * numb = array[currentIndex];
if (numb.floatValue < compareNumber.floatValue) {
low = currentIndex + 1;
}
else if (numb.floatValue > compareNumber.floatValue) {
high = currentIndex - 1;
}
else if (numb.floatValue == compareNumber.floatValue) {
return numb;
}
lastNumber = numb;
}
if (lastNumber.floatValue > compareNumber.floatValue && currentIndex != 0) {
lastNumber = array[currentIndex - 1];
}
return lastNumber;
}
I'm really bored right now, so I'm trying to test the fastest method. Here's how I did it.
NSMutableArray * numbersArray = [NSMutableArray new];
for (int i = 0; i < 100000; i++) {
float floater = i / 100.0;
[numbersArray addObject: #(floater)];
}
// courtesy #RobMayoff
NSDate * binaryDate = [NSDate date];
NSNumber * closestNumberBinary = [self binaryClosestLowestNumberForNumber:#4.4 inArray:numbersArray];
NSLog(#"Found closest number binary: %# in: %f seconds", closestNumberBinary, -[binaryDate timeIntervalSinceNow]);
// The Quartal Version
NSDate * quartalDate = [NSDate date];
NSNumber * closestNumberQuartal = [self quartalClosestLowestNumberForNumber:#4.4 inArray:numbersArray];
NSLog(#"Found closest number quartal: %# in: %f seconds", closestNumberQuartal, -[quartalDate timeIntervalSinceNow]);
// The incremental version
NSDate * incrementalDate = [NSDate date];
NSNumber * closestNumberIncremental = [self incrementalClosestLowestNumberForNumber:#4.4 inArray:numbersArray];
NSLog(#"Found closest number incremental: %# in: %f seconds", closestNumberIncremental, -[incrementalDate timeIntervalSinceNow]);
And here's the output:
Found closest number binary: 4.4 in: 0.000030 seconds
Found closest number quartal: 4.4 in: 0.000015 seconds
Found closest number incremental: 4.4 in: 0.000092 seconds
And another test case:
Found closest number binary: 751.48 in: 0.000030 seconds
Found closest number quartal: 751.48 in: 0.000016 seconds
Found closest number incremental: 751.48 in: 0.013042 seconds

Optimize apple system logs

+ (NSArray *)systemLogDictionariesForAppName:(NSString *)appName {
aslmsg q = asl_new(ASL_TYPE_QUERY);
asl_set_query(q, ASL_KEY_SENDER, [appName cStringUsingEncoding:NSASCIIStringEncoding], ASL_QUERY_OP_EQUAL);
aslresponse r = asl_search(NULL, q);
aslmsg m;
uint32_t i;
const char *key, *val;
NSMutableArray *systemLogDictionaries = [NSMutableArray array];
while (NULL != (m = aslresponse_next(r)))
{
NSMutableDictionary *dictionary = [NSMutableDictionary dictionary];
for (i = 0; (NULL != (key = asl_key(m, i))); i++)
{
val = asl_get(m, key);
NSString *stringKey = [NSString stringWithCString:key encoding:NSUTF8StringEncoding];
NSString *stringVal = [NSString stringWithCString:val encoding:NSUTF8StringEncoding];
[dictionary setObject:stringVal forKey:stringKey];
}
[systemLogDictionaries addObject:dictionary];
}
aslresponse_free(r);
return systemLogDictionaries;
}
Above code will get apple system log. Problem is, it take around 8second to pull all the logs from Apple System Log (ASL). Is there any way to optimize asl_set_query to get data faster or any other way which I am missing.
Note: Can we create a ASL query which will take time stamp and we can get less number of data to process. This will solve the problem I think.
ASL supports a few different logging levels, so you could specify a more restrictive level.
For example you can add another query (according to the man page they are joined via the logical AND):
// ...
asl_set_query(q, ASL_KEY_SENDER, [appName cStringUsingEncoding:NSASCIIStringEncoding], ASL_QUERY_OP_EQUAL);
// 3 is error messages
asl_set_query(q, ASL_KEY_LEVEL, "3", ASL_QUERY_OP_LESS_EQUAL | ASL_QUERY_OP_NUMERIC);
//-- Check for time --//
/* A dumped entry with your code looks like:
ASLMessageID = 1825403;
"CFLog Local Time" = "2013-07-20 08:33:12.943";
"CFLog Thread" = 951f;
Facility = "com.apple.Safari";
GID = 20;
Host = "XXX.local";
Level = 4;
Message = "CFPropertyListCreateFromXMLData(): Old-style plist parser: missing semicolon in dictionary on line 3. Parsing will be abandoned. Break on _CFPropertyListMissingSemicolon to debug.";
PID = 183;
ReadUID = 501;
Sender = Safari;
Time = 1374305592;
TimeNanoSec = 943173000;
UID = 501;
Time is a Unix timestamp, so you can use it in your query with ASL_KEY_TIME and one of these operators: ASL_QUERY_OP_EQUAL, ASL_QUERY_OP_GREATER, ASL_QUERY_OP_GREATER_EQUAL, ASL_QUERY_OP_LESS, ASL_QUERY_OP_LESS_EQUAL, ASL_QUERY_OP_NOT_EQUAL
The code below, generates a unix timestamp for yesterday and dumps all messages that occurred yesterday or later.
(Nevermind the dirty/hacky way I generate the timestamp, that was just for testing purposes)
*/
NSDate *yesterday = [NSDate dateWithTimeIntervalSinceNow: -(60.0f*60.0f*24.0f)];
NSString *theDate = [NSString stringWithFormat:#"%d", (int)[yesterday timeIntervalSince1970]];
asl_set_query(q, ASL_KEY_TIME, [theDate cStringUsingEncoding:NSASCIIStringEncoding], ASL_QUERY_OP_GREATER_EQUAL | ASL_QUERY_OP_NUMERIC);
aslresponse r = asl_search(NULL, q);
//...
For some more information on the different error levels check: http://www.cocoanetics.com/2011/03/accessing-the-ios-system-log/
Note that, depending on the level you set and the level your log messages are, further filtering may have no effect (ie. if all messages that are actually logged for your app are of the same level)
Further note, unlike the debug level querying, I haven't yet used the timestamp querying in any productive code, but in a test it seems to work perfectly fine and is doing what it is supposed to do.

How to correctly add large dataset in CoreData?

I have a huge NSArray (4.000.000 objects) that I want to save into Core Data.
Because I use ARC and the autorelease pool may get too big, I partitioned the process into multiple loops (so autorelease pools can have the chance to drain themselves).
In the following code I use a manager (clMan) to add items from the dictionaries inside the array(regions). The dictionaries contain two string fields which are parsed into scalar integers.
Code for partitioning the data into multiple loops
int loopSize = 50000;
int loops = 0;
int totalRepetitions = regions.count;
loops = totalRepetitions / loopSize;
int remaining = totalRepetitions % loopSize;
loops += 1;
for (int i = 0; i < loops; i++) {
int k = 0;
if (i == 0) k = 1;
if (i == (loops - 1))
{
// Last loop
for (long j = i * loopSize + k; j < i * loopSize + remaining; j++) {
[clMan addItemWithData:[regions objectAtIndex:j]];
}
[clMan saveContext];
break;
}
// Complete loops before the last one
for (long j = i * loopSize + k; j < (i + 1) * loopSize; j++) {
[clMan addItemWithData:[regions objectAtIndex:j]];
}
[clMan saveContext];
NSLog(#"Records added : %d", i * loopSize);
}
NSLog(#"Finished adding into core data");
Code for adding the data into core data:
-(void)addItemWithData:(NSDictionary *)data
{
MyRegion *region = [NSEntityDescription
insertNewObjectForEntityForName:#"MyRegion"
inManagedObjectContext:self.context];
region.index = [((NSString *)[data objectForKey:REGION_INDEX]) intValue];
region.id = [((NSString *)[data objectForKey:REGION_ID]) intValue];
}
The program crashes when it reaches the 1 500 000 index. The crash does not seem to happen because of parsing issues / logic.
Can anyone tell me if my logic is bad or what is the correct way to add this amount of data in CoreData?
After each loop, try calling NSManagedObjectContext.reset to "forget" the local copies in the MOC. Otherwise these might not be cleared and are causing a problem.
The WWDC 2012 code samples on iCloud have a method called seedStore where they migrate a local core data SQL database to the iCloud one - using a batch size of 5000 records and there it explicitly states that:
if (0 == (i % batchSize)) {
success = [moc save:&localError];
if (success) {
/*
Reset the managed object context to free the memory for the inserted objects
The faulting array used for the fetch request will automatically free
objects with each batch, but inserted objects remain in the managed
object context for the lifecycle of the context
*/
[moc reset];
} else {
NSLog(#"Error saving during seed: %#", localError);
break;
}
}
(Here i is the current index of the batch, thus i % batchSize == 0 if we start a new batch)

Calculating percent for 1,000s of objects slow in xcode

can anyone suggest a faster approach to the following:
I have an array of 5,000 managed objects (faulted) (an array of car.h objects)
Each object has a set of items (toCarParts.h). This set can have any number of objects.
Now i want to sort these out by the most matches in my search query carpart array.
I search for wheel, seat, window, mirror.
The method will go through each car and find the closest match, and calculate a percentage. So if car a has wheel, seat, window, mirror, mat, tire, wiper, pipe --> the % should be 50%. (Matched 4/8 parts.
This is simple enough, but the problem is with 5,000 items the search takes a long time (even using coredata).
The logic i am using goes something like: (Pseudocode)
For each Car*car in array.
NSMutableArray *x=[car tocarparts]allobjects];
For the count of objects in x.
Carpart*part=objectatindex...i.
If the name of this matches one of my parts
add a count to my counter.
At the end of the loop counter/[x count] =%.car.percent=%.
There has to be a better way, any suggestions? (I think its the dividing and checking each part that takes forever.
Thank you in advance.
Edited, added code below:.
-(NSMutableArray*)calculatePercentagePerFind:(NSMutableArray*)CarArray:(NSMutableArray*)partsArray{
NSArray*defaultParts =[NSArray arrayWithArray:[[[HelperMethods alloc]init]getObjectUserDefault:#"AvailableDefaultParts"]];
int lowestPercentMatchInt=[[[HelperMethods alloc]init]getIntegerUserDefault:#"lowestPercentageMatch"];
NSMutableArray*partsFromCarArray=[[NSMutableArray alloc]init];
NSMutableArray*returnArray=[[NSMutableArray alloc]init];
NSMutableArray *partsWithDefaultParts =[NSMutableArray arrayWithArray:partsArray];
[partsWithDefaultParts addObjectsFromArray:defaultParts];
for (int i=0; i<[CarArray count]; i++) {
double matchCount=0;
Car *CarResult =(Car*)[CarArray objectAtIndex:i];
//Check if it will at least be 30% match
double number1 = [partsWithDefaultParts count];
number1 =(number1/[CarResult.numberOfParts doubleValue])*100;
if (number1>lowestPercentMatchInt) {
partsFromCarArray =[NSMutableArray arrayWithArray:[[CarResult toParts]allObjects]];
NSMutableArray *faultedParts=[[NSMutableArray alloc]init];
for (int i =0; i<[partsFromCarArray count]; i++) {
CarPart*part = (CarPart*)[partsFromCarArray objectAtIndex:i];
[faultedParts addObject:part.name];
}
// for each part in the Car
for (NSString *partInCar in partsWithDefaultParts){
//if the search parts contain that part, add one to count
if ([faultedParts containsObject:partInCar]) {
matchCount++;
}
}
//Calculate percent match
double percentMatch = matchCount;
percentMatch =(percentMatch/[CarResult.numberOfParts doubleValue])*100;
//if at least 30%(user default) then add the percent match to Car result
if (percentMatch >lowestPercentMatchInt) {
if (percentMatch>100) {
CarResult.percentMatch = [NSNumber numberWithDouble:100.00];
}else{
CarResult.percentMatch = [NSNumber numberWithDouble:percentMatch];
}
[returnArray addObject:CarResult];
}
}
}
NSLog(#"Percent Matched Cars = %i",[returnArray count]);
return [self arrangeByHighestPercentMatch:returnArray];
}
Try this, which I believe will minimize the strain on core data.
NSSet *selectionSet; // contains the selected parts
NSPredicate *filter = [NSPredicate predicateWithFormat:
#"self IN %#", selectionSet];
float percentageSum = 0;
NSSet *parts;
for (Car *car in fetchedObjects) {
parts = car.parts; // so the relationship is retrieved only once
percentageSum +=
[parts filteredSetUsingPredicate:predicate].count*1.0f
/ (parts.count*1.0f);
}
return percentageSum/fetchedObjects.count;
This would average out the percentages across all cars. There are other methods to weigh the parts differently in the aggregate.
It is not clear from your question, but if you do not need a total percentage but one percentage for each car there would be no need to loop through all cars - you could just calculate the percentage on the fly when displaying it (e.g. with a transient property).

Parsing json returned by foursquare for iPhone gives Unrecognised leading character

I'm trying to get the nearby places using the foursquare api.
Here's the json data that is returned from
NSDictionary *results = [jsonString JSONValue];
NSLog(#"%#", results);
(
{
code = 200;
errorDetail = "This endpoint will stop returning groups in the future. Please use a current version, see http://bit.ly/lZx3NU.";
errorType = deprecated;
},
{
groups = (
{
items = (
{
categories = (
{
icon = "https://foursquare.com/img/categories/parks_outdoors/default.png";
id = 4bf58dd8d48988d163941735;
name = Park;
parents = (
"Great Outdoors"
);
pluralName = Parks;
primary = 1;
shortName = Park;
}
);
Then I try to get the list of the groups in an array with
NSArray *groups = [ (NSDictionary *)results objectForKey:#"groups"];
This returns the following error
2011-11-05 11:42:12.907 XperienzApp[1972:207] No of results returned: 0 Results : (null)
2011-11-05 11:42:13.225 XperienzApp[1972:207] -JSONValue failed. Error trace is: (
"Error Domain=org.brautaset.JSON.ErrorDomain Code=3 \"Unrecognised leading character\" UserInfo=0x5849cd0 {NSLocalizedDescription=Unrecognised leading character}"
)
2011-11-05 11:42:13.225 XperienzApp[1972:207] No of results returned: 0 Results : (null)
How should I parse this?
Edit:
I tried the suggested technique, this gives me an array
id groups = [[(NSDictionary *)results objectForKey:#"response"] objectForKey:#"groups"];
if ([results count] > 1){
NSLog(#"groups class %#\ngroups %# %d", groups, [groups class], [groups count]);
The log output is of the form:
{
categories = (
{
icon = "https://foursquare.com/img/categories/nightlife/danceparty.png";
id = 4bf58dd8d48988d11f941735;
name = Nightclub;
parents = (
"Nightlife Spots"
);
pluralName = Nightclubs;
primary = 1;
shortName = Nightclub;
}
);
contact = {
};
hereNow = {
count = 0;
};
id = 4eb33ba561af0dda8f673c1b;
location = {
address = "144 Willow St 4R";
city = Brooklyn;
crossStreet = Pierrepont;
distance = 462;
lat = "40.696864";
lng = "-73.996409";
postalCode = 11201;
state = NY;
};
name = "Entertainment 720, Ltd.";
stats = {
checkinsCount = 3;
tipCount = 0;
usersCount = 1;
};
verified = 0;
}
);
name = Nearby;
type = nearby;
}
)
groups __NSArrayM 1
This is again not json and is hard to parse, how do I get the output in json.
I'm the iPhone lead at foursquare. I'll try to take a stab at what's going on here.
First of all, I highly recommend you use JSONKit for your parser. It's lightweight and insanely fast: https://github.com/johnezang/JSONKit
It appears that you are parsing the JSON properly and getting the dictionary properly. Then you are logging the parsed object, not the original JSON. The output you are seeing is how Objective-C chooses to serialize the parsed dictionary to text. It is definitely not JSON. Using JSONKit, you could send the JSONString selector to your parsed result and convert it back to JSON and log that.
If you could provide some details on the problem you are trying to solve, I might be able to help you out more. And as Maudicus said, please pay attention to the error you are getting back. You don't want your app to break when we make the change to the API.
If the output below NSLog(#"%#", results); is your log statement. It appears your results variable is an array of dictionary objects.
Try to log the class of results to verify that NSLog(#"%#", [results class]);
If it is an array your groups object is the second object.
if ([results count] > 1)
id groups = [results objectAtIndex:1];
NSLog(#"groups class %#\ngroups %#", [groups class], groups);
Keep doing this until you understand the format of your data
Also the line
errorDetail = "This endpoint will stop returning groups in the future. Please use a current version, see http://bit.ly/lZx3NU.";
should be cause for concern. Check the documentation on foursquare for the current way of getting groups.

Resources