I've been having this bug for several weeks already. I searched on many forums every reply on duplicates and I implemented some of the normal approaches and still it doesn't work properly.
So to give you some context I'm working on a recipe application that scraps html recipes from the web and stores it in core data, simple right? Well when the client asked adding support for iCloud Sync I though it was going to be easy specially working on iOS 7 only which solves most of the problems for you.
The problems arises when the app populates initial data in the application. I have two related entities called MainCategory[e1] and Category[e2], there is one to many relationship between them (e1 <->>> e2).
The first the app starts it will create 5 Main Categories and for each Main Category it will add 5 Categories
+ (BOOL)initialLoad
{
DLog(#"Initial Load");
//Create main and sub categories to database
NSDictionary * categoriesDic = #{
CAT_MEAL_TYPE: #[C_STARTER,C_MAINS,C_DESSERT,C_SOUPS,C_SALAD],
CAT_INGREDIENT: #[C_BEEF,C_CHICKEN,C_PASTA,C_SALMON,C_CHOCOLATE],
CAT_CUISINE : #[C_CHINESE,C_FRENCH,C_INDIAN,C_ITALIAN,C_MOROCCAN],
CAT_SEASON : #[C_CHRISTMAS,C_SUNDAY_ROAST,C_DINNER,C_BBQ,C_NIBBLES],
CAT_DIET : #[C_WHEATFREE,C_VEGETARIAN,C_LOW_FAT,C_LOW_GI,C_DAIRY_FREE]
};
NSArray * mainCategoryKeys = #[CAT_MEAL_TYPE,CAT_INGREDIENT,CAT_CUISINE,CAT_SEASON,CAT_DIET];
for(NSString * eachMainCategoryName in mainCategoryKeys)
{
//Create Main category
MainCategory * eachMainCategory = [MainCategory mainCategoryWithName:eachMainCategoryName];
NSArray * subCategories = [categoriesDic objectForKey:eachMainCategoryName];
//Create Sub categories and adds them to main category
for(NSString * eachCategoryName in subCategories)
{
/*Category got renamed to zCategory given it's a reserver name in the framework and
can not be used */
zCategory * eachCategory = [zCategory categoryWithName:eachCategoryName];
[eachMainCategory addCategoriesObject:eachCategory];
}
}
[((AppDelegate *)[UIApplication sharedApplication].delegate) saveContext];
return TRUE;
}`
Then after saving the context all this initial data will sync with the database in iCloud, so far so good. The problem comes when on the second device it runs the same initialLoad code and gets sync once again. The result is getting double MainCategories and Categories as many of you know this problem.
After reading several threads about how to remove them I used the dateCreated approach where you add a NSDate property to each entity so every time you create one instance it will have a timestamp to track which one is older and which one is newer. Then I simply add an observer from NSNotificationCenter checking the iCloud import notification NSPersistentStoreCoordinatorStoresDidChangeNotification and runs a timerCheck that after 5 seconds will execute on the mainThread a clean duplicates method.
- (void)checkTimer{
if(self.cleanTimer)
{
[self.cleanTimer invalidate];
self.cleanTimer = nil;
}//schedule timer to clean iCloud duplicates of database
self.cleanTimer = [NSTimer scheduledTimerWithTimeInterval:5 target:self selector:#selector(cleanDuplicates:) userInfo:nil repeats:FALSE];
}
- (void)cleanDuplicates:(NSTimer*)timer{
[self performSelectorOnMainThread:#selector(cleanCron) withObject:nil waitUntilDone:TRUE];}
I'm invalidating the timer every time checkTimer method gets call in order to restart it again because you normally get several NSPersistentStoreCoordinatorStoresDidChangeNotification when content gets updated/inserted/deleted, this way I know it will run once after all the notifications have gone through.
btw cleanCron just calls a class method cleanDuplicates
- (void)cleanCron
{
[CTFetchCoreData cleanDuplicates];
}`
Here is where the non magic happens, I get all the MainCategories which will be 10 given they have been duplicated and order them with the oldest ones at the beginning, then it iterates and save them in an dictionary with their name as the key so whenever it finds another MainCategory with the same name it just deletes it. Btw in the relationship e1<->>e2 there is a cascade delete rule so every time you delete a MainCategory item it deletes all the related Categories with it so there shouldn't be a problem.
+ (BOOL)cleanDuplicates
{
#synchronized(self){
//Fetch mainCategories from coreData
NSArray * mainCategories = [CTFetchCoreData fetchAllMainCategories];
// Clean duplicate Main Categories
NSMutableDictionary * uniqueMainCatDic = [NSMutableDictionary dictionary];
// Sorts the array with the oldest dateCreated one
mainCategories = [mainCategories sortedArrayUsingComparator:^NSComparisonResult(MainCategory* obj1,MainCategory * obj2) {
if(obj1.dateCreated == nil || obj2.dateCreated == nil)
{
DLog(#"ERROR Date Created");
}
return [obj1.dateCreated compare:obj2.dateCreated];
}];
// if there are more than five MainCategories it procedes the clenaup
if(mainCategories.count > 5)
{
for(MainCategory* eachMainCat in mainCategories)
{
MainCategory * originalMainCat = [uniqueMainCatDic objectForKey:eachMainCat.name];
if( originalMainCat == nil)
{
DLog(#"-> %# = %#",eachMainCat.name, eachMainCat.dateCreated);
[uniqueMainCatDic setObject:eachMainCat forKey:eachMainCat.name];
}else{
// Clean duplicate Categories
[[self managedObjectContext] deleteObject:eachMainCat];
DLog(#"x %# = %#",eachMainCat.name, eachMainCat.dateCreated);
}
}
DLog(#"Cleaning Main Categories");
}
}
[[AppDelegate sharedInstance] saveContext];
return TRUE;
}
It turns out that after I run it on the second device I will get this output :
Sesame[4145:60b] -> Cuisine = 2014-02-06 16:15:38 +0000
Sesame[4145:60b] -> Meal = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] x Meal = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] -> Ingredients = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] x Ingredients = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] x Cuisine = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] x Cuisine = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] -> Occasion = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] -> Diet = 2014-02-06 17:15:54 +0000
Sesame[4145:60b] x Diet = 2014-02-06 17:15:54 +0000
which means that the same MainCategories are getting deleted, they have the same timestamp! I'm wondering how iCloud gets the information merged.
Please if you know a better approach to clean duplicated apart from the dateCreated property please tell me because I've tried it a lot without luck, there should be a better approach.
Thanks in advance!
Update :
Finally I've managed to solve my problem after all, crazy as it sounds I was getting duplicate instances from iCloud! that's what the dates were the same. I just added an if to check if both dates are the same then don't delete the MainCategory, and so next time you open your app Core Data will refix the merge and update the database with the correct instances and different date values as it was supposed to be.
I can't see anything obviously wrong with your code, though I would recommend using UUIDs instead of dates for ordering your duplicates. But that is unlikely to be related to what you are seeing.
To be honest, it looks to me that Core Data is really messing things up. (Eg It also seems there are 3 Cuisine categories.)
I experienced this type of issue when working with Core Data sync if I tried to delete the cloud data files, and didn't give it time to thoroughly remove the files from all devices. You end up with old transaction logs in there, which trigger extra objects to be inserted.
Core Data also tries to handle all merging on its own. How that happens is anyone's guess.
Core Data + iCloud is a bit unusual in that it is one of the only sync frameworks which has no concept of global identity. There are actually good reasons for Apple not doing it, which are too subtle to discuss here, but it does make it difficult for developers. Deduping post-merge is an ugly solution IMO. Your store has to become invalid before it can become valid again.
I much prefer the approach of frameworks like Wasabi Sync, TICDS, and Ensembles, which all have a concept of global identity and do not require deduping as a result.
(Disclosure: I founded and develop the Ensembles framework)
Also avoid using this
NSMutableDictionary * uniqueMainCatDic = [NSMutableDictionary dictionary];
rather use
NSMutableDictionary * uniqueMainCatDic = [[NSMutableDictionary alloc] init];
I think your weirdness of duplicates may go away if you always alloc the mutable dictionary. It took me weeks to figure this out - not sure if its a bug.
Related
Our app has found great success recently. Part of the app consists of a huge list of products. Products that we have stored on Firebase’s Realtime Database. However our Data set for the list we populate has now exceeded 5,000+ objects. Which of course has put great strain on our Realtime Database, causing our loading time of these products to push 1 minute + on the initial load and 18 secs+ after caching using Firebase’s persistent data cache.
I’ve been reading other questions pertaining to huge data sets and utilizing Firebase and the short answer is “don’t do it”.
However, what wasn’t addressed, and the “pickle” we’re in is what “can” we do? Is there somewhere else where we can host our data that will query our data exponentially faster?
Because of our use-case, pagination isn’t an option. The JSON is around 12MB. Any advice or “We ran into the same roadblock and this is what we did” would be greatly appreciated! Thanks!
How we query for data:
[_reference
observeSingleEventOfType:FIRDataEventTypeValue
withBlock:^(FIRDataSnapshot *snapshot) {
self.dataArray = [NSMutableArray array];
self.postCountNew = 0;
for (snapshot in snapshot.children) {
[_dataArray addObject:snapshot.value];
int timeInterval =
[now timeIntervalSinceDate:[_dateFormatter dateFromString:snapshot.value[#"Date"]]];
if (timeInterval < 86400 && timeInterval >= 0 && timeInterval != 0) {
_postCountNew++;
}
}
[self.tableView reloadData];
completionBlock (YES);
}];
I try to save many objects in coredata, but get this crash:
Communications error: <OS_xpc_error: <error: 0x19b354af0> { count = 1, contents =
"XPCErrorDescription" => <string: 0x19b354e50> { length = 22, contents = "Connection interrupted" }
}>
Message from debugger: Terminated due to memory issue
I use MagicalRecord:
[MagicalRecord saveInBackgroundWithBlock:^(NSManagedObjectContext *localContext){
for (int i = 0; i < json.count; i++) {
[Product parseWithData:((NSMutableArray *)json)[i]];
}
}];
Product.m
+ (void)parseWithData:(NSDictionary *)dictionary {
NSString *xml_id = [dictionary[#"XML_ID"] isKindOfClass:[NSString class]] ? dictionary[#"XML_ID"] : #"";
Product *product = [Product getProductWithXML_id:xml_id];
if (!product)
product = [Product MR_createEntity];
product.xml_id = xml_id;
product.code = [dictionary[#"Code"] isKindOfClass:[NSString class]] ? dictionary[#"Code"] : #"";
...
}
Can you suggest me, how can i save it?
When i save my objects in loop to core data - memory grow very fast
It seems to be a memory issue.
Try surrounding the inner part of your for loop with
autoreleasepool {
...
}
You need to paginate the way you get the data and/or save it.
By paginate, i mean :
download the first 1000 (for example, it depends on the content really)
When its done, save the 1000 you just go
When that is done, get the next 1000, save it again, and so on.
You need to know the number you're trying to get and use the (if I remember correctly ) SetLimit: on the parse method, and SetSkip. The skip skips the X first elements, and the limit is the max number of items that will be downloaded.
That way, you skip 0 with limit 1000, then recall the method with skip += limit, and you'll get the second 1000 chunk, and so on. The last chunk will obviously be smaller than 1000.
Doing this will drastically increase the time taken, but that could be done seamlessly in the background ; but it will be spread over much enough to need less memory.
Do it, and see if it makes a big difference. If not, you could always reduce to 500 instead of 1000, or completely change your architecture ; maybe you don't even need ALL the items right now !
I have an iOS app that receives data from the PARSE.COM.
How did not know nothing about 'parse.com' , I used the tutorial "http://www.raywenderlich.com/15916/how-to-synchronize-core-data-with-a-web-service-part-1".
The synchronization occurs only from the server to the device (iOS), and one time the object is added to the device, it should not be inserted again.
Turns out I got 131 objects in a class and 145 in another, but the Parse.com always returns me the first 100 items, even those already are in the device (iOS).
The worst thing is that in my code I have a variable "limit" in "request" that should work, but does not work for me.
I need your help, please ...
Code:
- (NSMutableURLRequest *)GETRequestForAllRecordsOfClass:(NSString *)className updatedAfterDate:(NSDate *)updatedDate
{
NSMutableURLRequest *request = nil;
NSDictionary *paramters = nil;
if (updatedDate) {
NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];
[dateFormatter setDateFormat:#"yyyy-MM-dd'T'HH:mm:ss.'999Z'"];
[dateFormatter setTimeZone:[NSTimeZone timeZoneWithName:#"GMT"]];
NSString *jsonString = [NSString
stringWithFormat:#"{\"updatedAt\":{\"$gte\":{\"__type\":\"Date\",\"iso\":\"%#\"}}}",
[dateFormatter stringFromDate:updatedDate]];
//That's line of 'paramters' is from original tutorial Raywenderlich
// paramters = [NSDictionary dictionaryWithObject:jsonString forKey:#"where"];
//This line was add for the autors of tutorial in a comment from your blog, and he say that has work, but not for me =(
paramters = #{#"where" : jsonString, #"limit" : #(1000)};
}
request = [self GETRequestForClass:className parameters:paramters];
return request;
}
The print of variable "request" after this method is this:
URL: https://api.parse.com/1/classes/Substancia?where=%7B%22updatedAt%22%3A%7B%22%24gte%22%3A%7B%22__type%22%3A%22Date%22%2C%22iso%22%3A%222014-09-23T02%3A13%3A01.999Z%22%7D%7D%7D&limit=1000
Why do same having the variable "LIMIT = 1000", the parse.com every returns me 100 items?
And even that returns 100 items, why do in the next time he does the "request" he it does not catch the next 100 since the other 100 registers earlier have already been entered?
Can anyone help me?
(Answering here since I don't have enough reputation to comment.)
For the limit=1000 not seeming to work: perhaps the "where" clause (i.e. constraining to items with updatedAt value >= 2014-09-23T02:13:01.999Z) is limiting results to less than 1000?
(To Ian's point) There is a 'skip' parameter that tells the API how many items to skip ahead, for pagination. i.e. limit=100&skip=100 to see page 2.
I'm not sure, but I think this is what you're looking for. A great solution on how to retrieve all the objects from Parse instead of the max limit i.e. 1000.
All:
I am recording a movie, using AVCaptureMovieFileOutput. As various events occur, I wish to store the event's title/time in the QuickTime movie being written. Thus I might have 20-30 data points that I wish to associate with a particular movie.
My strategy is to use metadata, but I have not been having much luck. Can someone please tell me, first of all:
a) Can I store arbitrary metadata, or just those keys and values as defined in AVMetadataFormat.h? I would like to be able to store an array.
b) If I can store an arbitrary array, what key does the trick? If not, could I store my metadata in a comment field (ugly, but I could parse 20-30 points quickly enough).
c) The code shown below does not appear to work, as no matter what I put in for the item.key (AVMetadataQuickTimeMetadataKeyArtist, AVMetadataCommonKeyArtist, or all sorts of other things ending in Artist) I never see anything in iTune's Get Info window.
- (IBAction)recordEvent:(id)sender {
NSLog(#"Record a metadata point here ...");
// is there any metadata associated with the file yet?
NSArray * existingMetaData = self.aMovieFileOutput.metadata;
NSMutableArray * newMetadataArray = nil;
if(existingMetaData){
newMetadataArray = [existingMetaData mutableCopy];
} else {
newMetadataArray = [[NSMutableArray alloc]init];
}
AVMutableMetadataItem * item = [[AVMutableMetadataItem alloc]init];
item.keySpace = AVMetadataKeySpaceCommon;
item.key = AVMetadataQuickTimeMetadataKeyArtist;
item.value = #"Enya, really!"; // in practice this will be the title of (UIButton *)sender
item.time = CMTimeMake(0.0,1.0);
[newMetadataArray addObject:item];
self.aMovieFileOutput.metadata = newMetadataArray;
}
Any advice would be greatly appreciated.
Thanks!
Storing metadata in QuickTime file via AVCaptureMovieFileOutput & AVMutableDataItems only allows you to store values for keys predefined in AVMetadataKeySpaceCommon keyspace, i. e. AVMetadataCommonKey* like keys
All the other data is ignoring
I am working on a TV-guide app, and am trying to get the nearest 3 dates from an NSArray with NSDictionary's. So far so good, but I have been trying to figure out how I can do this the best way using as little memory as possible and with as little code (hence decreasing the likelihood of bugs or crashes). The array is already sorted.
I have a dictionary with all the channels shows for one day. The dictionary withholds an NSDate (called date).
Lets say a channel has 8 shows and the time is now 11:45. show #3 started at 11:00 and ends at 12:00, show #4 starts at 12:00 and ends at 13:00, show #5 at 13:00 to 14:00 ect.
How could I fetch show #3 (which started in the past!), #4 and #5 the fastest (memory wise) and easiest from my array of dictionaries?
Currently I am doing a for loop fetching each dictionary, and then comparing the dictionaries date with the current date. And thats where I am stuck. Or maybe I just have a brain-fag.
My current code (after a while of testing different things):
- (NSArray*)getCommingProgramsFromDict:(NSArray*)programs amountOfShows:(int)shows
{
int fetched = 0;
NSMutableArray *resultArray = [[NSMutableArray alloc] init];
NSDate *latestDate = [NSDate date];
for (NSDictionary *program in programs)
{
NSDate *startDate = [program objectForKey:#"date"];
NSLog(#"Program: %#", program);
switch ([latestDate compare:startDate]) {
case NSOrderedAscending:
NSLog(#"latestDate is older, meaning the show starts in the future from latestDate");
// do something
break;
case NSOrderedSame:
NSLog(#"latestDate is the same as startDate");
// do something
break;
case NSOrderedDescending:
NSLog(#"latestDate is more recent, meaning show starts in the past");
// do something
break;
}
// Now what?
}
return resultArray;
}
I am writing it for iOS 5, using ARC.
After your EDIT and explanation, here is another answer, hopefully fitting your question better.
The idea is to find the index of the show that is next (startDate after now). Once you have it, it will be easy to get the show at the previous index (on air) and the 2 shows after it.
NSUInteger indexOfNextShow = [arrayOfShows indexOfObjectPassingTest:^BOOL(id program, NSUInteger idx, BOOL *stop) {
NSDate* startDate = [program objectForKey:#"date"];
return ([startDate timeIntervalSinceNow] > 0); // startDate after now, so we are after the on-air show
}];
At that stage, indexOfNextShow contains the index of the show in your NSArray that will air after the current show. Thus what you want according to your question is objects at index indexOfNextShow-1 (show on air), indexOfNextShow (next show) and indexOfNextShow+1 (show after the next).
// in practice you should check the range validity before doing this
NSIndexSet* indexes = [NSIndexSet indexSetWithIndexesInRange:NSMakeRange(indexOfNextShow-1,3)];
NSArray* onAirShowAnd2Next = [arrayOfShows objectsAtIndexes:indexes];
Obviously in practice you should add some verifications (like indexOfNextShow being >0 before trying to access object at index indexOfNextShow-1 and indexOfNextShow+1 not being past the total number of shows in your array).
The advantage of this is that since your array of shows is sorted by startDate already, indexOfObjectPassingTest: returns the first object passing the test, and stop iterating as soon as it has found the right object. So this is both concise, easy-to-read code and relatively efficient.
.
I'm not sure I understood your model structure, you have an NSArray of shows, each show being a NSDictionary holding the NSDate of the show along with other info, right?
One idea then is to sort this NSArray of show according to the distance between the start time of the show and now.
NSArray* shows = ... // your arraw of NSDictionaries representing each show
NSArray* sortedShows = [shows sortedArrayUsingComparator:^(id show1, id show2) {
NSTimeInterval ti1 = fabs([[show1 objectForKey:#"startDate"] timeIntervalSinceNow]);
NSTimeInterval ti2 = fabs([[show2 objectForKey:#"startDate"] timeIntervalSinceNow]);
return (NSComparisonResult)(ti1-ti2);
}];
Then of course it is easy at that point to only take the 3 first shows of the sortedShows array.
If I've misunderstood your model structure, please edit your question to specify it, but I'm sure you can adapt my code to fit your model then
The question asks for the "fastest (memory wise)". Are you looking for the fastest or the most memory/footprint conscious? With algorithms there is often a space vs. time tradeoff so as you make it faster, you typically do it by adding indexes and other lookup data structures which increase the memory footprint.
For this problem the straight forward implementation would be to iterate through each channel and each item comparing each against the top 3 held in memory. But that could be slow.
With additional storage, you could have an additional array which indexes into time slots (one per 15 minutes granularity good enough?) and then daisy chain shows off of those time slots. Given the current time, you could index straight into the current times slot and then look up the next set of shows. The array would have pointers to the same objects that the dictionaries are pointing to. That's an additional data structure to optimize one specific pattern of access but it does it at a cost - more memory.
That would increase your foot print but would be very fast since it's just an array index offset.
Finally, you could store all your shows in a sqlite database or CoreData and solve your problem with one query. Let the sql engine do the hard work. that wold also keep your memory foot print reasonable.
Hope that sparks some ideas.
EDIT:
A crude example showing how you can construct a look table - an array with slots for every 15 minutes. It's instant to jump to the current time slot since it's just an array offset. Then you walk the absolute number of walks - the next three and you're out. So, it's an array offset with 3 iterations.
Most of the code is building date - the lookup table, finding the time slot and the loop is trivial.
NSInteger slotFromTime(NSDate *date)
{
NSLog(#"date: %#", date);
NSDateComponents *dateComponents = [[NSCalendar currentCalendar] components:(NSHourCalendarUnit | NSMinuteCalendarUnit) fromDate:date];
NSInteger hour = [dateComponents hour];
NSInteger minute = [dateComponents minute];
NSInteger slot = (hour * 60 + minute)/15;
NSLog(#"slot: %d", (int)slot);
return slot;
}
int main (int argc, const char * argv[])
{
// An array of arrays - the outer array is an index of 15 min time slots.
NSArray *slots[96];
NSDate *currentTime = [NSDate date];
NSInteger currentSlot = slotFromTime(currentTime);
// populate with shows into the next few slots for demo purpose
NSInteger index = currentSlot;
NSArray *shows1 = [NSArray arrayWithObjects:#"Seinfeld", #"Tonight Show", nil];
slots[++index] = shows1;
NSArray *shows2 = [NSArray arrayWithObjects:#"Friends", #"Jurassic Park", nil];
slots[++index] = shows2;
// find next three -jump directly to the current slot and only iterate till we find three.
// we don't have to iterate over the full data set of shows
NSMutableArray *nextShow = [[NSMutableArray alloc] init];
for (NSInteger currIndex = currentSlot; currIndex < 96; currIndex++)
{
NSArray *shows = slots[currIndex];
if (shows)
{
for (NSString *show in shows)
{
NSLog(#"found show: %#", show);
[nextShow addObject:show];
if ([nextShow count] == 3)
break;
}
}
if ([nextShow count] == 3)
break;
}
return 0;
}
This outputs:
2011-10-01 17:48:10.526 Craplet[946:707] date: 2011-10-01 21:48:10 +0000
2011-10-01 17:48:10.527 Craplet[946:707] slot: 71
2011-10-01 17:48:14.335 Craplet[946:707] found show: Seinfeld
2011-10-01 17:48:14.336 Craplet[946:707] found show: Tonight Show
2011-10-01 17:48:21.335 Craplet[946:707] found show: Friends