I need to synch up Core Data on phone with MYSQL database on server. I have gotten as far as capturing JSON feed in iOS in array. My question is, what is the best way to compare and sync up the items?
I know there are many SO posts and tutorials on syncing however the comparison is often brushed over. Once you have the JSON feed in an array and also have a managed object context, how do you compare one entry against the other?
My strategy is to give each locally saved object (record) a local id. Then when syncing occurs if the local object (record) has no server id, add it to the server table. However, I'm not sure how that works with the managed object context.
Do you iterate through the local records and compare them against server records?
Or do you iterate through the JSON array against the local managed object context?
for (NSDictionary *item in names) {
id serverid = [item objectForKey:#"serverid"];
// .... check against something in managed objectcontext
if (!items.serverid ) {
///insert on server and also provide obtained serverid to managedobjectcontext
}
}
Would appreciate any insights or suggestions.
Tim Isted addressed a similar issue in his exceptional talk at WWDC 2013.
You should watch the whole thing, because it's pretty awesome. However, the relevant part for this topic starts about 12:43 into the presentation.
Related
I have iOS app that takes data from the server as json and then serializes them into objects of different types. Types can be complicated, can contain subtypes, can inherit, so there is no any limitations. Another thing that makes everything even more complicated is some of types are stored as AnyObject? and only in run time they are being serialized into real types accordingly to the specific rules. Something like that:
class A {
var typeName: String?
var b: AnyObject?
}
Then when it's serialized it can be done something like that:
if let someClass = NSClassFromString(typeName) as? SomeGenericType.Type{
b = someClass.init()
}
Also querying should be done on all the data. Currently I'm trying to store all of them locally, then load into memory and query there from the code. I'm using User defaults, but they have some limitations, also I needed to provide custom coding to make it work, and each time when I add a new field it turned out that I missed something in coding and nothing works. So it's pain.
Ideally I would just do some magic command and all the objects are sent to local storage no matter how complicated they are. The same to extract them from this storage. Also, user change data so I can't just store primary Json. And I don't want to covert objects back to Jason as for it's pain too.
Any suggestions?
If you want to use sqlite then You can store whole object in one row! I means you can create table with 2 columns one is id and second is your dataobject(it's data type should be blob). Then convert your whole object into data. Then store in sqlite table and retrieve it as data then convert it to object when want to use. By this way your object will remains in same format as you asked
Firebase while meant for online synching and storage can also cache everything locally in case you are offline and perform query's against the local cache. It uses JSON.
CouchDB also has a mobile version for iOS.
Both of those are over kill if your dataset is small; you can just store it as a text file and read the JSON back in. See performance characteristics here. The graph is for a 7MB file so if you are significantly less than that your load time may be minimal.
NSKeyedArchiver.archivedData(withRootObject:) is great for storing custom objects as Data objects. The only thing you need to do to be able to use this is to make your custom objects conform to NSCoding. A great example can be found here:
Save custom objects into NSUserDefaults
Once you have the Data version of the object, it can easily be stored in UserDefaults, as a property in CoreData, or even in the app's keychain entries. Depending on your use case, sensitivity of data, and how much data you intend to store, you might want to use any number of storage methods. NSKeyedArchiver.archivedData(withRootObject:) allows you to pretty much use any of them.
Intro
I've read alot of tutorials and articles on Core Data concurrency, but I'm having an issue that is not often covered, or covered in a real-world way that I am hoping someone can help with. I've checked the related questions in SO and none give an answer to this particular question that I can find.
Background
We have an existing application which fetches data from an API (in the background thread) and then saves the records returned into core data. We also need to display these records in the application at the time.
So the process we currently go through is to:
Make a network request for data (background)
Parse the data and map the objects to NSManagedObjects and save (background)
In the completion handler (main thread) we fetch records from core data with the same order and limit that we requested from the API.
Most tutorials on core data concurrency follow this pattern of saving in one thread and then fetching in another, but most of them give examples like:
NSArray *listOfPeople = ...;
[NSManagedObjectHelper saveDataInBackgroundWithContext:^(NSManagedObjectContext *localContext){
for (NSDictionary *personInfo in listOfPeople)
{
PersonEntity *person = [PersonEntity createInContext:localContext];
[person setValuesForKeysWithDictionary:personInfo];
}
} completion:^{
self.people = [PersonEntity findAll];
}];
Source
So regardless of the amount of records you get back, you just fetch all content. This works for small datasets, but I want to be more efficient. I've read many times not to read/write data across threads, so fetching afterwards gets around this issue, but I don't want to fetch all, I just want the new records.
My Problem
So, for my real world example. I want to make a request to my API for the latest information (maybe anything older than my oldest record in core data) and save it, them I need the exact data returned from the API in the main thread ready for display.
So my question is, When I reach my completion handler, how do I know what to fetch? or what did the API return?. A couple of methods I've considered so far:
after saving each record, store the ID in a temporary array and then perform some fetch where id IN array_of_ids.
If I am asking for the latest records, I could just use the count of records returned, then use an order by and limit in my request to the latest x records.
My Question
I realize that the above could be answering my own question but I want to know if there is a better way, or is one of those methods much better to use than the other? I just have this feeling that I am missing something
Thanks
EDIT:
Neither answer below actually addresses the question, This is to do with fetching and saving data in the background and then using the returned data in the main thread. I know it's not a good idea to pass data between threads, so the common way around this is to fetch from core data after inserting. I want to work out the more efficient way.
Have you checked NSFetchedResultsController? Instead of fetching presented objects into array, you will use fetched controller in similar fashion. Through NSFetchedResultsControllerDelegate you would be notified about all the changes performed in background (rows added, removed, changed) and no manual tracking would be needed.
I feel You missing case with two silmultaneous API calls. Both storring ids and counting created enities wont work for that case. Consider adding timestamp property for each PersonEntity.
Assuming that Your intention is to display recently updated persons.
The calcutation of the oldest timestamp to display can look like this:
#property NSDate *lastViewRefreshTime;
#property NSDate *oldestEntityToDisplay;
(...)
if (self.lastViewRefreshTime.timeIntervalSinceNow < -3) {
self.oldestEntityToDisplay = self.lastViewRefreshTime;
}
self.lastViewRefreshTime = [NSDate date];
[self displayPersonsAddedAfter: self.oldestEntityToDisplay];
Now, if two API responses returns in period shorter than 3s their data will be displayed together.
I have a Core Data layer with several thousand entities, constantly syncing to a server. The sync process uses fetch requests to check for deleted_at for the purposes of soft-deletion. There is a single context performing save operations in a performBlockAndWait call. The relationship mapping is handled by the RestKit library.
The CoreDataEntity class is a subclass of NSManagedObject, and it is also the superclass for all our different core data object classes. It has some attributes that are inherited by all our entities, such as deleted_at, entity_id, and all the boilerplate fetch and sync methods.
My issue is some fetch requests seem to return inconsistent results after modifications to the objects. For example after deleting an object (setting deleted_at to the current date):
[CoreDataEntity fetchEntitiesWithPredicate:[NSPredicate predicateWithFormat:#"deleted_at==nil"]];
Returns results with deleted_at == [NSDate today]
I have successfully worked around this behavior by additionally looping through the results and removing the entities with deleted_at set, however I cannot fix the converse issue:
[CoreDataEntity fetchEntitiesWithPredicate:[NSPredicate predicateWithFormat:#"deleted_at!=nil"]];
Is returning an empty array in the same conditions, preventing a server sync from succeeding.
I have confirmed deleted_at is set on the object, and the context save was successful. I just don't understand where to reset whatever cache is causing the outdated results?
Thanks for any help!
Edit: Adding a little more information, it appears that once one of these objects becomes corrupted, the only way get it to register is modifying the value again. Could this be some sort of Core Data index not updating when a value is modified?
Update: It appears to be a problem with RestKit https://github.com/RestKit/RestKit/issues/2218
You are apparently using some sintactic sugar extension to Core Data. I suppose that in your case it is a SheepData, right?
fetchEntitiesWithPredicate: there implemented as follows:
+ (NSArray*)fetchEntitiesWithPredicate:(NSPredicate*)aPredicate
{
return [self fetchEntitiesWithPredicate:aPredicate inContext:[SheepDataManager sharedInstance].managedObjectContext];
}
Are you sure that [SheepDataManager sharedInstance].managedObjectContext receives all the changes that you are making to your objects? Is it receives notifications of saves, or is it child context of your save context?
Try to replace your fetch one-liner with this:
[<your saving context> performBlockAndWait:^{
NSFetchRequest *request = [NSFetchRequest fetchRequestWithEntityName:#"CoreDataEntity"];
request.predicate = [NSPredicate predicateWithFormat:#"deleted_at==nil"];
NSArray *results = [<your saving context> executeFetchRequest:request error:NULL];
}];
First, after a save have you looked in the store to make sure your changes are there? Without seeing your entire Core Data stack it is difficult to get a solid understanding what might be going wrong. If you are saving and you see the changes in the store then the question comes into your contexts. How are they built and when. If you are dealing with sibling contexts that could be causing your issue.
More detail is required as to how your core data stack looks.
Yes, the changes are there. As I mentioned in the question, I can loop through my results and remove all those with deleted_at set successfully
That wasn't my question. There is a difference between looking at objects in memory and looking at them in the SQLite file on disk. The questions I have about this behavior are:
Are the changes being persisted to disk before you query for them again
Are you working with multiple contexts and potentially trying to fetch from a stale sibling.
Thus my questions about on disk changes and what your core data stack looks like.
Threading
If you are using one context, are you using more than one thread in your app? If so, are you using that context on more than one thread?
I can see a situation where if you are violating the thread confinement rules you can be corrupting data like this.
Try adding an extra attribute deleted that is a bool with a default of false. Then the attribute is always set and you can look for entities that are either true or false depending on your needs at the moment. If the value is true then you can look at deleted_at to find out when.
Alternatively try setting the deleted_at attribute to some old date (like perhaps 1 Jan 1980), then anything that isn't deleted will have a fixed date that is too old to have been set by the user.
Edit: There is likely some issue with deleted_at having never been touched on some entities that is confusing the system. It is also possible that you have set the fetch request to return results in the dictionary style in which case recent changes will not be reflected in the fetch results.
I'm having some problems storing a change to a complex object. I've done a lot of digging and can't figure this out for the life of me.
From debugging, I can clearly see that the object is correct before storing, but when I retrieve the stored data, it's empty(say the increase of a stat). Specifically here is the breakdown below
StatSheet has ArrayList of Players
Player has ArrayList of Stats
ArrayList of StatSheets -> ArrayList of Players -> ArrayList of Stats
The ArrayList of Stat objects doesn't store after a change is made, no matter what I do. The arraylist of players seems to update fine which confuses me. I have tried changing the update depth to 2, 3, 4, 5, and beyond. I have also tried specifically setting cascadeOnUpdate to true. Can someone please help, I've been at this for days.
It's been a while that I looked at db4o and you didn't give a lot of details about your environment or code but maybe you can look at these solutions:
Do you use web environment? So look at this first answer:
A few questions about working with db4o
Do you use 'commit' when you store your objects? Because after storing and updating process you should commit the changes.
The array list of objects is store but db4o don't know what to do with the inner objects. The ArrayList isn't 'Activatable', so you can't retrieve yours objects.
You must put activationPurpose on every getter/setter of your stored object to enable the activation of object.
As you can't do this on native java objects, DB4O provide you some objects that have been tagged with activationPurpose on there getter/setter : like :
com.db4o.collections.ActivatableArrayList
So every java collection that should be store must be replace with it db4o equivalent (com.db4o.collections.*).
I'm trying to sync my data from a web service in a simple way. I download my data using AFNetworking, and using a unique identifier on each object, I want to either insert, delete or update that data.
The problem is that with Core Data you have to actually insert objects in the NSObjectManagedContext to instantiate NSManagedObjects. Like this:
MyModel *model = (MyModel *)[NSEntityDescription insertNewObjectForEntityForName:#"MyModel" inManagedObjectContext:moc];
model.value = [jsonDict objectForKey:#"value"];
So when I get the data from the web service, I insert them right away in Core Data. So there's no real syncing going on: I just delete everything beforehand and then insert what's being returned from my web service.
I guess there's a better way of doing this, but I don't know how. Any help?
You are running into the classic insert/update/delete paradigm.
The answer is, it depends. If you get a chunk of json data then you can use KVC to extract the unique ids from that chunk and do a fetch against your context to find out what exists already. From there it is a simple loop over the chunk of data, inserting and updating as appropriate.
If you do not get the data in a nice chunk like that then you will probably need to do a fetch for each record to determine if it is an insert or update. That is far more expensive and should be avoided. Batch fetching before hand is recommended.
Deleting is just about as expensive as fetching/updating since you need to fetch the objects to delete them anyway so you might as well handle updating properly instead.
Update
Yes there is an efficient way of building the dictionary out of the Core Data objects. Once you get your array of existing objects back from Core Data, you can turn it into a dictionary with:
NSArray *array = ...; //Results from Core Data fetch
NSDictionary *objectMap = [NSDictionary dictionaryWithObjects:array forKeys:[array valueForKey:#"identifier"]];
This assumes that you have an attribute called identifier in your Core Data entity. Change the name as appropriate.
With that one line of code you now have all of your existing objects in a NSDictionary that you can then look up against as you walk the JSON.
The easiest thing to do is to restore the Json to a entity that maps properly to it. Once you've mapped it, determine if a object matching the entities ID exists already, if so then fetch the entity and merge changes. If not, create a new entity in Core Data and restore the Json to it.
I'm building a app were I do client side syncing with Evernote. They keep a syncUpdate number on all of their objects and at the server level. So when I start my sync I check if my clients syncUpdate count is less than the servers. If so, I know I am out of sync. If my updateCount is at 400 and the server is at 410, I tell the server to provide me with all objects between updateCount 400 and 410. Then I check if I already have the objects or not and perform my update/create.
Every time a object is modified on the server, that objects updateCount is increments along with the servers.
The server also keeps a time stamp of the last update, which I can check against also.