Here's my setup:
On every UIViewController viewWillAppear method I fetch data from the server. Data is parsed into realm objects which later added to Realm DB. I've setup a notification block to report if any changes occur to the results. Now, the problem is that even though fetched objects are identical to the ones already written to DB, RealmCollectionChange still reports as if all objects were modified. Here's a sample code:
Fetching / Parsing:
realmDB.beginWrite()
for projectJSON in projectsArray {
let project = createObjectFromJson(projectJSON)
realmDB.add(project, update: true)
}
realmDB.commitWrite()
Change Observer:
notificationToken = projects.addNotificationBlock { changes in
switch changes {
case .Update(_, let deletions, let insertions, let modifications):
...
}
So here, modifications always return full list of indexes as if all objects have been updated.
Is that expected? Any way to avoid that behavior? Seems as add:update forces an update as opposed to skipping an update if objects are the same.
In your createObjectFromJson method, you are almost certainly setting object properties regardless of whether they have changed. Unfortunately, setting a property is detected as a modification even if the value was the same. I'm not sure if this is intended behavior, but one way to get around this is to only set the property if the new value is not equal to the old, though this may get ugly.
Related
We have an iOS application that uses Core Data to persist records fetched from a private web API. One of our API requests fetches a list of Project records, each of which has multiple associated Location records. ObjectMapper is used to deserialize the JSON response, and we have a custom transformer that assigns the nested Location attributes to a Core Data association on the Project entity.
The relevant part of the code looks like this. It's executed within a PromiseKit promise (hence the seal), and we save first to a background context and then propagate to the main context that gets used on the UI thread.
WNManagedObjectController.backgroundContext.perform {
let project = Mapper<Project>().map(JSONObject: JSON(json).object)!
try! WNManagedObjectController.backgroundContext.save()
WNManagedObjectController.managedContext.performAndWait {
do {
try WNManagedObjectController.managedContext.save()
seal.fulfill(project.objectID)
} catch {
seal.reject(error)
}
}
}
The problem we're having is that this insert process is saving each Location record to the database twice. Strangely, the duplicated Location records don't have any association with their parent Project record. That is to say, if Location records are looked up with an NSFetchRequest, or if I run a query on the underlying SQLite database, I can see that there are two entries for each Location, but project.locations only returns one copy of each Location. The same (or very similar) process applied to other record types with the same structure also results in duplicates.
I've tried several things so far to narrow down the problem:
Inspected the API JSON - no duplicates.
Inspected the state of the project.locations property immediately before the Core Data write. No duplicate records are present prior to the objects being persisted, indicating that the deserializer and custom nested attributes transformer are working correctly.
Removed the block that propagates the changes to the main thread managed object context, in case this was causing the insert to occur twice. Still get duplicates with solely the write to the background context.
Run the app with com.apple.CoreData.ConcurrencyDebug 1 set. No exception is thrown in this process, confirming that it's not a thread safety issue of some kind.
Run the app with com.apple.CoreData.SQLDebug 1 set. I can see in the logs that Core Data is inserting exactly twice as many Location rows as expected into the underlying SQLite database.
Implemented a uniqueness constraint on the entity. This fixes the problem in terms of what data gets persisted, but will still throw an error unless an NSMergePolicy is set.
The last item in that list effectively solves the problem, but it's treating the symptom, not the cause. Data integrity is important for our application, and I'm looking to understand what the underlying problem might be, or other options I might pursue for investigating it further.
Thanks!
A year and eight months later, I finally got to the bottom of this bug when a similar issue occurred with a different set of records. The problem was that I was calling ObjectMapper on each Location object twice. I was using ObjectMapper's mapArray method within a custom ObjectMapper TransformType to deserialize and persist the Location records associated with each Project, which worked as follows:
let locations = Mapper<Location>().mapArray(JSONObject: value as AnyObject)
However, what I had overlooked is that I was also overriding the constructor for Location and calling ObjectMapper again there:
required public init?(map: Map) {
let entity = NSEntityDescription.entity(forEntityName: "Location", in: WNManagedObjectController.backgroundContext)
super.init(entity: entity!, insertInto: WNManagedObjectController.backgroundContext)
mapping(map: map)
}
The line mapping(map: map) was unnecessary, and proved to be the culprit. In a similar scenario with two levels of one-to-many associations, this had the somewhat amusing consequence of quadrupling (!) the records at the second level - their parents had been duplicated, each copy of which subsequently duplicated its children. This was what ultimately led me to the cause of the bug.
I have a SwiftUI calendaring app with a UI similar to the built-in Calendar.app. I'm getting crashes whenever I try to delete events. The overall lifecycle of my app is as follows:
Download calendar data from server and populate models ([Events], [Users], [Responses] etc)
Transform the source data into a more structured format (see https://stackoverflow.com/a/58583601/2282313)
Render list view of events, each event linking to a Detail View and an Edit modal (very similar to calendar.app)
When an event is deleted, I tell the server to delete the event (if it's a recurring event, the server will delete multiple events), then refresh my data from the server by re-downloading the data, re-populating the models and re-generating the structured data (which causes the list to refresh).
When I do this, I get crashes coming from my calculated values because event data displayed in the detail view is no longer available. For example, I get the array index of a user's RSVP as follows:
var responseIndex: Int {
userData.responses.firstIndex(where: { $0.user == response.user && $0.occurrence == response.occurrence })!
}
I thought this was because I hadn't dismissed the view displaying the deleted event before updating the data, but even if I delay the data refresh until the view is no longer displayed, I still get the crash (SwiftUI seems to keep these views in memory).
What is the right way to handle data deletion? Do I need to keep deleted events in my UserData EnvironmentObject and just mark them as "deleted/hidden" to avoid this issue, or is there a better way to handle it?
There's quite a bit of code involved in this, so it's tricky to provide a sample I'm happy to add relevant bits if asked.
EDIT: I found this article which clarifies something really well: https://jasonzurita.com/swiftui-if-statement/
SwiftUI is perfectly happy to try and render nil views, it just draws nothing. Counter-intuitively, a good way to avoid crashes and make the compiler happy is to set your code up around this.
Original "answer" follows...
I don't know if this is the "right" way to do this, but I ended up making sure that none of my UserData is ever deleted to avoid the crashes. I added a "deleted" bool to my Occurrence (i.e. Event) object, and when I refresh my structured data, I get the latest data from the server, but check to see if any of the old ones are no longer present. Steps are:
Get latest list of occurrences from server
Create a second init() for my structured data which takes the existing data as an argument
Inside the new init(), flatten the structured data, check for deleted items against the new data, update data which hasn't been removed, cull duplicates, then merge in net new data. Once that's done, I call my original init() with the modified data to create new structured data
Code looks like this:
init(occurrences: [Occurrence], existing: [Day]) {
// Create a mutable copy of occurrences (useful so I can delete duplicates)
var occurrences = occurrences
// Flatten the structured data into a plan array of occurrences again
var existingOccurrences = existing.compactMap({ $0.occurrences }).flatMap { $0 }
// Go through existing occurrences and see if they still exist.
existingOccurrences = existingOccurrences.map {
occurrence -> Occurrence in
let occurrenceIndex: Int? = occurrences.firstIndex(where: { $0.id == occurrence.id })
// If the occurrence no longer exists, mark it as "deleted" in the original data
if occurrenceIndex == nil {
var newOccurrence = occurrence
newOccurrence.deleted = true
return newOccurrence
// If it still exists, replace the existing copy with the new copy
// (in case it has changed since the last pull from the server)
// Remove the event from the "new" data so you don't get duplicates
} else {
let newOccurrence = occurrences[occurrenceIndex!]
occurrences.remove(at: occurrenceIndex!)
return newOccurrence
}
}
// Merge the existing data (with deleted items marked) and the updated data (with deleted items removed)
let finalOccurrences = existingOccurrences + occurrences
// Re-initialize the strutured data with the new array of data
self = EventData(occurrences: finalOccurrences)
}
Once this was done, I had to update my code to make sure I'm always using my structured data as the source of truth (which I wasn't doing before because accessing the "source" flat data was often easier, and I've updated my ForEach in my list view to only render a row if deleted is false.
It works! It's perhaps a sub-optimal way to solve the problem, but no more crashes. Still interested to hear better ways to solve the problem.
I've got an app that uses a Firebase db containing 100,000 items. My app has to process through each of these items which takes several seconds.
What is happening is that every time the app is launched (from a terminated state) those 100,000 items are being processed each time (even if the contents of the db on the Firebase server have not changed). Obviously, I don't want the app to do this if not necessary. Here's some code:
if dbRef == nil {
FirebaseApp.configure();
Database.database().isPersistenceEnabled = true
...
let dbRef = Database.database().reference(withPath: kFirebaseDBName)
_ = spamRef.observe(DataEventType.value, with: { (theSnapshot) in
if let content = theSnapshot.value as? [String : AnyObject]
{
self.processContent(content: content)
}
Each time the app is started then the content snapshot contains the entire database reference contents.
Is there a way of, for example, getting the last date the database was updated (on the server), or only obtaining the delta of changed items between each app launch - can a query return just changed since last queried for example, or something similar?
I don't know how many items have changed so cannot call something like:
queryLimited(toLast: N))
As I don't know what value N is.
I've tried adding keepSynced as follows in the hope it might change things, but no.
if dbRef == nil {
FirebaseApp.configure();
Database.database().isPersistenceEnabled = true
...
let dbRef = Database.database().reference(withPath: kFirebaseDBName)
dbRef.keepSynced(true)
_ = dbRef.observe(DataEventType.value, with: { (theSnapshot) in
if let content = theSnapshot.value as? [String : AnyObject]
{
self.processContent(content: content)
}
I have no idea how much data might have changed so don't know what value to supply to something like toLast or similar to modify the observation parameters.
The database (which was not created nor updated with new content by me) has 100,000 items in a flat structure (i.e. one parent with 100,000 children) and any number of these children in any order might have been deleted and replaced since last time my app ran, but the total will still be 100,000. None of the children have an explicit timestamp or anything like that.
I was under the impression if Firebase kept a local cache of the data (due to isPersistenceEnabled) then next time it connects with the server it would only sync what had changed on the server. Therefore in order to do this Firebase itself must internally have some delta information somewhere, so I was hoping that delta information may available in some form to my app.
Note: My app does not need persistence to be enabled, the above code is doing so just as variations to see if anything will result in the behavior I desire with the observer.
UPDATE
So looking at the documentation more you can set a timestamp for the last time a user was connected to the server using:
lastOnlineRef.onDisconnectSetValue(ServerValue.timestamp())
Take a look at this question Frank explains some issues with persistence and listeners. The question is for Android but the principles are the same.
I still think the problem is your query. Since you already have the data persisted .value is not what you want since this returns all of the data.
I think you want to attach a .childChanged listener to your query. In this case the query will only return the data that has been changed. If you haven't heard of .childChanged before you can read about it here.
I didn't realize this problem is specifically related to persistence. I think you are looking for keepSynced(). Take a look at this.
ORIGINAL ANSWER
The problem is your query. You are asking for all of the data that's why you're getting all of the data. You want to look into limiting your queries using toFirst or toLast. Additionally, I don't think you can query for the last time the database was updated. You could check the last node in your data structure if you have the timestamp saved, but you might as well just get the newest data.
You want something like this:
ref.child("yourChild").queryLimited(toLast: 7).observeSingleEvent(of: .value, with: { snap in
// do something
})
Depending on how you're writing your data you'll want toLast or toFirst. Assuming the newest data is written last toLast is what you want. Also note that the numbers I am limiting to are arbitrary you can use any number that fits your project.
If you already have a key and you want to start querying above that key you can do something like this:
ref.child("YourChild").queryOrderedByKey().queryEnding(atValue: lastVisiblePostKey).queryLimited(toLast: 8).observeSingleEvent(of: .value, with: { snap in
// do something with more posts
})
You may also want to look into this question, this question and pagination.
When I call this observe function from in my viewcontroller, the .childadded immediately returns a object that was already stored instead of has just bin added like .childadded would suspect.
func observe(callback: RiderVC){
let ref = DBProvider.Instance.dbRef.child("rideRequests")
ref.observe(DataEventType.childAdded) { (snapshot: DataSnapshot) in
if let data = snapshot.value as? NSDictionary {
let drive = cabRide(ritID: ritID, bestemming: bestemming,
vanafLocatie: vanaf, taxiID: taxiID, status: status)
print(drive)
callback.alertForARide(title: "Wilt u deze rit krijgen?", message: "Van: \(vanaf), Naar: \(bestemming)", ritID: ritID)
}
}
}
When I try this function with .childchanged, I only get a alert when it is changed like it suppose to do, but when doing .chiladded, it just gets all the requests out of the database and those requests were already there.
When I add a new request, it also gives an alert. So it works, but how can I get rid of the not added and already there requests?
Does anybody know this flaw?
This is working exactly as promised. From the documentation:
Retrieve lists of items or listen for additions to a list of items.
This event is triggered once for each existing child and then again
every time a new child is added to the specified path. The listener is
passed a snapshot containing the new child's data.
That might seem weird at first, but this is generally what most developers want, as it's basically a way of asking for all data from a particular branch in the database, even if new items get added to it in the future.
If you want it to work the way you're describing, where you're only getting new items in the database after your app has started up, you'll need to do a little bit of work yourself. First, you'll want to add timestamps to the objects you're adding to the database. Then you'll want to do some kind of call where you're asking to query your database by those timestamps. It'll probably look something like this:
myDatabaseRef.queryOrdered(byChild: "myTimestamp").queryStarting(atValue: <currentTimestamp>)
Good luck!
I am trying to track changes to objects in a core data context, tracking the name of properties that have changed along with the old and new values.
I've registered for NSManagedObjectContextWillSaveNotification to receive a notification when a save is about to occur, and can pull out the inserted/updated/deleted objects from the context... I can then see the changed values using .changedValues.
However, I am having difficulties retrieving the old values...
As an example, I have an object that tracks a position, and so one of the changes comes back with:
po [obj changedValues]
{
originX = 260;
originY = 180;
}
This gives me the new values for the properties that have changed on the object. To try and get the old values, I'm then using changedValuesForCurrentEvent, which according to the docs should return
"a dictionary containing the keys and old values of persistent
properties that have changed since the last posting of
NSManagedObjectContextObjectsDidChangeNotification"
However, when I try this, it is coming back empty...:
po [obj changedValuesForCurrentEvent]
{
}
How can I capture the old and new values?
You're mixing up your notifications. NSManagedObjectContextObjectsDidChangeNotification gets called any time you change values on a managed object, even though you haven't saved changes yet. NSManagedObjectContextWillSaveNotification gets called later on when you save. So the sequence is:
You change some attributes --> NSManagedObjectContextObjectsDidChangeNotification is posted, and you can use changedValuesForCurrentEvent to see what changed.
Later, you save changes. NSManagedObjectContextWillSaveNotification is posted. You can call changedValuesForCurrentEvent, but it's not helpful because it returns changes since the last did-change notification. There are no changes since the last did-change notification. If there were, you would have received another one. That method is documented to be useful on a did-change notification, not on a will-save notification.
If you want the old values and you want to get them when the will-save notification is posted, you have a couple of options:
Listen for NSManagedObjectContextObjectsDidChangeNotification. Cache information about changes in some collection object (probably NSDictionary). Then when NSManagedObjectContextWillSaveNotification happens, look up those changes, process them, and clear the change cache. OR...
When you get NSManagedObjectContextWillSaveNotification, create a second local managed object context. Since this is a will save notification, you can still fetch the old values. So, fetch each object that's getting saved and compare the before and after values to see what's different.
Although this question is 4 years old, Eddie's answer was very helpful. I made a little change to his answer. All the credits goes to him.
object.setValuesForKeys(object.committedValues(forKeys: object.changedValues().map { $0.key }))
I know this question is old, but there is a better way than the accepted answer. You can access the previous values via committedValues(forKeys:) in combination with changedValues(). There is no need to handle NSManagedObjectContextObjectsDidChangeNotification or to create another managed object context.
Here is some sample code that I use:
// For some reason, the Swift compiler chokes on the type of object.changedValues().keys.
// It should be of type [String], but it complains that it is of type `Dictionary<String, Any>.Keys`
// which is useless. Ah, the joys of Apple programming...
// Work around that like so:
var changedKeys = [String]()
for (key, _) in object.changedValues() {
changedKeys.append(key)
}
let oldData = object.committedValues(forKeys: changedKeys)
Sounds like you should call "changedValuesForCurrentEvent" only when you receive your "NSManagedObjectContextWillSaveNotification" notification.
And if "changedValuesForCurrentEvent" still returns a null dictionary or object, check to see if the notification had anything useful in it's "userInfo" dictionary itself. It also may be that there has not been a NSManagedObjectContextObjectsDidChangeNotification" posted, like you posted from the docs up there.