I am learning pagination with Firebase. I am using a method in which I store the key of the last added item in the previous page, so the next page can continue from there.
The problem is that when using ref.queryStarting(at value: lastItemKey) to keep retrieving items from the last added key, the last item gets repeated twice (since queryStarting is inclusive).
And so if I limit to 5 the query I would end up with only 4 new items as 1 would be a duplicate.
The only solution I came up is requesting one more item and remove the repeated one, but I wonder if it´s efficient at all doing it this way. (since we are wasting one item in each query)
If it´s any help, my code looks like this:
// rest of the pages
if let lastItemID = lastItemKey {
itemPageRef = self.itemsRef.queryOrderedByKey().queryStarting(atValue: lastItemID)
.queryLimited(toFirst: UInt(amount))
} else {
// First page of data: we retrieve the first (amount) items
print("We are in the first page of DATA")
itemPageRef = self.itemsRef.queryOrderedByKey().queryLimited(toFirst: UInt(amount))
}
itemPageRef.observeSingleEvent(of: .value, with: { [weak self] (snapshot) in
Requesting an overlapping child node between the pages is the only way the Firebase API supports. Since there is no other way to do this, there isn't a more efficient way.
That said, it's typically quite efficient, especially if you use a page size of 25+ child nodes, which is also more reasonable on most use-cases I've seen.
Related
I have a notes app. I normally paginate the below posts-userIds node by childByAutoId which works fine. I allow users to make edits, at which point I just update everything and add an editDate to the post itself. But that editDate is at the posts ref, not the posts-usersIds ref.
How can I paginate the posts-usersIds ref by editDate? The issue is editDates are optional, meaning they might make 200 posts, but only 2 edits or even none at all. Either way if the user wants to see the editDates first they still need to see the postDates along with them
Order would be editDates first then postDates second or if there aren't any editDates just show them all of the postDates
I was thinking instead of using a 1 as the value maybe I should put a postDate or editDate (if they made one) as the value to the posts-userIds ref as in
-postId-123: 1 // this what I normally use, it has no meaning, just a 1 so that it has a value
-postId-123: replace the 1 with the postDate and change this to editDate when they make an edit
Maybe I could then use ref.queryOrderedByValue() but I'm not sure.
My structure is the following:
#posts
#postId-123 // childByAutoId
-uid: "userId-ABC"
-postDate: 1640898417.456389
-editDate: 1640814049.713224 // edit date is only there if the user made an edit
#posts-userIds
#userId-ABC
-postId-123: 1
I paginate
let ref = Database.database().reference().child("posts-userIds").child(Auth.auth().currentUser!.uid)
if startKey == nil {
ref.queryOrderedByKey()
.queryLimited(toLast: 20)
.observeSingleEvent(of: .value, with: { (snapshot) in
// get posts and set startKey
})
} else {
ref.queryOrderedByKey()
.queryEnding(atValue: startKey!)
.queryLimited(toLast: 21)
.observeSingleEvent(of: .value, with: { (snapshot) in
})
}
Firebase queries can only order/filter on a value that is (in a fixed path) under the node that it's considering to return. So you can't posts-userIds on values from under posts.
The solution is to duplicate the value that you want to order/filter on under posts-userIds, either directly as the value (in which case you'll use queryOrderedByValue for the order), or in a child property (in which case you'll use queryOrdered(byChild:) for the order. I tend to favor the latter, simply because once there's one value you want to order/filter on, there'll often be more of them down the line.
This sort of data duplicating is quite common in NoSQL databases, so if you're not yet comfortable with it, I recommend reading NoSQL data modeling, and watching Firebase for SQL developers.
I am attempting to build a tableview that displays the users who have liked the current user's posts. For example, "John Smith, Tom Jones, and Sally Hughes liked your post (display image of post in cell". I think I'm on the right track but I'm wondering if my data structure is going to make this unnecessarily difficult or if there is an easier way.
My Firestore data structure is just below. The "BmV..." is the userId, "-M0e..." is the postId, and the "Ff0..." and "nIh..." are the users who have liked the post.
"likeActivity" : {
"BmvRlWWuGRWApqFvtT8mXQlDWzz2" : {
"-M0efUXcZy43fDVXjTvT" : {
"Ff0CxZhQzYVHuqbnsiOKwRAB01D2" : true,
"nIhx1SnChjapy4cbrD5sC1WIZXM2" : true
}
},
}
My first question is if this is the best way to structure this data? Then, In the ActivityViewController using the following code to retrieve of the current user's posts with activity.
var activityDict = [String: [Any]]()
let newActivity = DataService.ds.REF_LIKE_ACTIVITY.child("\(uid)")
//print("NEW POST - \(newPost)")
newActivity.observe(.value, with: { (snapshot) in
self.posts = []
if let snapshot = snapshot.children.allObjects as? [DataSnapshot] {
for snap in snapshot {
print("ACTIVITY -- \(snap.key)")
let userLikeData = DataService.ds.REF_LIKE_ACTIVITY.child("\(uid)").child(snap.key)
userLikeData.observe(.value, with: { (snapshot) in
self.posts.append(snap.key)
print("SNAPSHOT VALUE -- \(snapshot)")
if self.activityDict["\(snap.key)"] != nil {
self.activityDict["\(snap.key)"]!.append(snapshot.value!)
} else {
self.activityDict["\(snap.key)"] = [snapshot.value!]
}
self.activityTableView.reloadData()
})
}
}
})
My next question is if the activity should be accumulated in an array of dictionaries like I have it? Is there a better way to organize this data?
My first question is if this is the best way to structure this data?
When using NoSQL databases there is no singular "best" way to store data. You'll instead store the data in a way that best supports your use-cases. And since you'll typically uncover more (details about your) use-cases as you implement and evolve your app, you data model evolves (adapting and expanding) with it.
I recommend reading NoSQL data modeling, and watching Firebase for SQL developers and Getting to know Cloud Firestore. The last one is for Cloud Firestore, but many of the techniques Todd discusses apply equally to other NoSQL databases.
should [the activies] be accumulated in an array of dictionaries like I have it?"
If that works for your use-cases, then it sounds fine.
The only thing of note is that you're using observe, which attaches a permanent listener, and then append the updated snapshot data to the array. This means that if an activity gets changed in the database, your closure will get called again with a snapshot for that activity.
This is great, because it allows you to show the updated activity in the UI. But since you're appending it to the array, you'll end up displaying the activity twice: once as it was stored in the database when you first loaded it, and then again as it exists after the update. If that is what you're aiming for then 👍, but it is more common to update the existing data in the array, instead of adding the updated data as a new item.
I have a SwiftUI calendaring app with a UI similar to the built-in Calendar.app. I'm getting crashes whenever I try to delete events. The overall lifecycle of my app is as follows:
Download calendar data from server and populate models ([Events], [Users], [Responses] etc)
Transform the source data into a more structured format (see https://stackoverflow.com/a/58583601/2282313)
Render list view of events, each event linking to a Detail View and an Edit modal (very similar to calendar.app)
When an event is deleted, I tell the server to delete the event (if it's a recurring event, the server will delete multiple events), then refresh my data from the server by re-downloading the data, re-populating the models and re-generating the structured data (which causes the list to refresh).
When I do this, I get crashes coming from my calculated values because event data displayed in the detail view is no longer available. For example, I get the array index of a user's RSVP as follows:
var responseIndex: Int {
userData.responses.firstIndex(where: { $0.user == response.user && $0.occurrence == response.occurrence })!
}
I thought this was because I hadn't dismissed the view displaying the deleted event before updating the data, but even if I delay the data refresh until the view is no longer displayed, I still get the crash (SwiftUI seems to keep these views in memory).
What is the right way to handle data deletion? Do I need to keep deleted events in my UserData EnvironmentObject and just mark them as "deleted/hidden" to avoid this issue, or is there a better way to handle it?
There's quite a bit of code involved in this, so it's tricky to provide a sample I'm happy to add relevant bits if asked.
EDIT: I found this article which clarifies something really well: https://jasonzurita.com/swiftui-if-statement/
SwiftUI is perfectly happy to try and render nil views, it just draws nothing. Counter-intuitively, a good way to avoid crashes and make the compiler happy is to set your code up around this.
Original "answer" follows...
I don't know if this is the "right" way to do this, but I ended up making sure that none of my UserData is ever deleted to avoid the crashes. I added a "deleted" bool to my Occurrence (i.e. Event) object, and when I refresh my structured data, I get the latest data from the server, but check to see if any of the old ones are no longer present. Steps are:
Get latest list of occurrences from server
Create a second init() for my structured data which takes the existing data as an argument
Inside the new init(), flatten the structured data, check for deleted items against the new data, update data which hasn't been removed, cull duplicates, then merge in net new data. Once that's done, I call my original init() with the modified data to create new structured data
Code looks like this:
init(occurrences: [Occurrence], existing: [Day]) {
// Create a mutable copy of occurrences (useful so I can delete duplicates)
var occurrences = occurrences
// Flatten the structured data into a plan array of occurrences again
var existingOccurrences = existing.compactMap({ $0.occurrences }).flatMap { $0 }
// Go through existing occurrences and see if they still exist.
existingOccurrences = existingOccurrences.map {
occurrence -> Occurrence in
let occurrenceIndex: Int? = occurrences.firstIndex(where: { $0.id == occurrence.id })
// If the occurrence no longer exists, mark it as "deleted" in the original data
if occurrenceIndex == nil {
var newOccurrence = occurrence
newOccurrence.deleted = true
return newOccurrence
// If it still exists, replace the existing copy with the new copy
// (in case it has changed since the last pull from the server)
// Remove the event from the "new" data so you don't get duplicates
} else {
let newOccurrence = occurrences[occurrenceIndex!]
occurrences.remove(at: occurrenceIndex!)
return newOccurrence
}
}
// Merge the existing data (with deleted items marked) and the updated data (with deleted items removed)
let finalOccurrences = existingOccurrences + occurrences
// Re-initialize the strutured data with the new array of data
self = EventData(occurrences: finalOccurrences)
}
Once this was done, I had to update my code to make sure I'm always using my structured data as the source of truth (which I wasn't doing before because accessing the "source" flat data was often easier, and I've updated my ForEach in my list view to only render a row if deleted is false.
It works! It's perhaps a sub-optimal way to solve the problem, but no more crashes. Still interested to hear better ways to solve the problem.
I've got an app that uses a Firebase db containing 100,000 items. My app has to process through each of these items which takes several seconds.
What is happening is that every time the app is launched (from a terminated state) those 100,000 items are being processed each time (even if the contents of the db on the Firebase server have not changed). Obviously, I don't want the app to do this if not necessary. Here's some code:
if dbRef == nil {
FirebaseApp.configure();
Database.database().isPersistenceEnabled = true
...
let dbRef = Database.database().reference(withPath: kFirebaseDBName)
_ = spamRef.observe(DataEventType.value, with: { (theSnapshot) in
if let content = theSnapshot.value as? [String : AnyObject]
{
self.processContent(content: content)
}
Each time the app is started then the content snapshot contains the entire database reference contents.
Is there a way of, for example, getting the last date the database was updated (on the server), or only obtaining the delta of changed items between each app launch - can a query return just changed since last queried for example, or something similar?
I don't know how many items have changed so cannot call something like:
queryLimited(toLast: N))
As I don't know what value N is.
I've tried adding keepSynced as follows in the hope it might change things, but no.
if dbRef == nil {
FirebaseApp.configure();
Database.database().isPersistenceEnabled = true
...
let dbRef = Database.database().reference(withPath: kFirebaseDBName)
dbRef.keepSynced(true)
_ = dbRef.observe(DataEventType.value, with: { (theSnapshot) in
if let content = theSnapshot.value as? [String : AnyObject]
{
self.processContent(content: content)
}
I have no idea how much data might have changed so don't know what value to supply to something like toLast or similar to modify the observation parameters.
The database (which was not created nor updated with new content by me) has 100,000 items in a flat structure (i.e. one parent with 100,000 children) and any number of these children in any order might have been deleted and replaced since last time my app ran, but the total will still be 100,000. None of the children have an explicit timestamp or anything like that.
I was under the impression if Firebase kept a local cache of the data (due to isPersistenceEnabled) then next time it connects with the server it would only sync what had changed on the server. Therefore in order to do this Firebase itself must internally have some delta information somewhere, so I was hoping that delta information may available in some form to my app.
Note: My app does not need persistence to be enabled, the above code is doing so just as variations to see if anything will result in the behavior I desire with the observer.
UPDATE
So looking at the documentation more you can set a timestamp for the last time a user was connected to the server using:
lastOnlineRef.onDisconnectSetValue(ServerValue.timestamp())
Take a look at this question Frank explains some issues with persistence and listeners. The question is for Android but the principles are the same.
I still think the problem is your query. Since you already have the data persisted .value is not what you want since this returns all of the data.
I think you want to attach a .childChanged listener to your query. In this case the query will only return the data that has been changed. If you haven't heard of .childChanged before you can read about it here.
I didn't realize this problem is specifically related to persistence. I think you are looking for keepSynced(). Take a look at this.
ORIGINAL ANSWER
The problem is your query. You are asking for all of the data that's why you're getting all of the data. You want to look into limiting your queries using toFirst or toLast. Additionally, I don't think you can query for the last time the database was updated. You could check the last node in your data structure if you have the timestamp saved, but you might as well just get the newest data.
You want something like this:
ref.child("yourChild").queryLimited(toLast: 7).observeSingleEvent(of: .value, with: { snap in
// do something
})
Depending on how you're writing your data you'll want toLast or toFirst. Assuming the newest data is written last toLast is what you want. Also note that the numbers I am limiting to are arbitrary you can use any number that fits your project.
If you already have a key and you want to start querying above that key you can do something like this:
ref.child("YourChild").queryOrderedByKey().queryEnding(atValue: lastVisiblePostKey).queryLimited(toLast: 8).observeSingleEvent(of: .value, with: { snap in
// do something with more posts
})
You may also want to look into this question, this question and pagination.
When I call this observe function from in my viewcontroller, the .childadded immediately returns a object that was already stored instead of has just bin added like .childadded would suspect.
func observe(callback: RiderVC){
let ref = DBProvider.Instance.dbRef.child("rideRequests")
ref.observe(DataEventType.childAdded) { (snapshot: DataSnapshot) in
if let data = snapshot.value as? NSDictionary {
let drive = cabRide(ritID: ritID, bestemming: bestemming,
vanafLocatie: vanaf, taxiID: taxiID, status: status)
print(drive)
callback.alertForARide(title: "Wilt u deze rit krijgen?", message: "Van: \(vanaf), Naar: \(bestemming)", ritID: ritID)
}
}
}
When I try this function with .childchanged, I only get a alert when it is changed like it suppose to do, but when doing .chiladded, it just gets all the requests out of the database and those requests were already there.
When I add a new request, it also gives an alert. So it works, but how can I get rid of the not added and already there requests?
Does anybody know this flaw?
This is working exactly as promised. From the documentation:
Retrieve lists of items or listen for additions to a list of items.
This event is triggered once for each existing child and then again
every time a new child is added to the specified path. The listener is
passed a snapshot containing the new child's data.
That might seem weird at first, but this is generally what most developers want, as it's basically a way of asking for all data from a particular branch in the database, even if new items get added to it in the future.
If you want it to work the way you're describing, where you're only getting new items in the database after your app has started up, you'll need to do a little bit of work yourself. First, you'll want to add timestamps to the objects you're adding to the database. Then you'll want to do some kind of call where you're asking to query your database by those timestamps. It'll probably look something like this:
myDatabaseRef.queryOrdered(byChild: "myTimestamp").queryStarting(atValue: <currentTimestamp>)
Good luck!