Using the Realtime Database it says here that if you want to scale beyond 200,000 simultaneous connections that you can create/shard another database. It also says:
Each query only runs against a single database instance. Realtime
Database doesn't support queries across database instances.
No sharing or duplication of data across database instances (or
minimal sharing or duplication).
Each app instance only connects to one database at any given moment.
Let's say in first database I have a Posts ref, a Users ref, and a Search Posts ref with 100K user objects, 200K post objects, and 200K search objects. I now decide to create/shard another database with the same exact refs.
When the next x amount of users signs up, if their User, Post, and Search Posts refs are in the new shard database, does that mean that they won't be able to access users or search those user's posts from the first database? Also vice versa the users from the first database won't be able to have access to users or their posts in the second database?
The point of sharding is to load balance your connections and is not related to quantity of data.
The RTDB makes no decisions about where data is stored, you do. One server would contain users, another would contain posts.
You would run user queries against the server with users and posts queries against the serve with posts.
All you're doing is pointing your app toward the server you want to query before running the query.
In other words, there would be no reason to add users to server 1 and then add more users to server 2 as the quantity of data doesn't matter.
Related
As of right now I am using Firebase Realtime Database to include chat functionality as part of an app I'm working on. The only issue I've seemingly run into is figuring out how to include a user's data (profile, username, birthday, etc.) so that if a user clicks on a chat, they can then seamlessly go to a user's profile page without needing to fetch more data from the backend. Here's the current structure I'm using in Firebase Realtime Database for this:
$chats
$chatId
id
users
0: some user id
1: some user id
lastMessage
$userChats
$userId
$chatId: true
$users
$userId
user info here
In my case what I would like to know is if it makes more sense to duplicate all the user data for each user into each chat within the users array or if I should just use the referencing userId and pull that data after in a separate request?
Considering I store my users primarily in a separate PostgreSQL database I wonder if I could do a separate query to that database and not even worry about storing the users in the realtime database as well (considering I have to include aggregate info for each user like counters).
If you are always going to be fetching user data alongside a chat, then you should store them together. No need to make more than one call unnecessarily.
However, if you will ever fetch user data and/or conversations separately, I would recommend storing the user data separately and not within the conversation.
Also, if you really want an "immediate" feel (beyond the already "realtime" database performance), you could also fetch the user data in the background as soon as a particular chat is opened. That way, if the user taps to view a profile, it'll have already been fetched and will give that "instantaneous" experience you are looking for.
Plus, you have to remember that Realtime Database charges on the amount of data being transferred, no matter how many calls it takes (as opposed to Firestore which charges on the number of queries), so storing it separately does not increase billing at all compared to one query, and actually saves money in the cases where you don't need both data sets.
I am writing an application that will help users connect to each other based on a number of attributes (i.e. location, interests, etc). I am using firebase firestore to store all of my user data, which has fields like name, hometown, interests, classes, and connections, which is a list of UUID of the users they are connected to.
When I suggest other users for a user to connect with, I want to do so by the number of their mutual connections, however this is not explicitly a field in the database, since I would have to calculate and maintain that for every user in relation to every other user. Is there a way I can use firestore to query or order by a calculated property rather than a field in the database, or another way to store the mutual connections?
I know in theory I could pull all the users from the database and then locally calculate the number of mutual connections and sort the array, but that sounds horribly inefficient.
I have searched for existing solutions such as how facebook or linkedin would suggest friends or people you might know, but I can't find any concrete answers on how they did this, but those companies are not using firestore either. I am hoping someone has had this problem in the past and can help me out.
There is no way to perform a calculation on the data in a Firestore query. If you want to order or filter in a calculated value, you'll have to store that value in the database.
Typically you can either do this:
As you write the data from the client, you also update the calculated values.
In a Cloud Function that triggers when you write the data, and that then updates the calculated values.
In a Cloud Function that runs periodically, and that then updates the calculated values for all modified data.
I have a small social website on rails (planing to port to phoenix) that use react on view and backend is just a JSON API,
with more o less 3000 users online at any moment. It runs with postgres/memcached
When user, for example, visits its feed page, I do:
Select activities from database (20 per page)
Select last 4 comments from each activity from database (justo 1 select)
Select all users referenced by activity or comment from database (select users.* from users where id in (1,3,4,5,...100) )
I have a cache layer (memcached) that when I will load users, first I try load from memcached, if it not exists I read from database and put it on cache.
BUT I also have some "listenners" on users model (and over others referenced models like address and profile) to invalidate cache if any field change.
The problem:
This cache demand a lot of code.
Sometimes cache run out of sync.
I hate to have this listeners and they are "side effects"
My question is: Any one is doing something like that??
I search A LOT over all google about cache layer to json api and looks like that everyone is just using database directly.
I know that Rails has it own soluction (and I gess that phoenix dont has one), but it always end up using update_at column, that means, I have to go to database anyway.
alternative:
Live with date, life is not pretty
Buy a more powerful postgres instance... any one is using memcached like that.
Remover listeners, put some expires_in (1 or 2 minutos... or more) and let app
show out of sync data for a couple of minutes.
thanks for any help!
I have a situation where a user has acess to multiple systems(10,000 for example). Each system is associated with a system_id.
We currently use a web service to determine what the list of systems the user has access to.
But, the issue I am facing is to retireve the list, I have to call this web service on each page of the app, which is likely to cause perfomance issue as more & more systems get added to the user.
One thought was, the first time when the user logs in, we get the list of system ids that user has access to and store in as an array of ids in a sessions or internal users table.
And going forward use that table on subsequent pages
Would that be a bad idea or can i get any other alternatives?
In a rails application how do you efficiently display the total number of users and current number of users online?
Sam,
There are many methods for tracking online users. For example, authlogic has a last_request_at column which tracks when they last made a request to the site. Though, it's not very efficient to run a query for that every page load. I personally use Redis for tracking that sort of activity.
Here is a great example: Redis in practice, who is online
Hopefully this helps.
The common way to get the list of your online users is to store sessions in the DB (use ActiveRecord SessionStore), then retrieve recently updated sessions from the DB, deserialize them and see which users they belong to.