I have a very specific app and I don't know how should I save all my data in the Firebase database. Is it ok if I'll save all my data in one child and then simply get the first 0-1000 childs, then 1000-2000, 2000-3000 and so on. Will it affect performance?
The following example shows how to build and commit a batch of writes in Android:
// Get a new write batch
WriteBatch batch = db.batch();
// Set the value of 'NYC'
DocumentReference nycRef = db.collection("cities").document("NYC");
batch.set(nycRef, new City());
// Update the population of 'SF'
DocumentReference sfRef = db.collection("cities").document("SF");
batch.update(sfRef, "population", 1000000L);
// Delete the city 'LA'
DocumentReference laRef = db.collection("cities").document("LA");
batch.delete(laRef);
// Commit the batch
batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
#Override
public void onComplete(#NonNull Task<Void> task) {
// ...
}
});
Related
I have a stream of Map<ItemId, Item> as following:
final _itemsStreamSubject = BehaviorSubject<Map<String, Item>>();
I want to filter/map this stream to get a Stream<Item> that only sends updates when specified itemId's content changes:
Stream<Item> itemStreamOf(String itemId) {
return _itemsStreamSubject.stream.map(...).where(...).distinct();
}
How to do this? And, will the listeners of itemStreamOf() stream receive an update when _itemsStreamSubject adds an update?
I ran into an issue where I was intermittently receiving an error:
An entity object cannot be referenced by multiple instances of IEntityChangeTracker
whenever trying to attach an entity to the DbContext.
UPDATE: Original post is below and is TL;DR. So here is a simplified version with more testing.
First I get the Documents collection. There are 2 items returned in this query.
using (UnitOfWork uow = new UnitOfWork())
{
// uncomment below line resolves all errors
// uow.Context.Configuration.ProxyCreationEnabled = false;
// returns 2 documents in the collection
documents = uow.DocumentRepository.GetDocumentByBatchEagerly(skip, take).ToList();
}
Scenario 1:
using (UnitOfWork uow2 = new UnitOfWork())
{
// This errors ONLY if the original `uow` context is not disposed.
uow2.DocumentRepository.Update(documents[0]);
}
This scenario works as expected. I can force the IEntityChangeTracker error by NOT disposing the original uow context.
Scenario 2:
Iterate through the 2 items in the documents collection.
foreach (Document document in documents)
{
_ = Task.Run(() =>
{
using (UnitOfWork uow3 = new UnitOfWork())
{
uow3.DocumentRepository.Update(document);
});
}
}
Both items fail to attach to the DbSet with the IEntityChangeTracker error. Sometimes one succeeds and only one fails. I assume this might be to do with the exact timings of the Task Scheduler. But even if they are attaching concurrently, they are different document entities. So they shouldn't be being tracked by any other context. Why am I getting the error?
If I uncomment ProxyCreationEnabled = false on the original uow context, this scenario works! So how are they still being tracked even thought the context was disposed? Why is it a problem that they are DynamicProxies, even though they are not attached to or tracked by any context.
ORIGINAL POST:
I have an entity object called Document, and it's related entity which is a collection of DocumentVersions.
In the code below, the document object and all related entities including DocumentVersions have already been eagerly loaded before being passed to this method - which I will demonstrate after.
public async Task PutNewVersions(Document document)
{
// get versions
List<DocumentVersion> versions = document.DocumentVersions.ToList();
for (int i = 0; i < versions.Count; i++)
{
UnitOfWork uow = new UnitOfWork();
try
{
versions[i].Attempt++;
//... make some API call that succeeds
versions[i].ContentUploaded = true;
versions[i].Result = 1;
}
finally
{
uow.DocumentVersionRepository.Update(versions[i]); // error hit in this method
uow.Save();
}
}
}
The Update method just attaches the entity and changes the state. It is part of a GenericRepository class that all my Entity Repositories inherit from:
public virtual void Update(TEntity entityToUpdate)
{
dbSet.Attach(entityToUpdate); // error is hit here
context.Entry(entityToUpdate).State = EntityState.Modified;
}
The document entity, and all related entities are loaded eagerly using a method in the Document entity repository:
public class DocumentRepository : GenericRepository<Document>
{
public DocumentRepository(MyEntities context) : base(context)
{
this.context = context;
}
public IEnumerable<Document> GetDocumentByBatchEagerly(int skip, int take)
{
return (from document in context.Documents
.Include(...)
.Include(...)
.Include(...)
.Include(...)
.Include(d => d.DocumentVersions)
.AsNoTracking()
orderby document.DocumentKey descending
select document).Skip(skip).Take(take);
}
}
The method description for .AsNoTracking() says that "the entities returned will not be cached in the DbContext". Great!
Then why does the .Attach() method above think that this DocumentVersion entity is already referenced in another IEntityChangeTracker? I am assuming this means it is referenced in another DbContext, i.e: the one calling GetDocumentByBatchEagerly(). And why does this issue only present intermittently? It seems that it happens less often when I am stepping through the code.
I resolved this by adding the following line to the above DocumentRepository constructor:
this.context.Configuration.ProxyCreationEnabled = false;
I just don't understand why this appears to resolve the issue.
It also means if I ever want to use the DocumentRepository for something else and want to leverage change tracking and lazy loading, I can't. There doesn't seem to be a 'per query' option to turn off dynamic proxies like there is with 'as no tracking'.
For completeness, here is how the 'GetDocumentsByBatchEagerly' method is being used, to demonstrate that it uses it's own instance of UnitOfWork:
public class MigrationHandler
{
UnitOfWork uow = new UnitOfWork();
public async Task FeedPipelineAsync()
{
bool moreResults = true;
do
{
// documents retrieved with AsNoTracking()
List<Document> documents = uow.DocumentRepository.GetDocumentByBatchEagerly(skip, take).ToList();
if (documents.Count == 0) moreResults = false;
skip += take;
// push each record into TPL Dataflow pipeline
foreach (Document document in documents)
{
// Entry point for the data flow pipeline which links to
// a block that calls PutNewVersions()
await dataFlowPipeline.DocumentCreationTransformBlock.SendAsync(document);
}
} while (moreResults);
dataFlowPipeline.DocumentCreationTransformBlock.Complete();
// await completion of each block at the end of the pipeline
await Task.WhenAll(
dataFlowPipeline.FileDocumentsActionBlock.Completion,
dataFlowPipeline.PutVersionActionBlock.Completion);
}
}
Hi i have a sample stored procedure created like this
Bulk insert
function bulkImport(items) {
var container = getContext().getCollection();
var containerLink = container.getSelfLink();
var items=[{
"UserAccountID":"1236",
"FirstName": "Sanjeev",
},{
"UserAccountID":"1235",
"FirstName": "Sanjeev",
}];
// The count of imported items, also used as current item index.
var count = 0;
// Validate input.
if (!items) throw new Error("The array is undefined or null.");
var itemsLength = items.length;
if (itemsLength == 0) {
getContext().getResponse().setBody(0);
}
// Call the create API to create an item.
tryCreate(items[count], callback);
function tryCreate(item, callback) {
var isAccepted = container.createDocument(containerLink, item, callback);
if (!isAccepted) getContext().getResponse().setBody(count);
}
function callback(err, item, options) {
if (err) throw err;
// One more item has been inserted, increment the count.
count++;
if (count >= itemsLength) {
// If we created all items, we are done. Just set the response.
getContext().getResponse().setBody(count);
} else {
// Create next document.
tryCreate(items[count], callback);
}
}
}
How to execute this sp in the cosmos emulator. How to set the partition key uniquely for each of the item.
Since the partition key should be unique we cannot provide a single partition key while executing.
Firstly,you could create non-partitioned collection in the cosmos db if you do not need it. The portal has some limitations but the sdk allows it. You could refer to my previous case:Is it still a good idea to create comos db collection without partition key?.
Secondly, the bulk import of stored procedure is not suitable for the partitioned collection because you need to provide the pk for every execution of a stored procedure. Please refer to this link: Azure Cosmos DB asking for partition key for stored procedure
Finally,there are also many other solutions to import data into cosmos db,such as Azure Cosmos DB Data migration tool which is worth trying for you.
Whenever I use addListenerForSingleValueEvent with setPersistenceEnabled(true), I only manage to get a local offline copy of DataSnapshot and NOT the updated DataSnapshot from the server.
However, if I use addValueEventListener with setPersistenceEnabled(true), I can get the latest copy of DataSnapshot from the server.
Is this normal for addListenerForSingleValueEvent as it only searches DataSnapshot locally (offline) and removes its listener after successfully retrieving DataSnapshot ONCE (either offline or online)?
Update (2021): There is a new method call (get on Android and getData on iOS) that implement the behavior you'll like want: it first tries to get the latest value from the server, and only falls back to the cache when it can't reach the server. The recommendation to use persistent listeners still applies, but at least there's a cleaner option for getting data once even when you have local caching enabled.
How persistence works
The Firebase client keeps a copy of all data you're actively listening to in memory. Once the last listener disconnects, the data is flushed from memory.
If you enable disk persistence in a Firebase Android application with:
Firebase.getDefaultConfig().setPersistenceEnabled(true);
The Firebase client will keep a local copy (on disk) of all data that the app has recently listened to.
What happens when you attach a listener
Say you have the following ValueEventListener:
ValueEventListener listener = new ValueEventListener() {
#Override
public void onDataChange(DataSnapshot snapshot) {
System.out.println(snapshot.getValue());
}
#Override
public void onCancelled(FirebaseError firebaseError) {
// No-op
}
};
When you add a ValueEventListener to a location:
ref.addValueEventListener(listener);
// OR
ref.addListenerForSingleValueEvent(listener);
If the value of the location is in the local disk cache, the Firebase client will invoke onDataChange() immediately for that value from the local cache. If will then also initiate a check with the server, to ask for any updates to the value. It may subsequently invoke onDataChange() again if there has been a change of the data on the server since it was last added to the cache.
What happens when you use addListenerForSingleValueEvent
When you add a single value event listener to the same location:
ref.addListenerForSingleValueEvent(listener);
The Firebase client will (like in the previous situation) immediately invoke onDataChange() for the value from the local disk cache. It will not invoke the onDataChange() any more times, even if the value on the server turns out to be different. Do note that updated data still will be requested and returned on subsequent requests.
This was covered previously in How does Firebase sync work, with shared data?
Solution and workaround
The best solution is to use addValueEventListener(), instead of a single-value event listener. A regular value listener will get both the immediate local event and the potential update from the server.
A second solution is to use the new get method (introduced in early 2021), which doesn't have this problematic behavior. Note that this method always tries to first fetch the value from the server, so it will take longer to completely. If your value never changes, it might still be better to use addListenerForSingleValueEvent (but you probably wouldn't have ended up on this page in that case).
As a workaround you can also call keepSynced(true) on the locations where you use a single-value event listener. This ensures that the data is updated whenever it changes, which drastically improves the chance that your single-value event listener will see the current value.
So I have a working solution for this. All you have to do is use ValueEventListener and remove the listener after 0.5 seconds to make sure you've grabbed the updated data by then if needed. Realtime database has very good latency so this is safe. See safe code example below;
public class FirebaseController {
private DatabaseReference mRootRef;
private Handler mHandler = new Handler();
private FirebaseController() {
FirebaseDatabase.getInstance().setPersistenceEnabled(true);
mRootRef = FirebaseDatabase.getInstance().getReference();
}
public static FirebaseController getInstance() {
if (sInstance == null) {
sInstance = new FirebaseController();
}
return sInstance;
}
Then some method you'd have liked to use "addListenerForSingleEvent";
public void getTime(final OnTimeRetrievedListener listener) {
DatabaseReference ref = mRootRef.child("serverTime");
ref.addValueEventListener(new ValueEventListener() {
#Override
public void onDataChange(DataSnapshot dataSnapshot) {
if (listener != null) {
// This can be called twice if data changed on server - SO DEAL WITH IT!
listener.onTimeRetrieved(dataSnapshot.getValue(Long.class));
}
// This can be called twice if data changed on server - SO DEAL WITH IT!
removeListenerAfter2(ref, this);
}
#Override
public void onCancelled(DatabaseError databaseError) {
removeListenerAfter2(ref, this);
}
});
}
// ValueEventListener version workaround for addListenerForSingleEvent not working.
private void removeListenerAfter2(DatabaseReference ref, ValueEventListener listener) {
mHandler.postDelayed(new Runnable() {
#Override
public void run() {
HelperUtil.logE("removing listener", FirebaseController.class);
ref.removeEventListener(listener);
}
}, 500);
}
// ChildEventListener version workaround for addListenerForSingleEvent not working.
private void removeListenerAfter2(DatabaseReference ref, ChildEventListener listener) {
mHandler.postDelayed(new Runnable() {
#Override
public void run() {
HelperUtil.logE("removing listener", FirebaseController.class);
ref.removeEventListener(listener);
}
}, 500);
}
Even if they close the app before the handler is executed, it will be removed anyways.
Edit: this can be abstracted to keep track of added and removed listeners in a HashMap using reference path as key and datasnapshot as value. You can even wrap a fetchData method that has a boolean flag for "once" if this is true it would do this workaround to get data once, else it would continue as normal.
You're Welcome!
You can create transaction and abort it, then onComplete will be called when online (nline data) or offline (cached data)
I previously created function which worked only if database got connection lomng enough to do synch. I fixed issue by adding timeout. I will work on this and test if this works. Maybe in the future, when I get free time, I will create android lib and publish it, but by then it is the code in kotlin:
/**
* #param databaseReference reference to parent database node
* #param callback callback with mutable list which returns list of objects and boolean if data is from cache
* #param timeOutInMillis if not set it will wait all the time to get data online. If set - when timeout occurs it will send data from cache if exists
*/
fun readChildrenOnlineElseLocal(databaseReference: DatabaseReference, callback: ((mutableList: MutableList<#kotlin.UnsafeVariance T>, isDataFromCache: Boolean) -> Unit), timeOutInMillis: Long? = null) {
var countDownTimer: CountDownTimer? = null
val transactionHandlerAbort = object : Transaction.Handler { //for cache load
override fun onComplete(p0: DatabaseError?, p1: Boolean, data: DataSnapshot?) {
val listOfObjects = ArrayList<T>()
data?.let {
data.children.forEach {
val child = it.getValue(aClass)
child?.let {
listOfObjects.add(child)
}
}
}
callback.invoke(listOfObjects, true)
}
override fun doTransaction(p0: MutableData?): Transaction.Result {
return Transaction.abort()
}
}
val transactionHandlerSuccess = object : Transaction.Handler { //for online load
override fun onComplete(p0: DatabaseError?, p1: Boolean, data: DataSnapshot?) {
countDownTimer?.cancel()
val listOfObjects = ArrayList<T>()
data?.let {
data.children.forEach {
val child = it.getValue(aClass)
child?.let {
listOfObjects.add(child)
}
}
}
callback.invoke(listOfObjects, false)
}
override fun doTransaction(p0: MutableData?): Transaction.Result {
return Transaction.success(p0)
}
}
In the code if time out is set then I set up timer which will call transaction with abort. This transaction will be called even when offline and will provide online or cached data (in this function there is really high chance that this data is cached one).
Then I call transaction with success. OnComplete will be called ONLY if we got response from firebase database. We can now cancel timer (if not null) and send data to callback.
This implementation makes dev 99% sure that data is from cache or is online one.
If you want to make it faster for offline (to don't wait stupidly with timeout when obviously database is not connected) then check if database is connected before using function above:
DatabaseReference connectedRef = FirebaseDatabase.getInstance().getReference(".info/connected");
connectedRef.addValueEventListener(new ValueEventListener() {
#Override
public void onDataChange(DataSnapshot snapshot) {
boolean connected = snapshot.getValue(Boolean.class);
if (connected) {
System.out.println("connected");
} else {
System.out.println("not connected");
}
}
#Override
public void onCancelled(DatabaseError error) {
System.err.println("Listener was cancelled");
}
});
When workinkg with persistence enabled, I counted the times the listener received a call to onDataChange() and stoped to listen at 2 times. Worked for me, maybe helps:
private int timesRead;
private ValueEventListener listener;
private DatabaseReference ref;
private void readFB() {
timesRead = 0;
if (ref == null) {
ref = mFBDatabase.child("URL");
}
if (listener == null) {
listener = new ValueEventListener() {
#Override
public void onDataChange(DataSnapshot dataSnapshot) {
//process dataSnapshot
timesRead++;
if (timesRead == 2) {
ref.removeEventListener(listener);
}
}
#Override
public void onCancelled(DatabaseError databaseError) {
}
};
}
ref.removeEventListener(listener);
ref.addValueEventListener(listener);
}
As mentioned in the title, how many methods are available?
I just have this case: I get a entity object from one ObjectContext, and then I detach the entity obejct from OjbectContext object, and return it.
Later, if I make some changes on this object, and I want to save the changes back to database. I think I should write like this, right? (Well, this works for me.)
public Url GetOneUrl()
{
Url u;
using(ServicesEntities ctx = new ServicesEntities())
{
u = (from t in ctx.Urls select t).FirstOrDefault<Url>();
ctx.Detach(u);
}
return u;
}
public void SaveToDB(Url url)
{
using(ServicesEntities ctx = new ServicesEntities())
{
var t = ctx.GetObjectByKey(_Url.EntityKey) as Url;
ctx.Detach(t);
ctx.Attach(url);
ctx.ObjectStateManager.ChangeObjectState(url, System.Data.EntityState.Modified);
ctx.SaveChanges();
}
}
Url url = GetOneUrl();
url.UrsString = "http://google.com"; //I just change the content.
SaveToDB(url);
OR
public void SaveToDB(Url url)
{
using(ServicesEntities ctx = new ServicesEntities())
{
var t = ctx.GetObjectByKey(_Url.EntityKey) as Url;
t = url; //this will make t.UrlString becomes "http://google.com"
ctx.ApplyCurrentValues<Url>("Urls", t);
ctx.SaveChanges();
}
}
This way is also works for me.
The first way will generate sql statement to update all the columns of Url table, but the second method will provide a sql script only update the "UrlString" Columns.
Both of them will have to retrieve a temp entity object from database which is the 't' in above code.
Are there any other methods to achieve this purpose? Or other better method you know about it? Or any official solution about this topic?
Many Thanks.
I don't understand your first example. Why do you first get entity from ObjectContext? It is not needed because you have just created new instance of the context. You can just use:
public void SaveToDB(Url url)
{
using(ServicesEntities ctx = new ServicesEntities())
{
ctx.Attach(url);
ctx.ObjectStateManager.ChangeObjectState(url, System.Data.EntityState.Modified);
ctx.SaveChanges();
}
}
In your second example you can just call:
public void SaveToDB(Url url)
{
using(ServicesEntities ctx = new ServicesEntities())
{
var t = ctx.GetObjectByKey(_Url.EntityKey) as Url; // Ensures that old values are loaded
ctx.ApplyCurrentValues<Url>("Urls", url);
ctx.SaveChanges();
}
}
Now the difference between two approaches is clear. First approach (Attach) does not need to query the DB first. Second approach (ApplyCurrentValues) needs to query the DB first to get old values.
You can use two additional approaches. First is just extension of your former approach. It allows you defining which properties were changed. Second approach is manual synchronization with loaded entity. This approach doesn't use any special methods. You will simply set loaded entity's properties to required values manually. This approach is useful if you work with object graph instead of single entity because EF is not able to automatically synchronize changes in relations.