When my application loads, I am exporting few breeze entities and storing those in local cache so that later when I have to make use of those entities again, I import them back from local storage and execute the query locally.
There is one entity that has 84389 rows. I have noticed that the importEntites take longer to execute for this entity. Is there a way I can speed up this ?
var entities = manager.getEntities('Client');
var exportedEntity = manager.exportEntities(entities, { includeMetadata: false });
I am storing exportedEntity in cache.
I am importing the above exportedEntity into entitymanager after fetching from cache.
entityManager.importEntities(exportedEntity);
The above statement takes longer for the Client entity. The client has 80K rows in table. I am not sure if that reduces the execution speed for importEntities method.
I am also executing query locally after the entity is imported.
I think you mean that you have an EntityType, Client, with 84k rows/entities.
That's a lot of objects. I would reconsider whether that data should be treated as rich Breeze entities or if they should be held in a simpler, more compact form as data objects. I'm leaning strongly to the latter, especially if they are read-only. There is less benefit to representing read-only data as entities and, at these volumes, bad performance and memory usage are more likely.
Remember that not all data have to be entities and Breeze is happy to fetch a mix of entity and raw data. Hybrid apps are quite common.
If you still want to explore treating them as entities, you might try the following export command which outputs all entities of a given type as JSON:
var exported = manager.exportEntities('Client', {asString:false, includeMetadata:false});
The result can be imported as before.
var imported = manager.importEntities(exported);
This may improve export and import speed (or not).
You don't have to query locally afterward. All the imported entities are in imported.entities.
Related
I have a process whereby I have an NHibernate session which I use to run a query against the database. I then iterate through the collection of results, and for each iteration, using the same NHibernate session, I call a SQL Stored Procedure (using CreateSQLQuery() & ExecuteUpdate()), which ends up performing an update on a field for that entity.
When it has finished iterating over the list (and calling the SP x times), if I check the database directly in SSMS, I can see that the UPDATE for each row has been applied.
However, in my code, if I then immediately run the same initial query again, to retrieve that list of entities, it does not reflect the updates that the SP made for each row - the value is still NULL.
I haven't got any cache behavior specified against the configuration of NHibernate in my application, and have experimented with different SetCacheMode() when calling the query, but nothing seems to make any difference - the values that I can see directly in the DB have been updated, are not being brought back as updated when I re-query (using Session.QueryOver()) the database (using that same session).
By calling CreateSQLQuery (to update database, single row or multiple rows does not matter), actually you are doing DML-style operation which does not update the in-memory state.
Any call to CreateSQLQuery or CreateQuery will not use/reflect tracking. These are considered out-of-the-scope of Unit Of Work.
These operations directly affect the underlying database neglecting any in-memory state.
14.3. DML-style operations
As already discussed, automatic and transparent object/relational mapping is concerned with the management of object state. This implies that the object state is available in memory, hence manipulating (using the SQL Data Manipulation Language (DML) statements: INSERT, UPDATE, DELETE) data directly in the database will not affect in-memory state. However, NHibernate provides methods for bulk SQL-style DML statement execution which are performed through the Hibernate Query Language (HQL). A Linq implementation is available too.
They (may) work on bulk data. They are necessary in some scenarios for performance reasons. With these, tracking does not work; so yes, in-memory state become invalid. You have to use them carefully.
if I then immediately run the same initial query again, to retrieve that list of entities, it does not reflect the updates that the SP made for each row - the value is still NULL.
This is due to first (session) level cache. This is always enabled by default and cannot be disabled with ISession.
When you first load the objects, its a database hit. You get the objects from database - loop through them - execute commands those are out of Unit Of Work (as explained above) - and again execute same query twice to load same objects under same ISession instance. Second call does not hit the database at all.
It just return the instances from memory. As your in-memory instances are not updated at all, you always get original instances.
To get the updated instances, close the first session and reload the instances with new session.
For more details, please refer to: How does Hibernate Query Cache work
I have a static table for settings where I want to pull some stuff from an entity in Core Data. The use case does not lend itself to a table of records as you usually see. Rather each row of the static table is really a field related to the user--as in a user profile. I have a feeling that in testing I may have created more than one record in the entity. I know there are programs that let you see the SQL lite database underneath, but my question assumes you do not have this tool and are relying just on Xcode.
My question is when you have more than one record in a Core Data entity/table, and you try to load data from the managed object context into a VC, one field into one element, what record is shown by default?
Related to this, if you don't know how many managed object or rows are in the database, is there anyway to specify which record you want since there are no auto ids as you would use in a traditional database?
The record that gets loaded from the fetch first. Depending on your sort that might be consistent or it might be random.
I'm looking for a way to store large objects within Drools for long periods of time (i.e. not like facts which are added and removed from within a session).
I've read that Drools works using KnowledgeBases and Sessions (Stateless & Stateful) and that KnowledgeBases contain application knowledge definitions but no runtime data.
In a case where I need to store, for example, a large dictionary (that won't change but will be referenced by more than one successive session), and have objects added to working memory and checked against this dictionary to have rules fired, where would it be best to have this stored?
Does everything just go into working memory (in which case, would I need to load the dictionary into memory each time I open a new session?) or am I just missing a crucial Drools basic principle? Would global variables be a good fix for this?
Not sure how large "large" is (of course there's always a performance tradeoff), but you could also use an inserted object to pull from a database (/cache) and have the rules access the values via method.
when
$y : AnObject (name == "car", lookupPrice > 10000 );
where AnObject.getLookupPrice() is a method that would pull a value out of the cached / stored dictionary.
If the object isn't too big you could codify as well (as an object) and use it the same way.
If I use breeze to load a partial entity:
var query = EntityQuery.from('material')
.select('Id, MaterialName, MaterialType, MaterialSubType')
.orderBy(orderBy.material);
return manager.executeQuery(query)
.then(querySucceeded)
.fail(queryFailed);
function querySucceeded(data) {
var list = partialMapper.mapDtosToEntities(
manager, data.results, entityNames.material, 'id');
if (materialsObservable) {
materialsObservable(list);
}
log('Retrieved Materials from remote data source',
data, true);
}
...and I also want to have another slightly different partial query from the same entity (maybe a few other fields for example) then I'm assuming that I need to do another separate query as those fields weren't retrieved in the first query?
OK, so what if I want to use the same fields retrieved in the first query (Id, Materialname, MaterialType, MaterialSubType) but I want to call those fields different names in the second query (Materialname becomes just "name", MaterialType becomes "masterType" and so on) then is it possible to clone the partial entity I already have in memory (assuming it is in memory?) and rename the fields or do I still need to do a completely separate query?
I think I would "union" the two cases into one projection if I could afford to do so. That would simplify things dramatically. But it's really important to understand the following point:
You do not need to turn query projection results into entities!
Backgound: the CCJS example
You probably learned about the projection-into-entities technique from the CCJS example in John Papa's superb PluralSight course "Single Page Apps JumpStart". CCJS uses this technique for a very specific reason: to simplify list update without making a trip to the server.
Consider the CCJS "Sessions List" which is populated by a projection query. John didn't have to turn the query results into entities. He could have bound directly to the projected results. Remember that Knockout happily binds to raw data values. The user never edits the sessions on that list directly. If displayed session values can't change, turning them into observable properties is a waste of CPU.
When you tap on a Session, you go to a Session view/edit screen with access to almost every property of the complete session entity. CCJS needs the full entity there so it looks for the full (not partial) session in cache and, if not found, loads the entity from the server. Even to this point there is no particular value in having previously converted the original projection results into (partial) session entities.
Now edit the Session - change the title - and save it. Return to the "Sessions List"
Question
How do you make sure that the updated title appears in the Sessions List?
If we bound the Sessions List HTML to the projection data objects, those objects are not entities. They're just objects. The entity you edited in the session view is not an object in the collection displayed in the Sessions List. Yes, there is a corresponding object in the list - one that has the same session id. But it is not the same object.
Choices
#1: Refresh the list from the server by reissuing the projection query. Bind directly to the projection data. Note that the data consist of raw JavaScript objects, not entities; they are not in the Breeze cache.
#2: Publish an event after saving the real session entity; the subscribing "Sessions List" ViewModel hears the event, extracts the changes, and updates its copy of the session in the list.
#3: Use the projection-into-entity technique so that you can use a session entity everywhere.
Pros and Cons
#1 is easy to implement. But it requires a server trip every time you enter the Sessions List view.
One of the CCJS design goals was that, once loaded, it should be able to operate entirely offline with zero access to the server. It should work crisply when connectivity is intermittent and poor.
CCJS is your always-ready guide to the conference. It tells you instantly what sessions are available, when and where so you can find the session you want, as you're walking the halls, and get there. If you've been to a tech conference or hotel you know that the wifi is generally awful and the app is almost useless if it only works when it has direct access to the server.
#1 is not well suited to the intended operating environment for CCJS.
The CCJS Jumpstart is part way down that "server independent" path; you'll see something much closer to a full offline implementation soon.
You'll also lose the ability to navigate to related entities. The Sessions List displays each session's track, timeslot and room. That's repetitive information found in the "lookup" reference entities. You'll either have to expand the projection to include this information in a "flattened" view of the session (fatter payload) or get clever on the client-side and patch in the track, timeslot and room data by hand (complexity).
#2 helps with offline/intermittent connectivity scenarios. Of course you'll have to set up the messaging system, establish a protocol about saved entities and teach the Sessions List to find and update the affected session projection object. That's not super difficult - the Breeze EntityManager publishes an event that may be sufficient - but it would take even more code.
#3 is good for "server independence", has a small projection payload, is super-easy, and is a cool demonstration of breeze. You have to manage the isPartial flag so you always know whether the session in cache is complete. That's not hard.
It could get more complicated if you needed multiple flavors of "partial entity" ... which seems to be where you are going. That was not an issue in CCJS.
John chose #3 for CCJS because it fit the application objectives.
That doesn't make it the right choice for every application. It may not be the right choice for you.
For example, if you always have a fast, low latency connection, then #1 may be your best choice. I really don't know.
I like the cast-to-entity approach myself because it is easy and works so well most of the time. I do think carefully about that choice before I make it.
Summary
You do not have to turn projection query results into entities
You can bind to projected data directly, without Knockout observable properties, if they are read-only
Make sure you have a good reason to convert projected data into (partial) entities.
CCJS has a good reason to convert projected query data into entities. Do you?
I have an hierarchical structure with millions of records.
I'm doing a recursive scan on the DB in order to update some of the connections and some of the data.
the problem is that I get an outofmemory exception since the entire DB is eventually loaded to the context (lazy). data that I no longer need stays in the context without any way of removing it.
I also can't use Using(context...) since I need the context alive because I'm doing a recursive scan.
Please take the recursion as a fact.
Thanks
This sort of an operation is really not handled well nor does it scale well using entities. I tend to resort to stored procedures for batch ops.
If you do want to remove/dump objects from context, I believe this post has some info (solution to your problem at the bottom).
just ran into the same problem. I've used NHibernate before I used EF as ORM tool and it had exactly the same problem. These frameworks just keep the objects in memory as long as the context is alive, which has two consequences:
serious performance slowdown: the framework does comparisons between the objects in memory (e.g. to see if an object exists or not). You will notice a gradual degradation of performance when processing many records
you will eventually run out of memory.
If possible I always try to do large batch operation on the database using pure SQL (as the post above states clearly), but in this case that wasn't an option. So to solve this, what NHibernate has is a 'Clear' method on the session, which throws away all object in memory that refer to database records (new ones, added ones, corrupt ones...)
I tried to mimic this method in entity framework as follows (using the post described above):
public partial class MyEntities
{
public IEnumerable<ObjectStateEntry> GetAllObjectStateEntries()
{
return ObjectStateManager.GetObjectStateEntries(EntityState.Added |
EntityState.Deleted |
EntityState.Modified |
EntityState.Unchanged);
}
public void ClearEntities()
{
foreach (var objectStateEntry in GetAllObjectStateEntries())
{
Detach(objectStateEntry.Entity);
}
}
}
The GetAllObjectStateEntries() method is taken separately because it's useful for other things. This goes into a partial class with the same name as your Entities class (the one EF generates, MyEntities in this example), so it is available on your entities instance.
I call this clear method now every 1000 records I process and my application that used to run for about 70 minutes (only about 400k entities to process, not even millions) does it in 25mins now. Memory used to peak to 300MB, now it stays around 50MB