I have two tables (there are more in the database but only two are involved here).
Account and AccountStatus, an account can have an AccountStatus (active,inactive etc).
I create a new Account and set a couple of properties but when I reach this code:
1. var status = db.AccountStatuses.SingleOrDefault(s => s.ID == (long)AccountStatusEnum.Active);
2. account.AccountStatus = status;
3. db.Accounts.AddObject(account);
The first line executes fine, but when I reach the second line it takes a REALLY long time, and when I step in to the code it seems that every single account is loaded from the database.
I don't see why it should even want to load all the accounts?
We use Entity Framework 4 and Poco and we have lazy loading enabled.
Any suggestions?
Cheers
/Jimmy
You have to be careful which constructs you use to fetch data, as some will pull in the whole set and filter afterword. (aside: the long time delay may be the database being created and seeded, if there isn't one already, it will occur the first time you touch it, likely with a query of some sort. Also remember that when you retrieve a whole dataset, you may in actuality only have what amounts to a compiled query that won't be evaluated until you interact with it).
Try this form instead and see if you have the same issue:
var status = db.AccountStatuses.Where(s => s.ID == (long)AccountStatusEnum.Active);
Related
I use code first in a web application where I have a form to upload text files and import the data into my database.
Each file may have up to 20.000+ records for import.
To speed things up I preload some entities so not to ask the DbContext every time. Then when I create an object for insert, I do for example:
myNewObject.Category = preloadedCategories.First(p => p.Code == code);
I have read some articles on the web because EF is extremey slow on batch inserts, so what I do is:
first use Configuration.AutoDetectChangesEnabled = false;
then every 1000 records I dispose the object and make a new one.
BUT! since the preloaded entities where loaded from a db context that was disposed, after making a new DbContext, I have a problem with preloadedCategories.First(p => p.Code == code). When I make a SaveChanges(), EF tries to also save the preloadedCategories.First(p => p.Code == code) object and fails.
So how can I achive this? I don't want to aks the DbContext every time to load some (non changing) objects. Is it possible?
thanks
When dealing with a large number of records in EF, a few things will help
As #janhartmann states, use .AsNoTracking()
As you stated, use Configuration.AutoDetectChangesEnabled = false, which will require the next point
Use context.Categories.Entry(category).State = EntityState.Modified to attach a disconnected entity to a context and mark is as modified
Also make check that preloadedCategories is no longer an IQuerable and that the data really is local and not trying to lazy load from the database.
If there are no changes to your Category object and you just want to link your myNewObject to an existing category, you have two options
Set the foreign key on myNewObject instead of the navigation property
Use context.Products.Entry(myNewObject).State = EntitySate.Added instead of context.Products.Add(myNewObject) to avoid it adding the entire graph of navigation properties
Good luck
We have a breeze client solution in which we show parent entities with lists of their children. We do hard deletes on some child entities. Now when the user is the one doing the deletes, there is no problem, but when someone else does, there seems to be no way to invalidate the children already loaded in cache. We do a new query with the parent and expanding to children, but breeze attaches all the other children it has already heard of, even if the database did not return them.
My question: shouldn't breeze realize we are loading through expand and thus completely remove all children from cache before loading back the results from the db? How else can we accomplish this if that is not the case?
Thank you
Yup, that's a really good point.
Deletion is simply a horrible complication to every data management effort. This is true no matter whether you use Breeze or not. It just causes heartache up and down the line. Which is why I recommend soft deletes instead of hard deletes.
But you don't care what I think ... so I will continue.
Let me be straight about this. There is no easy way for you to implement a cache cleanup scheme properly. I'm going to describe how we might do it (with some details neglected I'm sure) and you'll see why it is difficult and, in perverse cases, fruitless.
Of course the most efficient, brute force approach is to blow away the cache before querying. You might as well not have caching if you do that but I thought I'd mention it.
The "Detached" entity problem
Before I continue, remember the technique I just mentioned and indeed all possible solutions are useless if your UI (or anything else) is holding references to the entities that you want to remove.
Oh, you'll remove them from cache alright. But whatever is holding references to them now will continue to have a reference to an entity object which is in a "Detached" state - a ghost. Making sure that doesn't happen is your responsibility; Breeze can't know and couldn't do anything about it if it did know.
Second attempt
A second, less blunt approach (suggested by Jay) is to
apply the query to the cache first
iterate over the results and for each one
detach every child entity along the "expand" paths.
detach that top level entity
Now when the query succeeds, you have a clear road for it to fill the cache.
Here is a simple example of the code as it relates to a query of TodoLists and their TodoItems:
var query = breeze.EntityQuery.from('TodoLists').expand('TodoItems');
var inCache = manager.executeQueryLocally(query);
inCache.slice().forEach(function(e) {
inCache = inCache.concat(e.TodoItems);
});
inCache.slice().forEach(function(e) {
manager.detachEntity(e);
});
There are at least four problems with this approach:
Every queried entity is a ghost. If your UI is displaying any of the queried entities, it will be displaying ghosts. This is true even when the entity was not touched on the server at all (99% of the time). Too bad. You have to repaint the entire page.
You may be able to do that. But in many respects this technique is almost as impractical as the first. It means that ever view is in a potentially invalid state after any query takes place anywhere.
Detaching an entity has side-effects. All other entities that depend on the one you detached are instantly (a) changed and (b) orphaned. There is no easy recovery from this, as explained in the "orphans" section below.
This technique wipes out all pending changes among the entities that you are querying. We'll see how to deal with that shortly.
If the query fails for some reason (lost connection?), you've got nothing to show. Unless you remember what you removed ... in which case you could put those entities back in cache if the query fails.
Why mention a technique that may have limited practical value? Because it is a step along the way to approach #3 that could work
Attempt #3 - this might actually work
The approach I'm about to describe is often referred to as "Mark and Sweep".
Run the query locally and calculate theinCache list of entities as just described. This time, do not remove those entities from cache. We WILL remove the entities that remain in this list after the query succeeds ... but not just yet.
If the query's MergeOption is "PreserveChanges" (which it is by default), remove every entity from the inCache list (not from the manager's cache!) that has pending changes. We do this because such entities must stay in cache no matter what the state of the entity on the server. That's what "PreserveChanges" means.
We could have done this in our second approach to avoid removing entities with unsaved changes.
Subscribe to the EntityManager.entityChanged event. In your handler, remove the "entity that changed" from the inCache list because the fact that this entity was returned by the query and merged into the cache tells you it still exists on the server. Here is some code for that:
var handlerId = manager.entityChanged.subscribe(trackQueryResults);
function trackQueryResults(changeArgs) {
var action = changeArgs.entityAction;
if (action === breeze.EntityAction.AttachOnQuery ||
action === breeze.EntityAction.MergeOnQuery) {
var ix = inCache.indexOf(changeArgs.entity);
if (ix > -1) {
inCache.splice(ix, 1);
}
}
}
If the query fails, forget all of this
If the query succeeds
unsubscribe: manager.entityChanged.unsubscribe(handlerId);
subscribe with orphan detection handler
var handlerId = manager.entityChanged.subscribe(orphanDetector);
function orphanDetector(changeArgs) {
var action = changeArgs.entityAction;
if (action === breeze.EntityAction.PropertyChange) {
var orphan = changeArgs.entity;
// do something about this orphan
}
}
detach every entity that remains in the inCache list.
inCache.slice().forEach(function(e) {
manager.detachEntity(e);
});
unsubscribe the orphan detection handler
Orphan Detector?
Detaching an entity can have side-effects. Suppose we have Products and every product has a Color. Some other user hates "red". She deletes some of the red products and changes the rest to "blue". Then she deletes the "red" Color.
You know nothing about this and innocently re-query the Colors. The "red" color is gone and your cleanup process detaches it from cache. Instantly every Product in cache is modified. Breeze doesn't know what the new Color should be so it sets the FK, Product.colorId, to zero for every formerly "red" product.
There is no Color with id=0 so all of these products are in an invalid state (violating referential integrity constraint). They have no Color parent. They are orphans.
Two questions: how do you know this happened to you and what do your do?
Detection
Breeze updates the affected products when you detach the "red" color.
You could listen for a PropertyChanged event raised during the detach process. That's what I did in my code sample. In theory (and I think "in fact"), the only thing that could trigger the PropertyChanged event during the detach process is the "orphan" side-effect.
What do you do?
leave the orphan in an invalid, modified state?
revert to the equally invalid former colorId for the deleted "red" color?
refresh the orphan to get its new color state (or discover that it was deleted)?
There is no good answer. You have your pick of evils with the first two options. I'd probably go with the second as it seems least disruptive. This would leave the products in "Unchanged" state, pointing to a non-existent Color.
It's not much worse then when you query for the latest products and one of them refers to a new Color ("banana") that you don't have in cache.
The "refresh" option seems technically the best. It is unwieldy. It could easily cascade into a long chain of asynchronous queries that could take a long time to finish.
The perfect solution escapes our grasp.
What about the ghosts?
Oh right ... your UI could still be displaying the (fewer) entities that you detached because you believe they were deleted on the server. You've got to remove these "ghosts" from the UI.
I'm sure you can figure out how to remove them. But you have to learn what they are first.
You could iterate over every entity that you are displaying and see if it is in a "Detached" state. YUCK!
Better I think if the cleanup mechanism published a (custom?) event with the list of entities you detached during cleanup ... and that list is inCache. Your subscriber(s) then know which entities have to be removed from the display ... and can respond appropriately.
Whew! I'm sure I've forgotten something. But now you understand the dimensions of the problem.
What about server notification?
That has real possibilities. If you can arrange for the server to notify the client when any entity has been deleted, that information can be shared across your UI and you can take steps to remove the deadwood.
It's a valid point but for now we don't ever remove entities from the local cache as a result of a query. But.. this is a reasonable request, so please add this to the breeze User Voice. https://breezejs.uservoice.com/forums/173093-breeze-feature-suggestions
In the meantime, you can always create a method that removes the related entities from the cache before the query executes and have the query (with expand) add them back.
I am using Mygeneration tools to create the abstract classes responsible for dealing with database to perform CRUD operation as well as some other dooDad operations. Problem is I cant retrieve the auto number field (it is also Primary Key) of table using the code
Employees newObj = new Employees();
newObj.ConnectionString = connectionString;
newObj.AddNew();
// Your Properties will be different here
newObj.FirstName = "Joe";
newObj.LastName = "Plank Plank";
newObj.Save();
int staffid=newObj.StaffID;
The same thing is working fine in MS SQL server or other databases. Looks like auto number is not generated instantly which can be accessed once I added the entry. But, later, when I am checking the database, I found that auto number is generated there. Not sure, why this is happening. Anybody having expertise with dooDads, please help with info.
Edited:
The main problem is I cant access the autonumber field instantly after I create the fresh row entry. Looks like MS Access autonumber takes some time to show up and even in VS, you can see this phenomenon. How to fix this problem?
I have built many applications using Doodads , using MS Access , you have only to make the filed as autonumber .. and generate the stored procedures and other classes.
i.e your code should work ..
also I made modification to Dodads to return list of Objects
How to get list of objects from BusinessEntity using myGeneration?
I'm learning about concurrency in conjunction with EF4.0 and have a question about the locking pattern used.
Say I configure a fixed concurrency mode on a version number property.
Now say I fetch a record (entity) from the database (context) and edit some property. Version gets incremented and when SaveChanges is called on its context. If the current database (context) version matches the version of the original record (entity) the save continues, otherwise an OptimisticConcurrencyException gets thrown by EF.
Now, my point of interest is the following: between the check of the versions there's always a small period of time, however small, it is there. So in theory someone else could've just updated the record between the comparison and the actual save, thus possibly corrupting the data.
How does this get solved? It feels as if the problem just gets pushed forward.
There is no period of time between checking versions and updating record because the database command looks like:
UPDATE SomeTable
SET SomeColumn = 'SomeValue'
WHERE Id = #Id AND Version = #OldVersion
SELECT ##ROWCOUNT
The check and update is one atomic operation. Rowcount will return 0 if no record with Id = #Id and Version = #OldVersion exists and that zero is translated to the exception.
This can (and probably is) solved using locking hints.
For SQL Server, EF can query (SELECT) from the database WITH UPDLOCK.
This tells the Database Engine that, you want to read a/several records, and nobody else can change those records until you perform an update thereafter.
If you want to see this for yourself, check out the Sql Server Profiler which will show you the queries in real-time.
Hope that helps.
CAVEAT: I can't say for sure that this is the way EF handles this scenario because I haven't checked myself but, certainly if you were going to do it yourself, this is one way to do it.
i am devloping a site using .net MVC
i have a data access layer which basically consists of static list objects that are created from data within my database.
The method that rebuilds this data first clears all the list objects. Once they are empty it then add the data. Here is an example of one of the lists im using. its a method which generates all the UK postcodes. there are about 50 methods similar to this in my application that return all sorts of information, such as towns, regions, members, emails etc.
public static List<PostCode> AllPostCodes = new List<PostCode>();
when the rebuild method is called it first clears the list.
ListPostCodes.AllPostCodes.Clear();
next it re-bulilds the data, by calling the GetAllPostCodes() method
/// <summary>
/// static method that returns all the UK postcodes
/// </summary>
public static void GetAllPostCodes()
{
using (fab_dataContextDataContext db = new fab_dataContextDataContext())
{
IQueryable AllPostcodeData = from data in db.PostCodeTables select data;
IDbCommand cmd = db.GetCommand(AllPostcodeData);
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.SelectCommand = (SqlCommand)cmd;
DataSet dataSet = new DataSet();
cmd.Connection.Open();
adapter.FillSchema(dataSet, SchemaType.Source);
adapter.Fill(dataSet);
cmd.Connection.Close();
// crete the objects
foreach (DataRow row in dataSet.Tables[0].Rows)
{
PostCode postcode = new PostCode();
postcode.ID = Convert.ToInt32(row["PostcodeID"]);
postcode.Outcode = row["OutCode"].ToString();
postcode.Latitude = Convert.ToDouble(row["Latitude"]);
postcode.Longitude = Convert.ToDouble(row["Longitude"]);
postcode.TownID = Convert.ToInt32(row["TownID"]);
AllPostCodes.Add(postcode);
postcode = null;
}
}
}
The rebuild occurs every 1 hour. this ensures that every 1 hour the site will have fresh set of cached data.
the issue ive got is that occasionally if during a rebuild, the server will be hit by a request and an exception is thrown. The exception is "Index was outside the bounds of the array." it is due to when a list is being cleared.
ListPostCodes.AllPostCodes.Clear(); - // throws exception - although its not always in regard to this list.
Once this exception is thrown application dies, All users are affected. I have to restart the server to fix it.
i have 2 questions...
If i utilise caching instead of static objects would this help ?
Is there any way i can say "while the rebuild is taking place, wait for it to complete until accepting requests"
any help is most appricaiated ;)
truegilly
1 If i utilise caching instead of
static objects would this help ?
Yes, all the things you do are easier done by the caching functionality that is build into ASP.NET
Is there any way i can say "while the
rebuild is taking place, wait for it
to complete until accepting requests"
The common pattern goes like this:
You request data from the Data layer
If the Datlayer sees that there is data in the cache, then it serves the data from cache
If no data is in the cache the data is requested from the db and put into cache. After that it is served to the client
There are rules (CacheDependency and Timeout) when the cache is to be cleared.
The easiest solution would be you stick to this pattern: This way the first request would hit the database and other requests get served from the cache. You trigger the refresh by implementing an SQLCacheDependency
You have to make sure that your list is not modified by one thread while other threads are trying to use it. This would be a problem even if you used the ASP.NET cache since collections are just not thread-safe. One way you can do this is by using a SynchronizedCollection instead of a List. Then make sure to use code like the following when you access the collection:
lock (synchronizedCollection.SyncRoot) {
synchronizedCollection.Clear();
etc...
}
You will also have to use locking when you read the collection. If you are enumerating over it, you should probably make a copy before doing so as you don't want to lock for a long time. For example:
List<whatever> tempCollection;
lock (synchrnonizedCollection.SyncRoot) {
tempCollection = new List<whatever>(synchronizedCollection);
}
//use temp collection to access cached data
The other option would be to create a ThreadSafeList class that uses locking internally to make the list object itself thread-safe.
I agree with Tom, you will have to do synchronization to make this work. One thing that would improve the performance is not clearing the list until you actually receive the new values from the database:
// Modify your function to return a new list instead of filling the existing one.
public static List<PostCode> GetAllPostCodes()
{
List<PostCode> temp = new List<PostCode>();
...
return temp;
}
And when you rebuild the data:
List<PostCode> temp = GetAllPostCodes();
AllPostCodes = temp;
This makes sure that your cached list is still valid while GetAllPostCodes() is executing. It also has the advantage that you can use a read-only list which makes the synchronization a bit easier.
In your case you need to refresh the data every one hour.
1) IT should use cache with absolute expiration set to 1 hour, so it expires after every 1 hour. Check the Cache before using it, by doing a NULL check.If its NULL get the data from DB and populate the Cache.
2) With above approach the disadvantage is that data can be stale by 1 hour. So if u want most updated data at all times, use SQLCacheDependency (PUSH). so whenever there is a change in the select command u r using, cache will be refreshed from the database with updated data.