I am getting an error in my MVC-based website surrounding data concurrency in Linq to SQL:
"Row Not Found Or Changed"
After reading several posts on here it seems as though an accepted solution is to set all non-Primary key fields to UpdateCheck = false in the dbml designer. Before taking the plunge with this, I wanted to ask, will I be losing anything if I do this?
To be honest, it seems to me like this should always be the case, as using the primary key should be the fastest way to find a record anyway. This would be assuming that you don't have any composite PKs. I'm not terribly familiar with data concurrency scenarios, but am I missing something here?
Thanks for your time!
[EDIT]
Thanks for the feedback guys! To give some more detail, the specific scenario I have is this:
I have an Article table with a number of fields (title, content, author, etc.). One of the fields that gets updated very frequently (anytime anyone views the article), is a popularity field, which gets incremented with every click. I saw the original error mentioned above when I updated the article text in the database. Then navigated to that article on the live site (which attempted to update the popularity field).
For starters it sounds like I need to NOT be using a shared DataContext. Beyond that, maybe look into setting certain fields to UpdateCheck never. That being said, I definitely do not want the Article Popularity to not get update due to some concurrency issue. So with that being the case, is there a way for me to ensure this with optimistic concurrency?
Thanks again!!
I do sometimes change UpdateCheck to Never on all but the primary key(s) on entity classes in the designer myself. What you lose is notification of interim changes to the data since you loaded it; what you end up with is a "last change wins" type of scenario. If that is OK in your situation, then... fine.
There are situations where that is definitely not a good thing to do. For instance, checking/adjusting a bank account funds balance... or any situation where an alternation to a field is going to be based on a calculation with the current value as one of the operands.
set all non-Primary key fields to UpdateCheck = false in the dbml designer. Before taking the plunge with this, I wanted to ask, will I be losing anything if I do this?
The job of concurrency mechanisms is to prevent multiple users from stomping each others changes. Pessimistic concurrency achieves this by locking out changes to a record when it is read. Then unlocking the record when changes are complete. Optimistic concurrency achieves this by remembering what the record looked like when it was read, and then not modifying the record if it has changed.
Linq to Sql uses optimistic concurrency. If you set the UpdateCheck property that way, you are disabling optimistic concurrency checking.
Consider this record:
Customer (Name = "Bob", Job = "Programmer", Salary = 35000.00)
LinqToSql DataContext#1 reads the record
LinqToSql DataContext#2 reads the record and commits the following change (Salary = 50000.00)
LinqToSql DataContext#1 attempts to commit the following change (Job = "Janitor")
With optimistic concurrency, DataContext#1 now gets a concurrency exception to resolve. Without optimistic concurrency, we now have a janitor making 50k.
Related
Most of our tables have one or more columns which are set by the database, either by a trigger, or we want to use the database default value (which requires not sending the field at all in the insert or update)
This includes transaction dates set in the dB (so all the times are times stamped very accurately by a single source, not relying on the accuracy of the time on an arbitrary server or pc.)
The second VERY common use case is say if a customer record has his address and a last logged in field. the last logged in field (and number of failed logins) is set by another part of the system (e.g. by a web site server). The current overly simplistic CRUD system which GORM provides would overwrite such a field when an operator or customer edits their address for example. This is because GORM includes in its update and insert statements every field, even if it's null, or if it has not been changed.
We need a way to scrub the field from inserts and updates, but still have it used in the read calls.
I.e. a true "read only" attribute.
We tried this:
failedLogins editable: false, attributes: [readonly:true]
Which has no effect on the SQL generated (and doesn't even affect the scaffolded UIs - its still editable in create and edit, in grails 2.4.4 at least, but thats another story)
When we do want to explicitly write one of these fields, such as number of failed logins, we would resort to using embedded SQL.
I saw this post: Read-Only columns
Which asks exactly the same question, but only gives one solution, which is this plugin:
extended GORM mappings
Unfortunately, this plugin has not been updated since 2010, and only works with 1.3. We need something which works with 2.4.4.
Any grails app which has multiple systems which edits independent fields needs something like this, or to do extensive locking (Which is usually out of the question).
E.g. an operator opens the customer details for editing, edits something editable (e.g. address), then the operator fails a login on the website (a different grails or non-grails app), then the operator saves the player details. If the saving included the numberOfFailedLogins field, the system would fail. If opening the player details for editing locked the player, then the player would not be able to login, as updating the "lastLoggedIn" or "numFailedLogins" would fail to be able to write due to the lock. The solution is VERY simple - read only columns. Another way would be to put each read only type field in their own tables, but this would be untenable (and result in hundreds of one field tables)
Or we go back to using MyBatis, which has no such issues, and full control. Sadly, there is no good mybatis plugin for grails.
You can use derived properties for string and number properties:
class Batch {
String name
Integer timesRun
static mapping = {
timesRun formula: 'times_run' //times_run is a column in the "batch" table
}
}
In the code above, timesRun would be read in from the database but ignored in inserts and updates as Hibernate considers the column a calculated one.
Updated the example because the original one may have been misleading
This probably doesn't specifically answer your question, but you can use dynamicUpdates to tell GORM to only update the properties of the domain object that have changed during the current session. So as long as you don't change the "read-only" property in your code it won't be set in the SQL update statement generated by Grails. For added safety you could override (and noop) the setter so that your code can never change that property.
https://grails.github.io/grails-doc/latest/ref/Database%20Mapping/dynamicUpdate.html
One of the downsides of dynamicUpdates is that it might make the Hibernate query cache less useful. However, it seems that some Grails/Hibernate experts recommend that you disable the query cache anyway (at least in older versions of Grails). Not sure if that's true of Grails 2.4+
http://grails.github.io/grails-howtos/en/performanceTuning.html
http://www.anyware.co.uk/2005/2012/11/12/the-false-optimism-of-gorm-and-hibernate/
http://tech.puredanger.com/2009/07/10/hibernate-query-cache/
The Problem
I'm just trying to figure out exactly how much of my own security I need to implement on the server side when saving changes in Breeze. In particular, I'm thinking about how a malicious user could manually hack the SaveChanges request, or hack the javascript in the client, to bypass my normal business rules - for example, to maliciously alter foreign key IDs on my entities.
I want to understand exactly where I need to focus my security efforts; I don't want to waste time implementing layers of security that are not required.
I'm using Breeze with .net and Entity Framework on the server side.
Example
Here's a trivial example. ObjectA has a reference to an ObjectB, and ObjectA is owned by a particular User. So, my database looks like this:
ObjectA:
Id ObjectB_Id SomeField User_Id
1 1 Alice's ObjectA 1
2 2 Bob's ObjectA 2
ObjectB:
Id SomeOtherField
1 Foo
2 Bar
User:
Id Name
1 Alice
2 Bob
From this model, the security concerns I have are:
I don't want unauthenticated users to be changing any data
I don't want Bob to be able to make any changes to Alice's ObjectA
I don't want Alice to try to point her ObjectA at Bob's ObjectB.
I don't want Bob to try to change the User_Id on his ObjectA to be Alice.
The solution for (1) is trivial; I'll ensure that my SaveChanges method has an [Authorize] attribute.
I can easily use Fiddler to build a SaveChanges request to reproduce issues 2 to 4 - for example, I can build a request which changes Alice's ObjectA to point to Bob's ObjectB. This is what the message content might look like:
"entities":
[
{
"Id":1,
"ObjectB_Id":2,
"SomeField":"Alice's ObjectA",
"User_Id":1,
"entityAspect":
{
"entityTypeName":"ObjectA:#MyNamespace",
"defaultResourceName":"ObjectAs",
"entityState":"Modified",
"originalValuesMap":
{
"ObjectB_Id":"1"
},
"autoGeneratedKey":
{
"propertyName":"Id",
"autoGeneratedKeyType":"Identity"
}
}
}
],
As I'd expect, when no security is implemented on the server side, this persists the updated value for ObjectB_Id into the database.
However, I've also confirmed that if there is no entry for ObjectB_Id in the originalValuesMap, then even if I change the value for ObjectB_Id in the main body of the message it is NOT updated in the database.
General Rules?
So, I think this means that the general security rules I need to follow on the server are:
[Edited 4 July 2013 - rewritten for clarity]
In general:
Nothing in the message can be trusted: neither values in the originalValuesMap nor supposedly "unchanged" values
The only exception is the identity of the entity, which we can assume is correct.
Supposedly "unchanged" properties may have been tampered with even if they are not in the originalValuesMap
For "Unchanged" properties (properties which are not also on the originalValuesMap):
When "using" any "unchanged" property, we must NOT use the value from the message; we must retrieve the object from the database and use the value from that.
for example, when checking owenership of an object to ensure that the user is allowed to change it, we cannot trust a UserId on the message; we must retrieve the entity from the database and use the UserId value from that
For any other "unchanged" property, which we are not using in any way, we don't need to worry if it has been tampered with because, even if it has, the tampered value will not be persisted to the database
For changed properties (properties which are also on the originalValuesMap):
Business rules may prevent particular properties being changed. If this is the case, we should implement a check for each such rule.
If a value is allowed to be changed, and it is a foreign key, we should probably perform a security check to ensure that the new value is allowed to be used by the session identity
We must not use any of the original values in the originalValuesMap, as these may have been tampered with
[End of edit]
Implementing the Rules
Assuming that these rules are correct, I guess there are a couple of options to implement security around the changed foreign keys:
If the business rules do not allow changes to a particular field, I will reject the SaveChanges request
If the business rules DO allow changes to a particular field, I will check that the new value is allowed. In doing this, CANNOT use the originalValuesMap; I'll need to go to the database (or other trusted source, eg session Cookie)
Applying these rules to the security concerns that I gave above,
security concern (2). I'll need to check the user identity on the session against the User_ID on the ObjectA that is currently in the database. This is because I cannot trust the User_ID on the request, even if it is not in the originalValuesMap.
security concern (3). If the business rules allow a change of ObjectB, I will need to check who owns the new value of ObjectB_Id; I'll do this by retrieving the specified ObjectB from the database. If this ObjectB is not owned by ObjectA's owner, I probably want to reject the changes.
security concern (4). If the business rules allow a change of User, this is already covered by (2).
Questions
So, really, I'm looking for confirmation that I'm thinking along the right lines.
Are my general rules correct?
Does my implementation of the rules sound reasonable?
Am I missing anything?
Am I over complicating things?
Phil ... you are absolutely on the right track here. You've done a nice job of laying out the issues and the threats and the general approach to mitigating those threats. It is almost as if you had written the introduction to the Breeze security chapter ... which we haven't gotten to yet.
I do not think that you are "over complicating things"
Someone reading this might think "wow ... that's a lot of work ... that Breeze stuff must be insecure".
Well it is a lot of work. But it isn't Breeze that is making it difficult. This is the necessary thinking for every web application in existence. Authentication is only the first step ... the easiest step ... in securing an application.
You shouldn't trust any client request ... even if the client is authenticated. That means making sure the client is authorized to make the request and that the content entering and exiting the server is consistent with what the client is both claiming to do and is allowed to do. These are general principles that apply to all web applications, not just Breeze applications. Adhering to these principles is no more difficult in Breeze than in any other technology.
One Breeze technicality you may have overlooked. The EFContextProvider.Context should only hold the entities to save; don't use it to retrieve original entities.You'll need a separate DbContext to retrieve the original entities to compare with the change-set entities from the client.
We are working on samples that demonstrate ways to handle the issues you described. For example, we're recommending (and demo'ing) a "validation rules engine" that plugs into the BeforeSaveEntitiesDelegate; this "engine" approach makes it easier to write bunches of server-side rules and have them applied automatically.
Our samples and guidance aren't quite ready for publication. But they are coming along.
Meanwhile, follow your instincts as you've described them here. Blog about your progress. Tell us about it ... and we'll be thrilled to highlight your posts.
I've been looking for guidance on the same matter and I am very happy to find your brilliant analysis. In my opinion the answer to our problem is different though, assuming that we are talking about applications which are to be composed of more than a few modules and are to live longer than a year.
If rules become too complicated it means that we might be using inappropriate approach. I'm sure many brilliant developers would cope following these rules but the sad truth is that most of our peers would either get it wrong or would forget about some of them under pressure.
I'd say that we need to go back to Fowler's, Evans' and Nilssons' publications and repeat after them that in larger applications (and these have strong security requirements) the entity model is not something that should be exposed to the client at all (for other reasons than security too - e.g. maintainability).
On the other hand it is worth looking at revisions to these original ideas proposed later by Greg Young and Udi Dahan. These in essence say that model for reading does not have to and often is not the same as model for writing 'data'.
To sum this up I'd say that the base rule should be DON'T use Breeze for writing and DO use it for reading (with DTOs/Projections), provided you don't query the 'real' model but the model built specially for reading (e.g. Views not Tables).
All this quite naturally emerges if you follow your domain and use cases and above all if you follow Test-Driven approach. Would you really end up with BeforeSaveEntities solution for business rules while following Test-Driven-Development?
I am experiencing some bizarre problems with Nhibernate within my MVC web application.
There is not 1 consistent error, I keep getting loads of random ones:
Transaction not successfully started
New request is not allowed to start because it should come with valid transaction descriptor
Unexpected row count: -1; expected: 1
To give a little context to the setup, I am using Ninject to DI the sessions and other Nhibernate related objects, currently I am using RequestScope however I have tried SingletonScope. I have a large and complicated data model, which is read out as a whole, but persisted back in separate parts, as these can all be edited and saved individually.
An example would be having a Customer object, which contains a address object, a contact object, friends object, previous orders object etc etc...
So the whole object is read out, then mapped to the UI domain models and then displayed in different partials within the page. Each partial can be updated individually via ajax, so you may update 1 section or you could update them all together. It seems mainly to give me the problems when I try to persist them all together (so 2-4 simultanious ajax requests to persist chunks of the model).
Now I have integration tests that work fine, which just test the persistence and retrieval of entities. As a whole and individually and all pass fine, however in the web app they just seem to keep throwing random exceptions, and originally refused to persist outside of the Nhibernate cache. I found a way round this by wrapping most units of work within transactions, which got the data persisting but started adding new errors to the mix.
Originally I was thinking of just scrapping Nhibernate from the project, as although I really want its persistance/caching layer, it just didnt seem to be flexible enough for my domain, which seems odd as I have used it before without much problem, although it doesn't like 1-1 mappings.
So has anyone else had flakey transaction/nhibernate issues like this within an ASP MVC app... I know this may be a bit vague as the errors dont point to one thing, and it doesn't always error, so its like stabbing in the dark, but I am out of ideas so any help would be great!
-- Update --
I cannot post all relevant code as the project is huge, but the transaction bit looks like:
using (var transaction = sessionManager.Session.BeginTransaction(IsolationLevel.ReadUncommitted))
{
try
{
// Do unit of work
transaction.Commit();
}
catch (Exception)
{
transaction.Rollback();
throw;
}
}
Some of the main problems I have had on this project have stemmed from:
There are some 1-1 relationships with composite keys, but logically it makes sense
The Nhibernate domain entities go through a mapping layer to become the UI domain entities, then vice versa when saving. Problem here is that with the 1-1 mappings, when persisting the example Address I have to make a Surrogate Customer object with the correct Id then merge.
There is ALOT of Ajax that deals with chunks of the overall model (I talk like there is one single model, but there are quite a few top level models, just one that is most important)
Some notes that may help. I use windsor but imagine the concepts are the same. Sounds like there may be a combination of things.
SessionFactory should be created as singleton and session should be per web request. Something like:
Bind<ISessionFactory>()
.ToProvider<SessionFactoryBuilder>()
.InSingletonScope();
Bind<ISession>()
.ToMethod( context => context.Kernel.Get<ISessionFactory>().OpenSession() )
.InRequestScope();
Be careful of keeping transactions open for too long, keep them as short lived as possible to avoid deadlocks.
Check your queries are running as as expected by using a tool like NHProf. Often people load up too much of the graph which impacts performance and can create deadlocks.
Check your mappings for things like not.lazyload() and see if you actually need the additional data in the queries and keep results returned to a min. Check your queries execution plans and ensure adequate indexes are in place.
I have had issues with mvc3 action filters being cached, which meant transactions were not always started, but would attempt to be closed causing issues. Moved all my transaction commits into ActionResults in the controllers to keep transaction as short as possible and close to the action.
Check your cascades in your mappings and keep the updates to a minimum.
I like using LINQ to SQL. The only problem is that I don't like the default way of updating tables.
Let's say I have the following table with the following columns:
ID (primary key), value1, value2, value3, value4, value5
When I need to update something I call
UPDATE ... WHERE ID=#id
LINQ to SQL calls
UPDATE ... WHERE ID=#id and value1=#value1 and value2=#value2 and value3=#value3 and value4=#value4 and value5=#value5
I can override this behavior by adding
UpdateCheck=UpdateCheck.Never
to every column, but with every update of the DataContext class with the GUI, this will be erased. Is there any way to tell LINQ to use this way of updating data?
I'm confused by this statement:
but with every update of the DataContext class with the GUI, this will be erased. Is there any way to tell LINQ to use this way of updating data?
By "the GUI", do you mean the Linq to SQL designer? Because the property sheet for each member has an "Update Check" property that you can set to "Never". If you are manually editing the .designer.cs file, don't do that, instead change the Update Check setting in the actual designer.
Designer Screen http://img29.imageshack.us/img29/7912/updatecheckdesigner.png
Please note: The "default way" of updating used by Linq to SQL is called optimistic concurrency, and is a way of preventing conflicting updates from multiple users. If you turn this off by using the method above, you have to be prepared to deal with the fact that if two users have the same record open at the same time, the second user's changes will overwrite the first user's changes without any warning or confirmation. Be sure that this is the behaviour you really want.
Unfortunately, no, there's not. You have to edit the DBML manually after it is generated (or updated) - which is a pain (or use the Designer as already mentioned in the other answer).
When I last used L2S on a project, I wrote a quick utility which ran post-generation and fixed it up, but it's an unnecessary pain which (c)shouldn't be required IMHO.
Ran into this one myself. The trick is to change the way one generates the DBML--such as using l2st4. Then you can set that pesky UpdateCheck property to always be never by modifying the template.
That is how Linq works. Why don't you like this update behavior?
Read about optimistic concurrency
http://msdn.microsoft.com/en-us/library/bb399373.aspx
alt text http://img29.imageshack.us/img29/825/simplemodel.jpg
How you can see above, it's a kind of subclass a table in a RDBMS (in this case, my favorite: MySQL) so I handle it with Visual Subclassing a base form for tb_order_base with validating field data, etc.
This way, I'm free of repeating code and some other bothering problems, well, it seems a true OO aproach. But ...
Now I got a Big problem with subclassed form i.e. tb_order_service with a master/detail approach, when I post the tb_order_base dataset, instead of Delphi first post it and get the PK ID from RDBMS and then post the TB_ORDER_PRODUCT with id filled, it does the opposed, posting the detail tb_order_product dataset first then master tb_order_base, so I get a big foreing key constraint error.
Does anyone has any idea to how to by-pass this amazing problem?
I have asked it before, but with few details in The Master/Detail Behavior
One thing I have done in such a condition was to break away from doing DBMS direct calls, and to instead use a memory dataset (or client dataset) for the form. When the user presses the save button, I would validate my edits and return any errors. If no errors then I would begin a database transaction, commit the master record (and then read the master record key back if its an insert), then run thru each table commiting the child records, followed by a commit of the transaction (or rollback if there were any problems saving child data).
I also stay away from using data-aware components. They work great for simple utility type programs, but when you start creating a complex system using them you will find little gotchas along the way that are easily solved by using a standard edit and a function to push/pull data to/from the database. The only exception I generally make would be for grids, but I only use the grid for selection...the actual editing is done using non data-aware components.