MVC EF and large datasets with complex queries - asp.net-mvc

Warning : Noob Question !
I am being pushed into using MVC & EF for my projects; both of which are new to me.
The MVC part I'm not too stressed about, but EF bothers me, there are some key concepts that my head just isn't around yet.
My database is a repository of millions of records which users will need to perform very complex searches upon.
Seems that the majority of articles I read are hooking the DB up to the Application with EF, and filtering data client-side. That's just a 'No' for me, and I can't imagine that I am the only one.
Then I have seen some articles that talk of pushing the filters back to the DB using LINQ/IQueryables.
That sounds more reasonable, at least I would not be bringing back more data than I need.
But my more complex queries would be nightmare to build in LINQ (I imagine).
Still, what bothers me, is that I have SQL Server to do all that heavy lifting for me. Store Procedures that can utilize most powerful mechanisms for the most complex queries imaginable. Isn't that what SQL Server was born to do ?
I am still thinking to bulldoze ahead with my old Skool brain.
Build Models that reflect my data tables/stucture.
Populate my Models 'manually' via Stored Procedures
Can anyone point out where I might be viewing all this wrongly ?
I know that there is heaps of info on Google, but most just jump into how to technically do 'A'... with little discussion over 'why' we are doing 'A' or what the alternatives are.
Thanks for any comment !

Point 1 :
Yes,you can start it as you mentioned.That is Build Models that reflect my data tables/structure.That means you're going to create everything by hand manually. Which is very tricky task hence you're already having the db.So I would like to suggest to use EntityFramework Reverse POCO Generator for generating the required models from your db.
Point 2 :
Yes,You can use Stored procedures with the EF code first.But be careful not to use EF core at this moment. B'cos the SP support is at the beginning level right now.So you can start with EF 6.X and later with the maturity of the EF core where you can move easily to the EF core world.
Point 3 :
You don't need to worry about the size of the records which you have to show on the UI. B'cos there having so many sophisticated UI components which are build for showing massive data on the UI. You can choose either from free or paid once.

Related

Cobination of mvc 4,entity framework, stored procedures is the right way?

We are doing a new project, for all devices and browsers compatibility we have decided to use asp.net mvc 4, Html5, css 3, for communicating with Database Entity Framework we want to use.
Our senior members(Manager, DBA(they are also new to mvc 4, EF)) in the team asking us to write every thing will be in the stored procedures while communicating Database so that maintenance becomes easy.
Is it the correct match if we go like that(MVC4+ EF + stored procedures)? Will i not get maintenance and performance if i go with Code first reverse engineering(because database tables are ready i want to do like that), Please reply.
Below is the flow we want to do, please correct me
As Database is already ready, so first we will write the stored procedures for communication with DB.
New Mvc 4 project and will add .edmx file(EF) and select tables and Stored procedures
in mvc controller or web api we write the consuming stored procedures
There is nothing technically wrong with ASP.NET MVC + EF + Stored Procedures approach, from the first sight.
But my experience show, is that typically it's huge overkill. The common problem I see is the conflicting interests between developers and DBA's. In most worst scenarios all DB releated stuff are controlled by DBA, so if developer what to add/change some feature he needs to wait for implementation of it by DBA (or wait for approve, which could also take long).
So, I personally see that as more bureaucratic way of development.
My own perpective is to be more agile on development and tools like Code First matches that. Stored Procedures could still play major role, while code/performance optimization, but not something to start with.
I agree that using stored procedures in the database is a good approach. Centralizing data validation and calculations in the database ensures data integrity. Client-side validation is important for the user experience but you must also ensure that you test the data validity in the database.
Using Entity Framework, you can generate entities which relate directly to tables in your database, or else you can design entities which use procedures for insert/update/delete operations rather than simple table updates.
In MVC you will use the entities as models to manage your data interactions.
Good luck
This is my personal view. I am sure others might have different ones. Since you are asking this question I am hoping you are open for discussions, otherwise I wouldn’t have bothered as this topic is like a religious discussion as lots of people have very strong opinions and are not likely to change them.
Personally I don't think stored procedures are meant to write business logic. They should be used for writing data access logic. I would only use a stored procedure if I want to optimize an expensive query such as a dynamic search but nothing else. You will get slightly less performance if you have your logic in the domain model, but its not even noticeable in most situations.
One of the strong arguments for writing business logic in stored procedures is because you can easily change some logic by changing your stored procedure. But should we really go and change the business logic of a deployed application without doing proper testing. What will happen if you accidently do a mistake? Doing a deployment is not such a big deal now with continuous builds and I don’t think as a professional developer you should take that risk.
When you decide to write your logic in stored procedures, you give up all the object oriented concepts and you end up writing some procedural code that we wrote maybe 10 years ago. C# language has come a long way now and you will not be able to use those new language features in heart of your application which is business logic. You also loose the visual studio features to refactor code, advanced and easy debugging features etc.
I also don’t like the idea of having triggers as it’s not visible in source code. Imagine someone new in your team trying to add a new feature some time later and if he doesn’t know that a trigger exists, he might write some incorrect logic.
If your application contains some complex business logic, (I am sure most applications do) you should have a domain model that contains not only just properties of your entities, but also your logic. Otherwise you will be falling in to the anti-pattern called anemic data model.
You will not be able to test your business logic by writing unit testing if you have your logic in stored procedures.
You will also not be able to deploy your business logic to multiple servers if you have them in stored procedures if your site becomes really successful.
You will also not be using all powerful capabilities of Entity framework and LINQ if you have all your logic in your stored procedures. You actually don’t need an ORM Mapper if that is the approach you are going to take.
This is what I would recommend for your project.
Even though you already have the database, you can still use code first approach of Entity framework. You can download the EF code first reverse engineer power tool and have the code first code auto generated for you. This is going to be a one off thing and after than if you have any more changes, you can directly do to the database and update the code first code accordingly. Fluent API is bit confusing at first, but you can easily learn that from the generated code.
Do not access your data context from the controller. Have a repository layer that will contain all your data access logic. You can access the repository from your controller. (This allows you to unit test your code by mocking the repository). There are lots of video tutorials on how to use the repository pattern on asp.net site.
Your domain model is going to be the entities that got generated from the Entity framework. Try to have your business logic in those models. It takes a little while to get use to the domain model pattern. But one you get used to it you will start to appreciate its benefits.
Hope this helps.

Is entity framework a bad choice for multiple websites and large aplications?

Scenario : We currently have a website and are working on building couple of websites with an admin website. We are using asp.net-mvc , SQL Server 2005 and Entity Framework 4. So, currently we have a single solution that has all the websites and all the websites are using the same entity framework model. The Model currently has over 70 tables and will potentially have a lot more in the future... around 400?
Questions : Is Entity Framework model going to be slower when it is going to grow bigger? I have read quite a few articles where they say it is pretty slow due to the additional layers of mapping when as compared to say ado.net? Also , we thought of having multiple models but it seems that it is a bad practice too and is LINQ useful when we are not using any ORM?
So, we are just curious what and how all the large websites using a similiar technology as we have achieve good performance while using an ORM like EF or do they never opt for an ORM ? I have also worked on a LINQ to SQL application that had over 150 tables and we encurred a huge startup penalty, site took 15-20 seconds to respond when first loaded. I am pretty sure this was due to large startup cost of LINQ to SQL ORM. It would be great if someone can share their experience and thoughts regarding this ? What are the best practices to follow and I know it depends on every application but if performance is a concern then what are the best steps to be taken ?
I don't have a definite answer for you, but I have found this SO post: ORM performance cost, it will probably be informative for you, expecially the second highest answer mentioning this site:
http://ormbattle.net/
My personal experience is that for any ORM mapper I have seen so far, Joel's law of leaky abstraction applies heavily. So if you are going to use EF, make sure you have alternatives for optimization at hand.
I think you can certainly get EF4 to work in a performant way with a database with a large number of tables. That said, you will certainly have to overcome a number of hurdles that are specific to EF.
I don't think LinqToSql is a good alternative since Microsoft has stopped enhancing it for the most part.
What other alternatives have you considered? ADO.NET? NHibernate? Stored Procedures?
I know NHibernate may have trouble establishing the SessionFactory for 400 tables quickly, but that only happens once when the website application starts, which should be fairly rare if the application is used heavily. Each web request generally has a new Session and creating sessions from the session factory is very quick and inexpensive.
My biggest concern with EF is the management of the thing, if you have multiple models, then you're suddenly going to have multiple work to do maintaining them, making sure you never update the wrong model for the right database, or vice versa. This is a problem for us at the moment, and looks to only get worse.
Personally, I like to write SQL rather than rely on an abstraction on top of an abstration. The DB knows SQL and so I keep it happy with hand-crafted stored procedures, or hand-crafted SQL in some cases. One huge benefit to this is that I can reply code to see what it was trying to do, and see the resulting data by c&p the sql from the log to the sql query editor. That, in my opinion, makes support so much easier it entirely invalidates any programmer benefit you might have from using an ORM in the first place (especially as EF generates absolutely unreadable SQL).
In fact, come to think of it, the only benefit an ORM gives you is that you can code a bit quicker (once you have everything set up and are not changing the schema, of course), and ultimately, I don't think the benefit is worth the cost, not when you consider that I spend most of my coding time thinking about what I'm going to do as the 'doing it' part is relatively small cmpared to the design, test, support and maintain parts.

Any MVC 3 tutorials on *Real World* developement situations?

MVC total noobe here but long time web developer - 10 + years. The tutorials for MVC 3 (and earlier versions) are great but as usual they lack a ton for real on the job type scenarios.
For example, how often do you find yourself in a situation where you are going to create a new database from scratch with no stored procs so that you could actually use EF Code-First. I don't know about you but in my career it has been NEVER.
The usual story is that you are creating a new app or enhancing an existing app with new functionality that will connect to an existing very mature database with tons of stored procs, user defined functions and views and you are required either by management or time restrictions to use it all. And of course you may get to create some new tables but they usually will have joins to existing tables or in the least your app will have to query existing tables for some of the data.
To see a tutorial based on that scenario would be WORTH IT'S WEIGHT IN GOLD. Especially the stored procedure scenario.
Thank you for any advice
Most of the earlier examples (e.g. the original NerdDinner) were based on either Linq to Sql or Entity Framework (without CodeFirst). Since CodeFirst is the 'new hotness' most of the latest examples use it.
The interesting part of the question, though, is that it highlights an important point: "it doesn't matter". Your data access strategy (EF, EF code first, NHibernate, L2S, raw SQL) is totally irrelevant to MVC. By that I don't mean it's unimportant, I mean that MVC doesn't place any constraints on you at all in that regard.
You will generally (in a well-designed MVC app) pass your controllers interfaces which let them call data access methods of various sorts (or perhaps another layer of indirection with services that do other things before hitting the storage layer). The implementation of the data access, if it is using an ORM like EF or Nhibernate, will then have mechanisms for you to either use a query syntax of some sort (e.g. LINQ) or to call stored procedures (possible in all major ORMs that Iv'e used) or to push raw SQL if hte situation calls for it.

Model for ASP.NET MVC

I just started with ASP.NET MVC 1.0. I've read through some tutorials but I don't have any good ideas on how to build my model.
I've been experimenting with LINQ to SQL. Can't say I like it that much, but I'll give it a try. I prefer using SQL Stored Procedures, but it doesn't seem to work very good together with the optional parameters in my stored procedures.
The project is a prototype, but I would like to start with a good design right away.
The database will probably contain around 50 tables and stored procedures will be used for all queries and updates. Some procedures will also return multiple results. How would the best way to design this model?
All and any ideas are welcome. Should I even use LINQ to SQL? Should I design my stored procedures to only return one result? I feel a little bit lost.
You can use stored procedures with LINQ. You simply need to replace the autogenerated statements on your entities with your stored procs. Look at the properties for the entity (table) in the designer to change these. Your stored procedures won't be able to return multiple results as they need to map onto a collection of either one of your entities or an autogenerated entity, i.e, onto a collection of a single class.
Having said all that, I'd give it a go without using stored procedures. The LINQ code is fully parameterized so you already have this benefit. You'll find that LINQ allows you to easily build up queries alleviating your need for the optional parameters in your stored procs and, likely, resulting in more efficient queries as you don't need to build extra logic into your query to handle the optional parameters. Once you start using LINQ I think you'll find that the power and expressiveness in the code will make you forget about using your stored procedures.
You may still have a need for stored procs or table-valued functions for complex queries -- that's ok. If they map onto an existing entity, you can simply drag the proc/function onto the entity in the designer and you get a nice method on your data context that runs your SQL and returns collections of that entity. If not, I often create a view for the proc/function schema, use that as an entity, and attach the proc/function to it.
May I ask why you need to use stored procedures for this project? They have almost no value today if you are working with a modern database. Parameterized SQL using a good ORM will give you the same cached execution plans and security benefits that you are looking for with stored procedures.
I would recommend NHibernate because it's a far better way to achieve persistence ignorance. And IMHO this is the goal of anyone designing a model. You should be thinking object first instead of data first.
The best resource around is a screencast series by Steve Bohlen "The summer of nhibernate"
Also LINQ to NHibernate is available and works great for any dynamic query that you might need to write.
The only pain you will notice with NHibernate is the XML, but if you really enjoy the ORM you could write a fluent interface instead using FNH.
I found NHibernate to be the best tool around for modeling a domain because your objects are 100% infrastructure free and this allows you to do anything you want with them without having to worry about the database. Also if you are into unit testing of any kind this will make your life much easier :P
Edit
Part of the reason I replied to your question is that I have built many a system using stored procedures and found to regret that decision later. But in my situation I had a requirement by a dba so I couldn't step away from the norm.
If you are already looking at LINQ to SQL I would simply encourage you to look at NHibernate as another great option to achieve the goal of persistence ignorance.
When I got away from thinking about the problem data first it allowed me to work at a much higher level and take advantage of the .net platform for solving problems. TSQL has its place but if you are writing an application and want to maintain it - try to think about how your domain objects will work together in memory first.

ASP.NET MVC + LINQ to SQL or Entities?

When linq to entities was released, everybody say that linq to sql is dead.
But most books and sample projects by microsoft employees, use mvc+linq to sql.
There is some reason for that? The linq to sql seems better for POCO, would it?
Linq to SQL had its origins in a thought experiment. Linq needed a proof of concept, and Linq to SQL provided that.
Since then, many in the user community (including a few prominent Microsoft employees) have embraced Linq to SQL as the lightweight SQL Server ORM of choice, especially for small projects like NerdDinner.
The furor began with this post by the ADO.NET team. In it, they said that Linq to SQL will be maintained, and features added based on user community feedback, but that the Entity Framework would be the primary focus of future data access efforts.
Many in the user community interpreted this move as "Microsoft is killing Linq to SQL."
Microsoft didn't help matters when it disabled the provider model for Linq to SQL by sealing some critical classes, effectively making Linq to SQL usable only with SQL Server. The Entity Framework will be the ORM of choice for multiple data providers.
Unfortunately, the Entity Framework seems not quite ready for prime time.
So it depends on who you believe. Do you believe the speculations (and they are just speculations) of many bloggers who say that Linq to SQL is truly dead and buried, or do you believe people like Damien Guard of Microsoft, who says that Linq to SQL has a long life ahead of it?
Linq-to-SQL is a great technology and very easy to use, so it's extremely well geared towards demos, and it's a perfect choice for smaller, more straightforward projects, too, where you have more or less a straight 1:1 mapping from your tables in SQL Server to your objects in memory.
And those are the most restrictive limitations of Linq-to-SQL:
SQL Server only
straight 1:1 mapping from tables to objects in memory
If that's okay with you, go ahead and use Linq-to-SQL ! By all means - MS is still adding features and fixing it for .NET 4.0 - it's not dead by a long shot.
Entity Framework in its v1 version available right now has some warts, as other posters have rightfully mentioned (no POCO support, no "domain design first" approach and many others). But the EF v4 feature set looks very compelling, and will give Linq-to-SQL a run for its money!
The main advantages for EF vs. Linq-to-SQL in an enterprise environment are its database independence (you can hook it up to Oracle, Firebird, DB2, many others) which can be crucial, and it's ability to present you with an object model that looks quite different from the physical storage model in the database (by virtue of the mapping between the conceptual layer and the physical storage layer).
So it's really a matter of what is it you need: do you need a quick way to get started (demos, simpler apps) - then your choice clearly is Linq-to-SQL (at least for now), or do you need a flexible mapping approach against a non-SQL Server backend - then your choice will be Entity Framework.
Marc
The only reason the samples were written with LINQ-to-SQL is because setting it up is fast and simple.
Otherwise you can use any ORM to suit you, whether it be EF or NHibernate or whatever.
Personally, I'd start getting used to the Entity Framework now. Since it has more features, is provider-independent, and being developed (perhaps) more aggressively. The two most common complaints I hear are lack of POCO and lack of Lazy Loading. It would appear that both of these are addressed in .NET 4.0 http://blogs.msdn.com/adonet/archive/2009/05/28/poco-in-the-entity-framework-part-2-complex-types-deferred-loading-and-explicit-loading.aspx
Besides being potentially faster (to get started with) or simpler, I'm not sure of any particular advantage to use LINQ to SQL, in the long run. Is anyone aware of any?
The nerddinner sample chapter written by ScottGu uses Linq-To-SQL, he does not endorse LINQ-to-SQL but does not promote LINQ-To-Entities either. With ViewModel (Stephen Walther has a nice post on this) & the repository pattern, LINQ-To-SQL is perfect for developing ASP.NET MVC applications
In my opinion, Linq to Entity is not a good candidate at the moment. One of the things that I've come to dislike about Linq To Entity is the lack of lazy loading, which drove me up the wall every time i wrote my repositories. I think, that's what killed it for me. The other thing is the whole issue with attaching and detaching entities which are being used by multiple instances of the object context. That was another killer for me, which resulted in a lot of hacking trying to recreate entity relationships and etc. Frankly, I think a better thing to do would be to wait for the next release of this framework. I think good things are coming for it :)

Resources