Im trying to populate my domain models and child entities with 1 SQL Store Proceedure execution. Perhaps this is answered here. Im pretty certain it's not possible but I though I would throw the question out there to find possible work arounds.
I have quite complex domain models and im looking for a more efficient way of loading my data rather than query a customer and then lazy load its children. I have presented a simple example of what im trying to achive below;
public class Customer{
public int Id { get; set; }
public virtual Address Address { get; set; }
}
public class Address{
public int Id { get; set; }
}
var customer = this.Database.SqlQuery< Customer >("exec SP_Name")
I know in EF5 you can return multiple data contexts but im hopeful I can resolve muliple child entities.
I hope ive made sense. Im lacking alot of sleep so apologies if it doesn't. Following a sport in a timezone 10 hours behind makes it difficult! :(
Stored procedures in EF don't offer eager loading. They can only load single level of entities. You can either use stored procedure with multiple result sets as mentioned in linked article but that works only with EDMX and you must execute mapped function import instead of SqlQuery. You can also simply use eager loading with LINQ query instead of stored procedure to avoid lazy loading:
var customers = context.Set<Customer>()
.Include(c => c.Address)
.FirstOrDefault(c => c.Name == someName);
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I'll try to describe a problem that I see very often in my workplace, but I couldn't find a way or a reasonable solution to it. I searched a lot about this and all I could find is what I already have implemented. The scenario is this:
I have an ASP.NET MVC app using Entity Framework which follows the Repository Pattern. I will use a simple Student/Teacher database structure to exemplify
Entities
public class Student : BaseEntity
{
public int Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public DateTime DateOfBirth { get; set; }
public Teacher Tutor { get; set; }
}
public class Teacher : BaseEntity
{
public int Id { get; set; }
public string Name { get; set; }
public ICollection<Student> Students { get; set; }
}
public class BaseEntity
{
public byte State { get; set; }
public Datetime CreateDate { get; set; }
public Datetime UpdateDate { get; set; }
}
Now, let's say I need to expose a method to return a list of student names along with its teacher's name:
var students = context.Students.Include(x => x.Teacher).ToList();
The query above is bad and we all know it, since it returns all columns. Now, if we refactor to this:
var students = context.Students.Include(x => x.Teacher)
.Select(x => new
{
Name = x.Name,
TeacherName = x.Teacher.Name
}).ToList();
I'll have a performant query that select only the fields I want. That's good. But now I need to make a choice. To return this list to my controller, I can either: Create a StudentDTO, or populate a Student instance with only the columns I selected in my query.
DTO approach
If I create a DTO and pass it along to my controller all will be "fine". No worries with extra unnecessary fields.
Return EF Entity Student
If I return the database entity to my controller as is, even though I loaded it with only the fields I wanted I will get all the other empty fields. So I create a ViewModel to return only the data I want. (This is what we are doing right now in most methods along with AutoMapper)
Problem:
See the pattern? Either way I will ended up creating a class to map my return. I'm fine with this most of the time, but in the project I'm working right now, we have several methods to expose and each of them has a different return structure. For example, what if I wanted to return a list of students that contains only student Id and Name? Create another DTO or ViewModel?
The project has a team of 5 developers and the ViewModel folder is getting pretty stuffed with classes. I started instructing them to return an anonymous object inside a controller when a method return is very specific. I only create View Models now when I'm sure it can be re-used elsewhere.
But this anonymous approach had become also bad because now I have to do this in several controllers and I feel I'm repeating myself.
Is there any other approach I can use to solve this? Of course the project I'm talking about is much more complex than this example, with several entities and pretty complex queries. I feel this happens a lot in other projects and I failed to find a decent way out of this.
I wish I would have stumbled upon this question a year ago. Hopefully this info helps you or someone else even though its been awhile.
You are exactly right. You see a problem and your gut is already telling you there is an architecture problem and you are right.
I know it is a pain but the DTO approach is the correct architecture in my opinion. These classes are a pain to create and use and we have all tried to shorcut around them at some point. But, you should be inheriting from base classes to share properties between objects to ease your pain. Also, use a 3rd party open source tool like automapper to move data easily between your data models and your dto objects.
Developers make this architectural mistake all the time (trying to bypass the DTO object) especially if they are working on smaller software systems and not large corporate enterprise systems where the above architecture will fail fast while it may succeed forever in a small system.
using your first example:
var students = context.Students.Include(x => x.Teacher).ToList();
You would then copy students into a dto object (perhaps StudentDTO) and then pass this up to your UX and translate it into a viewmodel object. If Student and StudentDTO are identical then they can inherit from the same base class and StudentDTO gets created with about one line of code.
Using your second example
var students = context.Students.Include(x => x.Teacher)
.Select(x => new
{
Name = x.Name,
TeacherName = x.Teacher.Name
}).ToList();
If you do not do this and instead translate everything into a ViewModel then you will end up with endless viewModels as you have.
However, even more importantly here, you shouldn't be able to move it into a viewmodel because ViewModels should sit in your UX layer and the fact that you can do this would indicate a small architecture issue on its own. ViewModels are only applicable to the UX and not anywhere else so keep them in the UX layer.
If you move your viewmodels into the UX you would find that you could not move your EF objects into your View Models and this would have steered you down the DTO architecture approach instead.
So why dont you just pass EF models up to the UX?
Well, you can. Works good in simple scenarios when you are just editing a single student for instance. This is tempting because its so easy to do and it works so why not do this?
For very simple, small systems please do. I think this is fine. you can pass an EF model up to the UX, translate it into a viewmodel and then the viewmodel back to the EF. To do this your EF models should sit outside your data and ux layers in their own layer. I typically use a dll called something like "MySoftwaremName.Common". I place all interfaces, data models and dtos in this layer and reference this layer from the UX, Service and Data layers so that DTOs and interfaces can easily be passed up and down the architecture.
My primary real world experience has shown that the primary reason you do not pass EF models directly up to the UX is Caching.
In small systems developers may not deal with this so this may work just fine. But in large systems where we are getting hit 50,000 to 100,000+ times a second caching is key.
What if you wanted to cache your student object so that it could be retrieved for the next 5 minutes without you hitting the db over and over again.
You cannot cache the EF models in ASP.NET or you will eventually encounter severe issues that are occuring randomly in your system that are hard to trace down.
You can stick the EF objects in Cache and even retrieve them from cache. However, EF entities are attached to Database Context objects and that context eventually goes out of scope and gets garbage collected by .NET. If you still have your Student object in cache after this happens and you retrieve it and try to use it (using your own example again) to get the Teacher subclass "Student.Teacher"
you will get the yellow screen of death if that Teacher object is not eager loaded and is lazy loaded By EF. Ef will try to use the context to go fetch that Teacher and there isn't one and it will blow up.
To prevent all this it is best to move your EF objects into DTOs and then you can safely tranfer this data, cache it etc without any worry.
The only pain in this architecture comes in translating DTOs to viewmodels and EF models exactly as you have stated. But as I mentioned above, using proper base classes to share fields and using something like AutoMapper or another Mapping engine will help you map quickly from DTOs to ViewModels and EF objects.
Always create a ViewModel. The ViewModel is a UI concern. your entity framework model is a domain concern. ViewModels aren't about re-use. They are about defining only what you need to satisfy the View that uses them.
I've read an article that in Entity Framework, the query will be sent to database after we call .ToList(), Single(), or First()
I have thousands of data so rather than load all the data I'd like to return data in paged. So I'm using PagedList to create paging in MVC. If it doesn't wrong when we called for example products.ToPagedList(pageNumber, 10), it will take only 10 records of data, not the whole data. Am I right?
Next, I'm using automapper to map from entities to viewmodel.
List<ProductViewModel> productsVM = Mapper.Map<List<Product>, List<ProductViewModel>>(products);
return productsVM.ToPagedList(pageNumber, 10);
As you can see in the snippet code above, does it take only 10 records before called .ToPagedList()? If when we do mapping, it will call .ToList() inside, I think it will call all of the data from the database then return 10 records. How to trace it?
The easiest way to see what is going on at database level is to use Sql Server Profiler. Then you will be able to see the sql queries that the entity framework is executing.
If you are using Sql Express then you can use Sql Express Profiler to do the same thing.
No, it doesn't take 10 records before paged list. The way your code is shown, AutoMapper will cause a deferred execution of the query, before it reaches paged list, which means it will return all data (let's suppose, 1000 records). Then PagedList will properly retrieve 10 of the already materialized List, and recognize the total amount of record was 1000.
I think you want to filter 10 in database, which has better performance, so you should use PagedList in the IQueryable of your database entities like this:
List<Product> filteredProducts = dbContext.Products.OrderBy(p => p.ProductId).ToPagedList(pageNumber, 10);
return Mapper.Map<List<Product>, List<ProductViewModel>>(filteredProducts);
The OrderBy is mandatory for PagedList.
BE CAREFUL WITH AUTOMAPPER
Consider the following scenario. What if your Product entity had a child relationship with ProductReview (a ICollection<ProductReview>) like this:
public class ProductReview
{
public int ProductId { get; set; }
public string Description { get; set; }
public int ReviewerId { get; set; }
public double Score { get; set; }
}
public class Product
{
public int ProductId { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public virtual ICollection<ProductReview> Reviews { get; set; }
}
...and your ProductViewModel had an int property ReviewsCount to show in your view?
When Automapper would map and transform your entities into view model, it would access the Reviews property of each Product in the List (let's suppose, 10 in your case), one by one, and get the Reviews.Count() to fill ReviewsCount in your ProductViewModel.
Considering my example, where I never eager loaded Reviews of products, if Lazy Load was on, AutoMapper would execute ten queries (one per product) to count Reviews. Count is a fast operation and ten products are just a few. but if instead of count you were actually mapping the ProductReview to a ProductReviewViewModel, this would be kinda heavy. If Lazy Load was turned off, we would get an exception, since Reviews would be null.
One possible solution, is to eager load all child you might need during mapping, like this:
List<Product> filteredProducts = dbContext.Products.Include("Reviews").OrderBy(p => p.ProductId).ToPagedList(pageNumber, 10);
return Mapper.Map<List<Product>, List<ProductViewModel>>(filteredProducts);
...so 10 products and their reviews would be retrieved in just one query, and no other queries would be executed by AutoMapper.
But.......I just need a count, do I really need to retrieve ALL Reviews just to avoid multiple queries?
Isn't it also heavy to load all reviews and all their expensive fields like Description which may have thousands of characters???
Yes, absolutely. Avoid mixing PagedList with AutoMapper for these scenarios.
Just do a projection like this:
List<Product> filteredProducts = dbContext.Products
.Select(p => new ProductViewModel
{
ProductId = p.ProductId,
ProductName = p.Name,
ProductDescription = p.Description,
ReviewsCount = p.Reviews.Count(),
ScoreAverage = p.Reviews.Select(r => r.Score).DefaultIfEmpty().Average()
})
.OrderBy(p => p.ProductId).ToPagedList(pageNumber, 10);
Now you are loading your 10 products, projecting them into ProductViewModel, calculating the Reviews count and score average, without retrieving all Reviews from database.
Of course there are scenarios where you might really need all child entities loaded/materialized, but other than that, projection ftw.
You can also put the Select() part inside an extension class, and encapsulate all your projections in extension methods, so you can reuse them like you would to with AutoMapper.
I'm not saying AutoMapper is evil and you shouldn't use it, I use it myself in some situations, you just need to use it when it's appropriate.
EDIT: AUTOMAPPER DOES SUPPORT PROJECTION
I found this question where #GertArnold explains the following about AutoMapper:
...the code base which added support for projections that get translated
into expressions and, finally, SQL
So be happy, just follow his suggestion.
Loading related object in MVC can be pretty confusing.
There are lots of terms you need to be aware of and learn if you really want to know what you're doing when writing your entity model classes and controllers.
A couple of questions that I've had for a very long time are: How does the virtual keyword work and when should I use it? And how does the Include extension method work and when should I use it?
Here's what I'm talking about;
virtual keyword:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
namespace LazyLoading.Models
{
public class Brand
{
public int BrandId { get; set; }
public string Name { get; set; }
public int ManufacturerId { get; set; }
public virtual Manufacturer Manufacturer { get; set; }
}
}
And the Include extension method:
using System;
using System.Collections.Generic;
using System.Data;
using System.Data.Entity;
using System.Linq;
using System.Web;
using System.Web.Mvc;
using LazyLoading.Models;
namespace LazyLoading.Controllers
{
public class LazyLoadingStoreController : Controller
{
private UsersContext db = new UsersContext();
//
// GET: /LazyLoadingStore/
public ActionResult Index()
{
var brands = db.Brands.Include(b => b.Manufacturer);
return View(brands.ToList());
}
... Other action methods ...
Notice that the Index() action method is autogenerated by Visual Studio. Yes, Visual Studio automatically added the .Include(b => b.Manufacturer). That's pretty nice.
Note: It took me too long to write this answer to just discard it when two other appeared ...
Virtual keyword works together with two properties DbContext.Configuration:
ProxyCreationEnabled - allows EF crating dynamic proxy when the object is created by the EF
LazyLoadingEnabled - allows dynamic proxy to load related entities when the navigation property is used for the first time
Lazy loading is transparently implemented through dynamic proxy. Dynamic proxy is class derived from your entity which is created and compiled by EF at runtime. It overrides your virtual navigation properties and implements logic checking if related entities were already loaded or not. If not it triggers a loading on the context (issues a new query to the database). Lazy loading can be executed only in the scope of the context to which the entity is attached - if you dispose the context you cannot use it.
Dynamic proxy creation is controlled by the first mentioned property. Once the entity instance is created as proxied you cannot "remove" the proxy. Also if you create entity without proxy (for example by calling constructor yourselves) you cannot add it later (but you can use DbSet<T>.Create instead of constructor to get the proxied instance).
The second property can be changed during live time of your entity instances so you can avoid unnecessary queries to the database when working with your entities by changing it to false (it is sometimes very useful).
Include represents eager loading. Eager loading loads related entities together with the main entity and it is executed as part of the main entity's query (it adds SQL joins to the query and builds a big result sets).
Benefit of eager loading is to get all data upfront with one roundtrip to the database. Especially if you know that you will need all of them it can be a way to go. Disadvantage of eager loading are very big result sets if you use too many includes as well as some limitations (you cannot add ordering or filtering to loaded entities - it always loads all related objects).
Benefit of lazy loading is to load data only when you really need them. It is helpful if you don't know upfront if you will really need them. Disadvantages are additional queries generated by EF in some scenarios when you don't expect them (any first access to the property will trigger the lazy loading - even Count on navigation collection will trigger loading of all data to be able to do count in your application instead of querying count from the database - this is called extra lazy loading and it is not natively supported by EF yet). Another big issue is N+1 problem. If you load several brands and you will access their manufacturer property by looping through all loaded brands without using eager loading, you will generate N+1 queries to database (where N is number of brands) - one for loading all brands and one for a manufacturer of each brand.
There is another option called explicit loading. It is like lazy loading but it is not transparently executed for you. You must execute it yourselves by using context class:
context.Entry(brand).Reference(b => b.Manufacturer).Load();
It is not very useful in this case but it would be useful if you have Brands navigation property on the Manufacturer class because you can do this:
var dataQuery = context.Entry(manufacturer).Collection(m => m.Brands).Query();
Now you have a IQueryable<Brand> instance and you can add any condition, ordering or even additional eager loading and execute it against the database.
I created a test MVC4 internet application.
Here's what I found:
First, create your entity model classes - notice the virtual keyword for the Manufacturer property:
public class Manufacturer
{
public int ManufacturerId { get; set; }
public string Name { get; set; }
public ICollection<Brand> Brands { get; set; }
}
public class Brand
{
public int BrandId { get; set; }
public string Name { get; set; }
public int ManufacturerId { get; set; }
public virtual Manufacturer Manufacturer { get; set; }
}
Next, create your controller - I created (autogenerated using create new controller dialog) mine with CRUD action methods and views. Notice the Include extension method that was autogenerated automatically by Visual Studio thanks to the relationship in you Brand model class.
public class LazyLoadingStoreController : Controller
{
private UsersContext db = new UsersContext();
//
// GET: /LazyLoadingStore/
public ActionResult Index()
{
var brands = db.Brands.Include(b => b.Manufacturer);
return View(brands.ToList());
}
Let's remove the Include part for now so that our action method looks like this:
public ActionResult Index()
{
var brands = db.Brands;
return View(brands.ToList());
}
And this is how the Index view will look in Page Inspector after adding a couple of Brand objects - notice that Visual Studio automatically added the dropdown for Manufacturer and how it automatically scaffolded the Name column for Manufacturer - sweet!:
The Create action method:
//
// GET: /LazyLoadingStore/Create
public ActionResult Create()
{
ViewBag.ManufacturerId = new SelectList(db.Manufacturers, "ManufacturerId", "Name");
return View();
}
Awesome. Everything was autogenerated for us!
Now, what happens if we remove the virtual keyword from our Manufacturer property?
public class Brand
{
public int BrandId { get; set; }
public string Name { get; set; }
public int ManufacturerId { get; set; }
public Manufacturer Manufacturer { get; set; }
}
This is what will happen - our Manufacturer data is gone:
Okay, makes sense. What if I add back the Include extension method (with virtual still removed from the Manufacturer property)?
public ActionResult Index()
{
var brands = db.Brands.Include(b => b.Manufacturer);
return View(brands.ToList());
}
This is the result from adding back the Include extension method - The Manufacturer data is back!:
So that's how all that stuff work.
Next thing would be to explain what kind of T-SQL that gets generated behind the scenes in both cases (Lazy loading and Eager loading). That I'll leave to someone else. :)
Note: Visual Studio automatically generates Include(b => b.Manufacturer) wheather you add the virtual keyword or not.
Note2: Oh, yeah. Almost forgot. Here are some links to some good Microsoft resources.
http://msdn.microsoft.com/en-us/data/jj574232.aspx
http://msdn.microsoft.com/en-us/library/vstudio/bb896272(v=vs.100).aspx
The second link talks about performance considerations which the other link lacks if that is something that gets you going.
Lazy loading
Brand is a POCO (plain old CLR object). It is persistence ignorant. In other words: it does not know it is created by an Entity Framework data layer. It knows even less how to load its Manufacturer.
Still, when you do this
var brand = db.Brands.Find(1);
var manufacturer = brand.Manufacturer;
the Manufacturer is loaded on the fly ("lazily"). If you monitor the SQL that's sent to the database you'll see that a second query is emitted to fetch the Manufacturer.
That is because under the hood, EF does not create a Brand instance, but a derived type, a proxy, that's stuffed with wiring to execute lazy loading. That's why the virtual modifier is required for lazy loading to be enabled: the proxy must be able to override it.
Lazy loading is typically used in smart client applications where a context has a relatively long life span (e.g. context per form). Although I must say that even in smart client applications using short-lived contexts is beneficial and perfectly possible.
Eager loading
Eager loading means that you load an object with adhering objects (parents and/or children) in one take. That's what the Include method is for. In the example
db.Brands.Include(b => b.Manufacturer)
you'll see that EF creates a SQL query with a join and that accessing a Brand's Manufacturer does not spawn a separate query any more.
Eager loading is the way to go in most situations (esp. in disconnected scenarios), because the recommended way to deal with context instances is to use them and dispose them for each unit of work. Thus, lazy loading is not an option, because for a navigation property to be lazy loaded the context must be alive.
I want to pass around an EF entity in a property of an View Model, but I am worried that it could incur performance issues since some of the column values of the DB records are big. I then realised that unless I access those particular fields then I would not incur that penalty due to lazy loading. Is this correct?
So to elaborate, in the below example the book property may be called ABSTRACT which is a STRING which is VARCHAR(MAX) in the DB. It may also have ID, AUTHOR and TITLE:
public class vmTest
{
public Book MyBook { get; set; }
}
If I only wished to access ID and TITLE then if my theory is correct then the ABSTRACT column will not be loaded into RAM and therefore cause no performance issues.
Many thanks in advance for any help.
As far as I am aware, the moment you ask for MyBook, all ints, strings will be loaded from the database. Lazy loading works only with referenced entities.
This actually the ideal situation for ViewModel. Create ViewModel and load it only with data you need. So in controller:
var toView = context.Books.Select(e => new YourViewModel { BookID = e.ID, Title = e.Title}).SingleOrDefault(vm => YourFilter(vm));
public class YourViewModel
{
public int BookID { get; set;}
public string Title { get; set;}
}
Doing so, you load only data you need as the select statement will do appropriate projection in db.
I am using Code First to automatically generate my database, and this works perfectly, generating an Orders table and an OrderLines table as expected when I add some test data.
I have the following Order class:
public class Order
{
public int OrderID { get; set; }
public void AddItem(string productCode, int quantity)
{
var existingLine = OrderLines.FirstOrDefault(x => x.ProductOption.ProductCode == item.ProductCode);
if (existingLine == null)
OrderLines.Add(new OrderLine { ProductOption = item, Quantity = quantity });
else
existingLine.Quantity += quantity;
}
public void RemoveItem(string productCode)
{
OrderLines.Remove(OrderLines.Where(x => x.ProductOption.ProductCode == productCode).FirstOrDefault());
}
public virtual ICollection<OrderLine> OrderLines { get; set; }
public Order()
{
OrderLines = new List<OrderLine>();
}
}
What I really want is to encapsulate the OrderLines collection, making it impossible for consumers of the class to directly add and remove items to/from it (using the Add / Remove methods of ICollection) and instead forcing them to use my custom AddItem and RemoveItem methods.
Normally I could just make the collection private, but I can't do that because it needs to be virtual for EF to correctly create the OrderLines table/foreign keys.
This answer seems to suggest that making the property internal would do the trick, but I tried, and in that case no OrderLines table is created.
Is there any way that this can be accomplished, or should I have designed this differently somehow? Any help much appreciated!
Update
After a bit more searching, I found this question which is rather more clearly stated than mine; however, it's still unanswered. The poster does link to this post which seems to suggest it can't really be done in the way I'm thinking of, but does anyone have any more up-to-date information?
I don't know if it's possible to do what you are asking or not, but I'm not sure it's the best design. The problem that I am seeing is you are firmly integrating your business logic into your business entities, and I think this will turn into confusion down the road.
Take the following scenario under consideration. Say you have a new requirement where you want users to be able to remove all items from an order. The only way to do it with your entity is to create a new RemoveAllItems() method to your Order class which does that. Now say you have a new requirement to Remove all items from an order that are in a specific category. That then means that you have to add yet another method.
This causes really bloated classes, and there is one major issue you will come up with. If you (or another developer) want to look at an entity and determine it's data structure, you can't at a glance because it's so intertwined with business logic.
What I would suggest is that you keep your entities as pure data structures, keeping all their relationships public. Then you need to create a service layer, which can consist of small or big classes (however you want to organize them) that actually perform the business functions. So for example, you can have a OrderItemService class, which has methods for adding, editing, and removing items from an order. All your business logic is performed in this class, and you just have to enforce that only service classes are allowed to interact with db entities.
Now, if you are looking for how a particular business process is performed, you know to look in the service layer classes, and if you want to look at how a data structure or entity is organized, you look at the entity. This keeps everything clean and very mantainable.
I'm far from an expert on code first and I haven't tried the following but is it possible to use the ReadOnlyCollectionBase and create a read only list similar to this MSDN article?
Well what you can do is set your collection as private and make the relationship using fluent API in the OnModelCreating, as shown below, I don't know if this will work, just make a try:
public class YourContext : DbContext
{
public DbSet<Order> Orders { get; set; }
public DbSet<OrderLine> OrderLines { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Order>()
.HasMany(o => o.OrderLines)
.WithRequired(l => l.OrderId)
.HasForeignKey(l => l.OrderId);
}
}
This will make your OrderLines as readonly:
public class YourContext : DbContext
{
public DbSet<Order> Orders { get; set; }
public DbSet<OrderLine> OrderLines
{
get { return set<OrderLine>(); }
}
}
I hope this can help you, please take a look a this blog post: EF Feature CTP5: Fluent API Samples