I have a Users table of 76 users and UserGroups table.
Using MVC, OData, a generic repository and EF, I am trying to optimize data retrieval when filtering based on the user group:
/api/Users?$filter=USERGROUPS/any(usergroup: usergroup/ID eq 'Group1')
On the client side, I get the right number of users - 71 (as OData is filtering based on the result), however I want to limit the number of records being returned form the actual query - ie. I do not want to return all records then filter (not optimal for very large data sets).
My API controller method is as follows:
[Queryable(AllowedQueryOptions = AllowedQueryOptions.All)]
public IQueryable<USER> Get()
{
var unitOfWork = new ATMS.Repository.UnitOfWork(_dbContext);
var users = unitOfWork.Repository<USER>()
.Query()
.Include(u => u.USERGROUPS)
.Get()
.OrderBy(order => order.USERNAME);
unitOfWork.Save(); // includes Dispose()
return users.AsQueryable();
}
I read in this post that:
Entity framework takes care of building dynamic query based on the
request.
However, using a SQL Server profiler, the query executed is requesting all the records, rather than a filtered query.
Adding a .Take() to the query does not accomplish the desired result, as we also need the actual number of records returned for paging purposes.
I was thinking of using the grabbing some properties through ODataQueryOptions, but that doesn't seem quite right either.
Is my implementation of Unit of Work and Repository incorrect, in relation to what I am trying to accomplish, and if so, how can this be corrected?
Simple - Just set the Page size for the Queryable atrribute [Queryable(PageSize=10)]
http://www.asp.net/web-api/overview/odata-support-in-aspnet-web-api/supporting-odata-query-options#server-paging
If You'd tell the EF where to apply the options, it would work.
Like this :
//[Queryable(AllowedQueryOptions = AllowedQueryOptions.All)]
public IQueryable<USER> Get(ODataQueryOptions<USER> options)
{
var users = options.ApplyTo(_dbContext.Set<USER>()
.Query()
.Include(u => u.USERGROUPS)
.Get()
.OrderBy(order => order.USERNAME));
return users;
}
Your code didn't work, because it tried to apply the options onto the last line "users.AsQueryable()", so what actually happened, is that EF pull the FULL dataset, and then applied the query onto the last line (that being a in memory collection). And that's why You didn't see that "filter" not being passed to the SQL.
The mechanics are such, that EF tries to apply the Query, to the IQueryable collection that it finds in the code (there's still a question how does it find the correct line).
Related
I have had to move queries off the main database in order to meet requirements for complex authorization - for example a user with a given authorization role can only view data for individuals in the same institution.
I am using the Breeze .net DocCode sample for guidance, and have copied the premise for the mapping of domain models to DTOs.
get { return ForCurrentUser(Context.Orders).Select(o => new Order {
OrderID = o.OrderID,
....
OrderDetails = o.OrderDetails.Select(od => new OrderDetail
{
ProductID = od.ProductID,
UnitPrice = od.UnitPrice
...
})
The problem is that which mapped properties to .include(entity framework method)/.expand (breeze method) is now a concern of the mapping function (for example, the above code will always return the OrderDetails collection, whether I want them or not). I would like to still only eagerly load/expand properties if the javascript client generated predicate has a .expand directive for that property.
Is this at all possible, or am I stuck with manually defining different mapping functions on the server, depending on what properties I want expanded? (I am happy to use tools such as automapper if that would solve or simplify the problem)
Thank you
You will need to use the ODataQueryOptions as a parameter to your controller method. This gives you the details of the query predicates in your server method, so that you can apply them as needed rather that having them applied automatically. This will let you expand, or not, based upon the query.
See this answer and this answer to see how it works.
This is using BreezeJS and a Breeze controller through to an EF provider. I have a couple of related Entities, lets call them Customer, which has a Navigation Property called Orders which links to a set of Order Entities for that customer.
What I'd like to display on the UI is a summary of Order Counts for a set of customers who match a partial name search. I can do this through returning all the Order objects, but they're quite large objects and I don't really want to return 100's of them when I don't have to. The inlineCount() method seems to always give the count of the top-level entity (Customer) rather than of the sub-Entities, no matter where I place it in the statement.
var predicate = breeze.Predicate.create('displayName', 'contains', partialName);
return this.entityQuery.from('Customers')
.where(predicate)
.orderBy('displayName')
.using(this.manager)
.expand('Orders')
.execute();
The documentation suggests that you can chain the expand in some way, but I have yet to find a syntax which is valid.
Ideally, I'd like to apply a where to the Orders by a property on Order called Status of say 0 (incomplete) and then give me just the count of those matching Orders. ie, return me all the Customer entities, but have a matching order count for each (rather than the whole list of Order objects and filter client-side).
Would appreciate any pointers in the right direction if it's possible to achieve. My current thinking is that I'll have to create a custom method on the server-side controller and do the work there, but before I make assumptions about what OData can support, I thought I'd check here for some confirmation.
So far, this is my best approach (maybe someone can correct me if there's a better way).
On the server, add this method:
public IQueryable<object> CustomerSummaries()
{
return Context.Customers.Select(p => new
{
Customer = p,
ActiveOrderCount = p.Orders.Count(o => o.Status == 1)
});
}
Then on the client end:
var predicate = breeze.Predicate.create('customer.displayName', 'contains', partialName);
return this.entityQuery.from('CustomerSummaries')
.where(predicate)
.using(this.manager)
.execute();
I am missing two topics in the Breeze documentation, Security and how to set limits on the data returned.
I don't want someone to delete everything in the database and don't want to return everything.
Should I use OData? with Odata I can set the restrictions I want.
"Security" is a huge, sprawling topic about which there is never enough to say. I hope to chip away at it.
As for limiting the max records, I'd use the [Queryable] attribute's PageSize option. The [BreezeQueryable] attribute inherits this option so you could write your Web API controller method or supporting repository method like this:
[HttpGet]
[BreezeQueryable(PageSize=100)] // returns a maximum of 100 orders
public IQueryable Orders() {
return ContextProvider.Context.Orders;
}
You might also be able to limit the results by adding top to the query inside the method.
[HttpGet]
public IQueryable Orders() {
return ContextProvider.Context.Orders.take(100);
}
However, certain query conditions may not get through. Try it first.
Getting an error client side with breeze: "Cannot call method 'map' of undefined" when trying to pull over some data. The difference between this action and one that works is that this action is calling a stored procedure and returning ObjectResult<T> instead of DbSet<T>.
Might this be why I get an error? Using Chrome Developer tools, I do see that the breeze controller is returning json data.
I have created a complex model type in the edmx for mapping the rows returned from the stored procedure.
The action in the breeze controller has a return type of IEnumerable<T>.
I experienced the same error when using an EF complex type. A workaround was to create a view in my database instead of using a complex type, set the stored procedure to return a type of the new view which had a primary key and then it worked. It would seem that breeze requires entities to have a primary key defined.
Hm... not quite sure what is happening, so just guessing here, but try adding an AsQueryable() to the result returned, and changing the result type to a IQueryable.
We don't have any stored proc tests for breeze yet, but this is impetus for me to add some :)
I had the very same issue, but thank God I figured out a solution. Instead of using a stored procedure, you should use a view, as Breeze recognizes views as DbSet<T>, just like tables. Say you have a SQL server table that contains two tables Customers and Orders.
Customers (**CustomerId**, FirstName, LastName)
Orders (OrderId, #CustomerId, OrderDate, OrderTotal)
Now, say you want a query that returns orders by CustomerId. Usually, you would do that in a stored procedure, but as I said, you need to use a view instead. So the query will look like this in the view.
Select o.OrderId, c.CustomerId, o.OrderDate, o.OrderTotal
from dbo.Orders o inner join dbo.Customers c on c.CustomerId = o.CustomerId
Notice there is no filtering (where ...). So:
i. Create a [general] view that includes the filtering key(s) and name it, say, OrdersByCustomers
ii. Add the OrdersByCustomers view to the entity model in your VS project
iii. Add the entity to the Breeze controller, as such:
public IQueryable<OrdersByCustomers> OrdersByCustomerId(int id)
{
return _contextProvider.Context.OrdersByCustomers
.Where(r => r.CustomerId == id);
}
Notice the .Where(r => r.CustomerId == id) filter. We could do it in the data service file, but because we want the user to see only his personal data, we need to filter from the server so it only returns his data.
iv. Now, that the entity is set in the controller, you may invoke it in the data service file, as such:
var getOrdersByCustomerId = function(orderObservable, id)
{
var query = breeze.EntityQuery.from('OrdersByCustomerId')
.WithParameters({ CustomerId: id });
return manager.executeQuery(query)
.then(function(data) {
if (orderObservable) orderObservable(data.results);
}
.fail(function(e) {
logError('Retrieve Data Failed');
}
}
v. You probably know what to do next from here.
Hope it helps.
I have an ASP.NET MVC website. In my backend I have a table called People with the following columns:
ID
Name
Age
Location
... (a number of other cols)
I have a generic web page that uses model binding to query this data. Here is my controller action:
public ActionResult GetData(FilterParams filterParams)
{
return View(_dataAccess.Retrieve(filterParams.Name, filterParams.Age, filterParams.location, . . .)
}
which maps onto something like this:
http://www.mysite.com/MyController/GetData?Name=Bill .. .
The dataAccess layer simply checks each parameter to see if its populated to add to the db where clause. This works great.
I now want to be able to store a user's filtered queries and I am trying to figure out the best way to store a specific filter. As some of the filters only have one param in the queryString while others have 10+ fields in the filter I can't figure out the most elegant way to storing this query "filter info" into my database.
Options I can think of are:
Have a complete replicate of the table (with some extra cols) but call it PeopleFilterQueries and populate in each record a FilterName and put the value of the filter in each of field (Name, etc)
Store a table with just FilterName and a string where I store the actual querystring Name=Bill&Location=NewYork. This way I won't have to keep adding new columns if the filters change or grow.
What is the best practice for this situation?
If the purpose is to save a list of recently used filters, I would serialise the complete FilterParams object into an XML field/column after the model binding has occurred. By saving it into a XML field you're also giving yourself the flexibility to use XQuery and DML should the need arise at a later date for more performance focused querying of the information.
public ActionResult GetData(FilterParams filterParams)
{
// Peform action to get the information from your data access layer here
var someData = _dataAccess.Retrieve(filterParams.Name, filterParams.Age, filterParams.location, . . .);
// Save the search that was used to retrieve later here
_dataAccess.SaveFilter(filterParams);
return View(someData);
}
And then in your DataAccess Class you'll want to have two Methods, one for saving and one for retrieving the filters:
public void SaveFilter(FilterParams filterParams){
var ser = new System.Xml.Serialization.XmlSerializer(typeof(FilterParams));
using (var stream = new StringWriter())
{
// serialise to the stream
ser.Serialize(stream, filterParams);
}
//Add new database entry here, with a serialised string created from the FilterParams obj
someDBClass.SaveFilterToDB(stream.ToString());
}
Then when you want to retrieve a saved filter, perhaps by Id:
public FilterParams GetFilter(int filterId){
//Get the XML blob from your database as a string
string filter = someDBClass.GetFilterAsString(filterId);
var ser = new System.Xml.Serialization.XmlSerializer(typeof(FilterParams));
using (var sr = new StringReader(filterParams))
{
return (FilterParams)ser.Deserialize(sr);
}
}
Remember that your FilterParams class must have a default (i.e. parameterless) constructor, and you can use the [XmlIgnore] attribute to prevent properties from being serialised into the database should you wish.
public class FilterParams{
public string Name {get;set;}
public string Age {get;set;}
[XmlIgnore]
public string PropertyYouDontWantToSerialise {get;set;}
}
Note: The SaveFilter returns Void and there is no error handling for brevity.
Rather than storing the querystring, I would serialize the FilterParams object as JSON/XML and store the result in your database.
Here's a JSON Serializer I regularly use:
using System.IO;
using System.Runtime.Serialization.Json;
using System.Text;
namespace Fabrik.Abstractions.Serialization
{
public class JsonSerializer : ISerializer<string>
{
public string Serialize<TObject>(TObject #object) {
var dc = new DataContractJsonSerializer(typeof(TObject));
using (var ms = new MemoryStream())
{
dc.WriteObject(ms, #object);
return Encoding.UTF8.GetString(ms.ToArray());
}
}
public TObject Deserialize<TObject>(string serialized) {
var dc = new DataContractJsonSerializer(typeof(TObject));
using (var ms = new MemoryStream(Encoding.UTF8.GetBytes(serialized)))
{
return (TObject)dc.ReadObject(ms);
}
}
}
}
You can then deserialize the object and pass it your data access code as per your example above.
You didn't mention about exact purpose of storing the filter.
If you insist to save filter into a database table, I would have following structure of the table.
FilterId
Field
FieldValue
An example table might be
FilterId Field FieldValue
1 Name Tom
1 Age 24
1 Location IL
3 Name Mike
...
The answer is much more simple than you are making it:
Essentially you should store the raw query in its own table and relate it to your People table. Don't bother storing individual filter options.
Decide on a value to store (2 options)
Store the URL Query String
This id be beneficial if you like open API-style apps, and want something you can pass nicely back and forth from the client to the server and re-use without transformation.
Serialize the Filter object as a string
This is a really nice approach if your purpose for storing these filters remains entirely server side, and you would like to keep the data closer to a class object.
Relate your People table to your Query Filters Table:
The best strategy here depends on what your intention and performance needs are. Some suggestions below:
Simple filtering (ex. 2-3 filters, 3-4 options each)
Use Many-To-Many because the number of combinations suggests that the same filter combos will be used lots of times by lots of people.
Complex filtering
Use One-To-Many as there are so many possible individual queries, it less likely they are to be reused often enough to make the extra-normalization and performance hit worth your while.
There are certainly other options but they would depend on more detailed nuances of your application. The suggestions above would work nicely if you are say, trying to keep track of "recent queries" for a user, or "user favorite" filtering options...
Personal opinion
Without knowing much more about your app, I would say (1) store the query string, and (2) use OTM related tables... if and when your app shows a need for further performance profiling or issues with refactoring filter params, then come back... but chances are, it wont.
GL.
In my opinion the best way to save the "Filter" is to have some kind of json text string with each of the "columns names"
So you will have something in the db like
Table Filters
FilterId = 5 ; FilterParams = {'age' : '>18' , ...
Json will provide a lot of capabilities, like the use of age as an array to have more than one filter to the same "column", etc.
Also json is some kind of standard, so you can use this "filters" with other db some day or to just "display" the filter or edit it in a web form. If you save the Query you will be attached to it.
Well, hope it helps!
Assuming that a nosql/object database such as Berkeley DB is out of the question, I would definitely go with option 1. Sooner or later you'll find the following requirements or others coming up:
Allow people to save their filters, label, tag, search and share them via bookmarks, tweets or whatever.
Change what a parameter means or what it does, which will require you to version your filters for backward compatibility.
Provide auto-complete functions over filters, possibly using a user's filter history to inform the auto-complete.
The above will be somewhat harder to satisfy if you do any kind of binary/string serialization where you'll need to parse the result and then process them.
If you can use a NoSql DB, then you'll get all the benefits of a sql store plus be able to model the 'arbitrary number of key/value pairs' very well.
Have thought about using Profiles. This is a build in mechanism to store user specific info. From your description of your problem its seems a fit.
Profiles In ASP.NET 2.0
I have to admit that M$ implementation is a bit dated but there is essentially nothing wrong with the approach. If you wanted to roll your own, there's quite a bit of good thinking in their API.