I've created a WebApi project in VS 2012, using NHibernate as my ORM and I intend to enable Odata support on it. So I've created a test controller with a single Get method that returns a list of entities from a table on my database.
Everything works fine, I can use OData to filter and order my results, etc. The problem is I couldn't find a way to limit the amount of data that's being returned from the database to the controller, and this table has millions of records in it.
Using the PageSize property of the Queryable attribute only seems to be limiting the amount of data returned to the client, but no the amount of Data returned from the DB.
I've tried applying a Take(n) on the IQueryable inside the get method before returning it, and it limits the results brought back from the DB, but it breaks the OData filtering, since if you try to query an entity that's not in the first n results, it just returns an empty collection.
I know you can use the $Top parameter on OData to accomplish this, but I would like not to depend on the client/consumer providing it in order to ensure that I'm not unnecessarily bringing thousands or even million of records that I'm not going to use.
I've also tried to manually check if the client provided a Top parameter on the query string, apply the OData transformation to my Queryable and then applying the Take(n) method over the transformed query. This approach enabled me to filter for any entity through OData, but it breaks pagination, because if I use the $Skip=n parameter, it again returns an empty collection.
So, is there any way to reliably limit the results fetched from the DB while not breaking the OData support?
We recently found that too. We are not applying a Take(pageSize) when server driven paging is enabled as we have to figure out if a next page link should be generated or not. We just enumerate the result set for pageSize number of entities and check if there are more entities or not. We thought that most providers generally bring a partial set of results as IQueryable is generally a lazy implementation. Turns out that is not true. Also, the database can optimize the query if it knows only pageSize number of results are required.
This is the issue that was opened for it. Good news is Youssef fixed it already :). This is the commit that fixed it. So, if you grab the nightly builds you should be good.
Related
I am using TableGateways along with the Object Mapper which uses the exchangeArray() method for getting database results. This method is fine for getting data in one go.
However, I am struggling to render a table of results where I want to get another resultset in the foreach loop for each row by performing a query.
In regular PHP, I would simply write a loop for the first set of data then write another loop within the first loop that would execute a query for each record. This does not follow Zend's MVC approach so how would I achieve the same thing?
This isn't anything to do with joins.
I have a table of invoices and payments. I am outputting a table of payments and for each payment, I want to output the invoices that are tied to that specific payment. I am trying to achieve this using Zend's MVC and TableGateway patterns.
Thanks.
In regular PHP, I would simply write a loop for the first set of data then write another loop within the first loop that would execute a query for each record. This does not follow Zend's MVC approach so how would I achieve the same thing?
Well, the thing is that the MVC pattern is going to add a level of complexity to something like this and that is not necessarily a bad thing.
You can still do the same as you did before, loop through the results & fetch the associated data. However, instead of doing it as you're displaying the data, you would do it in the controller or preferably in a service or DataMapper object. This is the added level of complexity since you'll have to loop through the results twice. Once to gather the associated data & once to display it. Again, this is not necessarily a bad thing since it makes your code more maintainable by not allowing you to change what the data looks like without needing to change how it's gather or vice versa.
Alternatively you could use an ORM such as Doctrine to give you the relationship mapping in one function call (under the hood it's doing essentially the same two loops). Of course, this means you will need to learn how the ORM works and its API.
I am relatively new to oData service and I am trying to explore if oData is feasible for my project.
From all the examples / demos that I have come across,every demo always loads up all data into the repository and then oData filters are applied over the data.
Is there a way to not load up all data (apply the filters to SQL from oData) from SQL which will obviously be highly inefficient for N number of requests coming in /second ?
So for example if I had a movies service :
localhost:4502/OdataService/movies(55)
The above example is actually just filtering for movie id 55 from an "entire" set of movies.Is there a way to make this filter happen at SQL level instead of bloating the memory first with all movies and then allowing oData to filter it?
Can anyone guide me in the right direction?
I found out after doing a small POC that Entity framework takes care of building dynamic query based on the request.
My objective:
I have built a working controller action in MVC which takes user input for various filter criteria and, using PredicateBuilder (part of LinqKit - sorry, I'm not allowed enough links yet) builds the appropriate LINQ query to return rows from a "master" table in SQL with a couple hundred thousand records. My implementation of the predicates is totally inelegant, as I'm new to a lot of this, and under a very tight deadline, but it did make life easier. The page operates perfectly as-is.
To this, I need to add a Full-Text search filter. Understanding the way LINQ translates Contains to LIKE(%%), using the advice in Simon Blog: LINQ-to-SQL - Enabling Full-Text Searching, I've already prepared Table Functions in SQL to run Freetext queries on the relevant columns. I have 4 functions, to match the query against 4 separate tables.
My approach:
At the moment, I'm building the predicates (I'll spare you) for the initial IQueryable data object, running a LINQ command to return them, like so:
var MyData = DB.Master_Items.Where(outer);
Then, I'm attempting to further filter MyData on the Keys returned by my full-text search functions:
var FTS_Matches_Subtable_1 = (from tbl in DB.Subtable_1
join fts in DB.udf_Subtable_1_FTSearch(KeywordTerms)
on tbl.ID equals fts.ID
select tbl.ForeignKey);
... I have 4 of those sets of matches which I've tried to use to filter my original dataset in several ways with no success. For instance:
MyNewData = MyData.Where(d => FTS_Matches_Subtable_1.Contains(d.Key) ||
FTS_Matches_Subtable_2.Contains(d.Key) ||
FTS_Matches_Subtable_3.Contains(d.Key) ||
FTS_Matches_Subtable_4.Contains(d.Key));
I just get the error: The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Too many parameters were provided in this RPC request. The maximum is 2100.
I get that it's because I'm trying to pass a relatively large set of data into the Contains function and LINQ is converting each record into a separate parameter, exceeding the limit.
I just don't know how to get around it.
I found another post linq expression to return property value which seemed SO promising. I tried ifwdev's solution (2nd highest ranked answer): using LinqKit to build an extension that will break up the queries into manageable chunks. But I can't figure out how to implement it. Out of my depth right now maybe?
Is there another approach that I'm missing? Some simpler way to accomplish this that I've overlooked?
Sorry for the long post. But thank you for any help you can provide!
This is a perfect time to go back to raw ado.net.
Twisting things around just to use linq to sql is probably just as time consuming if you wrote the query and hydration by hand.
This is probably a very simple question that I am working through in an MVC project. Here's an example of what I am talking about.
I have an rdml file linked to a database with a table called Users that has 500,000 rows. But I only want to find the Users who were entered on 5/7/2010. So let's say I do this in my UserRepository:
from u in db.GetUsers() where u.CreatedDate = "5/7/2010" select u
(doing this from memory so don't kill me if my syntax is a little off, it's the concept I am looking for)
Does this statement first return all 500,000 rows and then filter it or does it only bring back the filtered list?
It filters in the database since your building your expression atop of an ITable returning a IQueryable<T> data source.
Linq to SQL translates your query into SQL before sending it to the database, so only the filtered list is returned.
When the query is executed it will create SQL to return the filtered set only.
One thing to be aware of is that if you do nothing with the results of that query nothing will be queried at all.
The query will be deferred until you enumerate the result set.
These folks are right and one recommendation I would have is to monitor the queries that LinqToSql is creating. LinqToSql is a great tool but it's not perfect. I've noticed a number of little inefficiencies by monitoring the queries that it creates and tweaking it a bit where needed.
The DataContext has a "Log" property that you can work with to view the queries created. I created a simple HttpModule that outputs the DataContext's Log (formatted for sweetness) to my output window. That way I can see the SQL it used and adjust if need be. It's been worth its weight in gold.
Side note - I don't mean to be negative about the SQL that LinqToSql creates as it's very good and efficient almost every time. Another good side effect of monitoring the queries is you can show your friends that are die-hard ADO.NET - Stored Proc people how efficient LinqToSql really is.
I have silverlight 3.0 project that has a listbox that is databound to a list of items. What I want to do is limit the number of items displayed in the listbox to be <= 10. I originally accomplished this by limiting the data bound to the list to 10 items by doing a .Take(10) on my orignal data and databinding the result.
The problem w/ the .Take(10) approach is that the original datasource may change and since .Take() returns a reference (or copy not sure) of the original data I sometimes do not see changes in the data reflected in my UI.
I'm trying to figure out a better way of handling this rather than the .Take() approach. It seems you shouldn't 'filter' your data using LINQ functions if you have more than one UI element bound to the same data. My only thought on how to do this better is to make a custom container that will limit the count, but that seems like it might be a mountain of work to make a custom stackpanel or equivalent.
Take(10) does not make a copy, it just appends another step to the LINQ query. But all execution is still deferred till someone pulls the items of the query.
If you were setting the items statically, a copy would indeed be created, by running the query once. But since you set the constructed query as the ItemsSource property of the list box, it can run and update it any time, so it is the right approach.
The real reason why you sometimes do not see changes in the data reflected in the UI is that the list box has no way to determine why the data returned by the query have changed and it surely doesn't want to keep constantly trying to refetch the data and maybe update itself. You need to let it know.
How can you let it know? The documentation for ItemsSource says that "you should set the ItemsSource to an object that implements the INotifyCollectionChanged interface so that changes in the collection will be reflected (...).". Apparently the default way of doing things by .Net itself does not work in your case.
So there are some examples how to implement that yourself e.g. in this SO answer. If even the top-level source data collection (over which you are doing the LINQ query) does not support these notifications (which you would just forward), you might need to update the list box manually from your other code which changes the underlying data.