I'm retrieving MandatesHistory collection on the Mandate Entity like this:
var query = breeze.EntityQuery.from("MandatesHistory")
.where("Mandate.Id", "==", mandatId)
.expand("Mandate").skip(offset).take(pageSize).inlineCount(true);
return manager.executeQuery(query.using(service));
It works great the first time and the collection is populated with 10 records (that's the value of pageSize).
But then the second time, the next 10 records are added to the collection instead of replacing the previous 10. So the collection keeps growing as I scroll through pages.
Obviously my pagination no longer works either, because it only expects 10 records per page, that is 10 records in the collection at any given time.
Is that expected behaviour ? If I specify skip and take, I'd rather that the collection gets overwritten after each query.
Not entirely sure I understand the question. I'm not sure what 'collection' you are referring to, but I think part of the issue is a misunderstanding about the results of a query vs the contents of the EntityManager's cache. When you execute a query with skip and take, the results returned by the query, typically in the then... method, will in fact be what you want. i.e. in your case exactly 10 records. ( assuming a pageSize of 10).
var query = breeze.EntityQuery.from("MandatesHistory")
.where("Mandate.Id", "==", mandatId)
.expand("Mandate").skip(offset).take(pageSize).inlineCount(true);
manager.executeQuery(query.using(service)).then(data) {
var next10items = data.results; // this will always be 10 ( or less) records.
});
The EntityManager's cache on the other hand will grow with each 'new' page of data that it sees. But in general you shouldn't care about this. You should just be working with the results of the query each time.
Does this make sense?
Related
I want to query some objects from the database using a WHERE clause similar to the following:
#monuments = Monument.where("... lots of SQL ...").limit(6)
Later on, in my view I use methods like #monuments.first, then I loop through #monuments, then I display #monuments.count.
When I look at the Rails console, I see that Rails queries the database multiple times, first with a limit of 1 (for #monuments.first), then with a limit of 6 (for looping through all of them), and finally it issues a count() query.
How can I tell ActiveRecord to only execute the query once? Just executing the query once with a limit of 6 should be enough to get all the data I need. Since the query is slow (80ms), repeating it costs a lot of time.
In your situation you'll want to trigger the query before you your call to first because while first is a method on Array, it's also a “finder method” on ActiveRecord objects that'll fetch the first record.
You can prompt this with any method that requires data to work with. I prefer using to_a since it's clear that we'll be dealing with an array after:
#moments = Moment.where(foo: true).to_a
# SQL Query Executed
#moments.first #=> (Array#first) <Moment #foo=true>
#moments.count #=> (Array#count) 42
In this case, you can also use first(6) in place of limit(6), which will also trigger the query. It may be less obvious to another developer on your team that this is intentional, however.
AFAIK, #monuments.first should not hit the db, I confirmed it on my console, maybe you have multiple instance with same variable or you are doing something else(which you haven't shared here), share the exact code and query and we might debug.
Since, ActiveRecord Collections acts as array, you can use array analogies to avoid querying the db.
Regarding first you can do,
#monuments[0]
Regarding the count, yes, it is a different query which hits the db, to avoid it you can use length as..
#monuments.length
In Breeze.js inlineCount(http://www.breezejs.com/sites/all/apidocs/classes/EntityQuery.html#method_inlineCount) implements the odata inlinecount parameter (http://www.odata.org/documentation/uri-conventions#InlinecountSystemQueryOption)
so it shows the total count after filtering but without paging.
But is there an "inline" way in either Breeze or odata to also get the full count without filtering in a single breeze query? i.e. the count of all records that would have been returned without the specified filtering. I know I could do a seperate query for this, but was hoping for a way to do it in a single operation.
Apologies if this is a silly question; I'm still getting up to speed with odata and Breeze. The reason I ask is I'm using datatables.net and it expects both the total count, filtered count and number of records so it can display something like
Showing 1 to 10 of 48 entries (filtered from 148 total entries)
I cannot come up with an easy way to do this with a single query.
If I understand your question, you want basically three counts for each query you submit
The count of the query results actually returned - ( easily
obtained via 'results.length' )
The count of the query results that would have been returned
without any skip or take ( obtained via the ".inlinecount()"
method on the query.
The count of the query results that would have been returned
without any filter or skip or take. i.e. the count of the entire
"resource". ( This one is difficult without a separate query).
If this is correct, then I think that you are stuck with performing two queries, one to get the count for the entire "resource" (i.e. #3 above) and another using the "inlinecount" method for #1 and #2
I've got a coreData with sqlite backend of a table with two NSDate columns: eventStart and eventEnd. I would like to perform quite a complicated select and sort on it.
For one of the main displays in the application i'd like to retrieve two things:
10 events who's duration (eventEnd - eventStart) was smaller than specified value
10 events who's duration was larger than the specified value
The events have to be sorted correctly based on how far from the specified value they are
Two problems i've hit straight away is I can't find a way select a column from an expression (the date calculation). The second was NSSortDescriptor only seems to work on columns, not expressions. This is contrary to how SQLite works, and i'm wondering if it would be easier to just break out the raw SQL.
I should mention that the data i'm going to be working with will be too large to fit into memory for things like sorting, especially since because the sort is on an expression the query would have to return all data for sorting.
If you fetch the objects first, you should then be able to sort the result in memory using any key you want, including a method that returns the interval you mentioned. So, you'd create an NSSortDescriptor using the method name that returns this time interval, create a new array with it, then simply call
[originalFetchedArray sortedArrayUsingDescriptors:sortDescriptors];
which will return a new sorted array. If you're starting with an NSMutableArray, you can sort that in place using a similar method.
When I had a very similar problem, I just created a new column for the calculated value (in your case it will be duration). After that, the retrieval becomes trivial.
The problem with your typical rails pagination gem is that it does 2 queries: one for the page you're on and one for the total count. When you don't care about how many pages there are (e.g. in an endless scroll), that 2nd query is unnecessary (just add 1 to your LIMIT clause in the 1st query and you know if there are more or not).
Is there a gem that'll do pagination without the 2nd query? The 2nd query is expensive when applying non-indexed filters in my WHERE clause on large datasets and indexing all my various filters is unacceptable because I need my inserts to be fast.
Thanks!
Figured it out. When using the will_paginate gem, you can supply your own total_entries option to AR:Base.paginate. This makes it so the 2nd query doesn't run.
This works for sufficiently large datasets where you only care about recent entries.
This isn't necessarily acceptable if you actually expect to hit the end of your list because if the list size is divisible by per_page you're going to query an empty set on your last query. With endless scroll, this is fine. With a manual "load more" button, you'll be displaying "load more" at the very end when there are no more items to load.
The standard approach, as you've identified, is to fetch N+1 records when you need N and if you get more than N records in the response, there is at least one additional page of results you can display.
The only reason you'd want to do an explicit COUNT(*) call is if you need to know specifically how many more records you will need to fetch. On some engines this can take a good chunk of time to compute so it is best avoided especially if the value is never directly used.
Since this is so simple, you really don't need a plugin to do it. Plugins like will_paginate is more concerned with the number of pages available so it does the count operation.
I am working on a ASP.NET MVC application where we have to write our own code for paging.
And I see elements getting repeated on different pages. This is happening because the order of elements in the IQueryable varies randomly and one have to query the database for each page.
I solved this problem by ordering the elements by date of creation and then the order I wanted. But I feels its heavy. Is there a better way to solve this problem.
If you fetch rows from a database via IQuerable and order on one column, if there are rows that have the same value in that column the order of those rows may vary, you need to do as you do now, that is order by two columns, first the one you are really interested in and then the second (like date), there should not be that big a performance hit since the sorting is done in the database.
But what need to do is specify the ordering like this:
Context.Products
.Where(p => p.ProductID > 100)
.OrderBy(p => p.CategoryID)
.ThenBy(p => p.Date).ToList();
Notice the ThenBy, that will generate the correct SQL.
No, that is the correct way. A database might give back rows in a non-determistic order if you do not tell it to order by something.
Have a look at: http://srtsolutions.com/blogs/billwagner/archive/2006/10/14/paging-output-with-linq2sql.aspx of Bill Wagner.
He uses Take() and Skip() to create paged output.