I am trying to filter some records from a TADOQuery. I set the filtered property to true and when I set the filter to field='value', all works fine. I would like to dynamically build this filter by appending
<space>AND field='value'
to a value always true, and I thought 1=1 would do the trick. So I would have 1=1 as the default filter and then just append AND field='value' to it as necessary.
This, however, does not work. The error message reads:
Arguments are of the wrong type, are out of acceptable range, or are in conflict with one another.
Could anyone please tell me what could I use as a versatile always-true expression for this filter?
I suppose it goes without saying, but it depends on the OLE DB provider whether or not this works. When you set a filter on an existing record set, it ends up going through a different OLE DB interface (IViewFilter if I remember correctly). So even if a filter works in a WHERE clause on an SQL statement, it does not necessarily mean that it will work as a filter. The filter that you set ends up getting parsed apart into the component pieces and then passed to the OLE DB interface. It may be that the provider's implementation is not expecting a filter of the form "constant = constant". As a workaround, you might try setting it all in the WHERE clause of the SQL statement.
You have to set the 'Filtered' property to False if you are not filtering something, and set it True and your condition when you want the resultset to be filtered.
I would dynamically build the correct SQL property though so that you always exactly know what is being send to the database (and you are sure that only those records you want is received by your program).
The 1=1 trick works fine in the where clause of a query, but not in the filtered property. If you want to disable the filter, set filtered to false and all records will be returned.
The problem with filtering is that it is done client side. If you are using a database engine such as SQL Server and expect to have a large set of records to filter, then your better served by changing the SQL Query which will allow the database server to return only the records requested. Just remember to close your TAdoQuery first, change the SQL then re-open.
A trick I use to avoid returning the entire dataset (used for large datasets) is to consider a maximum number of records I want to display, then use the TOP SQL Syntax to return one more than the number of records I wanted to display 'n' ...if I reach that number, then I notify the user that there were more than n-1 records returned and to adjust the search/filter criteria.
Related
I'd like to know the best way to retrieve the schema for a query's result WITHOUT executing the query. By Schema I want to know the name of the fields that will be returned and if they are boolean, literal or float, ...
For those familiar with SQL Server I want the same kind of behavior than using a FMTONLY.
At the moment my best approach is to perform a LIMIT 0 ... but anyway I'm nowhere to know the type of the returned fields.
What you are seeking is not generally possible.
Unlike SQL, neo4j is "schemaless", and the "schema" of a query result is generally dependent on the data in the DB. This means that, generally, a query's result schema cannot be known without actually executing the query.
So a slightly weird one that I can't find any cause for really.
My app is set up to basically run almost all queries through one standard method that handles things like querying against the local cache etc. So essentially the queries are all pretty standardised.
Then I have just one, with a strange orderby issue. The query includes a specific orderby clause, and if I run the query first time, the cache is checked, no results found, queries the remote data source, get data, all correct and ordered.
When I return to the page, the query is executed again, and the query is executed against the local cache, where it does find the data and returns it... the weird parts is the order is reversed. Bear in mind the parameters going in are exactly the same, the only difference is the query is executed with executeQueryLocally, and results are found, and returned (in the first query, it is still executed with executeQueryLocally, it's just that no results are found and it goes on to execute it remotely).
I really can't see any specific issue as to why the results are reversed (I say they are reversed, I can't actually guarantee that - they might just be unordered and happen to come out in a reversed order)
This isn't really causing a headache, it's just weird, especially as it appears to be only one query where this happens).
Thoughts?
Server side queries and client side queries are not guaranteed to return results in any specific order UNLESS you have an "orderBy" clause specified. The reason that order may be different without the "orderBy" clause is that the data is being stored very differently on the server vs the client and unless a specific order is specified both will attempt to satisfy the query as efficiently as possible given the storage implementation.
One interesting side note is that per the ANSI 92 SQL standard, even your SQL database is not required to return data in the same order for the same query ( again unless you have an ORDER BY clause). It's just that it's very rare to see it happen.
I am scanning an SQLite database looking for all matches and using
OneFound:=False;
if tbl1.FieldByName('Name').AsString = 'jones' then
begin
OneFound:=True;
tbl1.Next;
end;
if OneFound then // Do something
or should I be using
if not(OneFound) then OneFound:=True;
Is it faster to just assign "True" to OneFound no matter how many times it is assigned or should I do the comparison and only change OneFuond the first time?
I know a better way would be to use FTS3, but for now I have to scan the database and the question is more on the approach to setting OneFound as many times as a match is encountered or using the compare-approach and setting it just once.
Thanks
Your question is, which is faster:
if not(OneFound) then OneFound:=True;
or
OneFound := True;
The answer is probably that the second is faster. Conditional statements involve branches which risks branch mis-prediction.
However, that line of code is trivial compared to what is around it. Running across a database one row at a time is going to be outrageously expensive. I bet that you will not be able to measure the difference between the two options because the handling of that little Boolean is simply swamped by the rest of the code. In which case choose the more readable and simpler version.
But if you care about the performance of this code you should be asking the database to do the work, as you yourself state. Write a query to perform the work.
It would be better to change your SQL statement so that the work is done in the database. If you want to know whether there is a tuple which contains the value 'jones' in the field 'name', then a quicker query would be
with tquery.create (nil) do
begin
sql.add ('select name from tbl1 where name = :p1 limit 1');
sql.params[0].asstring:= 'jones';
open;
onefound:= not isempty;
close;
free
end;
Your syntax may vary regarding the 'limit' clause but the idea is to return only one tuple from the database which matches the 'where' statement - it doesn't matter which one.
I used a parameter to avoid problems delimiting the value.
1. Search one field
If you want to search one particular field content, using an INDEX and a SELECT will be the fastest.
SELECT * FROM MYTABLE WHERE NAME='Jones';
Do not forget to create an INDEX on the column, first!
2. Fast reading
But if you want to search within a field, or within several fields, you may have to read and check the whole content. In this case, what will be slow will be calling FieldByName() for each data row: you should better use a local TField variable.
Or forget about TDataSet, and switch to direct access to SQLite3. In fact, using DB.pas and TDataSet requires a lot of data marshalling, so is slower than a direct access.
See e.g. DiSQLite3 or our DB classes, which are very fast, but a bit of higher level. Or you can use our ORM on top of those classes. Our classes are able to read more than 500,000 rows per second from a SQLite3 database, including JSON marshalling into objects fields.
3. FTS3/FTS4
But, as you guessed, the fastest would be indeed to use the FTS3/FTS4 feature of SQlite3.
You can think of FTS4/FTS4 as a "meta-index" or a "full-text index" on supplied blob of text. Just like google is able to find a word in millions of web pages: it does not use a regular database, but full-text indexing.
In short, you create a virtual FTS3/FTS4 table in your database, then you insert in this table the whole text of your main records in the FTS TEXT field, forcing the ID field to be the one of the original data row.
Then, you will query for some words on your FTS3/FTS4 table, which will give you the matching IDs, much faster than a regular scan.
Note that our ORM has dedicated TSQLRecordFTS3 / TSQLRecordFTS4 kind of classes for direct FTS process.
I've created a WebApi project in VS 2012, using NHibernate as my ORM and I intend to enable Odata support on it. So I've created a test controller with a single Get method that returns a list of entities from a table on my database.
Everything works fine, I can use OData to filter and order my results, etc. The problem is I couldn't find a way to limit the amount of data that's being returned from the database to the controller, and this table has millions of records in it.
Using the PageSize property of the Queryable attribute only seems to be limiting the amount of data returned to the client, but no the amount of Data returned from the DB.
I've tried applying a Take(n) on the IQueryable inside the get method before returning it, and it limits the results brought back from the DB, but it breaks the OData filtering, since if you try to query an entity that's not in the first n results, it just returns an empty collection.
I know you can use the $Top parameter on OData to accomplish this, but I would like not to depend on the client/consumer providing it in order to ensure that I'm not unnecessarily bringing thousands or even million of records that I'm not going to use.
I've also tried to manually check if the client provided a Top parameter on the query string, apply the OData transformation to my Queryable and then applying the Take(n) method over the transformed query. This approach enabled me to filter for any entity through OData, but it breaks pagination, because if I use the $Skip=n parameter, it again returns an empty collection.
So, is there any way to reliably limit the results fetched from the DB while not breaking the OData support?
We recently found that too. We are not applying a Take(pageSize) when server driven paging is enabled as we have to figure out if a next page link should be generated or not. We just enumerate the result set for pageSize number of entities and check if there are more entities or not. We thought that most providers generally bring a partial set of results as IQueryable is generally a lazy implementation. Turns out that is not true. Also, the database can optimize the query if it knows only pageSize number of results are required.
This is the issue that was opened for it. Good news is Youssef fixed it already :). This is the commit that fixed it. So, if you grab the nightly builds you should be good.
This is probably a very simple question that I am working through in an MVC project. Here's an example of what I am talking about.
I have an rdml file linked to a database with a table called Users that has 500,000 rows. But I only want to find the Users who were entered on 5/7/2010. So let's say I do this in my UserRepository:
from u in db.GetUsers() where u.CreatedDate = "5/7/2010" select u
(doing this from memory so don't kill me if my syntax is a little off, it's the concept I am looking for)
Does this statement first return all 500,000 rows and then filter it or does it only bring back the filtered list?
It filters in the database since your building your expression atop of an ITable returning a IQueryable<T> data source.
Linq to SQL translates your query into SQL before sending it to the database, so only the filtered list is returned.
When the query is executed it will create SQL to return the filtered set only.
One thing to be aware of is that if you do nothing with the results of that query nothing will be queried at all.
The query will be deferred until you enumerate the result set.
These folks are right and one recommendation I would have is to monitor the queries that LinqToSql is creating. LinqToSql is a great tool but it's not perfect. I've noticed a number of little inefficiencies by monitoring the queries that it creates and tweaking it a bit where needed.
The DataContext has a "Log" property that you can work with to view the queries created. I created a simple HttpModule that outputs the DataContext's Log (formatted for sweetness) to my output window. That way I can see the SQL it used and adjust if need be. It's been worth its weight in gold.
Side note - I don't mean to be negative about the SQL that LinqToSql creates as it's very good and efficient almost every time. Another good side effect of monitoring the queries is you can show your friends that are die-hard ADO.NET - Stored Proc people how efficient LinqToSql really is.