I'm working with ADO connecting to SQL Server 2005.
My TADODataSet selects 1 million records. using a TDBGrid and setting the TADODataSet.CursorLocation to clUseServer works. but the TDBGrid chokes!
How can I select 1 million records, avoid paging, and still be able to display records in the grid without fetching ALL records to the client side, letting the Grid read ahead as I scroll up and down?
SQL enterprise manager can execute a query and select 1 million records asynchronously without any problems (also MS-ACCESS).
TGrid is not your problem. Your problem is TADODataset is trying to load all the records. If you must run a query that returns so many records, you should set ExecuteOptions, try eoAsyncExecute and eoAsyncFetch. It may also help to set CacheSize.
Why do you need to fetch 1M records into a grid? No human being can look at so many records. Usually is far better to reduce the number of records before loading them into a UI.
If you have a good reason to show so many records into a grid, you'd need a dataset that 1) doesn't load the whole recordset when opened 2) doesn't cache previous record or it could run out of memory (under 32 bit Windows especially) far before reaching the end of the recordset if the record size is not small enough. To obtain such result beyond CursorLocation you have to set CursorType and CacheSize properly.
You can use a TClientDataset to implement incremental fetching setting the ADO dataset CursorType to ForwardOnly and CacheSize to a suitable value. Because TClientDataset caches read records, you want to avoid the source dataset to load all of them as well. A standard DB grid needs a bidirectional cursor, thereby it won't work with an unidirectional one. With so many records the client dataset cache can exhaust memory anyway. I'd suggest to use the Midas Speed Fix unit if you're using a Delphi version before 2010.
To avoid "out of memory" errors you may need to implement some kind of pagination. Anyway, check if the behaviour of other CursorType can help you.
You can try AnyDAC and TADTable. Its Live Data Window mode solves your and similar issues. The benefits are:
minimizes memory usage and allows to work with large data volumes, similar to an unidirectional dataset;
enables bi-directional navigation, in contrast to an unidirectional dataset;
gives always fresh data, reducing the need to refresh dataset;
does not delay to fetch all result set records, required to perform sorting, record location, jumping to last record, etc.
Related
For some reason, I need to use FDTable in a Delphi Project to Fetch a large number of records (Interbase Database), unfortunately, to open the FDTable takes too much time (up to 2min and sometimes more) even worse when to ApplyUpdate, I tried everything possible by changing the fetch options: Recsmax, Rowsize, Mode, etc. as mention on some pages, Like: https://docwiki.embarcadero.com/RADStudio/Sydney/en/Fetching_Rows_(FireDAC)
Set the RecsMax Option to a small value (50 or 100) helps a lot with the performance but it will not fetch 1 record with Filter applied even with FetchAll.
As I mention before I need to do this with FDtable, FDQuery is not an option as we all know dealing with queries is better.
Is there a recommendation to smoothly open and fetch the data (100k+ records)?
It's Possible to fetch records with Filter + RecsMax?
The query table must have Primary key.
You can configure TFDTable as follows
FDTable1.FetchOptions.Items := [fiMeta]; // at least
FDTable1.FetchOptions.Unidirectional := False;
FDTable1.FetchOptions.CursorKind := ckAutomatic; // or ckDynamic
FDTable1.CachedUpdates := False;
// LiveWindowParanoic
// When it is false there are problems with Locate and Recno
FDTable1.FetchOptions.LiveWindowParanoic := False // Play with True/False.
This is the configuration for the best performance
FDTable component is there mainly for BDE compatibility.
FDQuery with "select * from Table1" is exactly the same thing as using FDTable, but you can filter resultset on server side.
100k+ records is a lot of data to transfer, blobs add additional overhead, as they're usually fetched separately.
I suggest you rethink the functionality design.
I'm usign the dbExpress components within the Embarcadero C++Builder XE environment.
I have a relatively large table with something between 20k and 100k records, which I display in a DBGrid.
I am using a DataSetProvider, which is connected to a SQLQuery and a ClientDataSet, which is connected to the DataSetProvider.
I also need to analyze the data and therefore I need to run through the whole table. For smaller tables I always used code, which is basically something like this:
Form1->ClientDataSet1->First();
while(!Form1->ClientDataSet1->Eof){
temp=Form1->ClientDataSet1->FieldByName("FailReason")->AsLargeInt;
//do something with temp
Form1->ClientDataSet1->Next();
}
Of course this works out, but it is very slow, when I need to run through the whole DBGrid. For some 50000 records in can take up to some minutes. My suspicion is that the most perform is lost since the DBGrid needs to be repainted as the actual Dataset increments its address.
Therefore I am looking for a method which allows me to either read the data without manipulating the actual ClientDataSet. Maybe a method which copies the data of a column into a variable, or another way to run through the datasets, which is more efficient. I am sure if I would have a copy in a variable the operation would take less than a few seconds...
I googled now for hours, but didn't find anything useful so far.
Best regards,
Bodo
if your cds is connected to some db-aware control(-s) (via TDataSource) then first of all consider using DisableControls()
another option would be to avoid utilizing FieldByName within the loop
Well, I'm studying the "packetRecord" property (TClientDataSet) and i have a doubt about it. I will explain how i think this works and if i'm wrong, correct me please.
1 - If i configure th packetRecord = 50, when i do "SELECT * FROM history", if table history have 200k of rows, the TClientDataSet will do something like this: "SELECT * FROM history limit 50", so when i need more 50 rows the ClientDataSet will search more 50 in Database.
The property packetRecord just make senses if TClientDataSet don't get all rows in Database, at least for me.
Am i correct ?
It will probably execute the entire query and ask to return just 50 records, but that is an implementation detail that I think is not chosen by the ClientDataSet but rather by the provider, the dataset or the driver.
But in general, that is more or less how it works, yes.
Did some browsing through the code. If the ClientDataSet is linked to a (local) TDataSetProvider, that provider just opens the dataset it is connected to. After opening it, it sets the DataSet.BlockReadSize property to the number of records to retrieve (=packetRecords).
So in the end it comes down on the implementation of BlockReadSize and the dsBlockRead state of the TDataSet that is used.
With a client-server setup this must be the same thing. In fact, there doesn't even have to be a dataset or even a database. There's also an TXMLTransformProvider, and people could implement custom providers too. TXMLTransformProvider ignores this value completely.
So, like I said above, there is no general rule on how this works and even if this properties has any effect.
see TDataPacketWriter.WriteDataSet. no matter whether underlying dataset supports block read mode or not it (datapacket writer) will stop writing data packet as soon as requested amount of records processed (or Eof state reached)
Quick question (hopefully)
I have a large dataset (>100,000 records) that I would like to use as a lookup to determine existence or non-existence of multiple keys. The purpose of this is to find FK violations before trying to commit them to the database to try and avoid the resultant EDatabaseError messing up my transaction.
I had been using TClientDataSet/TDatasetProvider with the FindKey method, as this allowed a client-side index to be set up and was faster (2s to scan each key rather than 10s for ADO). However, moving to large datasets the population of the CDS is starting to take far more time than the local index is saving.
I see that I have a few options for alternatives:
client cursor with TADOQuery.locate method
ADO SELECT statements for each check (no client cache)
ADO SEEK method
Extend TADOQuery to mimic FindKey
The Locate method seems easiest and doesn't spam the server with the SELECT/SEEK methods. I like the idea of extending the TADOQuery, but was wondering whether anyone knew of any ready-made solutions for this rather than having to create my own?
I would create a temporary table in the database server. Insert all 100,000 records into this temp table. Do bulk inserts of say 3000 records at a time, to minimise round trips to the server. Then run select statements on this temp table to check for foreign key violations etc. If all okay, do an insert SQL from the temp table to the main table.
My question is very simple. I have a TClientDataSet that is linked to a TADOQuery via a TDataSetProvider. I can put data into the TClientDataSet from the TADOQuery, but how do I get data from the TClientDataSet back into the TADOQuery?
Data is automatically transferred from the TADOQuery to the TClientDataSet when I run a query and then set the TClientDataSet's Active property to True, but if I deactivate the TADOQuery and then activate it again, how can I get the data back from the TClientDataSet?
I am running the same query on several databases and using the TClientDataSet to concatenate the results. This is working fine. My problem now is that I need to get the concatenated result set back from the TClientDataSet into the TADOQuery so that I can use the TADOQuery's SaveToFile procedure (for compatibility reasons). How can I do this?
I don't do TADOQuery as I use dbExpress, but I imagine that one needs to use the same technique. After you have posted your changes to TClientDataSet, call 'ApplyUpdates (0)', which transfers the data from the clientdataset to its provider.
You could always write the dataset back out to a temp table and then query it. Ouch!!
I've just about finished looking into this. My application allows the user to generate reports by querying their databases. I can get this to work and it is very efficient for small result sets - however, as this is a reporting application, and it's entirely possible that hundreds of thousands of records can be returned, using a ClientDataSet gives massive performance problems. Once you get above around 50,000 records (reasonable, given the customer base), processing starts to increase exponentially, so this is now basically moot.