I'm usign the dbExpress components within the Embarcadero C++Builder XE environment.
I have a relatively large table with something between 20k and 100k records, which I display in a DBGrid.
I am using a DataSetProvider, which is connected to a SQLQuery and a ClientDataSet, which is connected to the DataSetProvider.
I also need to analyze the data and therefore I need to run through the whole table. For smaller tables I always used code, which is basically something like this:
Form1->ClientDataSet1->First();
while(!Form1->ClientDataSet1->Eof){
temp=Form1->ClientDataSet1->FieldByName("FailReason")->AsLargeInt;
//do something with temp
Form1->ClientDataSet1->Next();
}
Of course this works out, but it is very slow, when I need to run through the whole DBGrid. For some 50000 records in can take up to some minutes. My suspicion is that the most perform is lost since the DBGrid needs to be repainted as the actual Dataset increments its address.
Therefore I am looking for a method which allows me to either read the data without manipulating the actual ClientDataSet. Maybe a method which copies the data of a column into a variable, or another way to run through the datasets, which is more efficient. I am sure if I would have a copy in a variable the operation would take less than a few seconds...
I googled now for hours, but didn't find anything useful so far.
Best regards,
Bodo
if your cds is connected to some db-aware control(-s) (via TDataSource) then first of all consider using DisableControls()
another option would be to avoid utilizing FieldByName within the loop
Related
Well, I'm studying the "packetRecord" property (TClientDataSet) and i have a doubt about it. I will explain how i think this works and if i'm wrong, correct me please.
1 - If i configure th packetRecord = 50, when i do "SELECT * FROM history", if table history have 200k of rows, the TClientDataSet will do something like this: "SELECT * FROM history limit 50", so when i need more 50 rows the ClientDataSet will search more 50 in Database.
The property packetRecord just make senses if TClientDataSet don't get all rows in Database, at least for me.
Am i correct ?
It will probably execute the entire query and ask to return just 50 records, but that is an implementation detail that I think is not chosen by the ClientDataSet but rather by the provider, the dataset or the driver.
But in general, that is more or less how it works, yes.
Did some browsing through the code. If the ClientDataSet is linked to a (local) TDataSetProvider, that provider just opens the dataset it is connected to. After opening it, it sets the DataSet.BlockReadSize property to the number of records to retrieve (=packetRecords).
So in the end it comes down on the implementation of BlockReadSize and the dsBlockRead state of the TDataSet that is used.
With a client-server setup this must be the same thing. In fact, there doesn't even have to be a dataset or even a database. There's also an TXMLTransformProvider, and people could implement custom providers too. TXMLTransformProvider ignores this value completely.
So, like I said above, there is no general rule on how this works and even if this properties has any effect.
see TDataPacketWriter.WriteDataSet. no matter whether underlying dataset supports block read mode or not it (datapacket writer) will stop writing data packet as soon as requested amount of records processed (or Eof state reached)
I'm working with ADO connecting to SQL Server 2005.
My TADODataSet selects 1 million records. using a TDBGrid and setting the TADODataSet.CursorLocation to clUseServer works. but the TDBGrid chokes!
How can I select 1 million records, avoid paging, and still be able to display records in the grid without fetching ALL records to the client side, letting the Grid read ahead as I scroll up and down?
SQL enterprise manager can execute a query and select 1 million records asynchronously without any problems (also MS-ACCESS).
TGrid is not your problem. Your problem is TADODataset is trying to load all the records. If you must run a query that returns so many records, you should set ExecuteOptions, try eoAsyncExecute and eoAsyncFetch. It may also help to set CacheSize.
Why do you need to fetch 1M records into a grid? No human being can look at so many records. Usually is far better to reduce the number of records before loading them into a UI.
If you have a good reason to show so many records into a grid, you'd need a dataset that 1) doesn't load the whole recordset when opened 2) doesn't cache previous record or it could run out of memory (under 32 bit Windows especially) far before reaching the end of the recordset if the record size is not small enough. To obtain such result beyond CursorLocation you have to set CursorType and CacheSize properly.
You can use a TClientDataset to implement incremental fetching setting the ADO dataset CursorType to ForwardOnly and CacheSize to a suitable value. Because TClientDataset caches read records, you want to avoid the source dataset to load all of them as well. A standard DB grid needs a bidirectional cursor, thereby it won't work with an unidirectional one. With so many records the client dataset cache can exhaust memory anyway. I'd suggest to use the Midas Speed Fix unit if you're using a Delphi version before 2010.
To avoid "out of memory" errors you may need to implement some kind of pagination. Anyway, check if the behaviour of other CursorType can help you.
You can try AnyDAC and TADTable. Its Live Data Window mode solves your and similar issues. The benefits are:
minimizes memory usage and allows to work with large data volumes, similar to an unidirectional dataset;
enables bi-directional navigation, in contrast to an unidirectional dataset;
gives always fresh data, reducing the need to refresh dataset;
does not delay to fetch all result set records, required to perform sorting, record location, jumping to last record, etc.
I've got a DB Express TSimpleDataset connected to a Firebird database. I've just added several thousand rows of data to the dataset, and now it's time to call ApplyUpdates.
Unfortunately, this results in several thousand database hits as it tries to INSERT each row individually. That's a bit disappointing. What I'd really like to see is the dataset generate a single transaction with a few thousand INSERT statements in it and send the whole thing at once. I could set that up myself if I had to, but first I'd like to know if there's any method for it built in to the dataset or the DBX framework.
Don't know if it is possible with a TSimpleDataset (never used it), but surely you can if you use a TClientDataset + TDatasetProvider + <put your db dataset here>. You can write a BeforeUpdateRecord to handle the apply process yourself. Basically, it allows you to bypass the standard apply process, access the dataset delta with changes made to records, and then use your own code and components to apply changes to the database. For example you could call stored procedures to modify data, and so on.
However, there is a difference between a transaction and what is called "array DML", "bulk insert" or the like. Even if you use a single transaction (and an "apply" AFAIK happens in a single transaction), within the transaction you may still need to send "n" INSERTs. Some databases supports a way of sending a single INSERT (or update, delete) with an array of parameters to be inserted, reducing the number of single statements to be used - but that may be very database specific and AFAIK dbExpress/Datasnap do not support it - you still could use the BeforeUpdateRecord event to take advantage of specific database capabililties.
My question is very simple. I have a TClientDataSet that is linked to a TADOQuery via a TDataSetProvider. I can put data into the TClientDataSet from the TADOQuery, but how do I get data from the TClientDataSet back into the TADOQuery?
Data is automatically transferred from the TADOQuery to the TClientDataSet when I run a query and then set the TClientDataSet's Active property to True, but if I deactivate the TADOQuery and then activate it again, how can I get the data back from the TClientDataSet?
I am running the same query on several databases and using the TClientDataSet to concatenate the results. This is working fine. My problem now is that I need to get the concatenated result set back from the TClientDataSet into the TADOQuery so that I can use the TADOQuery's SaveToFile procedure (for compatibility reasons). How can I do this?
I don't do TADOQuery as I use dbExpress, but I imagine that one needs to use the same technique. After you have posted your changes to TClientDataSet, call 'ApplyUpdates (0)', which transfers the data from the clientdataset to its provider.
You could always write the dataset back out to a temp table and then query it. Ouch!!
I've just about finished looking into this. My application allows the user to generate reports by querying their databases. I can get this to work and it is very efficient for small result sets - however, as this is a reporting application, and it's entirely possible that hundreds of thousands of records can be returned, using a ClientDataSet gives massive performance problems. Once you get above around 50,000 records (reasonable, given the customer base), processing starts to increase exponentially, so this is now basically moot.
I have defined a Delphi TTable object with calculated fields, and it is used in a grid on a form. I would like to make a copy of the TTable object, including the calculated fields, open that copy, do some changes to the data with the copy, close the copy, and then refresh the original copy and thusly the grid view. Is there an easy way to get a copy of a TTable object to be used in such a way?
The ideal answer would be one that solves the problem as generically as possible, i.e., a way of getting something like this:
newTable:=getACopyOf(existingTable);
You can use the TBatchMove component to copy a table and its structure.
Set the Mode property to specify the desired operation. The Source and Destination properties indicate the datasets whose records are added, deleted, or copied. The online help has additional details.
(Although I reckon you should investigate a TClientDataSet approach - it's certainly more scalable and faster).
Let me propose several things:
Let us suppose that you want to make changes programmatically. You could then use DisableControls and EnableControls methods of the TTable to disallow screen updates during that time.
If you want to have two screens with the same data (f.e. to compare data during online changes), you could actually create the same screen twice, with the TTable object being on the screen itself. It will have the exact same configuration (but not carry over previously made changes on the first screen but read the data from the database). Changes made on one screen will not be automatically refreshed on the other.
Another way: Try using TDataSetProvider with TTable as Dataset (source) feeding a TClientDataSet. ApplyUpdates would feed back the changes to the TTable. Since the calculated fields are read only, they are not affected. (untested, but should work)
I believe that the second approach (TClientDataset) is probably the best method to use in this scenario. An alternative would be to use a memory table (kbmMemTable for instance). Either way, you would clone your original table and then after making your changes loop thru the memory version of your dataset and update your original table.
You should be able to select the table on the form, copy it using Ctrl-C, then paste it into any text editor. You will get the text version of the object's properties which you can then edit as needed. When you are done, select all the text again and you can copy it to the clipboard and paste it back onto a form.