I'm working on modernizing and fixing bugs in the codebase of a Delphi 4-era program written by someone else. A lot of the code is kinda scary by modern standards, and I can't help but wonder if some of the things I'm seeing are there because the original author didn't know about certain standard library features, or if they weren't available.
One of the more obnoxious "patterns" I see all over the app looks like this:
table := TClientDataset.Create;
with table do
begin
CloneCursor(dmDatabase.OriginalTable, false, true);
filtered := true;
active := true;
first;
while not EOF do
begin
if fieldByName('whatever').AsString = 'some criteria' then break;
next;
end;
if EOF then exit;
//do something based on the current row of the dataset
table.free;
end;
Almost every one of these groups could be replaced by a one-line call to either Lookup or Locate on the original dataset, with no need for an intermediary CDS at all. That makes me wonder, were these methods available back in the D4 days? When were Lookup and Locate first added?
Lookup and Locate were introduced in Delphi 2. It looks like the original author simply didn't take advantage of them.
Seems the Original programmer wanted to make sure that the row pointer is not changed
at all. Doing Locate (or Lookup) would change the row pointer, provoking all kinds
of data events (Datasource.OnDataChange, Dataset.AfterScroll and so on).
Doing the search with TClientDataset.CloneCursor, none of the these events ger triggered on the dmDatabase.OriginalTable and there's no need to reload the data from database.
Seems to me that is the intention. TClientDataset was presented on D3. And cloned cursors are a kind
of advanced feature - and need the dmDatabase.OriginalTable to be a CDS too.
Related
I have a code some thing like this
dxMemOrdered : TdxMemData;
while not qrySandbox2.EOF do
begin
dxMemOrdered.append;
dxMemOrderedTotal.asCurrency := qrySandbox2.FieldByName('TOTAL').asCurrency;
dxMemOrdered.post;
qrySandbox2.Next;
end;
this code executes in a thread. When there are huge records say "400000" it is taking around 25 minutes to parse through it. Is there any way that i can reduce the size by optimizing the loop? Any help would be appreciated.
Update
Based on the suggestions i made the following changes
dxMemOrdered : TdxMemData;
qrySandbox2.DisableControls;
while not qrySandbox2.Recordset.EOF do
begin
dxMemOrdered.append;
dxMemOrderedTotal.asCurrency := Recordset.Fields['TOTAL'].Value;
dxMemOrdered.post;
qrySandbox2.Next;
end;
qrySandbox2.EnableControls;
and my output time have improved from 15 mins to 2 mins. Thank you guys
Without seeing more code, the only suggestion I can make is make sure that any visual control that is using the memory table is disabled. Suppose you have a cxgrid called Grid that is linked to your dxMemOrdered memory table:
var
dxMemOrdered: TdxMemData;
...
Grid.BeginUpdate;
try
while not qrySandbox2.EOF do
begin
dxMemOrdered.append;
dxMemOrderedTotal.asCurrency := qrySandbox2.FieldByName('TOTAL').asCurrency;
dxMemOrdered.Post;
qrySandbox2.Next;
end;
finally
Grid.EndUpdate;
end;
Some ideas in order of performance gain vs work to do by you:
1) Check if the SQL dialect that you are using lets you use queries that directly SELECT from/INSERT to. This depends on the database you're using.
2) Make sure that if your datasets are not coupled to visual controls, that you call DisableControls/EnableControls around this loop
3) Does this code have to run in the main program thread? Maybe you can send if off to a separate thread while the user/program continues doing something else
4) When you have to deal with really large data, bulk insertion is the way to go. Many databases have options to bulk insert data from text files. Writing to a text file first and then bulk inserting is way faster than individual inserts. Again, this depends on your database type.
[Edit: I just see you inserting the info that it's TdxMemData, so some of these no longer apply. And you're already threading, missed that ;-). I leave this suggestions in for other readers with similar problems]
It's much better to let SQL do the work instead of iterating though a loop in Delphi. Try a query such as
insert into dxMemOrdered (total)
select total from qrySandbox2
Is 'total' the only field in dxMemOrdered? I hope that it's not the primary key otherwise you are likely to have collisions, meaning that rows will not be added.
There's actually a lot you could do to speed up your thread.
The first would be to look at the problem in a broader perspective:
Am I fetching data from a cached / fast disk, possibly moved in memory?
Am I doing the right thing, when aggregating totals by hand? SQL engines are expecially optimized to do those things, all you'd need to do is to define an additional logical field where to store the SQL aggregated result.
Another little optimization that may bring an improvement over large amounts of looping is to not use constructs like:
Recordset.Fields['TOTAL'].Value
Recordset.FieldByName('TOTAL').Value
but to add the fields with the fields editor and then directly accessing the right field. You'll save a whole loop through the fields collection, that otherwise is performed on every field, on every next record.
I am working with multiple databases within the same application. I am using drivers from two different companies. Both companies have tTable and tQuery Descendants that work well.
I need a way to have generic access to the data, regardless of which driver/tQuery component I am using to return a set of data. This data would NOT tie to components, just to my logic.
For example...(pseudocode) Let's create a function which can run against either tQuery component
function ListAllTables(NameOfDatabase : String) :ReturnSet??
begin
If NameOfDataBase = 'X' then use tQuery(Vendor 1)
else use tQuery(Vendor 2)
RunQuery;
Return Answer...
end;
When NORMALLY running a query, I do
Query.Open;
While not Query.EOF do
begin
Read my rows..
next;
end;
If I am CALLING ListAllTables, what is my return type so that I can iterate through the rows? Each tQuery Vendor is different, so I can't use that (can I, and if so, would I want to?) I could build a Memory Table, and pass that back, but that seems like extra work for ListAllRows to build a memory table, and then to pass it back to the calling routine so that it can "un-build", i.e. iterate through the rows...
What are your ideas and suggestions?
Thanks
GS
Almost all Delphi datasets descend from TDataset, and most useful behavior is defined on TDataset.
Therefore, if you assign each table or query to a variable of type TDataset, you should be able to perform your logic on that dataset in a vendor neutral fashion.
I would also isolate the production of the datasets into a set of factory functions that only create the vendor-specific dataset and return it as a TDataset. Each factory function goes in it's own unit. Then, only those small units need have any knowledge of the vendor specific components.
You can use IProviderSupport to generalize the query execution. I am expecting, that used query's are supporting IProviderSupport. This interface allows to set command text, parameters, execute commands, etc.
The common denominator for used query's is TDataSet. So, you will need pass TDataSet reference.
For example:
var
oDS: TDataSet;
...
if NameOfDataBase = 'X' then
oDS := T1Query.Create(nil)
else
oDS := T2Query.Create(nil);
(oDS as IProviderSupport).PSSetCommandText('select * from mytab');
oDS.Open;
while not oDS.Eof do begin
//
oDS.Next;
end;
Perhaps you could consider a universal data access component such as UniDAC or AnyDAC. This allows you to use only one component set to access different databases in a consistent way.
You might also be interested in DataAbstract from RemObjects. A powerful data abstraction, multi-tier, remoting solution with a lot of features. Not inexpensive, but excellent value for the money.
Relevant links:
http://www.da-soft.com/anydac/
http://www.devart.com/unidac/
http://www.remobjects.com/da/
A simple approach would be to have ListAllTables return (or populate) a stringlist.
What do you people use for generating unique account numbers?
Some use Autoinc field, others something else...
What would be the proper way i.e to get an account number before I run the insert query?
If you are using a SQL database, use a Generator. If you want to use an independent mechanism you could consider using a GUID.
You haven't told us what database system you are using, but from the sound of it, you're talking about the Paradox tables in Delphi. If so, an autoInc column can work, although if I recall correctly, you have to be very careful when moving your data around with Paradox autoInc columns because they re-generate from zero when moved.
As has been mentioned, you can use GUIDs - sysutils.function CreateGUID(out Guid: TGUID): HResult; - they will always be unique, but the downside in GUIDS is that ordering by these keys will not be intuitive and probably be meaningless, so you'll need a timestamp column of some sort to maintain the order of your inserts, which can be important. Also, a GUID is a rather long character string and will not be very efficient for use as an account#, which assumedly will be a primary or foreign key in many tables.
So I'd stick to autoInc if you want something automatic, but if you have to move data around and you need to maintain your original keys, load your original autoincs as integer columns in their new location or you could end up corrupting your entire database. (I believe there are other scenarios that also cause autoIncs to reset in Paradox tables - research this if it's relevant - been a long time since I've used Pdox, and it may not be a problem with other flat file databases)
If you are indeed using a database server - SQLServer, Oracle, Interbase, etc, they all have autoInc/indentity or generator functionality, sometimes in conjuction with a trigger - that is your best option.
Dorin's answer is also an excellent solution if you want to handle this yourself from within your Delphi code. Create a global, thread safe function to implement it - that will ensure a very high level of safety.
HTH
Depending on how long you want the number, you can go with Jamies MD5 conversion or:
var
LDateTime: TDateTime;
LBytes: array[0..7] of Byte absolute LDateTime;
LAccNo: string;
Index: Integer;
begin
LDateTime := Now;
LAccNo := EmptyStr;
for Index := 0 to 7 do
LAccNo := LAccNo + IntToHex( LBytes[ Index ], 2 );
// now you have a code in LAccNo, use it wisely (:
end;
I use this PHP snippet to generate a decent account number:
$account_number = str_replace(array("0","O"),"D",strtoupper(substr(md5(time()),0,7)));
This will create a 7 digit varchar string that doesn't contain 0's or o's (to avoid errors on the phone or transcribing them in e-mails, etc.) You get something like EDB6DA6 or 76337D5 or DB2E624.
In a Delphi application we are working on we have a big structure of related objects. Some of the properties of these objects have values which are calculated at runtime and I am looking for a way to cache the results for the more intensive calculations. An approach which I use is saving the value in a private member the first time it is calculated. Here's a short example:
unit Unit1;
interface
type
TMyObject = class
private
FObject1, FObject2: TMyOtherObject;
FMyCalculatedValue: Integer;
function GetMyCalculatedValue: Integer;
public
property MyCalculatedValue: Integer read GetMyCalculatedValue;
end;
implementation
function TMyObject.GetMyCalculatedValue: Integer;
begin
if FMyCalculatedValue = 0 then
begin
FMyCalculatedValue :=
FObject1.OtherCalculatedValue + // This is also calculated
FObject2.OtherValue;
end;
Result := FMyCalculatedValue;
end;
end.
It is not uncommon that the objects used for the calculation change and the cached value should be reset and recalculated. So far we addressed this issue by using the observer pattern: objects implement an OnChange event so that others can subscribe, get notified when they change and reset cached values. This approach works but has some downsides:
It takes a lot of memory to manage subscriptions.
It doesn't scale well when a cached value depends on lots of objects (a list for example).
The dependency is not very specific (even if a cache value depends only on one property it will be reset also when other properties change).
Managing subscriptions impacts the overall performance and is hard to maintain (objects are deleted, moved, ...).
It is not clear how to deal with calculations depending on other calculated values.
And finally the question: can you suggest other approaches for implementing cached calculated values?
If you want to avoid the Observer Pattern, you might try to use a hashing approach.
The idea would be that you 'hash' the arguments, and check if this match the 'hash' for which the state is saved. If it does not, then you recompute (and thus save the new hash as key).
I know I make it sound like I just thought about it, but in fact it is used by well-known softwares.
For example, SCons (Makefile alternative) does it to check if the target needs to be re-built preferably to a timestamp approach.
We have used SCons for over a year now, and we never detected any problem of target that was not rebuilt, so their hash works well!
You could store local copies of the external object values which are required. The access routine then compares the local copy with the external value, and only does the recalculation on a change.
Accessing the external objects properties would likewise force a possible re-evaluation of those properties, so the system should keep itself up-to-date automatically, but only re-calculate when it needs to. I don't know if you need to take steps to avoid circular dependencies.
This increases the amount of space you need for each object, but removes the observer pattern. It also defers all calculations until they are needed, instead of performing the calculation every time a source parameter changes. I hope this is relevant for your system.
unit Unit1;
interface
type
TMyObject = class
private
FObject1, FObject2: TMyOtherObject;
FObject1Val, FObject2Val: Integer;
FMyCalculatedValue: Integer;
function GetMyCalculatedValue: Integer;
public
property MyCalculatedValue: Integer read GetMyCalculatedValue;
end;
implementation
function TMyObject.GetMyCalculatedValue: Integer;
begin
if (FObject1.OtherCalculatedValue <> FObjectVal1)
or (FObject2.OtherValue <> FObjectVal2) then
begin
FMyCalculatedValue :=
FObject1.OtherCalculatedValue + // This is also calculated
FObject2.OtherValue;
FObjectVal1 := FObject1.OtherCalculatedValue;
FObjectVal2 := Object2.OtherValue;
end;
Result := FMyCalculatedValue;
end;
end.
In my work I use Bold for Delphi that can manage unlimited complex structures of cached values depending on each other. Usually each variable only holds a small part of the problem. In this framework that is called derived attributes. Derived because the value is not saved in the database, It just depends on on other derived attributes or persistant attributes in the database.
The code behind such attribute is written in Delphi as a procedure or in OCL (Object Constraint Language) in the model. If you write it as Delphi code you have to subscribe to the depending variables. So if attribute C depends on A and B then whenever A or B changes the code for recalc C is called automatically when C is read. So the first time C is read A and B is also read (maybe from the database). As long as A and B is not changed you can read C and got very fast performance. For complex calculations this can save quite a lot of CPU-time.
The downside and bad news is that Bold is not offically supported anymore and you cannot buy it either. I suppose you can get if you ask enough people, but I don't know where you can download it. Around 2005-2006 it was downloadable for free from Borland but not anymore.
It is not ready for D2009 as someone have to port it to Unicode.
Another option is ECO with dot.net from Capable Objects. ECO is a plugin in Visual Studio. It is a supported framwork that have the same idea and author as Bold for Delphi. Many things are also improved, for example databinding is used for the GUI-components. Both Bold and ECO use a model as a central point with classes, attributes and links. Those can be persisted in a database or a xml-file. With the free version of ECO the model can have max 12 classes, but as I remember there is no other limits.
Bold and ECO contains lot more than derived attributes that makes you more productive and allow you to think on the problem instead of technical details of database or in your case how to cache values. You are welcome with more questions about those frameworks!
Edit:
There is actually a download link for Embarcadero registred users for Bold for Delphi for D7, quite old... I know there was updates for D2005, ad D2006.
I have a fairly complex and large application that hands loads and loads of data. Is there a speedy way to add items to ComboBox that doesn't take so long? On my P3 3.2ghz, the following snippet takes just under a second to add around 32,000 items. (MasterCIList is a StringList with strings typically 20 - 30 bytes long).
with LookupComboBox do
begin
Items.BeginUpdate;
Items.Clear;
for i := 0 to MasterCIList.Count - 1 do
Items.Add(MasterCIList[i]);
Items.EndUpdate;
end;
Drilling down into the VCL, it appears that in TComboBoxStrings.Add, there is a call to
Result := SendMessage(ComboBox.Handle, CB_ADDSTRING, 0, Longint(PChar(S)));
I'm guessing this is really taking up time (okay, I know it is). Is there another way to populate the Items that is speedier? Any high-speed combo boxes available? I have the TMS components but they seem to be extensions of TComboBox.
For instance, I have the PlusMemo which seems to be a total rewrite of a TMemo. I can easily add a million line in a second to a PlusMemo. A TMemo, I don't think so.
Thank you so much for your time!
Sorry if I'm a nuisance, but I doubt a TComboBox with 32,000 items is even remotely ''usable'' --- I'd say there's a reason why it's slow: it was never meant to do this :)
Would there be a possibility to filter the data, and only load a subset? To be more concrete, in one particular database application I've been working on, the user can search for a person. We let the user type at least 3 or 4 characters of the name, and only then begin to return results in a listbox. This has greatly increased usability of the search form, also greatly speeding up the whole process.
Would you be able to follow a similar approach?
Or, on a completely different take, perhaps you could take a look at the VirtualTreeView component --- either for direct use, or for inspiration.
I agree that 32K items is a ridiculous amount to have in a combobox... That being said, you might try adding the items to a TStringList first and then use a combination of Begin/EndUpdate and AddStrings:
SL := TStringList.Create;
try
// Add your items to the stringlist
ComboBox.Items.BeginUpdate;
try
ComboBox.Items.AddStrings(YourStringList);
finally
ComboBox.Items.EndUpdate;
end;
finally
SL.Free;
end;
The code compiles, but I didn't test it further than that; I've never felt the need to add more than a few dozen items to a combobox or listbox. If any more items might be needed than that, I find a way to filter before populating the list so there are fewer items.
Just out of curiosity, how do you expect a user to sort through that many items to make a decision?
var
Buffer: TStringList;
begin
Buffer := TStringList.Create;
try
// --> Add items to Buffer here <--
ComboBox.Items := Buffer;
finally
FreeAndNil(Buffer);
end;
end;
This is the fastest way we've found to update a visual control.
The VCL does BeginUpdate, Clear, and EndUpdate internally.
If you don't believe me, profile it.
perhaps cmb.Items.Assign(myStringList) will help.
here's a wild idea: i haven't tried it but you might check to see if there's a way to virtually load the combobox by setting the number of items and then owner drawing. please pardon this crazy idea but i think i've heard of this being available somehow. irrelevant: this is how it's done in Palm OS...where the faster way to load the combobox is to not load it all... ;-)
Not an answer, but why on earth would you want 32,000 items in a combo box? That is a terrible way to store that much data.
i agree; it's a bad practice...
It's me again. I'm adding 32,000 items cause I need to. That's one of many controls in my application that has "lots" of items. I need to have that many items. It works just fine looking things up. Perfectly in fact. I'm just trying to optimize things. The users find things just fine since they are in a certain logical order.
Everything I've seem so far with Assign and AddStrings is that they eventually end up in Add with the SendMessage call. So I'll keep looking.
Thanks for the feedback.
use backgroundworker for adding MasterCIList items.after complete adding items use only AddStrings.
procedure TForm2.BackgroundWorker1Work(Worker: TBackgroundWorker);
var
I: Integer;
begin
MasterCIList.BeginUpdate;
try
MasterCIList.Capacity := 32 * 1024; // if derminate count of items
for I := 1 to 32 * 1024 do
begin
MasterCIList.Add(IntToStr(I));
{ if need progess }
if I mod 300 = 0 then
Worker.ReportProgress((I * 100) div MasterCIList.Capacity);
{ if need cancelable }
if (I mod 100 = 0) and Worker.CancellationPending then
Break;
end;
finally
MasterCIList.EndUpdate;
end;
end;
procedure TForm2.BackgroundWorker1WorkComplete(Worker: TBackgroundWorker;
Cancelled: Boolean);
begin
LookupComboBox.Items.AddStrings(MasterCIList );
// AddStrings use beginupdate..endupdate in itself
end;
Maybe you can try this?
"Speeding up adding items to a combobox or listbox"
http://blogs.msdn.com/b/oldnewthing/archive/2004/06/10/152612.aspx
Perhaps you can use a database engine in the back-end and use a data aware component. Then the things will be much more quicker and usable. Perhaps if you'll try to describe what do you try to accomplish we'll help you further. In any case, your UI design is, let's say, odd. And for this perhaps the Embarcadero forums will help you better.
I implement this in a different manner. First i removed the combobox control and take textbox control and assign it autocomplete to custom source where the custom source string collection is 32k items.I get the selected value from a new query on controls validation.
So it can replace combobox functionality. Mostly about 32k items people dont scroll but they keep entering key strokes and is catched by our custom auto complete source..