Efficiently populate combobox in Delphi - delphi

Need to add many items (more than 10k) in TComboBox (I know that TComboBox is not supposed to hold many items but its not up to me to change this) without adding duplicates.
So I need to search the full list before adding. I want to avoid TComboBox.items.indexof as I need a binary search but the binary find is not available in TStrings.
So I created a temporary Tstringlist, set sorted to true and used find. But now assigning the temporary Tstringlist back to TComboBox.Items
(myCB.Items.AddStrings(myList))
is really slow as it copies the whole list. Is there any way to move the list instead of copying it? Or any other way to efficient populate my TComboBox?

There is no way to "move" the list into the combo box because the combo box's storage belongs to the internal Windows control implementation. It doesn't know any way to directly consume your Delphi TStringList object. All it offers is a command to add one item to the list, which TComboBox then uses to copy each item from the string list into the system control, one by one. The only way to avoid copying the many thousands of items into the combo box is to avoid the issue entirely, such as by using a different kind of control or by reducing the number of items you need to add.
A list view has a "virtual" mode where you only tell it how many items it should have, and then it calls back to your program when it needs to know details about what's visible on the screen. Items that aren't visible don't occupy any space in the list view's implementation, so you avoid the copying. However, system combo boxes don't have a "virtual" mode. You might be able to find some third-party control that offers that ability.
Reducing the number of items you need to put in the combo box is your next best option, but only you and your colleagues have the domain knowledge necessary to figure out the best way to do that.

As Rudy Velthuis already mentioned in the comments and assuming you are using VCL, the CB_INITSTORAGE message could be an option:
SendMessage(myCB, CB_INITSTORAGE, myList.Count, 20 * myList.Count*sizeof(Integer));
where 20 is your average string length.
Results (on a i5-7200U and 20K items with random length betwen 1 and 50 chars):
without CB_INITSTORAGE: ~ 265ms
with CB_INITSTORAGE: ~215ms
So while you can speed up things a little by preallocating the memory, the bigger issue seems to be the bad user experience. How can a user find the right element in a combobox with such many items?

Notwithstanding that 10k items is crazy to keep in a TComboBox, an efficient strategy here would be to keep a cache in a separate object. For example, declare :
{ use a TDictionary just for storing a hashmap }
FComboStringsDict : TDictionary<string, integer>;
where
procedure TForm1.FormCreate(Sender: TObject);
var
i : integer;
spw : TStopwatch;
begin
FComboStringsDict := TDictionary<string, integer>.Create;
spw := TStopwatch.StartNew;
{ add 10k random items }
for i := 0 to 10000 do begin
AddComboStringIfNotDuplicate(IntToStr(Floor(20000*Random)));
end;
spw.Stop;
ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;
function TForm1.AddComboStringIfNotDuplicate(AEntry: string) : boolean;
begin
result := false;
if not FComboStringsDict.ContainsKey(AEntry) then begin
FComboStringsDict.Add(AEntry, 0);
ComboBox1.Items.Add(AEntry);
result := true;
end;
end;
Adding 10k items initially takes about 0.5s this way.
{ test adding new items }
procedure TForm1.Button1Click(Sender: TObject);
var
spw : TStopwatch;
begin
spw := TStopwatch.StartNew;
if not AddComboString(IntToStr(Floor(20000*Random))) then
ListBox1.Items.Add('Did not add duplicate');
spw.Stop;
ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;
But adding each subsequent item is very fast <1ms. This is a clumsy implementation, but you could easily wrap this behaviour into a custom class. The idea is to keep your data model as separate from the visual component as possible - keep them in sync when adding or removing items but do your heavy searches on the dictionary where the lookup is fast. Removing items would still rely on .IndexOf.

Related

How to auto delete clientdataset records that have not been updated

I have a clientdataset in RAM no database, that maintains a list of active nodes in a network.
Nodes continuously report back confirming they are alive, thus keeping the dataset updated.
The dataset is displayed in a dbgrid.
When a node stops reporting status it is deleted from the database after a few seconds inactivity.
I do this by updating a timeout field when a field is updated.
Every second I iterate through the dataset deleting outdated records.
This works, but the grid sometimes flicker when OnDrawColumnCell refreshes a single line grid to customize the column colors. I call DisableControls/EnableControls, but there seems to be a small delay until OnDrawCell redraws the grid causing the flicker.
If I disable the iteration to delete the outdated records the flicker stops.
Is there a better way to do this?
A way to minimise the flicker in your grid is to use a 'trick' which makes use of a special feature of ClientDataSets, namely that you can copy data between them by assigning their Data properties, as in
cdsDestination.Data := cdsSource.Data;
So what you can do is to have two CDSs, one which you use for display purposes only, and the other which processes your network nodes. This means that changes to the copy CDS are kept to the absolute minimum, and you can do pretty much whatever you like with your source CDS, and take as long as you like about it (as long, of course, as you can get it done before the next destination CDS update). Something like this:
const
NodeCount = 1000;
procedure TForm1.DoDataUpdate;
begin
// do something to CDS1's data here
cdsCopy.Data := CDS1.Data;
end;
procedure TForm1.FormCreate(Sender: TObject);
var
i : Integer;
begin
CDS1.CreateDataSet;
for i := 1 to NodeCount do
CDS1.InsertRecord([i, Now]);
CDS1.First;
DBGrid1.DataSource := DataSource1;
DataSource1.DataSet := cdsCopy;
end;
procedure TForm1.Timer1Timer(Sender: TObject);
begin
DoDataUpdate;
end;

Pascal: How to randomize two images on Turbo Delphi

I have a task of having two images (Two TImage), one a head and the other a tail (Coins), on my screen with a TButton to randomize both of them.
It is to be that when you press the button, the two images go random choosing Heads or Tails.
I know its kind of a easy question, but I am just learning. I just don't know what to use!
You need to sample from a discrete uniform distribution with two possible values. So like this:
function IsHead: Boolean;
begin
Result := Random()<0.5;
end;
Or like this:
function IsHead: Boolean;
begin
Result := Random(2)=0;
end;
You'll want to call Randomize somewhere in the startup of your program to make sure that you don't get the same sequence of pseudo-random numbers each time you run the program.
I'm assuming that you already know how to write button OnClick event handlers, and switch visibility of TImage controls.

How long have the Lookup and Locate methods of TDataset been around?

I'm working on modernizing and fixing bugs in the codebase of a Delphi 4-era program written by someone else. A lot of the code is kinda scary by modern standards, and I can't help but wonder if some of the things I'm seeing are there because the original author didn't know about certain standard library features, or if they weren't available.
One of the more obnoxious "patterns" I see all over the app looks like this:
table := TClientDataset.Create;
with table do
begin
CloneCursor(dmDatabase.OriginalTable, false, true);
filtered := true;
active := true;
first;
while not EOF do
begin
if fieldByName('whatever').AsString = 'some criteria' then break;
next;
end;
if EOF then exit;
//do something based on the current row of the dataset
table.free;
end;
Almost every one of these groups could be replaced by a one-line call to either Lookup or Locate on the original dataset, with no need for an intermediary CDS at all. That makes me wonder, were these methods available back in the D4 days? When were Lookup and Locate first added?
Lookup and Locate were introduced in Delphi 2. It looks like the original author simply didn't take advantage of them.
Seems the Original programmer wanted to make sure that the row pointer is not changed
at all. Doing Locate (or Lookup) would change the row pointer, provoking all kinds
of data events (Datasource.OnDataChange, Dataset.AfterScroll and so on).
Doing the search with TClientDataset.CloneCursor, none of the these events ger triggered on the dmDatabase.OriginalTable and there's no need to reload the data from database.
Seems to me that is the intention. TClientDataset was presented on D3. And cloned cursors are a kind
of advanced feature - and need the dmDatabase.OriginalTable to be a CDS too.

VirtualStringTree hide node(s)

is that possible to hide specific nodes in VirtualStringTree?
I'm implementing "filtering" feature (the VST acts as a list with columns), and I'd like to avoid reloading content each time the filter is changed - instead, much faster would be to tell VST not to render specific items ... any solutions?
VirtualTree.IsVisible[Node] := False;
There are problem using .IsVisible[] or .IsFiltered[] and it is that is very slow, i've probe filter in a tree with amoung of 25,000 nodes and is too slow.
I've found one aproach that is faster and solves the problem with scrollbar size when using Include(Node.states,vsFiltered) or (Node.States,vsVisible) is used, it consists on change manually the Node.TotalHeight value acording with the number of visible nodes (not Filtered).
For example i'm filtering 25,000 nodes and the code i was using is the follow :
procedure TFC_Articulo.Filtrar(Filtro:String);
var
Node:PVirtualNode;
Data:PArticulo;
begin
Node := TV.GetFirst;
TV.RootNode.TotalHeight:=TV.DefaultNodeHeight; // The Trick
while Assigned(Node) do
begin
Data:=TV.GetNodeData(Node);
Exclude(Node.States,vsFiltered); // By default all nodes wil be Visible
if ComparationHereForDetermineFiltering then
Include(Node.States,vsFiltered) // This node wil be filtered
else
Inc(TV.RootNode.TotalHeight,Node.NodeHeight); // Determine the Height of scrollbar
Node:=TV.GetNext(Node);
end;
TV.RootNode.TotalHeight:=TV.RootNode.TotalHeight+TV.BottomSpace;
TV.UpdateScrollBars(True);
end;
Hope this helps
Sorry bad english...

Delphi Adding Items to ComboBox Speed

I have a fairly complex and large application that hands loads and loads of data. Is there a speedy way to add items to ComboBox that doesn't take so long? On my P3 3.2ghz, the following snippet takes just under a second to add around 32,000 items. (MasterCIList is a StringList with strings typically 20 - 30 bytes long).
with LookupComboBox do
begin
Items.BeginUpdate;
Items.Clear;
for i := 0 to MasterCIList.Count - 1 do
Items.Add(MasterCIList[i]);
Items.EndUpdate;
end;
Drilling down into the VCL, it appears that in TComboBoxStrings.Add, there is a call to
Result := SendMessage(ComboBox.Handle, CB_ADDSTRING, 0, Longint(PChar(S)));
I'm guessing this is really taking up time (okay, I know it is). Is there another way to populate the Items that is speedier? Any high-speed combo boxes available? I have the TMS components but they seem to be extensions of TComboBox.
For instance, I have the PlusMemo which seems to be a total rewrite of a TMemo. I can easily add a million line in a second to a PlusMemo. A TMemo, I don't think so.
Thank you so much for your time!
Sorry if I'm a nuisance, but I doubt a TComboBox with 32,000 items is even remotely ''usable'' --- I'd say there's a reason why it's slow: it was never meant to do this :)
Would there be a possibility to filter the data, and only load a subset? To be more concrete, in one particular database application I've been working on, the user can search for a person. We let the user type at least 3 or 4 characters of the name, and only then begin to return results in a listbox. This has greatly increased usability of the search form, also greatly speeding up the whole process.
Would you be able to follow a similar approach?
Or, on a completely different take, perhaps you could take a look at the VirtualTreeView component --- either for direct use, or for inspiration.
I agree that 32K items is a ridiculous amount to have in a combobox... That being said, you might try adding the items to a TStringList first and then use a combination of Begin/EndUpdate and AddStrings:
SL := TStringList.Create;
try
// Add your items to the stringlist
ComboBox.Items.BeginUpdate;
try
ComboBox.Items.AddStrings(YourStringList);
finally
ComboBox.Items.EndUpdate;
end;
finally
SL.Free;
end;
The code compiles, but I didn't test it further than that; I've never felt the need to add more than a few dozen items to a combobox or listbox. If any more items might be needed than that, I find a way to filter before populating the list so there are fewer items.
Just out of curiosity, how do you expect a user to sort through that many items to make a decision?
var
Buffer: TStringList;
begin
Buffer := TStringList.Create;
try
// --> Add items to Buffer here <--
ComboBox.Items := Buffer;
finally
FreeAndNil(Buffer);
end;
end;
This is the fastest way we've found to update a visual control.
The VCL does BeginUpdate, Clear, and EndUpdate internally.
If you don't believe me, profile it.
perhaps cmb.Items.Assign(myStringList) will help.
here's a wild idea: i haven't tried it but you might check to see if there's a way to virtually load the combobox by setting the number of items and then owner drawing. please pardon this crazy idea but i think i've heard of this being available somehow. irrelevant: this is how it's done in Palm OS...where the faster way to load the combobox is to not load it all... ;-)
Not an answer, but why on earth would you want 32,000 items in a combo box? That is a terrible way to store that much data.
i agree; it's a bad practice...
It's me again. I'm adding 32,000 items cause I need to. That's one of many controls in my application that has "lots" of items. I need to have that many items. It works just fine looking things up. Perfectly in fact. I'm just trying to optimize things. The users find things just fine since they are in a certain logical order.
Everything I've seem so far with Assign and AddStrings is that they eventually end up in Add with the SendMessage call. So I'll keep looking.
Thanks for the feedback.
use backgroundworker for adding MasterCIList items.after complete adding items use only AddStrings.
procedure TForm2.BackgroundWorker1Work(Worker: TBackgroundWorker);
var
I: Integer;
begin
MasterCIList.BeginUpdate;
try
MasterCIList.Capacity := 32 * 1024; // if derminate count of items
for I := 1 to 32 * 1024 do
begin
MasterCIList.Add(IntToStr(I));
{ if need progess }
if I mod 300 = 0 then
Worker.ReportProgress((I * 100) div MasterCIList.Capacity);
{ if need cancelable }
if (I mod 100 = 0) and Worker.CancellationPending then
Break;
end;
finally
MasterCIList.EndUpdate;
end;
end;
procedure TForm2.BackgroundWorker1WorkComplete(Worker: TBackgroundWorker;
Cancelled: Boolean);
begin
LookupComboBox.Items.AddStrings(MasterCIList );
// AddStrings use beginupdate..endupdate in itself
end;
Maybe you can try this?
"Speeding up adding items to a combobox or listbox"
http://blogs.msdn.com/b/oldnewthing/archive/2004/06/10/152612.aspx
Perhaps you can use a database engine in the back-end and use a data aware component. Then the things will be much more quicker and usable. Perhaps if you'll try to describe what do you try to accomplish we'll help you further. In any case, your UI design is, let's say, odd. And for this perhaps the Embarcadero forums will help you better.
I implement this in a different manner. First i removed the combobox control and take textbox control and assign it autocomplete to custom source where the custom source string collection is 32k items.I get the selected value from a new query on controls validation.
So it can replace combobox functionality. Mostly about 32k items people dont scroll but they keep entering key strokes and is catched by our custom auto complete source..

Resources