Need to add many items (more than 10k) in TComboBox (I know that TComboBox is not supposed to hold many items but its not up to me to change this) without adding duplicates.
So I need to search the full list before adding. I want to avoid TComboBox.items.indexof as I need a binary search but the binary find is not available in TStrings.
So I created a temporary Tstringlist, set sorted to true and used find. But now assigning the temporary Tstringlist back to TComboBox.Items
(myCB.Items.AddStrings(myList))
is really slow as it copies the whole list. Is there any way to move the list instead of copying it? Or any other way to efficient populate my TComboBox?
There is no way to "move" the list into the combo box because the combo box's storage belongs to the internal Windows control implementation. It doesn't know any way to directly consume your Delphi TStringList object. All it offers is a command to add one item to the list, which TComboBox then uses to copy each item from the string list into the system control, one by one. The only way to avoid copying the many thousands of items into the combo box is to avoid the issue entirely, such as by using a different kind of control or by reducing the number of items you need to add.
A list view has a "virtual" mode where you only tell it how many items it should have, and then it calls back to your program when it needs to know details about what's visible on the screen. Items that aren't visible don't occupy any space in the list view's implementation, so you avoid the copying. However, system combo boxes don't have a "virtual" mode. You might be able to find some third-party control that offers that ability.
Reducing the number of items you need to put in the combo box is your next best option, but only you and your colleagues have the domain knowledge necessary to figure out the best way to do that.
As Rudy Velthuis already mentioned in the comments and assuming you are using VCL, the CB_INITSTORAGE message could be an option:
SendMessage(myCB, CB_INITSTORAGE, myList.Count, 20 * myList.Count*sizeof(Integer));
where 20 is your average string length.
Results (on a i5-7200U and 20K items with random length betwen 1 and 50 chars):
without CB_INITSTORAGE: ~ 265ms
with CB_INITSTORAGE: ~215ms
So while you can speed up things a little by preallocating the memory, the bigger issue seems to be the bad user experience. How can a user find the right element in a combobox with such many items?
Notwithstanding that 10k items is crazy to keep in a TComboBox, an efficient strategy here would be to keep a cache in a separate object. For example, declare :
{ use a TDictionary just for storing a hashmap }
FComboStringsDict : TDictionary<string, integer>;
where
procedure TForm1.FormCreate(Sender: TObject);
var
i : integer;
spw : TStopwatch;
begin
FComboStringsDict := TDictionary<string, integer>.Create;
spw := TStopwatch.StartNew;
{ add 10k random items }
for i := 0 to 10000 do begin
AddComboStringIfNotDuplicate(IntToStr(Floor(20000*Random)));
end;
spw.Stop;
ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;
function TForm1.AddComboStringIfNotDuplicate(AEntry: string) : boolean;
begin
result := false;
if not FComboStringsDict.ContainsKey(AEntry) then begin
FComboStringsDict.Add(AEntry, 0);
ComboBox1.Items.Add(AEntry);
result := true;
end;
end;
Adding 10k items initially takes about 0.5s this way.
{ test adding new items }
procedure TForm1.Button1Click(Sender: TObject);
var
spw : TStopwatch;
begin
spw := TStopwatch.StartNew;
if not AddComboString(IntToStr(Floor(20000*Random))) then
ListBox1.Items.Add('Did not add duplicate');
spw.Stop;
ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;
But adding each subsequent item is very fast <1ms. This is a clumsy implementation, but you could easily wrap this behaviour into a custom class. The idea is to keep your data model as separate from the visual component as possible - keep them in sync when adding or removing items but do your heavy searches on the dictionary where the lookup is fast. Removing items would still rely on .IndexOf.
I have a code some thing like this
dxMemOrdered : TdxMemData;
while not qrySandbox2.EOF do
begin
dxMemOrdered.append;
dxMemOrderedTotal.asCurrency := qrySandbox2.FieldByName('TOTAL').asCurrency;
dxMemOrdered.post;
qrySandbox2.Next;
end;
this code executes in a thread. When there are huge records say "400000" it is taking around 25 minutes to parse through it. Is there any way that i can reduce the size by optimizing the loop? Any help would be appreciated.
Update
Based on the suggestions i made the following changes
dxMemOrdered : TdxMemData;
qrySandbox2.DisableControls;
while not qrySandbox2.Recordset.EOF do
begin
dxMemOrdered.append;
dxMemOrderedTotal.asCurrency := Recordset.Fields['TOTAL'].Value;
dxMemOrdered.post;
qrySandbox2.Next;
end;
qrySandbox2.EnableControls;
and my output time have improved from 15 mins to 2 mins. Thank you guys
Without seeing more code, the only suggestion I can make is make sure that any visual control that is using the memory table is disabled. Suppose you have a cxgrid called Grid that is linked to your dxMemOrdered memory table:
var
dxMemOrdered: TdxMemData;
...
Grid.BeginUpdate;
try
while not qrySandbox2.EOF do
begin
dxMemOrdered.append;
dxMemOrderedTotal.asCurrency := qrySandbox2.FieldByName('TOTAL').asCurrency;
dxMemOrdered.Post;
qrySandbox2.Next;
end;
finally
Grid.EndUpdate;
end;
Some ideas in order of performance gain vs work to do by you:
1) Check if the SQL dialect that you are using lets you use queries that directly SELECT from/INSERT to. This depends on the database you're using.
2) Make sure that if your datasets are not coupled to visual controls, that you call DisableControls/EnableControls around this loop
3) Does this code have to run in the main program thread? Maybe you can send if off to a separate thread while the user/program continues doing something else
4) When you have to deal with really large data, bulk insertion is the way to go. Many databases have options to bulk insert data from text files. Writing to a text file first and then bulk inserting is way faster than individual inserts. Again, this depends on your database type.
[Edit: I just see you inserting the info that it's TdxMemData, so some of these no longer apply. And you're already threading, missed that ;-). I leave this suggestions in for other readers with similar problems]
It's much better to let SQL do the work instead of iterating though a loop in Delphi. Try a query such as
insert into dxMemOrdered (total)
select total from qrySandbox2
Is 'total' the only field in dxMemOrdered? I hope that it's not the primary key otherwise you are likely to have collisions, meaning that rows will not be added.
There's actually a lot you could do to speed up your thread.
The first would be to look at the problem in a broader perspective:
Am I fetching data from a cached / fast disk, possibly moved in memory?
Am I doing the right thing, when aggregating totals by hand? SQL engines are expecially optimized to do those things, all you'd need to do is to define an additional logical field where to store the SQL aggregated result.
Another little optimization that may bring an improvement over large amounts of looping is to not use constructs like:
Recordset.Fields['TOTAL'].Value
Recordset.FieldByName('TOTAL').Value
but to add the fields with the fields editor and then directly accessing the right field. You'll save a whole loop through the fields collection, that otherwise is performed on every field, on every next record.
I'm working on modernizing and fixing bugs in the codebase of a Delphi 4-era program written by someone else. A lot of the code is kinda scary by modern standards, and I can't help but wonder if some of the things I'm seeing are there because the original author didn't know about certain standard library features, or if they weren't available.
One of the more obnoxious "patterns" I see all over the app looks like this:
table := TClientDataset.Create;
with table do
begin
CloneCursor(dmDatabase.OriginalTable, false, true);
filtered := true;
active := true;
first;
while not EOF do
begin
if fieldByName('whatever').AsString = 'some criteria' then break;
next;
end;
if EOF then exit;
//do something based on the current row of the dataset
table.free;
end;
Almost every one of these groups could be replaced by a one-line call to either Lookup or Locate on the original dataset, with no need for an intermediary CDS at all. That makes me wonder, were these methods available back in the D4 days? When were Lookup and Locate first added?
Lookup and Locate were introduced in Delphi 2. It looks like the original author simply didn't take advantage of them.
Seems the Original programmer wanted to make sure that the row pointer is not changed
at all. Doing Locate (or Lookup) would change the row pointer, provoking all kinds
of data events (Datasource.OnDataChange, Dataset.AfterScroll and so on).
Doing the search with TClientDataset.CloneCursor, none of the these events ger triggered on the dmDatabase.OriginalTable and there's no need to reload the data from database.
Seems to me that is the intention. TClientDataset was presented on D3. And cloned cursors are a kind
of advanced feature - and need the dmDatabase.OriginalTable to be a CDS too.
is that possible to hide specific nodes in VirtualStringTree?
I'm implementing "filtering" feature (the VST acts as a list with columns), and I'd like to avoid reloading content each time the filter is changed - instead, much faster would be to tell VST not to render specific items ... any solutions?
VirtualTree.IsVisible[Node] := False;
There are problem using .IsVisible[] or .IsFiltered[] and it is that is very slow, i've probe filter in a tree with amoung of 25,000 nodes and is too slow.
I've found one aproach that is faster and solves the problem with scrollbar size when using Include(Node.states,vsFiltered) or (Node.States,vsVisible) is used, it consists on change manually the Node.TotalHeight value acording with the number of visible nodes (not Filtered).
For example i'm filtering 25,000 nodes and the code i was using is the follow :
procedure TFC_Articulo.Filtrar(Filtro:String);
var
Node:PVirtualNode;
Data:PArticulo;
begin
Node := TV.GetFirst;
TV.RootNode.TotalHeight:=TV.DefaultNodeHeight; // The Trick
while Assigned(Node) do
begin
Data:=TV.GetNodeData(Node);
Exclude(Node.States,vsFiltered); // By default all nodes wil be Visible
if ComparationHereForDetermineFiltering then
Include(Node.States,vsFiltered) // This node wil be filtered
else
Inc(TV.RootNode.TotalHeight,Node.NodeHeight); // Determine the Height of scrollbar
Node:=TV.GetNext(Node);
end;
TV.RootNode.TotalHeight:=TV.RootNode.TotalHeight+TV.BottomSpace;
TV.UpdateScrollBars(True);
end;
Hope this helps
Sorry bad english...
I've heard many programmers, particularly Delphi programmers scorn the use of 'with'.
I thought it made programs run faster (only one reference to parent object) and that it was easier to read the code if used sensibly (less than a dozen lines of code and no nesting).
Here's an example:
procedure TBitmap32.FillRectS(const ARect: TRect; Value: TColor32);
begin
with ARect do FillRectS(Left, Top, Right, Bottom, Value);
end;
I like using with. What's wrong with me?
One annoyance with using with is that the debugger can't handle it. So it makes debugging more difficult.
A bigger problem is that it is less easy to read the code. Especially if the with statement is a bit longer.
procedure TMyForm.ButtonClick(...)
begin
with OtherForm do begin
Left := 10;
Top := 20;
CallThisFunction;
end;
end;
Which Form's CallThisFunction will be called? Self (TMyForm) or OtherForm? You can't know without checking if OtherForm has a CallThisFunction method.
And the biggest problem is that you can make bugs easy without even knowing it. What if both TMyForm and OtherForm have a CallThisFunction, but it's private. You might expect/want the OtherForm.CallThisFunction to be called, but it really is not. The compiler would have warned you if you didn't use the with, but now it doesn't.
Using multiple objects in the with multiplies the problems. See http://blog.marcocantu.com/blog/with_harmful.html
I prefer the VB syntax in this case because here, you need to prefix the members inside the with block with a . to avoid ambiguities:
With obj
.Left = 10
.Submit()
End With
But really, there's nothing wrong with with in general.
It would be great if the with statement would be extented the following way:
with x := ARect do
begin
x.Left := 0;
x.Rigth := 0;
...
end;
You wouldn't need to declare a variable 'x'. It will be created by the compiler. It's quick to write and no confusion, which function is used.
It is not likely that "with" would make the code run faster, it is more likely that the compiler would compile it to the same executable code.
The main reason people don't like "with" is that it can introduce confusion about namespace scope and precedence.
There are cases when this is a real issue, and cases when this is a non-issue (non-issue cases would be as described in the question as "used sensibly").
Because of the possible confusion, some developers choose to refrain from using "with" completely, even in cases where there may not be such confusion. This may seem dogmatic, however it can be argued that as code changes and grows, the use of "with" may remain even after code has been modified to an extent that would make the "with" confusing, and thus it is best not to introduce its use in the first place.
In fact:
procedure TBitmap32.FillRectS(const ARect: TRect; Value: TColor32);
begin
with ARect do FillRectS(Left, Top, Right, Bottom, Value);
end;
and
procedure TBitmap32.FillRectS(const ARect: TRect; Value: TColor32);
begin
FillRectS(ARect.Left, ARect.Top, ARect.Right, ARect.Bottom, Value);
end;
Will generate exactly the same assembler code.
The performance penalty can exist if the value of the with clause is a function or a method. In this case, if you want to have good maintenance AND good speed, just do what the compiler does behind the scene, i.e. create a temporary variable.
In fact:
with MyRect do
begin
Left := 0;
Right := 0;
end;
is encoded in pseudo-code as such by the compiler:
var aRect: ^TRect;
aRect := #MyRect;
aRect^.Left := 0;
aRect^.Right := 0;
Then aRect can be just a CPU register, but can also be a true temporary variable on stack. Of course, I use pointers here since TRect is a record. It is more direct for objects, since they already are pointers.
Personally, I used with sometimes in my code, but I almost check every time the asm generated to ensure that it does what it should. Not everyone is able or has the time to do it, so IMHO a local variable is a good alternative to with.
I really do not like such code:
for i := 0 to ObjList.Count-1 do
for j := 0 to ObjList[i].NestedList.Count-1 do
begin
ObjList[i].NestedList[j].Member := 'Toto';
ObjList[i].NestedList[j].Count := 10;
end;
It is still pretty readable with with:
for i := 0 to ObjList.Count-1 do
for j := 0 to ObjList[i].NestedList.Count-1 do
with ObjList[i].NestedList[j] do
begin
Member := 'Toto';
Count := 10;
end;
or even
for i := 0 to ObjList.Count-1 do
with ObjList[i] do
for j := 0 to NestedList.Count-1 do
with NestedList[j] do
begin
Member := 'Toto';
Count := 10;
end;
but if the inner loop is huge, a local variable does make sense:
for i := 0 to ObjList.Count-1 do
begin
Obj := ObjList[i];
for j := 0 to Obj.NestedList.Count-1 do
begin
Nested := Obj.NestedList[j];
Nested.Member := 'Toto';
Nested.Count := 10;
end;
end;
This code won't be slower than with: compiler does it in fact behind the scene!
By the way, it will allow easier debugging: you can put a breakpoint, then point your mouse on Obj or Nested directly to get the internal values.
What you save in typing, you lose in readability.
Many debuggers won't have a clue what you're referring to either so debugging is more difficult.
It doesn't make programs run faster.
Consider making the code within your with statement a method of the object that you're refering to.
It's primarily a maintenance issue.
The idea of WITH makes reasonable sense from a language point of view, and the argument that it keeps code, when used sensibly, smaller and clearer has some validity. However the problem is that most commercial code will be maintained by several different people over it's lifetime, and what starts out as a small, easily parsed, construct when written can easily mutate over time into unwieldy large structures where the scope of the WITH is not easily parsed by the maintainer. This naturally tends to produce bugs, and difficult to find ones at that.
For example say we have a small function foo which contains three or four lines of code which have been wrapped inside a WITH block then there is indeed no issue. However a few years later this function may have expanded, under several programmers, into 40 or 50 lines of code still wrapped inside a WITH. This is now brittle, and is ripe for bugs to be introduced, particularly so if the maintainer stars introducing additional embedded WITH blocks.
WITH has no other benefits - code should be parsed exactly the same and run at the same speed (I did some experiments with this in D6 inside tight loops used for 3D rendering and I could find no difference). The inability of the debugger to handle it is also an issue - but one that should have been fixed a while back and would be worth ignoring if there were any benefit. Unfortunately there isn't.
This debate happens in Javascript a lot too.
Basically, that With syntax makes it very hard to tell at a glance which Left/Top/etc property/method you're calling on.You could have a local variable called Left, and a property (it's been a while since I've done delphi, sorry if the name is wrong) called Left, perhaps even a function called Left. Anyone reading the code who isn't super familiar with the ARect structure could be very very lost.
I do not like it because it makes debbuging a hassle. You cannot read the value of a variable or the like by just hovering over it with a mouse.
There's nothing wrong with it as long as you keep it simple and avoid ambiguities.
As far as I'm aware, it doesn't speed anything up though - it's purely syntactic sugar.
At work we give points for removing Withs from an existing Win 32 code base because of the extra effort needed to maintain code that uses them. I have found several bugs in a previous job where a local variable called BusinessComponent was masked by being within a With begin block for an object that a published property BusinessComponent of the same type. The compiler chose to use the published property and the code that meant to use the local variable crashed.
I have seen code like
With a,b,c,d do {except they are much longer names, just shortened here)
begin
i := xyz;
end;
It can be a real pain trying to locate where xyz comes from. If it was c, I'd much sooner write it as
i := c.xyz;
You think it's pretty trivial to understand this but not in a function that was 800 lines long that used a with right at the start!
You can combine with statements, so you end up with
with Object1, Object2, Object3 do
begin
//... Confusing statements here
end
And if you think that the debugger is confused by one with, I don't see how anyone can determine what is going on in the with block
It permits incompetent or evil programmers to write unreadble code. Therefor, only use this feature if you are neither incompetent nor evil.
... run faster ...
Not necessarily - your compiler/interpreter is generally better at optimizing code than you are.
I think it makes me say "yuck!" because it's lazy - when I'm reading code (particularly someone else's) I like to see explicit code. So I'd even write "this.field" instead of "field" in Java.
We've recently banned it in our Delphi coding stnadards.
The pros were frequently outweighing the cons.
That is bugs were being introduced because of its misuse. These didn't justify the savings in time to write or execute the code.
Yes, using with can led to (mildly) faster code execution.
In the following, foo is only evaluated once:
with foo do
begin
bar := 1;
bin := x;
box := 'abc';
end
But, here it is evaluated three times:
foo.bar := 1;
foo.bin := x;
foo.box := 'abc';
For Delphi 2005 is exist hard error in with-do statement - evaluate pointer is lost and repace with pointer up. There have to use a local variable, not object type directly.