Related
I need to find a certain string, in a text file, from bottom (the end of the line).
Once the string has been found, the function exit.
Here is my code, which is working fine. But, it is kind of slow.
I meant, I run this code every 5 seconds. And it consumes about 0.5% to 1% CPU time.
The text file is about 10 MB.
How to speed this up? Like, really fast and it doesn't consume much CPU time.
function TMainForm.GetVMem: string;
var
TS: TStrings;
sm: string;
i: integer;
begin
TS := TStringList.Create;
TS.LoadFromFile(LogFileName);
for i := TS.Count-1 downto 0 do
begin
Application.ProcessMessages;
sm := Trim(TS[i]);
if Pos('Virtual Memory Total =', sm) > 0 then
begin
Result := sm;
TS.Free;
exit;
end;
end;
Result := '';
TS.Free;
end;
You can use a TMemoryStream and use LoadFromFile to load the complete file content.
The you can cast the property Memory to either PChar or PAnsiChar or other character type depending on file content.
When you have the pointer, you can use it to check for content. I would avoid using string handling because it is much slower than pointer operation.
You can move the pointer from the end of memory (use Stream.Size) and search backward for the CR/LF pair (or whatever is used as line delimiter). Then from that point check for the searched string. If found, you are done, if not, loop searching the previous CR/LF.
That is more complex than the method you used but - if done correctly - will be faster.
If the file is to big to fit in memory, specially in a x32 application, you'll have to resort to read the file line by line, keeping only one line, going to the end of file. Each time you find the searched string, then save his position. At the end of file, this saved position - if any - will be the last searched file.
If the file is really very large, and the probability that the searched string is near the end, you may read the file backward (Setting TStream.position to have direct access). block by block. Then is each block, use the previous algorithm. Pay attention that the searched string may be split in two blocks depending on the block size.
Again, depending on the file size, you may split the search in several parallel searches using multi threading. Do not create to much thread neither. Pay attention that the searched string may be split in two blocks assigned to different threads depending on the block size.
Need to add many items (more than 10k) in TComboBox (I know that TComboBox is not supposed to hold many items but its not up to me to change this) without adding duplicates.
So I need to search the full list before adding. I want to avoid TComboBox.items.indexof as I need a binary search but the binary find is not available in TStrings.
So I created a temporary Tstringlist, set sorted to true and used find. But now assigning the temporary Tstringlist back to TComboBox.Items
(myCB.Items.AddStrings(myList))
is really slow as it copies the whole list. Is there any way to move the list instead of copying it? Or any other way to efficient populate my TComboBox?
There is no way to "move" the list into the combo box because the combo box's storage belongs to the internal Windows control implementation. It doesn't know any way to directly consume your Delphi TStringList object. All it offers is a command to add one item to the list, which TComboBox then uses to copy each item from the string list into the system control, one by one. The only way to avoid copying the many thousands of items into the combo box is to avoid the issue entirely, such as by using a different kind of control or by reducing the number of items you need to add.
A list view has a "virtual" mode where you only tell it how many items it should have, and then it calls back to your program when it needs to know details about what's visible on the screen. Items that aren't visible don't occupy any space in the list view's implementation, so you avoid the copying. However, system combo boxes don't have a "virtual" mode. You might be able to find some third-party control that offers that ability.
Reducing the number of items you need to put in the combo box is your next best option, but only you and your colleagues have the domain knowledge necessary to figure out the best way to do that.
As Rudy Velthuis already mentioned in the comments and assuming you are using VCL, the CB_INITSTORAGE message could be an option:
SendMessage(myCB, CB_INITSTORAGE, myList.Count, 20 * myList.Count*sizeof(Integer));
where 20 is your average string length.
Results (on a i5-7200U and 20K items with random length betwen 1 and 50 chars):
without CB_INITSTORAGE: ~ 265ms
with CB_INITSTORAGE: ~215ms
So while you can speed up things a little by preallocating the memory, the bigger issue seems to be the bad user experience. How can a user find the right element in a combobox with such many items?
Notwithstanding that 10k items is crazy to keep in a TComboBox, an efficient strategy here would be to keep a cache in a separate object. For example, declare :
{ use a TDictionary just for storing a hashmap }
FComboStringsDict : TDictionary<string, integer>;
where
procedure TForm1.FormCreate(Sender: TObject);
var
i : integer;
spw : TStopwatch;
begin
FComboStringsDict := TDictionary<string, integer>.Create;
spw := TStopwatch.StartNew;
{ add 10k random items }
for i := 0 to 10000 do begin
AddComboStringIfNotDuplicate(IntToStr(Floor(20000*Random)));
end;
spw.Stop;
ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;
function TForm1.AddComboStringIfNotDuplicate(AEntry: string) : boolean;
begin
result := false;
if not FComboStringsDict.ContainsKey(AEntry) then begin
FComboStringsDict.Add(AEntry, 0);
ComboBox1.Items.Add(AEntry);
result := true;
end;
end;
Adding 10k items initially takes about 0.5s this way.
{ test adding new items }
procedure TForm1.Button1Click(Sender: TObject);
var
spw : TStopwatch;
begin
spw := TStopwatch.StartNew;
if not AddComboString(IntToStr(Floor(20000*Random))) then
ListBox1.Items.Add('Did not add duplicate');
spw.Stop;
ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;
But adding each subsequent item is very fast <1ms. This is a clumsy implementation, but you could easily wrap this behaviour into a custom class. The idea is to keep your data model as separate from the visual component as possible - keep them in sync when adding or removing items but do your heavy searches on the dictionary where the lookup is fast. Removing items would still rely on .IndexOf.
It is important for a MIDI player to playback the notes as precisely as possible. I never quite succeeded in that, always blaming the timer (see a previous thread: How to prevent hints interrupting a timer). Recently I acquired ProDelphi and started measuring what exactly consumed that much time. The result was quite surprising, see the example code below.
procedure TClip_View.doMove (Sender: TObject; note, time, dure, max_time: Int32);
var x: Int32;
begin
{$IFDEF PROFILE}Profint.ProfStop; Try; Profint.ProfEnter(#self,1572 or $58B20000); {$ENDIF}
Image.Picture.Bitmap.Canvas.Pen.Mode := pmNot;
Image.Picture.Bitmap.Canvas.MoveTo (FPPos, 0);
Image.Picture.Bitmap.Canvas.LineTo (FPPos, Image.Height);
x := time * GPF.PpM div MIDI_Resolution;
Image.Picture.Bitmap.Canvas.Pen.Mode := pmNot;
Image.Picture.Bitmap.Canvas.MoveTo (x, 0);
Image.Picture.Bitmap.Canvas.LineTo (x, Image.Height);
FPPos := x;
// Bevel.Left := time * GPF.PpM div MIDI_Resolution;
{$IFDEF PROFILE}finally; Profint.ProfExit(1572); end;{$ENDIF}
end; // doMove //
The measurements are (without debug code on an Intel i7-920, 2,7Ghz):
95 microseconds for the code as shown
5.609 milliseconds when all is commented out except for the now commented out statement (Bevel.Left :=)
0.056 microseconds when all code is replaced by x := time * GPF.PpM div MIDI_Resolution;
Just moving around a Bevel costs 60 times as much CPU as just drawing on a Canvas. That surprised me. The results of measurement 1 are very audible (there is more going on than just this), but 2 and 3 not. I need some form of feedback to the user as what the player now is processing, some sort of line over a piano roll is the accepted way. In my never ending quest of reducing CPU cycles in the timed-event loop I have some questions:
Why does moving around a bevel cost that much time?
Is there a way to reduce more CPU cycles than in drawing on a bitmap?
Is there a way to reduce the flicker when drawing?
You won't be able to change the world, nor VCL nor Windows. I suspect you are asking to much to those...
IMHO you should better change a little bit your architecture:
Sound processing shall be in one (or more) separated thread(s), and should not be at all linked to the UI (e.g. do not send GDI messages from it);
UI refresh shall be made using a timer with a 500 ms resolution (half a second refresh sounds reactive enough), not every time there is a change.
That is, the sequencer won't refresh the UI, but the UI will ask periodically the sequencer what is its current status. This will be IMHO much smoother.
To answer your exact questions:
"Moving a bevel" is in fact sending several GDI messages, and the rendering will be made by the GDI stack (gdi32.dll) using a temporary bitmap;
Try to use a smaller bitmap, or try using a Direct X buffer mapping;
Try DoubleBuffered := true on your TForm.OnCreate event, or use a dedicated component (TPaintBox) with a global bitmap for the whole component content, with something like that for the message WM_ERASEBKGND.
Some code:
procedure TMyPaintBox.WMEraseBkgnd(var Message: TWmEraseBkgnd);
begin
Message.Result := 1; // no erasing is necessary after this method call
end;
I have the feeling that your bitmap buffering is wrong. When you move your clip it shouldn't have to be redrawn at all. you could try with This clip component structure:
TMidiClip = Class(TControl)
Private
FBuffer: TBitmap;
FGridPos: TPoint;
FHasToRepaint: Boolean;
Public
Procedure Paint; Override; // you only draw the bitmap on the control canvas
Procedure Refresh; // you recompute the FBuffer.canvas
End;
When you change some properties such as "clip tick length" you set "FHasToRepaint" to true but not when changing "FGridPos" (position on the grid). So most of the time, in your Paint event, you only have a copy of your FBuffer.
Actually, this is very dependent on the design of your grid and its children (clips).
I might be wrong but it seems that your design is not enough decomposed in Controls: the master grid should be a TControl, a clip should be a TControl, even the events on a clip should be some TControls...You can only define a strongly optimized bitmap-buffering system by this way (aka "double-buffering").
About the Timer: you should use a musical clock which processes per audio sample otherwise you can't have a good enough resolution. Such a clock can be implemented using the "Windows Audio driver" (mmsystem.pas) or an "Asio" driver (you have an interface in BASS, The Delphi Asio Vst project, for example).
By far the best way to tackle this is to stop the GUI message queue interfering with your MIDI player. Put the MIDI player on a background thread so that it can do its work without interruption from the main thread. Naturally this relies on you running on a machine with more than a single processor but it's not unreasonable to take that for granted nowadays.
Judging from your comments it looks like your audio processing is being blocked by the UI thread. Don't let that happen and your audio problems will disappear. If you are using something like TThread.Synchronize to fire VCL events from the audio thread then that will block on the UI thread. Use an asynchronous communication method instead.
Your proposed alternative of speeding up the VCL is not really viable. You can't modify the VCL and even if you could, the bottleneck could easily be the underlying Windows code.
I have a fairly complex and large application that hands loads and loads of data. Is there a speedy way to add items to ComboBox that doesn't take so long? On my P3 3.2ghz, the following snippet takes just under a second to add around 32,000 items. (MasterCIList is a StringList with strings typically 20 - 30 bytes long).
with LookupComboBox do
begin
Items.BeginUpdate;
Items.Clear;
for i := 0 to MasterCIList.Count - 1 do
Items.Add(MasterCIList[i]);
Items.EndUpdate;
end;
Drilling down into the VCL, it appears that in TComboBoxStrings.Add, there is a call to
Result := SendMessage(ComboBox.Handle, CB_ADDSTRING, 0, Longint(PChar(S)));
I'm guessing this is really taking up time (okay, I know it is). Is there another way to populate the Items that is speedier? Any high-speed combo boxes available? I have the TMS components but they seem to be extensions of TComboBox.
For instance, I have the PlusMemo which seems to be a total rewrite of a TMemo. I can easily add a million line in a second to a PlusMemo. A TMemo, I don't think so.
Thank you so much for your time!
Sorry if I'm a nuisance, but I doubt a TComboBox with 32,000 items is even remotely ''usable'' --- I'd say there's a reason why it's slow: it was never meant to do this :)
Would there be a possibility to filter the data, and only load a subset? To be more concrete, in one particular database application I've been working on, the user can search for a person. We let the user type at least 3 or 4 characters of the name, and only then begin to return results in a listbox. This has greatly increased usability of the search form, also greatly speeding up the whole process.
Would you be able to follow a similar approach?
Or, on a completely different take, perhaps you could take a look at the VirtualTreeView component --- either for direct use, or for inspiration.
I agree that 32K items is a ridiculous amount to have in a combobox... That being said, you might try adding the items to a TStringList first and then use a combination of Begin/EndUpdate and AddStrings:
SL := TStringList.Create;
try
// Add your items to the stringlist
ComboBox.Items.BeginUpdate;
try
ComboBox.Items.AddStrings(YourStringList);
finally
ComboBox.Items.EndUpdate;
end;
finally
SL.Free;
end;
The code compiles, but I didn't test it further than that; I've never felt the need to add more than a few dozen items to a combobox or listbox. If any more items might be needed than that, I find a way to filter before populating the list so there are fewer items.
Just out of curiosity, how do you expect a user to sort through that many items to make a decision?
var
Buffer: TStringList;
begin
Buffer := TStringList.Create;
try
// --> Add items to Buffer here <--
ComboBox.Items := Buffer;
finally
FreeAndNil(Buffer);
end;
end;
This is the fastest way we've found to update a visual control.
The VCL does BeginUpdate, Clear, and EndUpdate internally.
If you don't believe me, profile it.
perhaps cmb.Items.Assign(myStringList) will help.
here's a wild idea: i haven't tried it but you might check to see if there's a way to virtually load the combobox by setting the number of items and then owner drawing. please pardon this crazy idea but i think i've heard of this being available somehow. irrelevant: this is how it's done in Palm OS...where the faster way to load the combobox is to not load it all... ;-)
Not an answer, but why on earth would you want 32,000 items in a combo box? That is a terrible way to store that much data.
i agree; it's a bad practice...
It's me again. I'm adding 32,000 items cause I need to. That's one of many controls in my application that has "lots" of items. I need to have that many items. It works just fine looking things up. Perfectly in fact. I'm just trying to optimize things. The users find things just fine since they are in a certain logical order.
Everything I've seem so far with Assign and AddStrings is that they eventually end up in Add with the SendMessage call. So I'll keep looking.
Thanks for the feedback.
use backgroundworker for adding MasterCIList items.after complete adding items use only AddStrings.
procedure TForm2.BackgroundWorker1Work(Worker: TBackgroundWorker);
var
I: Integer;
begin
MasterCIList.BeginUpdate;
try
MasterCIList.Capacity := 32 * 1024; // if derminate count of items
for I := 1 to 32 * 1024 do
begin
MasterCIList.Add(IntToStr(I));
{ if need progess }
if I mod 300 = 0 then
Worker.ReportProgress((I * 100) div MasterCIList.Capacity);
{ if need cancelable }
if (I mod 100 = 0) and Worker.CancellationPending then
Break;
end;
finally
MasterCIList.EndUpdate;
end;
end;
procedure TForm2.BackgroundWorker1WorkComplete(Worker: TBackgroundWorker;
Cancelled: Boolean);
begin
LookupComboBox.Items.AddStrings(MasterCIList );
// AddStrings use beginupdate..endupdate in itself
end;
Maybe you can try this?
"Speeding up adding items to a combobox or listbox"
http://blogs.msdn.com/b/oldnewthing/archive/2004/06/10/152612.aspx
Perhaps you can use a database engine in the back-end and use a data aware component. Then the things will be much more quicker and usable. Perhaps if you'll try to describe what do you try to accomplish we'll help you further. In any case, your UI design is, let's say, odd. And for this perhaps the Embarcadero forums will help you better.
I implement this in a different manner. First i removed the combobox control and take textbox control and assign it autocomplete to custom source where the custom source string collection is 32k items.I get the selected value from a new query on controls validation.
So it can replace combobox functionality. Mostly about 32k items people dont scroll but they keep entering key strokes and is catched by our custom auto complete source..
I've heard many programmers, particularly Delphi programmers scorn the use of 'with'.
I thought it made programs run faster (only one reference to parent object) and that it was easier to read the code if used sensibly (less than a dozen lines of code and no nesting).
Here's an example:
procedure TBitmap32.FillRectS(const ARect: TRect; Value: TColor32);
begin
with ARect do FillRectS(Left, Top, Right, Bottom, Value);
end;
I like using with. What's wrong with me?
One annoyance with using with is that the debugger can't handle it. So it makes debugging more difficult.
A bigger problem is that it is less easy to read the code. Especially if the with statement is a bit longer.
procedure TMyForm.ButtonClick(...)
begin
with OtherForm do begin
Left := 10;
Top := 20;
CallThisFunction;
end;
end;
Which Form's CallThisFunction will be called? Self (TMyForm) or OtherForm? You can't know without checking if OtherForm has a CallThisFunction method.
And the biggest problem is that you can make bugs easy without even knowing it. What if both TMyForm and OtherForm have a CallThisFunction, but it's private. You might expect/want the OtherForm.CallThisFunction to be called, but it really is not. The compiler would have warned you if you didn't use the with, but now it doesn't.
Using multiple objects in the with multiplies the problems. See http://blog.marcocantu.com/blog/with_harmful.html
I prefer the VB syntax in this case because here, you need to prefix the members inside the with block with a . to avoid ambiguities:
With obj
.Left = 10
.Submit()
End With
But really, there's nothing wrong with with in general.
It would be great if the with statement would be extented the following way:
with x := ARect do
begin
x.Left := 0;
x.Rigth := 0;
...
end;
You wouldn't need to declare a variable 'x'. It will be created by the compiler. It's quick to write and no confusion, which function is used.
It is not likely that "with" would make the code run faster, it is more likely that the compiler would compile it to the same executable code.
The main reason people don't like "with" is that it can introduce confusion about namespace scope and precedence.
There are cases when this is a real issue, and cases when this is a non-issue (non-issue cases would be as described in the question as "used sensibly").
Because of the possible confusion, some developers choose to refrain from using "with" completely, even in cases where there may not be such confusion. This may seem dogmatic, however it can be argued that as code changes and grows, the use of "with" may remain even after code has been modified to an extent that would make the "with" confusing, and thus it is best not to introduce its use in the first place.
In fact:
procedure TBitmap32.FillRectS(const ARect: TRect; Value: TColor32);
begin
with ARect do FillRectS(Left, Top, Right, Bottom, Value);
end;
and
procedure TBitmap32.FillRectS(const ARect: TRect; Value: TColor32);
begin
FillRectS(ARect.Left, ARect.Top, ARect.Right, ARect.Bottom, Value);
end;
Will generate exactly the same assembler code.
The performance penalty can exist if the value of the with clause is a function or a method. In this case, if you want to have good maintenance AND good speed, just do what the compiler does behind the scene, i.e. create a temporary variable.
In fact:
with MyRect do
begin
Left := 0;
Right := 0;
end;
is encoded in pseudo-code as such by the compiler:
var aRect: ^TRect;
aRect := #MyRect;
aRect^.Left := 0;
aRect^.Right := 0;
Then aRect can be just a CPU register, but can also be a true temporary variable on stack. Of course, I use pointers here since TRect is a record. It is more direct for objects, since they already are pointers.
Personally, I used with sometimes in my code, but I almost check every time the asm generated to ensure that it does what it should. Not everyone is able or has the time to do it, so IMHO a local variable is a good alternative to with.
I really do not like such code:
for i := 0 to ObjList.Count-1 do
for j := 0 to ObjList[i].NestedList.Count-1 do
begin
ObjList[i].NestedList[j].Member := 'Toto';
ObjList[i].NestedList[j].Count := 10;
end;
It is still pretty readable with with:
for i := 0 to ObjList.Count-1 do
for j := 0 to ObjList[i].NestedList.Count-1 do
with ObjList[i].NestedList[j] do
begin
Member := 'Toto';
Count := 10;
end;
or even
for i := 0 to ObjList.Count-1 do
with ObjList[i] do
for j := 0 to NestedList.Count-1 do
with NestedList[j] do
begin
Member := 'Toto';
Count := 10;
end;
but if the inner loop is huge, a local variable does make sense:
for i := 0 to ObjList.Count-1 do
begin
Obj := ObjList[i];
for j := 0 to Obj.NestedList.Count-1 do
begin
Nested := Obj.NestedList[j];
Nested.Member := 'Toto';
Nested.Count := 10;
end;
end;
This code won't be slower than with: compiler does it in fact behind the scene!
By the way, it will allow easier debugging: you can put a breakpoint, then point your mouse on Obj or Nested directly to get the internal values.
What you save in typing, you lose in readability.
Many debuggers won't have a clue what you're referring to either so debugging is more difficult.
It doesn't make programs run faster.
Consider making the code within your with statement a method of the object that you're refering to.
It's primarily a maintenance issue.
The idea of WITH makes reasonable sense from a language point of view, and the argument that it keeps code, when used sensibly, smaller and clearer has some validity. However the problem is that most commercial code will be maintained by several different people over it's lifetime, and what starts out as a small, easily parsed, construct when written can easily mutate over time into unwieldy large structures where the scope of the WITH is not easily parsed by the maintainer. This naturally tends to produce bugs, and difficult to find ones at that.
For example say we have a small function foo which contains three or four lines of code which have been wrapped inside a WITH block then there is indeed no issue. However a few years later this function may have expanded, under several programmers, into 40 or 50 lines of code still wrapped inside a WITH. This is now brittle, and is ripe for bugs to be introduced, particularly so if the maintainer stars introducing additional embedded WITH blocks.
WITH has no other benefits - code should be parsed exactly the same and run at the same speed (I did some experiments with this in D6 inside tight loops used for 3D rendering and I could find no difference). The inability of the debugger to handle it is also an issue - but one that should have been fixed a while back and would be worth ignoring if there were any benefit. Unfortunately there isn't.
This debate happens in Javascript a lot too.
Basically, that With syntax makes it very hard to tell at a glance which Left/Top/etc property/method you're calling on.You could have a local variable called Left, and a property (it's been a while since I've done delphi, sorry if the name is wrong) called Left, perhaps even a function called Left. Anyone reading the code who isn't super familiar with the ARect structure could be very very lost.
I do not like it because it makes debbuging a hassle. You cannot read the value of a variable or the like by just hovering over it with a mouse.
There's nothing wrong with it as long as you keep it simple and avoid ambiguities.
As far as I'm aware, it doesn't speed anything up though - it's purely syntactic sugar.
At work we give points for removing Withs from an existing Win 32 code base because of the extra effort needed to maintain code that uses them. I have found several bugs in a previous job where a local variable called BusinessComponent was masked by being within a With begin block for an object that a published property BusinessComponent of the same type. The compiler chose to use the published property and the code that meant to use the local variable crashed.
I have seen code like
With a,b,c,d do {except they are much longer names, just shortened here)
begin
i := xyz;
end;
It can be a real pain trying to locate where xyz comes from. If it was c, I'd much sooner write it as
i := c.xyz;
You think it's pretty trivial to understand this but not in a function that was 800 lines long that used a with right at the start!
You can combine with statements, so you end up with
with Object1, Object2, Object3 do
begin
//... Confusing statements here
end
And if you think that the debugger is confused by one with, I don't see how anyone can determine what is going on in the with block
It permits incompetent or evil programmers to write unreadble code. Therefor, only use this feature if you are neither incompetent nor evil.
... run faster ...
Not necessarily - your compiler/interpreter is generally better at optimizing code than you are.
I think it makes me say "yuck!" because it's lazy - when I'm reading code (particularly someone else's) I like to see explicit code. So I'd even write "this.field" instead of "field" in Java.
We've recently banned it in our Delphi coding stnadards.
The pros were frequently outweighing the cons.
That is bugs were being introduced because of its misuse. These didn't justify the savings in time to write or execute the code.
Yes, using with can led to (mildly) faster code execution.
In the following, foo is only evaluated once:
with foo do
begin
bar := 1;
bin := x;
box := 'abc';
end
But, here it is evaluated three times:
foo.bar := 1;
foo.bin := x;
foo.box := 'abc';
For Delphi 2005 is exist hard error in with-do statement - evaluate pointer is lost and repace with pointer up. There have to use a local variable, not object type directly.