I'm working with Delphi XE, and writing an application that is using RemObjects SDK to communicate (in case that may be relevant). I have FastMM debug on, and sometimes (not always) when I close it gives a warning about a single "Unexpected Memory Leak". "An unexpected memory leak has occurred. The unexpected small block leaks are: 117-124 bytes: UnicodeString x 1". Very occasionally, I get x2 reported.
Now, my understanding is that Strings are reference counted, and since there is no other object involved to cause the leak, what might be the situation that could cause this to happen? In this StackOverflow question people cannot find a way to make a leak.
If there is no obvious way, then I will download the latest FastMM source (it appears not to be included with the XE source).
[Edit once resolved] The solution to finding this was to install FastMM source, and enable the FullDebugMode to get the stack trace.
When using typed constants and depending on finalization order, it used to be possible for FastMM to report a leak when there really isn't.
FastMM: Leaked memory reported where I believe it shouldn't.
In short, when the FinalizedFirst unit
get's finalized, the SString constant
get's freed. After finalization of the
unit is done, the finalization of
FinalizedLast get's called. In it is
finalization, it call's the method
LeakMemory of the FinalizedFirst
method. The SString variable gets
initialized again and doesn't get
freed as the finalization of
FinalizedFirst has already run.
FinalizedLast Unit
unit FinalizedLast;
interface
uses FinalizedFirst;
implementation
initialization LeakMemory;
finalization LeakMemory;
end.
FinalizedFirst Unit
unit FinalizedFirst;
interface
procedure LeakMemory;
implementation
uses FinalizedLast;
procedure LeakMemory;
const
SString: string = '';
begin
//***** SString will get initialized once or twice depending on the
// finalization order of units. If it get's initialized twice,
// a memory leak is reported.
if SString = '' then
SString := 'FooBar';
end;
end.
Project LeakMemory
program LeakMemory;
uses
FastMM4 in 'FastMM4.pas',
Forms,
FinalizedFirst in 'FinalizedFirst.pas',
FinalizedLast in 'FinalizedLast.pas';
{$R *.RES}
begin
Application.Initialize;
Application.Run;
end.
You can leak strings by freeing records on the heap using FreeMem instead of Dispose or if you overwrite a record using System.Move or FillChar. In the first case the finalization code isn't run and in the second if the string field was filled with a nil it will think it's already cleared it.
If you want to find the location for the leak download FastMM and turn on FullDebugMode. It will include a stack trace of where the leak occurred.
The only way that comes to mind where you can leak a string without deliberately breaking it (like manually incrementing the ref count or doing some messy pointer operation) is by using threadvar.
Like the help file states,
Dynamic variables that are ordinarily
managed by the compiler (long strings,
wide strings, dynamic arrays,
variants, and interfaces) can be
declared with threadvar, but the
compiler does not automatically free
the heap-allocated memory created by
each thread of execution. If you use
these data types in thread variables,
it is your responsibility to dispose
of their memory from within the
thread, before the thread terminates.
Beside that, there's nothing that comes to mind that wasn't already stated.
[Edit by questioner] This was indeed the issue, and the specific code was as follows:
threadvar
g_szAuthentication : String;
procedure TMyBase.SetAuthentication(szUserName, szPassword: String);
begin
g_szAuthentication := '?name=' + szUserName + '&pass=' + szPassword;
end;
I usually see string leaks when they are contained inside other objects that have not been destroyed properly. For example an object that has not been freed. However, you would expect to see that object reported too.
The way to resolve this is to download and use the full version of FastMM and configure it to report stack traces when it detects leaks. When you do that you will get a full stack trace of the code that allocated the leaked object and at that point it is usually clear what the problem is.
Related
I am having a memory problem that cannot explain root cause, please help me!
The problem is as follows:
My program throws exception when i exit.
I investigated and thought that the cause of the error was the use Copymemory function to copy the record containing the variable have string data type.
Below is my demo program by delphi 2009:
In *.dpr file, I added ShareMem unit for use BorlndMM.dll instead of using FastMM memory management.
I define a struct containing a String variable.
Allocate 1 array of 2048 PByte elements.
Use Memory Copy function to copy 2 struct.
finally, Free array 2048.
when i exit program, my program threw exception.
type
TMyStructure = record
F1: TMyStructure;
str1: string;
unit 1
procedure TForm9.FormCreate(Sender: TObject);
var
i: Integer;
begin
F1.str1 := 'This is a string to demo for copying the String data type
using the Copymemory method';
end;
unit 2
procedure TForm8.btn1Click(Sender: TObject);
var
Form9: TForm9;
i: Integer;
s1: array[0..2048-1] of PByte;
begin
// Alloc 1st 2048 segments, each segment gain 1KB
for i := 0 to Length(s1) - 1 do
begin
s1[i] := AllocMem(1024);
end;
// Create Form9
Form9 := TForm9.Create(Owner);
// -> ERROR
CopyMemory(#Self.F1, #Form9.F1, SizeOf(TMyStructure));
// -> OK
// Self.F1 := Form9.F1;
// Free memory s1 array
for i := 0 to Length(s1) - 1 do
begin
FreeMem(s1[i]);
end;
end;
Note:
I have known management in memory of string is reference counting.
I've been investigating because of the use of Memorycopy, so reference count of the string is not incremented although there are 2 references to it.
I investigated that when the string was free for the first time, its reference count dropped to zero, but that memory was not returned to the OS.However, it seems that when another free variable has a nearby address, that memory loses the reference, so when accessing the second free will cause an exception.
Since no source code can be found as well as documentation describing the operation mechanism of BorlndMM.dll should read the spec as well as GetMem.inc source code file to understand the mechanism of operation FastMM and speculate the root cause of the bug. I'm guessing when a free variable, BorlndMM find ahead and after that free space and the combination leads to the memory area that can not be referenced anymore.Of course, using Memorycopy to copy two string variables is a action wrong and must fix. However, I would like to understand how the Memory Management mechanism works to explain the root cause of the above phenomenon.
Expect for help!
If possible please explain the root cause of the above phenomenon for me. And, If there is a document / source explaining how the operation of Memory Management 2005-BorlndMM , please send it to me. Thanks you so much!
The problem with your code is that it bypasses string reference counting. By copying the structure using CopyMemory you end up with two variables, both referring to the same string object. But that object still has a reference count of one.
What happens after that is unpredictable. You may encounter runtime errors. Or you may not. The behaviour is not well defined.
From your question you appear to be trying to understand why different memory managers lead to different behaviours. The thing about undefined behaviour is that it is, well, not defined. There's no point trying to reason about the behaviour. It's not possible to reason about it.
You must fix your code by replacing the memory copy with simple assignment:
F1 := Form9.F1;
I'm using DUnit and FastMM to catch unfinalized memory blocks but there seems to be a Bug. I dunno if its in FastMM, DUnit or in Delphi itself, but here goes:
When my Test Case has internal strings, the test fails with memory leaks. If I run the same test again without closing the DUnit GUI, the test passes OK. The same occours with DUnit GUI Testing, I believe for the same reason. There are no Leaks in my app, the proof is that FastMM doesn't generate the leak report in those cases.
Question 1: Is there a way to ignore them without setting the AllowedMemoryLeakSize
Question 2: I'm using Delphi 7, any news if this fix in Delphi XE?
My actual test configuration:
test.FailsOnNoChecksExecuted := True;
test.FailsOnMemoryLeak := True;
test.FailsOnMemoryRecovery := False;
test.IgnoreSetUpTearDownLeaks := True;
Here's a sample code (implementation only)
procedure TTest.Setup;
begin
A := 'test';
end;
procedure TTest.TearDown;
begin
// nothing here :)
end;
procedure TTest.Test;
begin
CheckTrue(True);
end;
Thanks!!!!
UPDATE: The problem i'm facing is documented in http://members.optusnet.com.au/mcnabp/Projects/HIDUnit/HIDUnit.html#memoryleakdetection
But the same link doesn't present a solution other than running the same test again.
Actually, strictly speaking your test is leaking memory on the first run.
It's not a bug in FastMM, DUnit or in Delphi, the bug is in your test.
Let's start by clearing up misconceptions, and explaining some inner workings:
Misconception: FastMM proves there are no leaks in my app
The problem here is that FastMM can give you a false sense of security if it doesn't detect leaks. The reason is that any kind of leak detection has to look for leaks from checkpoints. Provided all allocations done after the Start checkpoint are recovered by the End checkpoint - everything's cool.
So if you create a global object Bin, and send all objects to the Bin without destroying them, you do have a memory leak. Keep running like and your application will run out of memory. However, if the Bin destroys all its objects before the FastMM End checkpoint, FastMM won't notice anything untoward.
What's happening in your test is FastMM has a wider range on its checkpoints than DUnit leak detection. Your test leaks memory, but that memory is later recovered by the time FastMM does its checks.
Each DUnit test gets its own instance for multiple runs
DUnit creates a separate instance of your test class for each test case. However, these instances are reused for each run of the test. The simplified sequence of events is as follows:
Start checkpoint
Call SetUp
Call the test method
Call TearDown
End checkpoint
So if you have a leak between those 3 methods - even if the leak is only to the instance, and will be recovered as soon as the object is destroyed - the leak will be reported. In your case, the leak is recovered when the object is destroyed. So if DUnit had instead created & destroyed the test class for each run, no leak would be reported.
NOTE This is by design, so you can't really call it a bug.
Basically DUnit is being very strict about the principle that your test must be 100% self contained. From SetUp to TearDown, any memory you allocate (directly/indirectly) must be recovered.
Constant strings are copied whenever they are assigned to a variable
Whenever you code StringVar := 'SomeLiteralString' or StringVar := SomeConstString or StringVar := SomeResourceString the value of the constant is copied (yes, copied - not reference counted)
Again, this is by design. The intention is that if the string was retrieved from a library, you don't that string to be trashed if the library is unloaded. So it's not really a bug, just an "inconvenient" design.
So the reason your test code leaks memory on the first run is that A := 'test' is allocating memory for a copy of "test". On the subsequent runs, another copy of "test" is made, and the previous copy is destroyed - but the net memory allocation is the same.
Solution
The solution in this particular case is trivial.
procedure TTest.TearDown;
begin
A := ''; //Remove the last reference to the copy of "test" and presto leak is gone :)
end;
And in general, you shouldn't have to do much more than that. If your test creates child objects that reference copies of constant strings, those copies will be destroyed when the child objects are destroyed.
However, if any of your tests pass references to strings to any global objects / singletons (naughty, naughty, you know you shouldn't be doing that), then you'll have leaked a reference and hence some memory - even if it is recovered later.
Some further observations
Going back to the discussion about how DUnit runs tests. It is possible for separate runs of the same test to interfere with each other. E.g.
procedure TTestLeaks.SetUp;
begin
FSwitch := not FSwitch;
if FSwitch then Fail('This test fails every second run.');
end;
Expanding on the idea, you can get your test to "leak" memory on the first and every second(even) run.
procedure TTestLeaks.SetUp;
begin
FSwitch := not FSwitch;
case FSwitch of
True : FString := 'Short';
False : FString := 'This is a long string';
end;
end;
procedure TTestLeaks.TearDown;
begin
// nothing here :( <-- note the **correct** form for the smiley
end;
This doesn't really result in overall consumption of memory increasing because each alternate run recovers the same amount of memory that is leaked on every second run.
The string copying results in some interesting (and perhaps unexpected) behaviour.
var
S1, S2: string;
begin
S1 := 'Some very very long string literal';
S2 := S1; { A pointer copy and increased ref count }
if (S1 = S2) then { Very quick comparison because both vars point to the same address, therefore they're obviously equal. }
end;
However....
const
CLongStr = 'Some very very long string literal';
var
S1, S2: string;
begin
S1 := CLongStr;
S2 := CLongStr; { A second **copy** of the same constant is allocated }
if (S1 = S2) then { A full comparison has to be done because there is no shortcut to guarantee they're the same. }
end;
This does suggest an interesting, though extreme and probably ill-advised workaround just due to the sheer absurdness of the approach:
const
CLongStr = 'Some very very long string literal';
var
GlobalLongStr: string;
initialization
GlobalLongStr := CLongStr; { Creates a copy that is safely on the heap so it will be allowed to be reference counted }
//Elsewhere in a test
procedure TTest.SetUp;
begin
FString1 := GlobalLongStr; { A pointer copy and increased ref count }
FString2 := GlobalLongStr; { A pointer copy and increased ref count }
if (FString1 = FString2) then { Very efficient compare }
end;
procedure TTest.TearDown;
begin
{... and no memory leak even though we aren't clearing the strings. }
end;
Finally / Conclusion
Yes, apparently this lengthy post is going to end.
Thank you very much for asking the question.
It gave me a clue as to a related problem I remember experiencing a while back. After I've had a chance to confirm my theory, I'll post a Q & A; as others might also find it useful.
I would try the current release from Subversion first (but this version does not work with Delphi 7, only 2007 and newer):
In the commit log, one version has a comment about a fix in the area
Revision 40 Modified Fri Apr 15 23:21:27 2011 UTC (14 months ago)
move JclStartExcetionTracking and JclStopExceptionTracking out of
DUnit recursion to prevent invalid memory leak reporting
I found a way to lessen the problem: instead of working with Strings, I used ShortStrings and WideStrings in the Test Classes. No leaks poped from them.
It's not the solution, which by the way seems to be solved in the newest Delphi versions.
Bottom line is that the detected leak may be irrelevant to the test case being executed but it is a legitimate leak at the time it is detected. The memory for the string was unallocated prior to entry into the SetUp procedure and it is not deallocated prior to exiting from the TearDown procedure. So it is a memory leak until either the string variable is reassigned or the test case is destroyed.
For strings and dynamic arrays you can use SetLength(<VarName>, 0) in the TearDown procedure.
For a long time I’ve noticed that the Win64 version of my server application leak memory. While the Win32 version works fine with a relatively stable memory footprint, the memory used by the 64 bit version increases regularly – maybe 20Mb/day, without any apparent reason (Needless to say, FastMM4 did not report any memory leak for both of them). The source code is identical between the 32bit and the 64bit version. The application is built around the Indy TIdTCPServer component, it is a highly multithreaded server connected to a database that processes commands sent by other clients made with Delphi XE2.
I spend a lot of time reviewing my own code and trying to understand why the 64 bit version leaked so much memory. I ended up by using MS tools designed to track memory leaks like DebugDiag and XPerf and it seems there is a fundamental flaw in the Delphi 64bit RTL that causes some bytes to be leaked each time a thread has detached from a DLL. This issue is particularly critical for highly multithreaded applications that must run 24/7 without being restarted.
I reproduced the problem with a very basic project that is composed by an host application and a library, both built with XE2. The DLL is statically linked with the host app. The host app creates threads that just call the dummy exported procedure and exit:
Here is the source code of the library:
library FooBarDLL;
uses
Windows,
System.SysUtils,
System.Classes;
{$R *.res}
function FooBarProc(): Boolean; stdcall;
begin
Result := True; //Do nothing.
end;
exports
FooBarProc;
The host application uses a timer to create a thread that just call the exported procedure:
TFooThread = class (TThread)
protected
procedure Execute; override;
public
constructor Create;
end;
...
function FooBarProc(): Boolean; stdcall; external 'FooBarDll.dll';
implementation
{$R *.dfm}
procedure THostAppForm.TimerTimer(Sender: TObject);
begin
with TFooThread.Create() do
Start;
end;
{ TFooThread }
constructor TFooThread.Create;
begin
inherited Create(True);
FreeOnTerminate := True;
end;
procedure TFooThread.Execute;
begin
/// Call the exported procedure.
FooBarProc();
end;
Here is some screenshots that show the leak using VMMap (look at the red line named "Heap"). The following screenshots were taken within a 30 minutes interval.
The 32 bit binary shows an increase of 16 bytes, which is totally acceptable:
The 64 bit binary shows an increase of 12476 bytes (from 820K to 13296K), which is more problematic:
The constant increase of heap memory is also confirmed by XPerf:
XPerf usage
Using DebugDiag I was able to see the code path that was allocating the leaked memory:
LeakTrack+13529
<my dll>!Sysinit::AllocTlsBuffer+13
<my dll>!Sysinit::InitThreadTLS+2b
<my dll>!Sysinit::::GetTls+22
<my dll>!System::AllocateRaiseFrame+e
<my dll>!System::DelphiExceptionHandler+342
ntdll!RtlpExecuteHandlerForException+d
ntdll!RtlDispatchException+45a
ntdll!KiUserExceptionDispatch+2e
KERNELBASE!RaiseException+39
<my dll>!System::::RaiseAtExcept+106
<my dll>!System::::RaiseExcept+1c
<my dll>!System::ExitDll+3e
<my dll>!System::::Halt0+54
<my dll>!System::::StartLib+123
<my dll>!Sysinit::::InitLib+92
<my dll>!Smart::initialization+38
ntdll!LdrShutdownThread+155
ntdll!RtlExitUserThread+38
<my application>!System::EndThread+20
<my application>!System::Classes::ThreadProc+9a
<my application>!SystemThreadWrapper+36
kernel32!BaseThreadInitThunk+d
ntdll!RtlUserThreadStart+1d
Remy Lebeau helped me on the Embarcadero forums to understand what was happening:
The second leak looks more like a definite bug. During thread
shutdown, StartLib() is being called, which calls ExitThreadTLS() to
free the calling thread's TLS memory block, then calls Halt0() to
call ExitDll() to raise an exception that is caught by
DelphiExceptionHandler() to call AllocateRaiseFrame(), which
indirectly calls GetTls() and thus InitThreadTLS() when it accesses a
threadvar variable named ExceptionObjectCount. That re-allocates the
TLS memory block of the calling thread that is still in the process
of being shut down. So either StartLib() should not be calling
Halt0() during DLL_THREAD_DETACH, or DelphiExceptionHandler should
not be calling AllocateRaiseFrame() when it detects a
_TExitDllException being raised.
It seems clear for me that there is an major flaw in the Win64 way to handle threads shutdown. A such behavior prohibits the development of any multithreaded server application that must run 27/7 under Win64.
So:
What do you think of my conclusions?
Do any of you have a workaround for this issue?
QC Report 105559
A very simple work around is to re-use the thread and not create and destroy them. Threads are pretty expensive, you'll probably get a perf boost too... Kudos on the debugging though...
In order to avoid the exception memoryleak trap, you could try to put an try/except around the FoobarProc. Maybe not for a definitive solution, but to see why the axception is raised in the first place.
I usually have something like this:
try
FooBarProc()
except
if IsFatalException(ExceptObject) then // checks for system exceptions like AV, invalidop etc
OutputDebugstring(PChar(ExceptionToString(ExceptObject))) // or some other way of logging
end;
I use Delphi 10.2.3 and the problem described seems to still exist, at least under the following circumstances.
// Remark: My TFooThread is created within the 64 Bit DLL:
procedure TFooThread.Execute;
begin
while not terminated do
try
ReadBlockingFromIndySocket();
ProcessData();
except on E:Exception do
begin
LogTheException(E.Message);
// Leave loop and thread
Abort;
end
end;
end;
This leaks memory whenever the loop/thread is left. MadExcept leak report shows that an exception object is not destroyed, in my case mostly an EIdConnClosedGracefully when the connection was closed remotely. The problem was found to be the Abort statement to leave the loop and thus the thread. Indications in the leak report seem to proof the observations of #RemyLebeau. Running the exact same code in the main program instead of the 64 Bit DLL does not leak any memory.
Solution: Exchange the Abort statement with Exit.
Conclusion: A thread execution function in a 64 Bit DLL must not be left with an exception (Abort is an exception as well), or else the exception causes a memory leak.
At least this worked for me.
This is not a major leak, but more of a would be nice to tidy I think, but I have found that my Delphi XE code can leak a String. This is because it is defined as a threadvar as it needs to be, but when the thread terminates, it is apparently not tidying up such variables.
Is there a way for me to manually tidy a string on termination of the thread? Do I just assign an empty string to it, or set it to nil or something?
Assign an empty string to it, set it to nil or call Finalize() on it. They are all equivalent and they will deallocate the storage thus removing your memory leak.
In response to Marco's comment, the documentation is explicit on this:
Dynamic variables that are ordinarily
managed by the compiler (long strings,
wide strings, dynamic arrays,
variants, and interfaces) can be
declared with threadvar, but the
compiler does not automatically free
the heap-allocated memory created by
each thread of execution. If you use
these data types in thread variables,
it is your responsibility to dispose
of their memory from within the
thread, before the thread terminates.
For example:
threadvar
S: AnsiString;
S := 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';
...
S := ; // free the memory used by S
Rather bizarrely the documentation contains a clear error in the final line which should read S := nil;
It is of course easy to see for yourself that thread local variables are not disposed automatically:
program LeakMe;
{$APPTYPE CONSOLE}
threadvar
s: string;
begin
ReportMemoryLeaksOnShutdown := True;
s := 'Leak me';
end.
Some other small solution - you can use ShortString instead of String here (or any other kind of fixed length array), then memory leak disappear
We are using a class called ODNCServer here - at initialization, an TAutoObjectFactory object is created:
initialization
pAutoObjectFactory := TAutoObjectFactory.Create(ComServer, TODNCServer, Class_ODNCServer, ciSingleInstance, tmApartment);
Now FastMM is complaining about a memory leak because this object isn't freed anywhere. If I add a finalization statement like this
finalization
if assigned(pAutoObjectFactory) then
TAutoObjectFactory(pAutoObjectFactory).Free;
then the object is freed, but after the FastMM dialog about the memory leak pops up, so actually, the OS seems to be unloading the DLL, not the program. Instances of ODNCServer are created like this
fODNCServer := TODNCServer.Create(nil);
//register into ROT
OleCheck(
RegisterActiveObject(
fODNCServer.DefaultInterface, // instance
CLASS_ODNCServer, // class ID
ACTIVEOBJECT_STRONG, //strong registration flag
fODNCServerGlobalHandle //registration handle result
));
and freed like this:
if ((assigned(fODNCServer)) and (fODNCServerGlobalHandle <> -1)) then
begin
Reserved := nil;
OleCheck(RevokeActiveObject(fODNCServerGlobalHandle,Reserved));
fDTRODNCServerGlobalHandle := -1;
end;
FreeAndNil(fODNCServer);
So, does anybody know what I have to change to get rid of that memory leak? By the way, I also tried using FastMM's RegisterExpectedMemoryLeaks to register and ignore the leak, but this doesn't seem to work. Additionally, even if, it would just be a workaround and I'd like to know the right way to do this.
Don't worry about it. It's not a "leak" in the strict sense. Yes you are creating an object that is never free'd, but the keyword is "an". Singular.
Your application/DLL will not "leak" memory in the sense that it will create numerous instances of these objects,continually increase it's memory use. Furthermore the memory used by that single factory object (and others like it) will be cleaned up when the process terminates anyway.
If you showed the code you are using to call RegisterExpectedMemoryLeak() it might be possible to determine why it is not working your specific case.