Let's start with this simple Delphi 2010 snippet:
var
StringList: TStringList;
begin
ReportMemoryLeaksOnShutdown := True;
StringList := TStringList.Create;
StringList.LoadFromFile('c:\fateh.txt');
RegisterExpectedMemoryLeak(StringList);
FastMM4 report the memory leak again and again even with Addr(StringList) as parameter
so how to Register Expected MemoryLeak and why the methods sitting above doesn't work
thank's in advance.
You only registered the leak of the string list object. You also need to register that you are leaking all the objects owned by the string list. In this case it owns StringList.Count instances of string objects. The memory manager does not know that those strings are owned by the string list object and so will also be leaked.
And that's much easier said than done. Because you need to find the start of the memory block that represents a string. That's at a fixed offset from the string's first character, and the offset depends on which Delphi version you use.
In Unicode Delphi, in 32 bit code, the offset is 12 bytes. So the following will register the leaked strings:
for i := 0 to StringList.Count-1 do
if StringList[i]<>'' then
RegisterExpectedMemoryLeak(PByte(StringList[i])-12);
Even when you do that you'll still get two reported memory leaks. At least one of those is explained by the dynamic array that is owned by the string list, TStringList.FList. If you wanted to register that leak, then you'll need to do some more hacking because again you'll have to rely on implementation details as to where that array is stored.
Related
I am having a memory problem that cannot explain root cause, please help me!
The problem is as follows:
My program throws exception when i exit.
I investigated and thought that the cause of the error was the use Copymemory function to copy the record containing the variable have string data type.
Below is my demo program by delphi 2009:
In *.dpr file, I added ShareMem unit for use BorlndMM.dll instead of using FastMM memory management.
I define a struct containing a String variable.
Allocate 1 array of 2048 PByte elements.
Use Memory Copy function to copy 2 struct.
finally, Free array 2048.
when i exit program, my program threw exception.
type
TMyStructure = record
F1: TMyStructure;
str1: string;
unit 1
procedure TForm9.FormCreate(Sender: TObject);
var
i: Integer;
begin
F1.str1 := 'This is a string to demo for copying the String data type
using the Copymemory method';
end;
unit 2
procedure TForm8.btn1Click(Sender: TObject);
var
Form9: TForm9;
i: Integer;
s1: array[0..2048-1] of PByte;
begin
// Alloc 1st 2048 segments, each segment gain 1KB
for i := 0 to Length(s1) - 1 do
begin
s1[i] := AllocMem(1024);
end;
// Create Form9
Form9 := TForm9.Create(Owner);
// -> ERROR
CopyMemory(#Self.F1, #Form9.F1, SizeOf(TMyStructure));
// -> OK
// Self.F1 := Form9.F1;
// Free memory s1 array
for i := 0 to Length(s1) - 1 do
begin
FreeMem(s1[i]);
end;
end;
Note:
I have known management in memory of string is reference counting.
I've been investigating because of the use of Memorycopy, so reference count of the string is not incremented although there are 2 references to it.
I investigated that when the string was free for the first time, its reference count dropped to zero, but that memory was not returned to the OS.However, it seems that when another free variable has a nearby address, that memory loses the reference, so when accessing the second free will cause an exception.
Since no source code can be found as well as documentation describing the operation mechanism of BorlndMM.dll should read the spec as well as GetMem.inc source code file to understand the mechanism of operation FastMM and speculate the root cause of the bug. I'm guessing when a free variable, BorlndMM find ahead and after that free space and the combination leads to the memory area that can not be referenced anymore.Of course, using Memorycopy to copy two string variables is a action wrong and must fix. However, I would like to understand how the Memory Management mechanism works to explain the root cause of the above phenomenon.
Expect for help!
If possible please explain the root cause of the above phenomenon for me. And, If there is a document / source explaining how the operation of Memory Management 2005-BorlndMM , please send it to me. Thanks you so much!
The problem with your code is that it bypasses string reference counting. By copying the structure using CopyMemory you end up with two variables, both referring to the same string object. But that object still has a reference count of one.
What happens after that is unpredictable. You may encounter runtime errors. Or you may not. The behaviour is not well defined.
From your question you appear to be trying to understand why different memory managers lead to different behaviours. The thing about undefined behaviour is that it is, well, not defined. There's no point trying to reason about the behaviour. It's not possible to reason about it.
You must fix your code by replacing the memory copy with simple assignment:
F1 := Form9.F1;
I think that C++ supports something on the lines of :
Object objects[100];
This would instantiate a 100 objects, right? Is it possible to do this in Delphi (specifically 2007)? Something other than:
for i:=0 to 99 do
currentObject = TObject.Create;
or using the Allocate function, with a passed size value a hundred times the TObject size, because that just allocates memory, it doesn't actually divide the memory and 'give' it to the objects.
If my assumption that the c++ instantiation is instant rather than under-the-hood-iterative, I apologize.
What you are looking for is impossible because
Delphi does not support static (stack allocated) objects.
Delphi objects do not have default constructors that can be automatically invoked by compiler.
So that is not a lack of 'sugar syntax'.
For the sake of complete disclosure:
Delphi also supports legacy 'old object model' (Turbo Pascal object model) which allows statically allocated objects;
Dynamic object allocation itself does not prevent automatic object instantiation syntax, but makes such a syntax undesirable;
Automatic object instantiation syntax is impossible because Delphi does not have default constructors: Delphi compiler never instantiate objects implicitly because it does not know what constructor to call.
While you can't do what you want using objects, if your objects are relatively simple, you may be able to get what you want by using an array of records.
Records in Delphi can have properties (including setters and getters), and class and instance methods. They are created automatically when declared, so declaring an array of them will create them all without iterating.
For more info: http://docwiki.embarcadero.com/RADStudio/XE3/en/Structured_Types#Records_.28advanced.29.
(I'm not sure when the new functionality was added to Delphi, it may well be after the 2007 version).
I don't know of any non-hacky way to do this besides iterating:
var
MyObjects: array[0..99] of TMyObject;
i: Integer;
begin
for i := 0 to 99 do
MyObjects[i] := TMyObject.Create;
end;
That declaration wouldn't create 100 objects, it would just give you an array of 100 object references that point to nothing useful.
Creating an object is a two step process. The first step is allocating memory (which your code also doesn't), the second step is calling the constructor (Create method) to initialize that memory, create additional objects, etc, etc.
The allocation part can be done without the loop, but the constructor needs to be called to intialize each instance.
Many VCL classes don't have an additional constructor. They just have the empty constructor that does nothing. In that case, there is no need to call it.
For instance, to fetch an array of stringlists, you can use the following code, adjusted from this example:
type
TStringListArray = array of TStringList;v
var
Instances: array of Byte;
function GetStringLists(Number: Integer): TStringListArray;
var
i: Integer;
begin
// Allocate instance memory for all of them
SetLength(Instances, Number * TStringList.InstanceSize);
// Zero the memory.
FillChar(Instances[0], Length(Instances), 0);
// Allocate array for object references.
SetLength(Result, Number);
for i := 0 to High(Result) do
begin
// Store object reference.
Result[i] := #Instances[i * TStringList.InstanceSize];
// Set the right class.
PPointer(Result[i])^ := TStringList;
// Call the constructor.
Result[i].Create;
end;
end;
And to get an array of 100 stringlists:
var
x: TStringListArray;
begin
x := GetStringLists(100);
So while this procedure may save you a neglectable amount of time, and may theoretically be more memory-efficient (less fragmentation), you will still need the loop. No easy way out.
It is somewhat possible in Delphi (but is not very practical in a Delphi environment, unless you are writing a custom memory manager). Creating an object instance is a two-step process - allocating memory for the object, and constructing the object's members inside of that memory. There is nothing requiring the memory of a given object to be allocated individually. You can allocate a larger block of memory and construct multiple objects inside of that block, taking advantage of a feature of Delphi that calls a constructor like a normal method if it is called from an instance variable instead of a class type, eg:
var
objects: array of Byte;
obj: TSomeClass;
begin
SetLength(objects, 100 * TSomeClass.InstanceSize);
FillChar(objects[0], 0, Length(objects));
for i := 0 to 99 do
begin
obj := TSomeClass.InitInstance(#objects[i * TSomeClass.InstanceSize]));
obj.Create;
end;
...
for i := 0 to 99 do
begin
obj := TSomeClass(#objects[i * TSomeClass.InstanceSize]);
obj.CleanupInstance;
end;
SetLength(objects, 0);
end;
This is not much different than what C++ does behind the scenes when declaring an array of object instances, only C++ supports declaring the array statically and it will call the constructors and destructors automatically, whereas Delphi does not support that.
There are a number of third-party implementations that allow you to allocate objects on the stack or in user-defined buffers, such as:
Objects on the Stack: A Delphi Relic
Allocate objects on stack, at specified memory address, or through any memory manager
Just to name a couple.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
TStringList of objects taking up tons of memory in Delphi XE
I have a program that takes a while to run and uses a fair amount of memory (starts out at about 800 mb, ends at about 1.1 GB).
The problem I'm having is that even though the amount of memory it's using is fine, the amount of memory commited goes over 4GB and gives me an "Out of Memory" error.
Here is a screen shot of Resource Monitor:
The main thing that takes up the memory is a TStringList filled with an object that has a bunch of different values and a few stringlists
Does anyone know why it is committing so much more memory than I need, and how can I make my program not do that?
The commit size is the amount of memory allocated by the program (but is not necessarily in physical memory). As mentioned in the comments, it is quite possible that the program has a memory leak (e.g., it creates objects possibly but does not delete them). So even if the amount of memory in use is "reasonable", it doesn't necessarily represent the amount actually allocated, and that amount in a 32-bit system can't "typically" exceed 4GB. This question discusses leak detection tools that might be useful.
Your software probably contains a memory leak. Put this little piece of code right after your program's "main" block (.dpr file - press F7 to get there).
{$WARN SYMBOL_PLATFORM OFF}
ReportMemoryLeaksOnShutdown := DebugHook <> 0;
So, the end result should look similar to this:
begin
{$WARN SYMBOL_PLATFORM OFF}
ReportMemoryLeaksOnShutdown := DebugHook <> 0;
Application.Initialize;
Application.CreateForm(TForm1, Form1);
Application.Run;
end.
Then, run your program until you get some big memory usage and simply quit it. Delphi will show a nice window giving you the size of the memory leak as well as some tips about where they might be happening.
Good luck!
You didn't showed code, but this effect appears to be the result of a memory leak.
So, this TStringList (I'll call it parent) have objects associated, which have its own child TStringLists (which I suppose have only strings).
The easier way to do this memory management is letting the TStringList do it for you:
If in the Delphi version you have TStringList has the OwnsObjects property, set it to true. With this, every call to Delete and Clear methods will call the destructor of the objects associated with the parent list - and this will work even if cll parent.Free. If it doesn't implement, you'll have to do it yourself - overriding the Delete, Clear and Destroy methods of TStringList in a descendant class.
Make sure the destructor of the child objects clean correctly the objects created inside it - if one of the stringlists in a child objects has owned objects, you could use the same strategy in the above item.
An (oversimplified) example:
TMyChildObject = class
private
fMyListOfMagazineNames,
// This one will have TBitmap instances associated with the dogs names
fMyListOfDogNamesAndPhotos : TStringList;
fName: String;
public
//[properties, methods, whatever]
property Name: string read fName write fName;
constructor Create; virtual;
destructor Destroy; override;
end;
constructor TMyChildObject.Create;
begin
inherited Create;
fMyListOfMagazineNames = TStringList.Create;
fMyListOfDogNamesAndPhotos = TStringList.Create;
fMyListOfDogNamesAndPhotos.OwnsObjects := True;
end;
destructor TMyChildObject.Destroy;
begin
fMyListOfMagazineNames.Free;
fMyListOfDogNamesAndPhotos.Free;
inherited Destroy;
end;
An example of use:
var
MyObjList: TStringList;
begin
MyObjList := TStringList.Create;
MyObjList.OwnsObjects := True;
// Create your child objects
// And Free them at once
MyObjList.Free;
end;
As was stated in your previous question - thanks for having selected one answer.
The main solution to your problem is not to allocate all objects in memory. If your 28 MB initial data consume more than 1 GB of memory, you have a lot of redundant information.
You shall modify your algorithm as such:
Use a SQL database instead to process your data without memory issue (with proper indexes) - may be SQLite3 or embedded Firebird will work very well;
Leave the data on the input files, the memory-map them instead of allocating a copy in memory: use getter methods for each properties, to retrieve the value on the fly;
Use a NoSQL database (like this) which may be faster than a SQL database for inserting rows, and consume less disk space and memory;
Or... consider compressing the objects content (a fast compression is a good option).
This is not a major leak, but more of a would be nice to tidy I think, but I have found that my Delphi XE code can leak a String. This is because it is defined as a threadvar as it needs to be, but when the thread terminates, it is apparently not tidying up such variables.
Is there a way for me to manually tidy a string on termination of the thread? Do I just assign an empty string to it, or set it to nil or something?
Assign an empty string to it, set it to nil or call Finalize() on it. They are all equivalent and they will deallocate the storage thus removing your memory leak.
In response to Marco's comment, the documentation is explicit on this:
Dynamic variables that are ordinarily
managed by the compiler (long strings,
wide strings, dynamic arrays,
variants, and interfaces) can be
declared with threadvar, but the
compiler does not automatically free
the heap-allocated memory created by
each thread of execution. If you use
these data types in thread variables,
it is your responsibility to dispose
of their memory from within the
thread, before the thread terminates.
For example:
threadvar
S: AnsiString;
S := 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';
...
S := ; // free the memory used by S
Rather bizarrely the documentation contains a clear error in the final line which should read S := nil;
It is of course easy to see for yourself that thread local variables are not disposed automatically:
program LeakMe;
{$APPTYPE CONSOLE}
threadvar
s: string;
begin
ReportMemoryLeaksOnShutdown := True;
s := 'Leak me';
end.
Some other small solution - you can use ShortString instead of String here (or any other kind of fixed length array), then memory leak disappear
Consider the following scenario:
type
PStructureForSomeCDLL = ^TStructureForSomeCDLL;
TStructureForSomeCDLL = record
pName: PAnsiChar;
end
function FillStructureForDLL: PStructureForSomeDLL;
begin
New(Result);
// Result.pName := PAnsiChar(SomeObject.SomeString); // Old D7 code working all right
Result.pName := Utf8ToAnsi(UTF8Encode(SomeObject.SomeString)); // New problematic unicode version
end;
...code to pass FillStructureForDLL to DLL...
The problem in unicode version is that the string conversion involved now returns a new string on stack and that's reclaimed at the end of the FillStructureForDLL call, leaving the DLL with corrupted data. In old D7 code, there were no intermediate conversion funcs and thus no problem.
My current solution is a converter function like below, which is IMO too much of an hack. Is there a more elegant way of achieving the same result?
var gKeepStrings: array of AnsiString;
{ Convert the given Unicode value S to ANSI and increase the ref. count
of it so that returned pointer stays valid }
function ConvertToPAnsiChar(const S: string): PAnsiChar;
var temp: AnsiString;
begin
SetLength(gKeepStrings, Length(gKeepStrings) + 1);
temp := Utf8ToAnsi(UTF8Encode(S));
gKeepStrings[High(gKeepStrings)] := temp; // keeps the resulting pointer valid
// by incresing the ref. count of temp.
Result := PAnsiChar(temp);
end;
One way might be to tackle the problem before it becomes a problem, by which I mean adapt the class of SomeObject to maintain an ANSI Encoded version of SomeString (ANSISomeString?) for you alongside the original SomeString, keeping the two in step in a "setter" for the SomeString property (using the same UTF8 > ANSI conversion you are already doing).
In non-Unicode versions of the compiler make ANSISomeString be simply a "copy" of SomeString string, which will of course not be a copy, merely an additional ref count on SomeString. In the Unicode version it references a separate ANSI encoding with the same "lifetime" as the original SomeString.
procedure TSomeObjectClass.SetSomeString(const aValue: String);
begin
fSomeString := aValue;
{$ifdef UNICODE}
fANSISomeString := Utf8ToAnsi(UTF8Encode(aValue));
{$else}
fANSISomeString := fSomeString;
{$endif}
end;
In your FillStructure... function, simply change your code to refer to the ANSISomeString property - this then is entirely independent of whether compiling for Unicode or not.
function FillStructureForDLL: PStructureForSomeDLL;
begin
New(Result);
result.pName := PANSIChar(SomeObject.ANSISomeString);
end;
There are at least three ways to do this.
You could change SomeObject's class
definition to use an AnsiString
instead of a string.
You could
use a conversion system to hold
references, like in your example.
You could initialize result.pname
with GetMem and copy the result of the
conversion to result.pname^ with
Move. Just remember to FreeMem it
when you're done.
Unfortunately, none of them is a perfect solution. So take a look at the options and decide which one works best for you.
Hopefully you already have code in your application to properly dispose off of all the dynamically allocated records that you New() in FillStructureForDLL(). I consider this code highly dubious, but let's assume this is reduced code to demonstrate the problem only. Anyway, the DLL you pass the record instance to does not care how big the chunk of memory is, it will only get a pointer to it anyway. So you are free to increase the size of the record to make place for the Pascal string that is now a temporary instance on the stack in the Unicode version:
type
PStructureForSomeCDLL = ^TStructureForSomeCDLL;
TStructureForSomeCDLL = record
pName: PAnsiChar;
// ... other parts of the record
pNameBuffer: string;
end;
And the function:
function FillStructureForDLL: PStructureForSomeDLL;
begin
New(Result);
// there may be a bug here, can't test on the Mac... idea should be clear
Result.pNameBuffer := Utf8ToAnsi(UTF8Encode(SomeObject.SomeString));
Result.pName := Result.pNameBuffer;
end;
BTW: You wouldn't even have that problem if the record passed to the DLL was a stack variable in the procedure or function that calls the DLL function. In that case the temporary string buffers will only be necessary in the Unicode version if more than one PAnsiChar has to be passed (the conversion calls would otherwise reuse the temporary string). Consider changing the code accordingly.
Edit:
You write in a comment:
This would be best solution if modifying the DLL structures were an option.
Are you sure you can't use this solution? The point is that from the POV of the DLL the structure isn't modified at all. Maybe I didn't make myself clear, but the DLL will not care whether a structure passed to it is exactly what it is declared to be. It will be passed a pointer to the structure, and this pointer needs to point to a block of memory that is at least as large as the structure, and needs to have the same memory layout. However, it can be a block of memory that is larger than the original structure, and contain additional data.
This is actually used in quite a lot of places in the Windows API. Did you ever wonder why there are structures in the Windows API that contain as the first thing an ordinal value giving the size of the structure? It's the key to API evolution while preserving backwards compatibility. Whenever new information is needed for the API function to work it is simply appended to the existing structure, and a new version of the structure is declared. Note that the memory layout of older versions of the structure is preserved. Old clients of the DLL can still call the new function, which will use the size member of the structure to determine which API version is called.
In your case no different versions of the structure exist as far as the DLL is concerned. However, you are free to declare it larger for your application than it really is, provided the memory layout of the real structure is preserved, and additional data is only appended. The only case where this wouldn't work is when the last part of the structure were a record with varying size, kind of like the Windows BITMAP structure - a fixed header and dynamic data. However, your record looks like it has a fixed length.
Wouldn't PChar(AnsiString(SomeObject.SomeString)) work?