This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
TStringList of objects taking up tons of memory in Delphi XE
I have a program that takes a while to run and uses a fair amount of memory (starts out at about 800 mb, ends at about 1.1 GB).
The problem I'm having is that even though the amount of memory it's using is fine, the amount of memory commited goes over 4GB and gives me an "Out of Memory" error.
Here is a screen shot of Resource Monitor:
The main thing that takes up the memory is a TStringList filled with an object that has a bunch of different values and a few stringlists
Does anyone know why it is committing so much more memory than I need, and how can I make my program not do that?
The commit size is the amount of memory allocated by the program (but is not necessarily in physical memory). As mentioned in the comments, it is quite possible that the program has a memory leak (e.g., it creates objects possibly but does not delete them). So even if the amount of memory in use is "reasonable", it doesn't necessarily represent the amount actually allocated, and that amount in a 32-bit system can't "typically" exceed 4GB. This question discusses leak detection tools that might be useful.
Your software probably contains a memory leak. Put this little piece of code right after your program's "main" block (.dpr file - press F7 to get there).
{$WARN SYMBOL_PLATFORM OFF}
ReportMemoryLeaksOnShutdown := DebugHook <> 0;
So, the end result should look similar to this:
begin
{$WARN SYMBOL_PLATFORM OFF}
ReportMemoryLeaksOnShutdown := DebugHook <> 0;
Application.Initialize;
Application.CreateForm(TForm1, Form1);
Application.Run;
end.
Then, run your program until you get some big memory usage and simply quit it. Delphi will show a nice window giving you the size of the memory leak as well as some tips about where they might be happening.
Good luck!
You didn't showed code, but this effect appears to be the result of a memory leak.
So, this TStringList (I'll call it parent) have objects associated, which have its own child TStringLists (which I suppose have only strings).
The easier way to do this memory management is letting the TStringList do it for you:
If in the Delphi version you have TStringList has the OwnsObjects property, set it to true. With this, every call to Delete and Clear methods will call the destructor of the objects associated with the parent list - and this will work even if cll parent.Free. If it doesn't implement, you'll have to do it yourself - overriding the Delete, Clear and Destroy methods of TStringList in a descendant class.
Make sure the destructor of the child objects clean correctly the objects created inside it - if one of the stringlists in a child objects has owned objects, you could use the same strategy in the above item.
An (oversimplified) example:
TMyChildObject = class
private
fMyListOfMagazineNames,
// This one will have TBitmap instances associated with the dogs names
fMyListOfDogNamesAndPhotos : TStringList;
fName: String;
public
//[properties, methods, whatever]
property Name: string read fName write fName;
constructor Create; virtual;
destructor Destroy; override;
end;
constructor TMyChildObject.Create;
begin
inherited Create;
fMyListOfMagazineNames = TStringList.Create;
fMyListOfDogNamesAndPhotos = TStringList.Create;
fMyListOfDogNamesAndPhotos.OwnsObjects := True;
end;
destructor TMyChildObject.Destroy;
begin
fMyListOfMagazineNames.Free;
fMyListOfDogNamesAndPhotos.Free;
inherited Destroy;
end;
An example of use:
var
MyObjList: TStringList;
begin
MyObjList := TStringList.Create;
MyObjList.OwnsObjects := True;
// Create your child objects
// And Free them at once
MyObjList.Free;
end;
As was stated in your previous question - thanks for having selected one answer.
The main solution to your problem is not to allocate all objects in memory. If your 28 MB initial data consume more than 1 GB of memory, you have a lot of redundant information.
You shall modify your algorithm as such:
Use a SQL database instead to process your data without memory issue (with proper indexes) - may be SQLite3 or embedded Firebird will work very well;
Leave the data on the input files, the memory-map them instead of allocating a copy in memory: use getter methods for each properties, to retrieve the value on the fly;
Use a NoSQL database (like this) which may be faster than a SQL database for inserting rows, and consume less disk space and memory;
Or... consider compressing the objects content (a fast compression is a good option).
Related
When I create TObjectDictionary with doOwnsValues and store TStringList as Value, after freeing the TObjectDictionary, I could still access the TStringList.
I modified the example given here:
Example for using Generics.Collections.TObjectDictionary
{$APPTYPE CONSOLE}
{$R *.res}
uses
Generics.Collections,
Classes,
System.SysUtils;
Var
MyDict : TObjectDictionary<String, TStringList>;
Sl : TStringList;
begin
ReportMemoryLeaksOnShutdown:=True;
try
//here i'm creating a TObjectDictionary with the Ownership of the
Values
//because in this case the values are TStringList
MyDict := TObjectDictionary<String, TStringList>.Create([doOwnsValues]);
try
//create an instance of the object to add
Sl:=TStringList.Create;
//fill some foo data
Sl.Add('Foo 1');
Sl.Add('Foo 2');
Sl.Add('Foo 3');
//Add to dictionary
MyDict.Add('1',Sl);
//add another stringlist on the fly
MyDict.Add('2',TStringList.Create);
//get an instance to the created TStringList
//and fill some data
MyDict.Items['2'].Add('Line 1');
MyDict.Items['2'].Add('Line 2');
MyDict.Items['2'].Add('Line 3');
//finally show the stored data
Writeln(MyDict.Items['1'].Text);
Writeln(MyDict.Items['2'].Text);
finally
//only must free the dictionary and don't need to worry for free the
TStringList assignated to the dictionary
MyDict.Free;
end;
Sl.Add('Foo added?');
Writeln(Sl[0]);
except
on E: Exception do
Writeln(E.ClassName, ': ', E.Message);
end;
Readln;
end.
I expect S1 should not have been accessible. Is it memory leak? or bug?
The string list object Sl has been freed by the dictionary. No memory is leaked. There is no bug in Delphi's RTL (the dictionary class) here.
Hence, when you do Sl.Add('Foo added?'); you are accessing an object that has been freed, and that is a bug in your code. You mustn't do that.
Just because you didn't get any visible symptoms of the bug this time doesn't mean it isn't a bug.
A simpler example:
type
TTest = class
Value: Integer;
end;
procedure TForm1.FormCreate(Sender: TObject);
var
Test: TTest;
begin
Test := TTest.Create;
try
Test.Value := 123;
finally
Test.Free;
end;
ShowMessage(Test.Value.ToString); // Bug!! Never do this!
end;
This is a bug. But chances are you might get a 123 message box anyway.
When you free an object, its memory area is declared as available so that other data might be stored there in the future. Thus, immediately after the destruction of your object, its bytes are likely intact where you left them. But eventually they will be overwritten with new data, and as time goes by, all bytes might change. You never know, and you shouldn't know: you don't own that part of the computer's memory any more.
Of course, destroying an object might do more than simply giving up its location in memory: OS handles might be closed, temporary files might be removed, etc. Hence, there are many things that might go wrong if you use a freed object, even if you use the object immediately after its destruction.
Even worse, the memory area of your old object might now partially or completely belong to new objects (or other kinds of data) in your application, so by accessing the members of your ghost object, you might be randomly changing bytes in these new objects. This might cause subtle random malfunction and errors and crashes at any later time in your application's life.
But you need not care about such details, because you'd never use a freed object anyway.
(Or, think of it this way: if you use a freed object, you are violating a contract, and basically anything can happen. One possibility is that nothing "strange" happens at all, but that is just dumb luck.)
The problem I am facing is as follows.Please note I'm no pro at delphi pascal
To save a Large amount of memory and time I created a procedure:
procedure TForm1.Placeholder(tspath: STRING);
begin //rgbholder :TImage globally declared (Dinamicly create image)
rgbholder.Free;//Free previous image
rgbholder := timage.Create(self);
rgbholder.Width := 10;
rgbholder.Height := 10;
rgbholder.Visible := false;
rgbholder.Bitmap.LoadFromFile(TSpath);
end;
procedure TForm1.Image3Click(Sender: TObject);
begin
placeholder('Data\Grass\Grassanim1low.png');
bitmaplistanimation5.Stop;
bitmaplistanimation5.Loop := true;
//bitmaplistanimation5.AnimationBitmap.LoadFromFile('Data\Grass\Grassanim1low.png');
bitmaplistanimation5.AnimationBitmap.Assign( rgbholder.Bitmap);
bitmaplistanimation5.AnimationCount := 22;
bitmaplistanimation5.AnimationRowCount := 2;
bitmaplistanimation5.Duration := 2.5;
bitmaplistanimation5.PropertyName :='bitmap';
end;
Now my problem is freeing up that memory of the bitmaplistanimation5.AnimationBitmap.Assign(rgbholder.Bitmap);
When this code executes
bitmaplistanimation5.Stop;
bitmaplistanimation5.Enabled := false;
bitmaplistanimation5.AnimationBitmap.Free;
Everything goes well until I close the executable with a close; or by simply closing it with the exit in window.
After closing it throws a Application Error Exception EInvalidPointer in module Project1.exe at 00007A55. Invalid Pointer Operation
I don't think I'm trying to free memory that is already freed, I'm also not trying to free memory that was allocated somewhere other than the memory manager unless the placeholder procedure counts. Unless I'm missing something or not understanding something I should.
As a general rule, you should only call Free on an object that you created. You did not create this object, the TBitmapListAnimation control created it. So this code is a clear mistake on your part:
bitmaplistanimation5.AnimationBitmap.Free;
You will have to remove that.
I suspect that the right way to clear the memory from an FMX bitmap is to set the Width and Height properties of the bitmap to 0.
I wonder whether or not you really need to release this memory. If your program stops using the memory, and the system is short of memory, if can swap your unused memory to disk. It is common to find programmers who don't have a good understanding of virtual memory, taking actions like this that give no benefit. I would not be at all surprised if by trying to release this memory you actually made your program's performance worse rather than better.
I think that C++ supports something on the lines of :
Object objects[100];
This would instantiate a 100 objects, right? Is it possible to do this in Delphi (specifically 2007)? Something other than:
for i:=0 to 99 do
currentObject = TObject.Create;
or using the Allocate function, with a passed size value a hundred times the TObject size, because that just allocates memory, it doesn't actually divide the memory and 'give' it to the objects.
If my assumption that the c++ instantiation is instant rather than under-the-hood-iterative, I apologize.
What you are looking for is impossible because
Delphi does not support static (stack allocated) objects.
Delphi objects do not have default constructors that can be automatically invoked by compiler.
So that is not a lack of 'sugar syntax'.
For the sake of complete disclosure:
Delphi also supports legacy 'old object model' (Turbo Pascal object model) which allows statically allocated objects;
Dynamic object allocation itself does not prevent automatic object instantiation syntax, but makes such a syntax undesirable;
Automatic object instantiation syntax is impossible because Delphi does not have default constructors: Delphi compiler never instantiate objects implicitly because it does not know what constructor to call.
While you can't do what you want using objects, if your objects are relatively simple, you may be able to get what you want by using an array of records.
Records in Delphi can have properties (including setters and getters), and class and instance methods. They are created automatically when declared, so declaring an array of them will create them all without iterating.
For more info: http://docwiki.embarcadero.com/RADStudio/XE3/en/Structured_Types#Records_.28advanced.29.
(I'm not sure when the new functionality was added to Delphi, it may well be after the 2007 version).
I don't know of any non-hacky way to do this besides iterating:
var
MyObjects: array[0..99] of TMyObject;
i: Integer;
begin
for i := 0 to 99 do
MyObjects[i] := TMyObject.Create;
end;
That declaration wouldn't create 100 objects, it would just give you an array of 100 object references that point to nothing useful.
Creating an object is a two step process. The first step is allocating memory (which your code also doesn't), the second step is calling the constructor (Create method) to initialize that memory, create additional objects, etc, etc.
The allocation part can be done without the loop, but the constructor needs to be called to intialize each instance.
Many VCL classes don't have an additional constructor. They just have the empty constructor that does nothing. In that case, there is no need to call it.
For instance, to fetch an array of stringlists, you can use the following code, adjusted from this example:
type
TStringListArray = array of TStringList;v
var
Instances: array of Byte;
function GetStringLists(Number: Integer): TStringListArray;
var
i: Integer;
begin
// Allocate instance memory for all of them
SetLength(Instances, Number * TStringList.InstanceSize);
// Zero the memory.
FillChar(Instances[0], Length(Instances), 0);
// Allocate array for object references.
SetLength(Result, Number);
for i := 0 to High(Result) do
begin
// Store object reference.
Result[i] := #Instances[i * TStringList.InstanceSize];
// Set the right class.
PPointer(Result[i])^ := TStringList;
// Call the constructor.
Result[i].Create;
end;
end;
And to get an array of 100 stringlists:
var
x: TStringListArray;
begin
x := GetStringLists(100);
So while this procedure may save you a neglectable amount of time, and may theoretically be more memory-efficient (less fragmentation), you will still need the loop. No easy way out.
It is somewhat possible in Delphi (but is not very practical in a Delphi environment, unless you are writing a custom memory manager). Creating an object instance is a two-step process - allocating memory for the object, and constructing the object's members inside of that memory. There is nothing requiring the memory of a given object to be allocated individually. You can allocate a larger block of memory and construct multiple objects inside of that block, taking advantage of a feature of Delphi that calls a constructor like a normal method if it is called from an instance variable instead of a class type, eg:
var
objects: array of Byte;
obj: TSomeClass;
begin
SetLength(objects, 100 * TSomeClass.InstanceSize);
FillChar(objects[0], 0, Length(objects));
for i := 0 to 99 do
begin
obj := TSomeClass.InitInstance(#objects[i * TSomeClass.InstanceSize]));
obj.Create;
end;
...
for i := 0 to 99 do
begin
obj := TSomeClass(#objects[i * TSomeClass.InstanceSize]);
obj.CleanupInstance;
end;
SetLength(objects, 0);
end;
This is not much different than what C++ does behind the scenes when declaring an array of object instances, only C++ supports declaring the array statically and it will call the constructors and destructors automatically, whereas Delphi does not support that.
There are a number of third-party implementations that allow you to allocate objects on the stack or in user-defined buffers, such as:
Objects on the Stack: A Delphi Relic
Allocate objects on stack, at specified memory address, or through any memory manager
Just to name a couple.
Version used: Delphi 7.
I'm working on a program that does a simple for loop on a Virtual ListView. The data is stored in the following record:
type TList=record
Item:Integer;
SubItem1:String;
SubItem2:String;
end;
Item is the index. SubItem1 the status of the operations (success or not). SubItem2 the path to the file. The for loop loads each file, does a few operations and then, save it. The operations take place in a TStringList. Files are about 2mb each.
Now, if I do the operations on the main form, it works perfectly.
Multi-threaded, there is a huge memory problem. Somehow, the TStringList doesn't seem to be freed completely. After 3-4k files, I get an EOutofMemory exception. Sometimes, the software is stuck to 500-600mb, sometimes not. In any case, the TStringList always return an EOutofMemory exception and no file can be loaded anymore. On computers with more memory, it takes longer to get the exception.
The same thing happens with other components. For instance, if I use THTTPSend from Synapse, well, after a while, the software cannot create any new threads because the memory consumption is too high. It's around 500-600mb while it should be, max, 100mb. On the main form, everything works fine.
I guess the mistake is on my side. Maybe I don't understand threads enough. I tried to free everything on the Destroy event. I tried FreeAndNil procedure. I tried with only one thread at a time. I tried freeing the thread manually (no FreeOnTerminate...)
No luck.
So here is the thread code. It's only the basic idea; not the full code with all the operations. If I remove the LoadFile prodecure, everything works good. A thread is created for each file, according to a thread pool.
unit OperationsFiles;
interface
uses Classes, SysUtils, Windows;
type
TOperationFile = class(TThread)
private
Position : Integer;
TPath, StatusMessage: String;
FileStringList: TStringList;
procedure UpdateStatus;
procedure LoadFile;
protected
procedure Execute; override;
public
constructor Create(Path: String; LNumber: Integer);
end;
implementation
uses Form1;
procedure TOperationFile.LoadFile;
begin
try
FileStringList.LoadFromFile(TPath);
// Operations...
StatusMessage := 'Success';
except
on E : Exception do StatusMessage := E.ClassName;
end;
end;
constructor TOperationFile.Create(Path : String; LNumber: Integer);
begin
inherited Create(False);
TPath := Path;
Position := LNumber;
FreeOnTerminate := True;
end;
procedure TOperationFile.UpdateStatus;
begin
FileList[Position].SubItem1 := StatusMessage;
Form1.ListView4.UpdateItems(Position,Position);
end;
procedure TOperationFile.Execute;
begin
FileStringList:= TStringList.Create;
LoadFile;
Synchronize(UpdateStatus);
FileStringList.Free;
end;
end.
What could be the problem?
I thought at one point that, maybe, too many threads are created. If a user loads 1 million files, well, ultimately, 1 million threads is going to be created -- although, only 50 threads are created and running at the same time.
Thanks for your input.
There are (probably) no leaks in the code you show in the question.
I say probably because an exception raised during the Execute could result in a leak. The lifetime of the string list should be protected by a finally block.
FileStringList:= TStringList.Create;
try
LoadFile;
Synchronize(UpdateStatus);
finally
FileStringList.Free;
end;
That said, I expect the exception swallow in LoadFile means that you don't leak the string list.
You say that perhaps thousands of threads are created. Each thread reserves memory for its stack, and the default stack size is 1MB. Once you have thousands of 1MB stacks reserved, you can easily exhaust or fragment address space.
I've seen problems due to cavalier creation of threads in the past. For example I had a program that failed when it created and destroyed threads, with never more than 256 threads in existence. This was on a 16 core machine with 4GB address space. You probably have 2GB address space available.
Although you state that no more than 50 threads are in existence at any one moment, I'm not sure how you can be sure of that. Not least, because you have set FreeOnTerminate to True and thereby surrendered control over the lifetime of your threads.
My guess is that your problems are related to the number of threads you create. One thread per processor will suffice. Re-use your threads. It's expensive to create and destroy a thread for a small task.
If this is not enough to solve your problems then you will need to show the code that manages thread lifetime.
Finally, I wonder how much benefit you will extract from threading this app. If it is IO bound then the threaded version may well be slower!
Based on the information given, it's not possible to reproduce your error.
Some hints are made by Remy and David which might help you.
Looking at the structure of your program, the flow can be divided into two classical solutions.
The first part where you are delegating tasks to different threads, is a Single-Producer-Multiple-Consumer problem.
Here it can be solved by creating a small number of threads, passing them a thread-safe object queue.
The main thread then pushes the task objects into the queue. The consumer threads takes care of the individual file checking tasks.
The second part where the result is to be transfered to the main thread is a Multiple-Producer-Single-Consumer problem.
If you pass a second thread-safe object queue to the threads at initialization, they can easily put the results into the queue.
Drain the result queue from the main thread within a timer event.
I am using following class
mycalss = class
public
Self.timer1: TTimer;
Self.abitmap: tbitmap;
Self.animage: TImage;
Self.formxyz:form;
//some other declaration
//also having my own constructor and destructor overridden
end;
when a button is clicked myclass1 will be created and when other button is clicked myclass1 will be destroyed
begin
FreeAndNil(Self.timer1) ;
FreeAndNil(Self.abitmap) ;
FreeAndNil(Self.animage);
//others
Self.formxyz.Close ;
FreeAndNil(Self.formxyz);
FreeAndNil(Self) ;
inherited Destroy;
end;
when ever i click on create button the windows task manager is showing 800 kb increase in memory consumption and whenever i click destroy there is only 500kb decrease ,
I am quite sure that the size increment is taking place when it is creating(not because of other side effects)as my app is a simple one.
Question :Can I expect the delphi application to remove all the objects(~800kb) created to be removed on destroy operation(If i have done freeing properly)?.
First only free object that you allocated in the class - do not destroy the self variable
this will have some nice side effects when calling inherited and that inherited class also wants to free things.
The only reliable thing to check for memory leaks is to enable ReportMemoryLeaksOnShutdown or
even better - download the FastMM sources from sourceforge.
Second I think your encountering a speciality of the internal memory manager - the internal memory manager only frees memory of very large allocated blocks. Looking at the internals it reveals that the memory manager basically consists of 3 block managers, a small, a medium and
a manager for large blocks. For the small and medium blocks the manager keeps the memory and does not deallocate it to windows - the likelyhood that it will be reused in near future is quite high. It only immediately deallocates the memory for large blocks.
Your constructor and destructor should look like this:
constructor TMyClass.Create(AOwner: TComponent);
begin
inherited;
timer1:= TTimer.Create;
Bitmap1:= TBitmap.Create;
AnImage:= TImage.Create(AOwner);
...
end;
destructor TMyClass.Destroy;
begin
timer1.Free;
abitmap.Free;
animage.Free;
inherited Destroy;
end;
Because you are not reusing any members it makes no sense to call FreeAndNil.