I have a MDI program. When It starts it takes 2-3MB of RAM. Then, in this program I create about 260 MDI child windows (each has a TStringGrid, a bitmap and some other controls) and display some data. The application needs about 500MB to load all those windows. If I close each MDI child manually, the application still uses 160MB of RAM. Why it doesn't return to few MB of RAM? Should I worry about this? 160MB it is A LOT for a system that has only 1GB or RAM!!
Note: I use the WORKING SET column in Task Manager to see RAM statistics. Maybe I need a better tool to read the RAM utilization. (Private Working Set is just a bit smaller than Working Set).
This is not a leak!
FastMM (set on aggressive) indicates no memory leak when I close the program. See my Answer post for additional evidence that it isn't a leak.
I release stuff
Many people told me that closing a child window only hides it. I know that. I use "Action:= caFree" to actually release the forms. Each form is responsible for releasing the controls it holds.
Answer
I have found that FastMM is responsible for this. See the answer I posted below.
Delphi 7, Win 7 32 bit
Similar posts:
Can memory be cleaned up?
When to call SetProcessWorkingSetSize? (Convincing the memory manager to release the memory)
Task Manager is not the right tool to detect memory leaks. Delphi allocates large blocks of memory and keeps them later for future use, so certain increase in allocated memory is expected even after you release all resources. Any other results (and more detailed answers) can be obtained only by using specialize memory analysis tools. AQTime is the first that comes to mind, or if you can find old but useful MemProof, it would help you a lot (MemProof is free and for memory analysis it's more handy than AQTime).
It is very well possible that FastMM does not show memory leaks upon application termination (for instance because all objects are TComponents that are owned, and the ownser frees them).
But in the mean time, while running those components can still be around, and not freed soon enough.
Did you use the FastMM unit that shows a form with the current memory usage?
< Edit >
This is the FastMMUsageTracker.pas in the directory ...\FastMM\Demos\Usage Tracker.
Use that unit, then call the ShowFastMMUsageTracker function in it.
You can refresh that form every once in a while to see how your memory consumption grows.
I have put a FastMMUsageTrackerProject sample on-line, including an update of FastMM4 that makes it easier to check and debug memory leaks:
the form in the FastMMUsageTracker unit is now resizable, and the controls in it anchor in the right way
there is a new FastMmBootstrapUnit unit making debugging specific memory leaks easier
Something I had at hand last week, was a 3rd party DLL, which was not written in Delphi.
The DLL had a memory leak using Windows GlobalAlloc calls, which are not tracked by FastMM.
NB: I'm about to post an update to FastMM on
--jeroen
Answer:
I just removed FastMM from my project and the program returned to few MB after freeing all those child windows. Many may argue that this is not a misbehavior and that FastMM is doing this in order to do some kind of kinky memory optimizations. They may be true. However, it may be good for MY application but it may not be good of other running applications.
So, at least we know who causes this. I worried to a whole day that may program is leaking RAM like an old bucket. I am relieved now.
UPDATE:
To confirm that this behavior is generated by FastMM (as suggested by Barry Kelly) I created a second program that allocated A LOT of RAM. As soon as Windows ran out of RAM, my program memory utilization returned to its original value.
(Note: I am not saying there is a bug in FastMM!)
My program is not leaking. Problem solved.
The main limitation of FastMM's memory leak tracing is that it can only run when you shut down the program. It could be that you're still holding references to objects or other data that gets cleaned up when you shut down the program, but stays around until then.
For example, when you close the MDI child windows, do you call Free or Release on them, or just make them disappear? If they're hidden but not freed, they'll still be in memory.
If you close an MDI form it is not freed automatically. Use Action = caFree (google for that) to make sure the form is also freed.
Related
I am trying to debug memory usage in a large application using Delphi 7. I was able to installed fastmm debug full dll and with it solve some leak problems.
I also installed the memory usage tracker, allowing me to see which blocks were allocated and of what size they are.
My question is, is there a way to find out where the blocks were allocated? I know it is possible because if the memory wasn't freed a stack trace gets printed. Is there a way to 'poke' at fastmm to get it to print the stack trace for a given allocation?
Side question: if the start address of an allocation is known, is there a way to find out which class the object is? (assuming that the allocation was for a object.
You can either:
try to use LogAllocatedBlocksToFile procedure. If its ALastAllocationGroupToLog param is less than AFirstAllocationGroupToLog or is zero, then all blocks along with their allocation call stacks are logged. However, if your app has many memory allocations, prepare to long waiting. I experienced about 4 hrs wait time and 1.5Gb resulting file. (Side-note: use glogg to view such large files)
modify FastMM4.pas so implementation's LogCallStack will be visible in interface. Or you can try to use it directly from FastMM_FullDebugMode.dll
On the side question: try to use DetectClassInstance function.
If you use FullDebugMode and enable the conditionals that will write the data to a file, then you should get exactly what you're asking for. It will write out a stack trace for every leaked allocation when the program shuts down. (If you're debugging a program with a lot of memory leaks, this can take a while. I've seem it make shutdowns last for 10 minutes or more if the leaking item is a container that holds lots of other objects.)
Considering you said in a comment that the app's memory gets cleaned up nicely on app's closing for me sounds you're looking for logical memory leaks - in other words: objects that are alive more time than needed but when the time arrive for finish the app they are cleaned because exist code to clean them.
Example:
Create an TForm using the Application as owner and the variable that references it is the global one that Delphi creates when create the form's unit.
Config the Form's CloseAction to caHide (in the OnClose event)
Show the form, operate on it
Close the form and never more use it until the app closes
Close the app, which makes Application clear all the objects it owns
So you have an logical memory leak but not an physical memory leak - which is the kind that FastMM can easily detect. Since you not intended that our hypothetical TForm live until the application finish, it semantically leaked but since it is referenced and there is code that destroys it at the end of application, to FastMM is a normal allocation.
That said appears you need not the memory dump of the memory manager, but an memory profiler.
I recently discovered that Delphi has a global variable called ReportMemoryLeaksOnShutdown, when set to True will detect Memory leaks when the Application closes. I found that information from reading some comments on another related question: What is the best tool to detect memory leaks in Delphi
So from the Project Source I put ReportMemoryLeaksOnShutdown := True;
Now when my Application is closed it is picking up a lot of Memory leaks. My immediate thought is to check that created Objects are freed correctly (try..finally..free etc).
I have gone over the code and I cannot see where the leaks could be coming from, and now I need to find them, because if Memory leaks are reported when the Application is exited then that very much means there are Memory leaks at runtime, which will grow in size and is bad!
From the the link above 3rd party tools were recommended such as Eureka Log. Is there a way using just the IDE and debugger to help me find and fix the problem areas?
UPDATE
I managed to get rid of about 6 memory leaks, I discovered it was to do with MDI Childs. The childs are holding some pointer data in listboxes, and when the main Application is closing, it was not freeing the childs correctly, that is now fixed.
I am now left with these 2 errors:
I found this post http://fgaillard.com/2011/02/when-the-debugger-leaks/ which may suggest the debugger is at fault with my above error?
First, make sure you get the full version of FastMM. It has some additional capabilities, such as FullDebugMode, which will help you here. Rebuild your project with the FullDebugMode and 'LogMemoryLeaksToFile' settings defined in the compiler options, and the FullDebugMode DLL in the same folder as your EXE. This will produce a file with detailed information on your memory leaks at program shtudown, in addition to the dialog box. The most useful information here will be a partial stack trace of each allocation.
Once you have this information, you can start fixing memory leaks. There's a bit of a trick to this: Remember that object ownership generally looks a lot like a tree: One object owns other objects, which each own other objects, and so on. So what you want to look for first is the leak type with the smallest number of leaks, since that's likely to be the root of the tree.
For example, if the report says you're leaking one TObjectList and 1000 TMyObject instances, it's likely that those TMyObject instances are assigned to the list and you just forgot to free the list. Fixing that would clear the whole report, so don't go hunting around for the individual child objects until you've ruled out other things.
The best way to do this is to get the tool to tell where where the allocation was made that led to a leak. To do this you need to download and use the full version of FastMM. The version supplied with Delphi does not have that capability.
When using the full FastMM, a report will be produced with all the gory details you need, including stack traces, to tell you what piece of code leaked.
One old application started to consume memory a lot after server update. Memory usage seems to rise with out limit until program hangs.
According to FastMM4 and EurekaLog, there's no memory leak (except 28 bytes), so I assume all memory is freed when application is shutdown.
Are there any tools or strategies suitable for tracking this kind of memory problem?
Since September 2012, there is a very simple and comfortable way to find this type of "run-time only" memory leaks.
FastMM4991 introduced a new method, LogMemoryManagerStateToFile:
Added the LogMemoryManagerStateToFile call. This call logs a summary of
the memory manager state to file: The total allocated memory, overhead,
efficiency, and a breakdown of allocated memory by class and string type.
This call may be useful to catch objects that do not necessarily leak, but
do linger longer than they should.
To discover the leak at run time, you only need these steps
add a call to LogMemoryManagerStateToFile('memory.log', '') in a place where it will be called in intervals
run the application
open the log file with a tail program (for example BareTail), which will auto-refresh when the file content changes
watch the first lines of the file, they will contain the memory allocations which occupy the highest amount of memory
if you see a class or memory type constantly has a growing number of instances, this can be the reason of your leak
The growing memory consumption is an application issue. It is not a bug, which can discover FastMM4 or EurekaLog. As from they point of view - application just correctly uses the memory.
Using AQTime, MemProof (hard to find, D7 is last supported version (?)), SleuthQA (similar to MemProof) or similar memory profilers, you can track the memory usage outside of application in real-time.
Using FastMM4, GetMemoryManagerState / GetMemoryManagerUsageSummary you can track memory usage from application. Output this information into trace file and analyze it after run. Or make simple wrapping function for one of the above procedures, which will return curent memory usage. And call it from IDE Debugger Evalute / Modify, add to Watches or call OutputDebugString, and see the current memory usage.
Note, if memory is eated by some DLL then you may not see her memory usage using (3). Use (2).
Analyzing the memory usage and the tasks performed by the application, you may discover what leads to raised memory usage.
AQTime (a commercial tool which is quite expensive) can report your memory usage, down to the line of source code that allocated each object. In the case of very large memory usage scenarios, you might want the AQTime functionality that can show the number of objects and the size (total plus individual instance size) for each object. AQTime worked great for me, starting with Delphi 7, and all later versions, including your version (2006) and the latest versions (XE and XE2).
As the program memory usage grows, AQTime can be used to grab "snapshots" of the runtime heap, you can use to understand memory usage of your application; What is being created, and how many of each object exists. Even when no leaks exist, understanding the runtime behaviour of your application in terms of the objects it creates and manages, is very important, and AQTime is the most powerful tool I know of for Delphi users.
If you are willing to upgrade to Delphi XE/XE2, you might have an included light version of AQTime already, if so, check it out. If not, I recommend you try their demo. I am unaware of any free or open source alternatives that can provide the same functionality.
Lesser functionality could be cobbled together manually by writing lots of trace messages, or using the FastMM full-debug-mode. If you could write a complete dump of your memory usage into a very large file, you might be able to write some tools to parse, and create a summary. The problem I have with FastMM in this case, is that you will be drowned in detail information, without the ability to extract exactly the summary information that helps you understand your situation. So, you can try to write your own tool to summarize the memory usage. In one application I had that used a series of components that I knew would use a lot of memory, I wrote a dialog box into my application that showed current memory usage by these large memory-blob-of-data objects.
Have you ever think about the Leak that is causing the IDE... it is so huge!!!
In my case (2GB of RAM) i do the next...
1. Open the IDE
2. Leave it minimized for near six hours
3. See how Physical memory is getting used
The result:
While IDE is oppened (remember i also do the test having it minimized) it is getting more and more RAM... till no more ram free.
It gets all 2GB RAM + all Pagefile hard disk space (i have it configured to a mas of 4GB)
In less that six hours (doing nothing on IDE) it tries to use more than 6GB.
That is called a Memory Leak casused by the IDE... i do not type any letter on IDE, do not compile anything, do not even open any project... just open IDE and minimize it... leave the computer without doing anything on it for about six hours and IDE is consuming 6GB of memory.
Of course, after that, the IDE start with annoying messages of SystemOutOfMemory... and i must kill it... then all that 6GB are freed!!!
When on the hell will this get fixed?
Please note i have all patches applied, i also tested without applying each patch/hotfix, etc...
The best i got was dissabling some options on Tools, like the one that underlines bad code, etc... so why on the hell that option has any influence... i am not typing anything on the IDE (on the tests)... and if i have it dissabled the memory leak gets reduced a lot...
Of course, if i use the IDE (write code on an opened project) without even compiling / running it... the thing goes much more worst... memory leak upto 6GB can got reached on less than an hour, sometimes occurs after 15 minutes of Copy/Paste source code.
Seems there will not be a solution in a short time!!!
So i got the next solution that works perfect:
-Close the IDE an reopen it each 15 minutes or less
Ugly solution, i know... but works!!!
I had an issue recently (see my last question) that led me to take a closer look at the memory management in my Delphi application. After my first exploration, I have two questions.
I've started playing with the FastMMUsageTracker, and noticed the following. When I open a file to be used by the app (which also creates a form etc...), there is a significant discrepancy between the variation in available virtual memory for the app, and the variation in "FastMM4 allocated" memory.
First off, I'm a little confused by the terminology: why is there some FastMM-allocated memory and some "System-allocated" (and reserved) memory? Since FastMM is the memory manager, why is the system in charge of allocating some of the memory?
Also, how can I get more details on what objects/structures have been allocated that memory? The VM chart is only useful in showing the amount of memory that is "system allocated", "system reserved", or "FastMM allocated", but there is no link to the actual objects requiring that memory. Is it possible for example to get a report, mid-execution, similar to what FastMM generates upon closing the application? FastMM obviously stores that information somewhere.
As a bonus for me, if people can recommend a good reference (book, website) on the subject, it would also be much appreciated. There are tons of info on the net, but it's usually very case-specific and experts-oriented.
Thanks!
PS: This is not about finding leaks, no problem there, just trying to understand memory management better and be pre-emptive for the future, as our application uses more and more memory.
Some of your questions are easy. Well, one of them anyway!
Why is there some FastMM-allocated
memory and some "System-allocated"
(and reserved) memory? Since FastMM is
the memory manager, why is the system
in charge of allocating some of the
memory?
The code that you write in Delphi is only part of what runs in your process. You use 3rd party libraries in the form of DLLs, most notably the Windows API. Anytime you create a Delphi form, for example, there are a lot of windows objects behind it that consume memory. This memory does not get allocated by FastMM and I presume is what is termed "system-allocated" in your question.
However, if you want to go any deeper then this very rapidly becomes an extremely complex topic. If you do want to go deeper into the implementation of Windows memory management then I think you need to consult a serious reference source. I suggest Windows Internals by Mark Russinovich, David Solomon and Alex Ionescu.
First off, I'm a little confused by the terminology: why is there some FastMM-allocated memory and some "System-allocated" (and reserved) memory? Since FastMM is the memory manager, why is the system in charge of allocating some of the memory?
Where do you suppose FastMM gets the memory to allocate? It comes from the system, of course.
When your app starts up, FastMM gets a block of memory from the system. When you ask for some memory to use (whether with GetMem, New, or TSomething.Create), FastMM tries to give it to you from that first initial block. If there's not enough there, FastMM asks for more (in one block if possible) from the system, and returns a chunk of that to you. When you free something, FastMM doesn't return that memory to the OS, because it figures you'll use it again. It just marks it as unused internally. It also tries to realign unused blocks so that they're as contiguous as possible, in order to try not to have to go back to the OS for more needlessly. (This realignment isn't always possible, though; that's where you end up with memory fragmentation from things like multiple resizing of dynamic arrays, lots of object creates and frees, and so forth.)
In addition to the memory FastMM manages in your app, the system sets aside room for the stack and heap. Each process gets a meg of stack space when it starts up, as room to put variables. This stack (and the heap) can grow dynamically as needed.
When your application exits, all of the memory it's allocated is released back to the OS. (It may not appear so immediately in Task Manager, but it is.)
Is it possible for example to get a report, mid-execution, similar to what FastMM generates upon closing the application?
Not as far as I can tell. Because FastMM stores it somewhere doesn't necessarily mean there's a way to access it during runtime from outside the memory manager. You can look at the source for FastMMUsageTracker to see how the information is retrieved (using GetMemoryManagerState and GetMemoryMap, in the RefreshSnapshot method). The source to FastMM4 is also available; you can look and see what public methods are available.
FastMM's own documentation (in the form of the readme files, FastMM4Options.inc comments, and the FastMM4_FAQ.txt file) is useful to some extent in explaining how it works and what debugging options (and information) is available.
For a detailed map of what memory a process is using, try VMMAP from www.sysinternals.com (also co-authored by Mark Russinovich, mentioned in David's answer). This also allows you to see what is stored in some of the locations (type control-T when a detail line is selected).
Warning: there is much more memory in use by your process than you might think. You may need to read the book first.
Somebody suggested recently ( My program never releases the memory back. Why? ) that my program leaks some memory. I have FastMM set to aggressive and it reports no memory leaks when I shutdown the program.
Anyway, I would like to know if there can be memory leaks that are no detected by FastMM?
Update: I don't personally use Win API to allocate memory. But I am afraid that some 3rd party components I use (not so many) may use it. Can you let me know all possible API calls that FastMM cannot intercept? I will do a search in my code for them. Thanks.
Delphi 7, Win 7 32 bit
FastMM 4.97
I am not interested about interfaces.
FastMM is a layer on top of Windows memory management. Obviously, if you (or some component or whatever) uses Windows APIs to allocate memory, then such allocation bypasses FastMM and you won't be able to track it. BTW Delphi memory managers themselves use that APIs to allocate chunks of memory. So if you need to see allocations on that level, FastMM is not enough - you must use tools like AQTime and similar (as I suggested in the previous question).
No, only memory leaks which memory was alocated by FastMM.
EDIT:
Maybe the answer looks wrapped but it is not! If anyone check the FastMM haw is made than can see that every pointer of memory alocation is pushed (and poped out at FreeMem) in to one of stacks (there is more stacks, depend of memory size) so at the end of closing application the FastMM only check stacks, if something in stacks, and if it is, than report memory leak!
I've never known FastMM to fail to detect a memory leak.
There are several possible causes: (which apply to any memory manager)
your main program loop leaks memory, but does so to something that is freed on shutdown
the simplest case is logging to a memo. The memo gets bigger and bigger, but is destroyed on shutdown.
the memory is allocated outside fastmm's control
directly allocating from windows
memory allocated in dlls etc.
Heapfragmentation. A memory manager keeps large blocks allocated (e.g. because it still contains a small % of allocations). Result: The app doesn't use it, but it is not release to the OS either. The runtime/memorymanager keeps it around.
fastmm should be more resilient against this phenomena, but in doubt try to print heapmanager info to see if this is the case.
There is a lot of good answer already, but one point that wasn't mentionned yet...
There is a lot of "leaks" that won't get detected by most memory leak detector since the memory IS freed, but way after it isn't used anymore. For exemple, object stacked in a TObjectList. Object are put in the object list, but after you are done using them, you don't free them. They will be destroyed when the object list is destroyed too (When application close for exemple, assuming OwnsObject=True). Since the objects are actually freed, objects are not "leaked", but still make your application use more and more memory over time.
FastMM won't report those as it only makes "full run" analysis. To detect those, you need a memory leak detector that allows to do partial runs, that is, analyzing what "leaked" between point A and point B during the execution. AQTime that Eugene mentionned allow that kind of checks. But be aware that is takes a bit of analysis because that will yield many false-positive (Pretty much all "realloc" operations will be marked as a leak).
FastMM does not detect leaks of memory allocations not going through FastMM.
This can include GlobalAlloc calls from 3rd party libraries or DLLs you use.
Edit: Microsoft's MSDN has a nice list of memory allocation methods.
This was in fact the problem I mentioned I mentioned in my answer to your previous FastMM question.
You can use a tool like VMMap to track the memory leaks that FastMM cannot detect.
--jeroen