I recently discovered that Delphi has a global variable called ReportMemoryLeaksOnShutdown, when set to True will detect Memory leaks when the Application closes. I found that information from reading some comments on another related question: What is the best tool to detect memory leaks in Delphi
So from the Project Source I put ReportMemoryLeaksOnShutdown := True;
Now when my Application is closed it is picking up a lot of Memory leaks. My immediate thought is to check that created Objects are freed correctly (try..finally..free etc).
I have gone over the code and I cannot see where the leaks could be coming from, and now I need to find them, because if Memory leaks are reported when the Application is exited then that very much means there are Memory leaks at runtime, which will grow in size and is bad!
From the the link above 3rd party tools were recommended such as Eureka Log. Is there a way using just the IDE and debugger to help me find and fix the problem areas?
UPDATE
I managed to get rid of about 6 memory leaks, I discovered it was to do with MDI Childs. The childs are holding some pointer data in listboxes, and when the main Application is closing, it was not freeing the childs correctly, that is now fixed.
I am now left with these 2 errors:
I found this post http://fgaillard.com/2011/02/when-the-debugger-leaks/ which may suggest the debugger is at fault with my above error?
First, make sure you get the full version of FastMM. It has some additional capabilities, such as FullDebugMode, which will help you here. Rebuild your project with the FullDebugMode and 'LogMemoryLeaksToFile' settings defined in the compiler options, and the FullDebugMode DLL in the same folder as your EXE. This will produce a file with detailed information on your memory leaks at program shtudown, in addition to the dialog box. The most useful information here will be a partial stack trace of each allocation.
Once you have this information, you can start fixing memory leaks. There's a bit of a trick to this: Remember that object ownership generally looks a lot like a tree: One object owns other objects, which each own other objects, and so on. So what you want to look for first is the leak type with the smallest number of leaks, since that's likely to be the root of the tree.
For example, if the report says you're leaking one TObjectList and 1000 TMyObject instances, it's likely that those TMyObject instances are assigned to the list and you just forgot to free the list. Fixing that would clear the whole report, so don't go hunting around for the individual child objects until you've ruled out other things.
The best way to do this is to get the tool to tell where where the allocation was made that led to a leak. To do this you need to download and use the full version of FastMM. The version supplied with Delphi does not have that capability.
When using the full FastMM, a report will be produced with all the gory details you need, including stack traces, to tell you what piece of code leaked.
Related
I've tried to read almost every decent tutorial in the internet, but still can't understand what is really happening here:
I've "Hide System Libraries" and "Invert the call tree", but I do not understand how to find actual code responsible for for example this leak. Any tips are appreciated. May be I am missing something obvious. I am getting hundreds of leaks, however I am using weak in closures, I do not have classes referencing each other etc. But it looks like I am missing something fundamental.
The problem shown in your screenshot is Instruments can't find your app's debug symbols. Instruments is showing memory addresses instead of function names. You are not going to be able to find the source of your memory leaks in Instruments without function names, even if you invert the call tree and hide system libraries.
Make sure your project is generating debug symbols. Check that the Generate Debug Symbols build setting is set to Yes. If your project is generating debug symbols, Instruments may be unable to find the dSYM file that contains the debug symbols. In Instruments choose Instrument > Call Tree > Locate dSYM. The dSYM is usually in the same directory as the release version's application bundle. The following article has additional information:
Instruments: Locating dSYM Files
Memory leaks can be difficult to track down. This is likely going to be a time consuming process, so be prepared. In the end, there is usually a lot of trial and error with debugging memory leaks. The "Memory Leaks" instrument has actually only detected one leak for me in the past. I've always had to track them down myself using the "Allocations" instrument.
One of the things that has helped me in the past is to start by trying to figure out what objects are actually being leaked. Click on the allocations instrument (the row above "Leak Checks"). Now try sorting by number of objects released or amount of memory used. See if there are any objects that have a count of 0 released when they shouldn't be sticking around. See if there is an object type that is taking an abnormal amount of memory.
Memory leaks are always due to developer mistakes with memory management. There are some minor memory leaks that exist in some of the lower level private APIs in Foundation and UIKit. At those lower levels, they are dealing with a lot more manual memory management, so its much easier to make tiny mistakes. You can't really do anything about those, but they are relatively rare.
If your application is working just fine, you may not need to worry about fixing these. There is some cost benefit analysis you want to do here. If this isn't impacting performance or stability, is the time investment in fixing these right now worth the minor benefits it will provide you and your users?
However it is worth nothing that memory leaks can add up, so if a user has your app open for a long time, the amount of leaked memory will eventually become a problem if you continue to leak more objects over time. At some point the application will crash and the user will have to re-open. But if your memory leaks are small enough that this doesn't become an issue unless the app has been open for HOURS, is it really that much of a problem anyways? That's always a judgment call on your part.
I am trying to debug memory usage in a large application using Delphi 7. I was able to installed fastmm debug full dll and with it solve some leak problems.
I also installed the memory usage tracker, allowing me to see which blocks were allocated and of what size they are.
My question is, is there a way to find out where the blocks were allocated? I know it is possible because if the memory wasn't freed a stack trace gets printed. Is there a way to 'poke' at fastmm to get it to print the stack trace for a given allocation?
Side question: if the start address of an allocation is known, is there a way to find out which class the object is? (assuming that the allocation was for a object.
You can either:
try to use LogAllocatedBlocksToFile procedure. If its ALastAllocationGroupToLog param is less than AFirstAllocationGroupToLog or is zero, then all blocks along with their allocation call stacks are logged. However, if your app has many memory allocations, prepare to long waiting. I experienced about 4 hrs wait time and 1.5Gb resulting file. (Side-note: use glogg to view such large files)
modify FastMM4.pas so implementation's LogCallStack will be visible in interface. Or you can try to use it directly from FastMM_FullDebugMode.dll
On the side question: try to use DetectClassInstance function.
If you use FullDebugMode and enable the conditionals that will write the data to a file, then you should get exactly what you're asking for. It will write out a stack trace for every leaked allocation when the program shuts down. (If you're debugging a program with a lot of memory leaks, this can take a while. I've seem it make shutdowns last for 10 minutes or more if the leaking item is a container that holds lots of other objects.)
Considering you said in a comment that the app's memory gets cleaned up nicely on app's closing for me sounds you're looking for logical memory leaks - in other words: objects that are alive more time than needed but when the time arrive for finish the app they are cleaned because exist code to clean them.
Example:
Create an TForm using the Application as owner and the variable that references it is the global one that Delphi creates when create the form's unit.
Config the Form's CloseAction to caHide (in the OnClose event)
Show the form, operate on it
Close the form and never more use it until the app closes
Close the app, which makes Application clear all the objects it owns
So you have an logical memory leak but not an physical memory leak - which is the kind that FastMM can easily detect. Since you not intended that our hypothetical TForm live until the application finish, it semantically leaked but since it is referenced and there is code that destroys it at the end of application, to FastMM is a normal allocation.
That said appears you need not the memory dump of the memory manager, but an memory profiler.
I had an issue recently (see my last question) that led me to take a closer look at the memory management in my Delphi application. After my first exploration, I have two questions.
I've started playing with the FastMMUsageTracker, and noticed the following. When I open a file to be used by the app (which also creates a form etc...), there is a significant discrepancy between the variation in available virtual memory for the app, and the variation in "FastMM4 allocated" memory.
First off, I'm a little confused by the terminology: why is there some FastMM-allocated memory and some "System-allocated" (and reserved) memory? Since FastMM is the memory manager, why is the system in charge of allocating some of the memory?
Also, how can I get more details on what objects/structures have been allocated that memory? The VM chart is only useful in showing the amount of memory that is "system allocated", "system reserved", or "FastMM allocated", but there is no link to the actual objects requiring that memory. Is it possible for example to get a report, mid-execution, similar to what FastMM generates upon closing the application? FastMM obviously stores that information somewhere.
As a bonus for me, if people can recommend a good reference (book, website) on the subject, it would also be much appreciated. There are tons of info on the net, but it's usually very case-specific and experts-oriented.
Thanks!
PS: This is not about finding leaks, no problem there, just trying to understand memory management better and be pre-emptive for the future, as our application uses more and more memory.
Some of your questions are easy. Well, one of them anyway!
Why is there some FastMM-allocated
memory and some "System-allocated"
(and reserved) memory? Since FastMM is
the memory manager, why is the system
in charge of allocating some of the
memory?
The code that you write in Delphi is only part of what runs in your process. You use 3rd party libraries in the form of DLLs, most notably the Windows API. Anytime you create a Delphi form, for example, there are a lot of windows objects behind it that consume memory. This memory does not get allocated by FastMM and I presume is what is termed "system-allocated" in your question.
However, if you want to go any deeper then this very rapidly becomes an extremely complex topic. If you do want to go deeper into the implementation of Windows memory management then I think you need to consult a serious reference source. I suggest Windows Internals by Mark Russinovich, David Solomon and Alex Ionescu.
First off, I'm a little confused by the terminology: why is there some FastMM-allocated memory and some "System-allocated" (and reserved) memory? Since FastMM is the memory manager, why is the system in charge of allocating some of the memory?
Where do you suppose FastMM gets the memory to allocate? It comes from the system, of course.
When your app starts up, FastMM gets a block of memory from the system. When you ask for some memory to use (whether with GetMem, New, or TSomething.Create), FastMM tries to give it to you from that first initial block. If there's not enough there, FastMM asks for more (in one block if possible) from the system, and returns a chunk of that to you. When you free something, FastMM doesn't return that memory to the OS, because it figures you'll use it again. It just marks it as unused internally. It also tries to realign unused blocks so that they're as contiguous as possible, in order to try not to have to go back to the OS for more needlessly. (This realignment isn't always possible, though; that's where you end up with memory fragmentation from things like multiple resizing of dynamic arrays, lots of object creates and frees, and so forth.)
In addition to the memory FastMM manages in your app, the system sets aside room for the stack and heap. Each process gets a meg of stack space when it starts up, as room to put variables. This stack (and the heap) can grow dynamically as needed.
When your application exits, all of the memory it's allocated is released back to the OS. (It may not appear so immediately in Task Manager, but it is.)
Is it possible for example to get a report, mid-execution, similar to what FastMM generates upon closing the application?
Not as far as I can tell. Because FastMM stores it somewhere doesn't necessarily mean there's a way to access it during runtime from outside the memory manager. You can look at the source for FastMMUsageTracker to see how the information is retrieved (using GetMemoryManagerState and GetMemoryMap, in the RefreshSnapshot method). The source to FastMM4 is also available; you can look and see what public methods are available.
FastMM's own documentation (in the form of the readme files, FastMM4Options.inc comments, and the FastMM4_FAQ.txt file) is useful to some extent in explaining how it works and what debugging options (and information) is available.
For a detailed map of what memory a process is using, try VMMAP from www.sysinternals.com (also co-authored by Mark Russinovich, mentioned in David's answer). This also allows you to see what is stored in some of the locations (type control-T when a detail line is selected).
Warning: there is much more memory in use by your process than you might think. You may need to read the book first.
I have a MDI program. When It starts it takes 2-3MB of RAM. Then, in this program I create about 260 MDI child windows (each has a TStringGrid, a bitmap and some other controls) and display some data. The application needs about 500MB to load all those windows. If I close each MDI child manually, the application still uses 160MB of RAM. Why it doesn't return to few MB of RAM? Should I worry about this? 160MB it is A LOT for a system that has only 1GB or RAM!!
Note: I use the WORKING SET column in Task Manager to see RAM statistics. Maybe I need a better tool to read the RAM utilization. (Private Working Set is just a bit smaller than Working Set).
This is not a leak!
FastMM (set on aggressive) indicates no memory leak when I close the program. See my Answer post for additional evidence that it isn't a leak.
I release stuff
Many people told me that closing a child window only hides it. I know that. I use "Action:= caFree" to actually release the forms. Each form is responsible for releasing the controls it holds.
Answer
I have found that FastMM is responsible for this. See the answer I posted below.
Delphi 7, Win 7 32 bit
Similar posts:
Can memory be cleaned up?
When to call SetProcessWorkingSetSize? (Convincing the memory manager to release the memory)
Task Manager is not the right tool to detect memory leaks. Delphi allocates large blocks of memory and keeps them later for future use, so certain increase in allocated memory is expected even after you release all resources. Any other results (and more detailed answers) can be obtained only by using specialize memory analysis tools. AQTime is the first that comes to mind, or if you can find old but useful MemProof, it would help you a lot (MemProof is free and for memory analysis it's more handy than AQTime).
It is very well possible that FastMM does not show memory leaks upon application termination (for instance because all objects are TComponents that are owned, and the ownser frees them).
But in the mean time, while running those components can still be around, and not freed soon enough.
Did you use the FastMM unit that shows a form with the current memory usage?
< Edit >
This is the FastMMUsageTracker.pas in the directory ...\FastMM\Demos\Usage Tracker.
Use that unit, then call the ShowFastMMUsageTracker function in it.
You can refresh that form every once in a while to see how your memory consumption grows.
I have put a FastMMUsageTrackerProject sample on-line, including an update of FastMM4 that makes it easier to check and debug memory leaks:
the form in the FastMMUsageTracker unit is now resizable, and the controls in it anchor in the right way
there is a new FastMmBootstrapUnit unit making debugging specific memory leaks easier
Something I had at hand last week, was a 3rd party DLL, which was not written in Delphi.
The DLL had a memory leak using Windows GlobalAlloc calls, which are not tracked by FastMM.
NB: I'm about to post an update to FastMM on
--jeroen
Answer:
I just removed FastMM from my project and the program returned to few MB after freeing all those child windows. Many may argue that this is not a misbehavior and that FastMM is doing this in order to do some kind of kinky memory optimizations. They may be true. However, it may be good for MY application but it may not be good of other running applications.
So, at least we know who causes this. I worried to a whole day that may program is leaking RAM like an old bucket. I am relieved now.
UPDATE:
To confirm that this behavior is generated by FastMM (as suggested by Barry Kelly) I created a second program that allocated A LOT of RAM. As soon as Windows ran out of RAM, my program memory utilization returned to its original value.
(Note: I am not saying there is a bug in FastMM!)
My program is not leaking. Problem solved.
The main limitation of FastMM's memory leak tracing is that it can only run when you shut down the program. It could be that you're still holding references to objects or other data that gets cleaned up when you shut down the program, but stays around until then.
For example, when you close the MDI child windows, do you call Free or Release on them, or just make them disappear? If they're hidden but not freed, they'll still be in memory.
If you close an MDI form it is not freed automatically. Use Action = caFree (google for that) to make sure the form is also freed.
Somebody suggested recently ( My program never releases the memory back. Why? ) that my program leaks some memory. I have FastMM set to aggressive and it reports no memory leaks when I shutdown the program.
Anyway, I would like to know if there can be memory leaks that are no detected by FastMM?
Update: I don't personally use Win API to allocate memory. But I am afraid that some 3rd party components I use (not so many) may use it. Can you let me know all possible API calls that FastMM cannot intercept? I will do a search in my code for them. Thanks.
Delphi 7, Win 7 32 bit
FastMM 4.97
I am not interested about interfaces.
FastMM is a layer on top of Windows memory management. Obviously, if you (or some component or whatever) uses Windows APIs to allocate memory, then such allocation bypasses FastMM and you won't be able to track it. BTW Delphi memory managers themselves use that APIs to allocate chunks of memory. So if you need to see allocations on that level, FastMM is not enough - you must use tools like AQTime and similar (as I suggested in the previous question).
No, only memory leaks which memory was alocated by FastMM.
EDIT:
Maybe the answer looks wrapped but it is not! If anyone check the FastMM haw is made than can see that every pointer of memory alocation is pushed (and poped out at FreeMem) in to one of stacks (there is more stacks, depend of memory size) so at the end of closing application the FastMM only check stacks, if something in stacks, and if it is, than report memory leak!
I've never known FastMM to fail to detect a memory leak.
There are several possible causes: (which apply to any memory manager)
your main program loop leaks memory, but does so to something that is freed on shutdown
the simplest case is logging to a memo. The memo gets bigger and bigger, but is destroyed on shutdown.
the memory is allocated outside fastmm's control
directly allocating from windows
memory allocated in dlls etc.
Heapfragmentation. A memory manager keeps large blocks allocated (e.g. because it still contains a small % of allocations). Result: The app doesn't use it, but it is not release to the OS either. The runtime/memorymanager keeps it around.
fastmm should be more resilient against this phenomena, but in doubt try to print heapmanager info to see if this is the case.
There is a lot of good answer already, but one point that wasn't mentionned yet...
There is a lot of "leaks" that won't get detected by most memory leak detector since the memory IS freed, but way after it isn't used anymore. For exemple, object stacked in a TObjectList. Object are put in the object list, but after you are done using them, you don't free them. They will be destroyed when the object list is destroyed too (When application close for exemple, assuming OwnsObject=True). Since the objects are actually freed, objects are not "leaked", but still make your application use more and more memory over time.
FastMM won't report those as it only makes "full run" analysis. To detect those, you need a memory leak detector that allows to do partial runs, that is, analyzing what "leaked" between point A and point B during the execution. AQTime that Eugene mentionned allow that kind of checks. But be aware that is takes a bit of analysis because that will yield many false-positive (Pretty much all "realloc" operations will be marked as a leak).
FastMM does not detect leaks of memory allocations not going through FastMM.
This can include GlobalAlloc calls from 3rd party libraries or DLLs you use.
Edit: Microsoft's MSDN has a nice list of memory allocation methods.
This was in fact the problem I mentioned I mentioned in my answer to your previous FastMM question.
You can use a tool like VMMap to track the memory leaks that FastMM cannot detect.
--jeroen