After spending time with FastMM4 in weeding out any pieces of code that could possibly cause memory leak as indicated by FastMM4, we have been test running our software for about a month non-stop on Windows 7. This is what I am seeing so far in the Task Manager for my software process.
> -CPU started out at 1% and 0%. Now it is bouncing around from 2% to 5%
> -VM usage started out at 11,852KB. Now it is at 4,900kb but bouncing
> around from 4,900kb to 5,000kb.
Does this mean we have memory leak in our software? I am confused and concerned.
Thanks in advance,
That sounds like pretty normal memory usage. The program does something that needs memory, and memory usage goes up. The program gets done with what it's doing and releases the memory, and memory usage goes back down. A memory leak is when memory usage goes up and keeps going up because you're not releasing the memory once you're done with it.
If you've got FastMM4, you shouldn't have to hunt around for things that might possibly cause a memory leak. Just turn on Full Debug Mode and the logging option and it will find any memory that leaks when you run it, and write out a file with types and stack traces for you.
No, when memory usage goes up and eventually your application uses all available memory, you have memory leak and reason to be concerned.
Even if an application seems to run fine for days, there still can be memory leaks hidden in code areas which are not (or not frequently) used. So they can be a problem, when this part of the application becomes active some time later.
To make sure that all code is tested for leaks, you can use FastMM4 with unit tests (with DUnit), ensuring to execute as many code paths as possible. Unit test code coverage can be measured for example with this open source Delphi tool, or Discover which recently became open source.
Also, DUnit in the trunk version (9.4.0) supports automatic memory leak detection (based on FastMM4) for every test case.
May be worth to think about the debugger? I've understood that you do not use them. I also had problems with memory leaks. I just had no guessed about it. Now use the "guardian" of the leaks - for me it became deleaker. And you can look such one, that will please you.
Related
This is a follow-up to this question: What could explain the difference in memory usage reported by FastMM or GetProcessMemoryInfo?
My Delphi XE application is using a very large amount of memory which sometimes lead to an out of memory exception. I'm trying to understand why and what is causing this memory usage and while FastMM is reporting low memory usage, when requesting for TProcessMemoryCounters.PageFileUsage I can clearly see that a lot of memory is used by the application.
I would like to understand what is causing this problem and would like some advise on how to handle it:
Is there a way to know what is contained in that memory and where it has been allocated ?
Is there some tool to track down memory usage by line/procedure in a Delphi application ?
Any general advise on how to handle such a problem ?
EDIT 1 : Here are two screenshots of FastMMUsageTracker indicating that memory has been allocate by the system.
Before process starts:
After process ends:
Legend: Light red is FastMM allocated and dark gray is system allocated.
I'd like to understand what is causing the system to use that much memory. Probably by understanding what is contained in that memory or what line of code or procedure did cause that allocation.
EDIT 2 : I'd rather not use the full version of AQTime for multiple reasons:
I'm using multiple virtual machines for development and their licensing system is a PITA (I'm already a registered user of TestComplete)
LITE version doesn't provide enough information and I won't waste money without making certain the FULL version will give me valuable information
Any other suggestions ?
Another problem might be heap fragmentation. This means you have enough memory free, but all the free blocks are to small. You might see it visually by using the source version of FastMM and use the FastMMUsageTracker.pas as suggested here.
You need a profiler, but even that won't be enough in lots of places and cases. Also, in your case, you would need the full featured AQTime, not the lite version that comes with Delphi XE and XE2. (AQTIME is extremely expensive, and annoyingly node-locked, so don't think I'm a shill for SmartBear software.)
The thing is that people often mistake AQTime Allocation Profiler as only a way to find leaks. It can also tell you where your memory goes, at least within the limits of the tool. While running, and consuming lots of memory, I click Run -> Get Results.
Here is one of my applications being profile in AQTime with its Allocation Profiler showing exactly what class is allocating how many instances on the heap and how much memory those use. Since you report low Delphi heap usage with FastMM, that tells me that most of AQTime's ability to analyze by delphi class name will also be useless to you. However by using AQTime's events and triggers, you might be able to figure out what areas of your application are causing you a "memory usage expense" and when those occur, what the expense is. AQTime's real-time instrumentation may be sufficient to help you narrow down the cause even though it might not find for you what function call is causing the most memory usage automatically.
The column names include "Object Name" which includes things like this:
* All delphi classes, and their instance count and heap usage.
* Virtual Memory blocks allocated via Win32 calls.
It can detect Delphi and C/C++ library allocations on the heap, and can see certain Windows-API level memory allocations.
Note the live count of objects, the amount of memory from the heap that is used.
I usually try to figure out the memory cost of a particular operation by measuring heap memory use before, and just after, some expensive operation, but before the cleanup (freeing) of the memory from that expensive operation. I can set event points inside AQTime and when a particular method gets hit or a flag gets turned on by me, I can measure before, and after values, and then compare them.
FastMM alone can not even detect a non-delphi allocation or an allocation from a heap that is not being managed by FastMM. AQTime is not limited in that way.
One old application started to consume memory a lot after server update. Memory usage seems to rise with out limit until program hangs.
According to FastMM4 and EurekaLog, there's no memory leak (except 28 bytes), so I assume all memory is freed when application is shutdown.
Are there any tools or strategies suitable for tracking this kind of memory problem?
Since September 2012, there is a very simple and comfortable way to find this type of "run-time only" memory leaks.
FastMM4991 introduced a new method, LogMemoryManagerStateToFile:
Added the LogMemoryManagerStateToFile call. This call logs a summary of
the memory manager state to file: The total allocated memory, overhead,
efficiency, and a breakdown of allocated memory by class and string type.
This call may be useful to catch objects that do not necessarily leak, but
do linger longer than they should.
To discover the leak at run time, you only need these steps
add a call to LogMemoryManagerStateToFile('memory.log', '') in a place where it will be called in intervals
run the application
open the log file with a tail program (for example BareTail), which will auto-refresh when the file content changes
watch the first lines of the file, they will contain the memory allocations which occupy the highest amount of memory
if you see a class or memory type constantly has a growing number of instances, this can be the reason of your leak
The growing memory consumption is an application issue. It is not a bug, which can discover FastMM4 or EurekaLog. As from they point of view - application just correctly uses the memory.
Using AQTime, MemProof (hard to find, D7 is last supported version (?)), SleuthQA (similar to MemProof) or similar memory profilers, you can track the memory usage outside of application in real-time.
Using FastMM4, GetMemoryManagerState / GetMemoryManagerUsageSummary you can track memory usage from application. Output this information into trace file and analyze it after run. Or make simple wrapping function for one of the above procedures, which will return curent memory usage. And call it from IDE Debugger Evalute / Modify, add to Watches or call OutputDebugString, and see the current memory usage.
Note, if memory is eated by some DLL then you may not see her memory usage using (3). Use (2).
Analyzing the memory usage and the tasks performed by the application, you may discover what leads to raised memory usage.
AQTime (a commercial tool which is quite expensive) can report your memory usage, down to the line of source code that allocated each object. In the case of very large memory usage scenarios, you might want the AQTime functionality that can show the number of objects and the size (total plus individual instance size) for each object. AQTime worked great for me, starting with Delphi 7, and all later versions, including your version (2006) and the latest versions (XE and XE2).
As the program memory usage grows, AQTime can be used to grab "snapshots" of the runtime heap, you can use to understand memory usage of your application; What is being created, and how many of each object exists. Even when no leaks exist, understanding the runtime behaviour of your application in terms of the objects it creates and manages, is very important, and AQTime is the most powerful tool I know of for Delphi users.
If you are willing to upgrade to Delphi XE/XE2, you might have an included light version of AQTime already, if so, check it out. If not, I recommend you try their demo. I am unaware of any free or open source alternatives that can provide the same functionality.
Lesser functionality could be cobbled together manually by writing lots of trace messages, or using the FastMM full-debug-mode. If you could write a complete dump of your memory usage into a very large file, you might be able to write some tools to parse, and create a summary. The problem I have with FastMM in this case, is that you will be drowned in detail information, without the ability to extract exactly the summary information that helps you understand your situation. So, you can try to write your own tool to summarize the memory usage. In one application I had that used a series of components that I knew would use a lot of memory, I wrote a dialog box into my application that showed current memory usage by these large memory-blob-of-data objects.
Have you ever think about the Leak that is causing the IDE... it is so huge!!!
In my case (2GB of RAM) i do the next...
1. Open the IDE
2. Leave it minimized for near six hours
3. See how Physical memory is getting used
The result:
While IDE is oppened (remember i also do the test having it minimized) it is getting more and more RAM... till no more ram free.
It gets all 2GB RAM + all Pagefile hard disk space (i have it configured to a mas of 4GB)
In less that six hours (doing nothing on IDE) it tries to use more than 6GB.
That is called a Memory Leak casused by the IDE... i do not type any letter on IDE, do not compile anything, do not even open any project... just open IDE and minimize it... leave the computer without doing anything on it for about six hours and IDE is consuming 6GB of memory.
Of course, after that, the IDE start with annoying messages of SystemOutOfMemory... and i must kill it... then all that 6GB are freed!!!
When on the hell will this get fixed?
Please note i have all patches applied, i also tested without applying each patch/hotfix, etc...
The best i got was dissabling some options on Tools, like the one that underlines bad code, etc... so why on the hell that option has any influence... i am not typing anything on the IDE (on the tests)... and if i have it dissabled the memory leak gets reduced a lot...
Of course, if i use the IDE (write code on an opened project) without even compiling / running it... the thing goes much more worst... memory leak upto 6GB can got reached on less than an hour, sometimes occurs after 15 minutes of Copy/Paste source code.
Seems there will not be a solution in a short time!!!
So i got the next solution that works perfect:
-Close the IDE an reopen it each 15 minutes or less
Ugly solution, i know... but works!!!
I'm finding AQTime hard to use because it interferes with the original program too much. If I have a program that uses, for example, 300MB of ram I can use AQTime's allocation profiler without a problem, and find out where most of the memory is being used. However I notice that running under AQTime, the original program uses more like 1GB while it's being profiled.
Right now I'm trying to reduce memory usage in a program which is using 1.4GB of memory. If I run it under AQTime, then the original program uses all of the 2GB address space and crashes. I can of course invent a smaller set of test data and estimate how the memory usage will scale with the full data set - but the reason I'm using a profiler in the first place is to try to avoid this sort of guesswork.
I already have AQTime set to 'Collect stack information - None' and all the check boxes to do with checking memory integrity are switched off, and I've tried restricting the area being profiled to just a few classes but this doesn't seem to improve anything. Is there a way to use AQTime that produces a smaller overhead? Or failing that, what other approaches are there to get a good idea of the memory being used?
The app is written in Delphi 2010 and I'm using AQTime 6.
NB: On top of the increased memory usage, running under AQTime slows the app down an awful lot, making the whole exercise not just impossible but impractical too :-P
AFAIK the allocation profiler will track memory block allocation regardless of profiling areas. Profiling areas are used to track classes instantiation. Of course memory-profiling an application that allocates a large amount of memory is a issue, you may try to use the LARGE_ADRESS_AWARE flag, and the /3GB boot switch, or use a 64 bit system (as long as you have at least 4GB of memory, or more). Also you can take snapshot of the application state before it crashes, to see where the memory is allocated. Profiling takes time, anyway, you may have to let it run for a while.
I have a MDI program. When It starts it takes 2-3MB of RAM. Then, in this program I create about 260 MDI child windows (each has a TStringGrid, a bitmap and some other controls) and display some data. The application needs about 500MB to load all those windows. If I close each MDI child manually, the application still uses 160MB of RAM. Why it doesn't return to few MB of RAM? Should I worry about this? 160MB it is A LOT for a system that has only 1GB or RAM!!
Note: I use the WORKING SET column in Task Manager to see RAM statistics. Maybe I need a better tool to read the RAM utilization. (Private Working Set is just a bit smaller than Working Set).
This is not a leak!
FastMM (set on aggressive) indicates no memory leak when I close the program. See my Answer post for additional evidence that it isn't a leak.
I release stuff
Many people told me that closing a child window only hides it. I know that. I use "Action:= caFree" to actually release the forms. Each form is responsible for releasing the controls it holds.
Answer
I have found that FastMM is responsible for this. See the answer I posted below.
Delphi 7, Win 7 32 bit
Similar posts:
Can memory be cleaned up?
When to call SetProcessWorkingSetSize? (Convincing the memory manager to release the memory)
Task Manager is not the right tool to detect memory leaks. Delphi allocates large blocks of memory and keeps them later for future use, so certain increase in allocated memory is expected even after you release all resources. Any other results (and more detailed answers) can be obtained only by using specialize memory analysis tools. AQTime is the first that comes to mind, or if you can find old but useful MemProof, it would help you a lot (MemProof is free and for memory analysis it's more handy than AQTime).
It is very well possible that FastMM does not show memory leaks upon application termination (for instance because all objects are TComponents that are owned, and the ownser frees them).
But in the mean time, while running those components can still be around, and not freed soon enough.
Did you use the FastMM unit that shows a form with the current memory usage?
< Edit >
This is the FastMMUsageTracker.pas in the directory ...\FastMM\Demos\Usage Tracker.
Use that unit, then call the ShowFastMMUsageTracker function in it.
You can refresh that form every once in a while to see how your memory consumption grows.
I have put a FastMMUsageTrackerProject sample on-line, including an update of FastMM4 that makes it easier to check and debug memory leaks:
the form in the FastMMUsageTracker unit is now resizable, and the controls in it anchor in the right way
there is a new FastMmBootstrapUnit unit making debugging specific memory leaks easier
Something I had at hand last week, was a 3rd party DLL, which was not written in Delphi.
The DLL had a memory leak using Windows GlobalAlloc calls, which are not tracked by FastMM.
NB: I'm about to post an update to FastMM on
--jeroen
Answer:
I just removed FastMM from my project and the program returned to few MB after freeing all those child windows. Many may argue that this is not a misbehavior and that FastMM is doing this in order to do some kind of kinky memory optimizations. They may be true. However, it may be good for MY application but it may not be good of other running applications.
So, at least we know who causes this. I worried to a whole day that may program is leaking RAM like an old bucket. I am relieved now.
UPDATE:
To confirm that this behavior is generated by FastMM (as suggested by Barry Kelly) I created a second program that allocated A LOT of RAM. As soon as Windows ran out of RAM, my program memory utilization returned to its original value.
(Note: I am not saying there is a bug in FastMM!)
My program is not leaking. Problem solved.
The main limitation of FastMM's memory leak tracing is that it can only run when you shut down the program. It could be that you're still holding references to objects or other data that gets cleaned up when you shut down the program, but stays around until then.
For example, when you close the MDI child windows, do you call Free or Release on them, or just make them disappear? If they're hidden but not freed, they'll still be in memory.
If you close an MDI form it is not freed automatically. Use Action = caFree (google for that) to make sure the form is also freed.
Somebody suggested recently ( My program never releases the memory back. Why? ) that my program leaks some memory. I have FastMM set to aggressive and it reports no memory leaks when I shutdown the program.
Anyway, I would like to know if there can be memory leaks that are no detected by FastMM?
Update: I don't personally use Win API to allocate memory. But I am afraid that some 3rd party components I use (not so many) may use it. Can you let me know all possible API calls that FastMM cannot intercept? I will do a search in my code for them. Thanks.
Delphi 7, Win 7 32 bit
FastMM 4.97
I am not interested about interfaces.
FastMM is a layer on top of Windows memory management. Obviously, if you (or some component or whatever) uses Windows APIs to allocate memory, then such allocation bypasses FastMM and you won't be able to track it. BTW Delphi memory managers themselves use that APIs to allocate chunks of memory. So if you need to see allocations on that level, FastMM is not enough - you must use tools like AQTime and similar (as I suggested in the previous question).
No, only memory leaks which memory was alocated by FastMM.
EDIT:
Maybe the answer looks wrapped but it is not! If anyone check the FastMM haw is made than can see that every pointer of memory alocation is pushed (and poped out at FreeMem) in to one of stacks (there is more stacks, depend of memory size) so at the end of closing application the FastMM only check stacks, if something in stacks, and if it is, than report memory leak!
I've never known FastMM to fail to detect a memory leak.
There are several possible causes: (which apply to any memory manager)
your main program loop leaks memory, but does so to something that is freed on shutdown
the simplest case is logging to a memo. The memo gets bigger and bigger, but is destroyed on shutdown.
the memory is allocated outside fastmm's control
directly allocating from windows
memory allocated in dlls etc.
Heapfragmentation. A memory manager keeps large blocks allocated (e.g. because it still contains a small % of allocations). Result: The app doesn't use it, but it is not release to the OS either. The runtime/memorymanager keeps it around.
fastmm should be more resilient against this phenomena, but in doubt try to print heapmanager info to see if this is the case.
There is a lot of good answer already, but one point that wasn't mentionned yet...
There is a lot of "leaks" that won't get detected by most memory leak detector since the memory IS freed, but way after it isn't used anymore. For exemple, object stacked in a TObjectList. Object are put in the object list, but after you are done using them, you don't free them. They will be destroyed when the object list is destroyed too (When application close for exemple, assuming OwnsObject=True). Since the objects are actually freed, objects are not "leaked", but still make your application use more and more memory over time.
FastMM won't report those as it only makes "full run" analysis. To detect those, you need a memory leak detector that allows to do partial runs, that is, analyzing what "leaked" between point A and point B during the execution. AQTime that Eugene mentionned allow that kind of checks. But be aware that is takes a bit of analysis because that will yield many false-positive (Pretty much all "realloc" operations will be marked as a leak).
FastMM does not detect leaks of memory allocations not going through FastMM.
This can include GlobalAlloc calls from 3rd party libraries or DLLs you use.
Edit: Microsoft's MSDN has a nice list of memory allocation methods.
This was in fact the problem I mentioned I mentioned in my answer to your previous FastMM question.
You can use a tool like VMMap to track the memory leaks that FastMM cannot detect.
--jeroen