Is it possible to think to have in future an automatic garbage collector in Delphi? There are many applications in which a higly detailed control on when to free an object is not so important and it is just an extra thing to care about.
For such applications having a kind of garbage collector that works like java's one would be interesting.
It could be set in the project options.
Will this possible or not?
Note: I don't mean to manually create it like explained here, I really mean a Delphi feature.
Stated in another way: is it possible to set FastMM to act as Garbage collector?
There are many applications in which a higly detailed control on when to free an object is not so important and it is just an extra thing to care about.
I believe there are almost none such applications. Most of the times you think you don't require a control on when your objects are destroyed, you're potentially introducing a bug.
Now, there indeed are cases when certain objects can be safely ignored to later be dealt with by automatic collector. But keep in mind that you need to think this through for every object you're planning on not destroying manually. What if it holds some locks? What if it has some files open, maybe in share-deny mode?
There's not much of a benefit in freeing you from thinking about destroying every object, when to program safely you still need to think about destroying every object.
The purpose of garbage collectors is not to free programmers from seeing this stuff through. It's to save a bit on reference counting and try/finally calls.
Garbage collection is possible in C and C++, so I see no reason why Delphi couldn't also have such a feature. If you cross your fingers and wait long enough, Delphi might get garbage collection. I don't see it as a priority for Embarcadero, though.
You can't set FastMM to act as a garbage collector because FastMM doesn't do garbage collection, so there's nothing to set. Delphi's hypothetical future garbage-collection feature would probably have to cooperate with the memory manager, so if such a feature ever exists, and FastMM is still the memory manager at that time, then FastMM will probably gain some settings.
you have both pros and cons with garbage collection i thing delphi is good even without GC(garbage collector). even delphi apps take less memory size than managed .net apps , some times garbage collection also slow down the process because it has to find the unwanted resources , confirm whether they are needed again and delete it.if it needed again it has to load again( app becomes slow) or an error there so delphi is good without GC manually freeing is good for a professional programmer
lasted versions of delphi coming with RTTI (this is also a reason for the big size of apps)
i think rtti(Runtime Type Information) can help us in future . because it hold some informations about the process going on so i think may be in future some similar function like garbage collector is possible but not sure
but delphi for dot net 2007 and other old delphi dot net have garbage collector + vcl but now deprecated (the garbage collector also dont work 100% well)
Related
i'm using WMI SMBios to get some hardware information
check uSMBios.pas
i don't wanna users see what is the used serial numbers in memory so i'm trying to clear it
when i call
SMBios:=TSMBios.Create;
//my code
SMBios.free;
the SMBios Object still in memory in many locations
i tried this code on Destroy Event
if Assigned(FRawSMBIOSData.SMBIOSTableData) then
begin
ZeroMemory(FRawSMBIOSData.SMBIOSTableData,FRawSMBIOSData.Length);
FreeMem(FRawSMBIOSData.SMBIOSTableData);
end;
it working great with GetSystemFirmwareTable API code in SMBios but in WMI it removes some memory but still i can find few blocks
wondering why after calling object.free or freeandnil the used memory not released
any idea how to force the application to free it ?
Memory is released, it is just not wiped. You maybe mistake two concepts: the memory is bound to some owner and cannot be given to another one, and memory is cleansed of all the information.
Look, when you go over fresh snow or over sands, you leave your footsteps behind you. You moved away, so the places you've been through are FREE now for anyone else to occupy. But your footsteps remain there until someone would overwrite them with his own ones.
Now, you may be paranoid and after every step you would turn back, take a brush and remove your fresh footstep. That is possible and might make sense, but it might be painfully slow.
Some objects might deal with sensitive data, like passwords, cipher keys, personal data in mass calculations, etc. For those objects there is the sense to be paranoid and brush out every their trace. So those objects are written in a way to wipe the memory they would no more need immediately after last use. And to do it once again in the destructor.
But when you just closed the form with the message like "file saved successfully" there ain't any secrets worth painting over. And that is the most of the program.
So now please decide if you really have some sensitive data like passwords. If you do - your code should overwrite it with different data before freeing. And you would have to learn how the data is kept for different types in Delphi, so pieces of the data would not be copied in other places of memory during your processing of them. But most probably you don't need the data actually destroyed, you only need to mark "this place is FREE for anyone to put their data over my garbage" and that is what freeing object on Delphi actually does. If that is enough for you just don't you bother to wipe the data (which is substituting random garbage instead of sensitive garbage, but a garbage still).
Now, few words about suggestions of LU RD and whosrdaddy. Yes, Delphi provides you means to hook into the way heap is managed and to explicitly wipe the data with garbage before marking the apartment free. However this is only a partial solution for sensitive data.
99,9% of times you would be clearing data that was not worth it. Strings, dynamic arrays, TList and other containers would be slow - and your program too.
your app consists of procedures, that have local variables. Many of those variables, like Short Strings, fixed size arrays, GUIDs, are allocated on stack rather than in heap. Those suggestions would not clean them, only free.
your objects typically allocate memory in Delphi heap. But they might also allocate it otherwise. In Windows heap, in some multithreading-aware pool, or whatever. That memory would not be wiped modifying default Delphi heap manager behavior.
Overall it is the same idea. Your procedure or your object knows which data is dangerous and where it is kept - that object or procedure is responsible of cleansing. Global Delphi-scale solutions would be both ineffective and unreliable.
Say for instance I write a program which allocates a bunch of large objects when it is initialized. Then the program runs for awhile, perhaps indefinitely, and when it's time to terminate, each of the large initialized objects are freed.
So my question is, will it take longer to manually deallocate each block of memory separately at the end of the program's life or would it be better to let the system unload the program and deallocate all of the virtual memory given to the program by the system at the same time.
Would it be safe and/or faster? Also, if it is safe, does the compiler do this when set to optimise anyway?
1) Not all systems will free a memory for you when application terminates. Of course most of the modern desktop systems will do this, so if you are going to run your program only on Linux or Mac(or Windows), you can leave the deallocation to the system.
2) Often it is needed to make some operations with the data on termination, not just to free the memory. So if you are going to develop such program design that makes it hard to deallocate objects at the end manually, then it can happen that later you will need to perform some code before exiting and you will face up with hard problem.
2') Sometimes even if you think that your program will need some objects all the way until dead, later you may want to make a library from you program or change a project to load and unload you big objects and the poor design of your program will make this hard or impossible.
3) Moreover, the program deallocation performance depends on the implementation of the allocator you are going to use in your program. The system deallocation depends on the system memory management and even for a single system there can be several implementations. So if you face with allocation/deallocation performance problems - you would like to develop better allocator rather then hope on the system.
4) So my opinion is: When you deallocate memory manually at the end - you are always on a right way. When you don't do this, perhaps you can get some ambiguous benefits in several cases, but likely you will just face with the problems sooner or later.
Well most OS will free the memory at exit if the program, but the bigger question is why would you want it to have to?
Is it faster? Hard to say with memory sometimes. I would guess not really and definitely not worth breaking good coding practices anyway.
Is it safe? Define safe... Will your OS crash? Probably not. Will your code be susceptible to memory leaks or other problems? Absolutely, it will. In fact you are basically telling it you want memory leaks.
Best practice is to always free your memory when you are done with it. With C and C++, every malloced or new block of memory should have a corresponding free or delete.
It is a bad idea to rely on the OS to free your memory because it not only makes your code look bad and makes it less portable, but if the program was ever integrated into another program, then you will likely be tracking down memory leaks for hours.
So, short answer, always do it manually.
Programs with a short maintenance life time are good candidates for memory deallocation by "exit() and let the kernel sort them out." However, if the program will last more than a few months you have to consider the maintenance burden.
For instance, consider that someone may realize that a subsequent stage is required in the program, and some of the data is not needed, or not needed in memory. They now have to go and find out how to deallocate the memory, properly removing stale references, etc.
I had an issue recently (see my last question) that led me to take a closer look at the memory management in my Delphi application. After my first exploration, I have two questions.
I've started playing with the FastMMUsageTracker, and noticed the following. When I open a file to be used by the app (which also creates a form etc...), there is a significant discrepancy between the variation in available virtual memory for the app, and the variation in "FastMM4 allocated" memory.
First off, I'm a little confused by the terminology: why is there some FastMM-allocated memory and some "System-allocated" (and reserved) memory? Since FastMM is the memory manager, why is the system in charge of allocating some of the memory?
Also, how can I get more details on what objects/structures have been allocated that memory? The VM chart is only useful in showing the amount of memory that is "system allocated", "system reserved", or "FastMM allocated", but there is no link to the actual objects requiring that memory. Is it possible for example to get a report, mid-execution, similar to what FastMM generates upon closing the application? FastMM obviously stores that information somewhere.
As a bonus for me, if people can recommend a good reference (book, website) on the subject, it would also be much appreciated. There are tons of info on the net, but it's usually very case-specific and experts-oriented.
Thanks!
PS: This is not about finding leaks, no problem there, just trying to understand memory management better and be pre-emptive for the future, as our application uses more and more memory.
Some of your questions are easy. Well, one of them anyway!
Why is there some FastMM-allocated
memory and some "System-allocated"
(and reserved) memory? Since FastMM is
the memory manager, why is the system
in charge of allocating some of the
memory?
The code that you write in Delphi is only part of what runs in your process. You use 3rd party libraries in the form of DLLs, most notably the Windows API. Anytime you create a Delphi form, for example, there are a lot of windows objects behind it that consume memory. This memory does not get allocated by FastMM and I presume is what is termed "system-allocated" in your question.
However, if you want to go any deeper then this very rapidly becomes an extremely complex topic. If you do want to go deeper into the implementation of Windows memory management then I think you need to consult a serious reference source. I suggest Windows Internals by Mark Russinovich, David Solomon and Alex Ionescu.
First off, I'm a little confused by the terminology: why is there some FastMM-allocated memory and some "System-allocated" (and reserved) memory? Since FastMM is the memory manager, why is the system in charge of allocating some of the memory?
Where do you suppose FastMM gets the memory to allocate? It comes from the system, of course.
When your app starts up, FastMM gets a block of memory from the system. When you ask for some memory to use (whether with GetMem, New, or TSomething.Create), FastMM tries to give it to you from that first initial block. If there's not enough there, FastMM asks for more (in one block if possible) from the system, and returns a chunk of that to you. When you free something, FastMM doesn't return that memory to the OS, because it figures you'll use it again. It just marks it as unused internally. It also tries to realign unused blocks so that they're as contiguous as possible, in order to try not to have to go back to the OS for more needlessly. (This realignment isn't always possible, though; that's where you end up with memory fragmentation from things like multiple resizing of dynamic arrays, lots of object creates and frees, and so forth.)
In addition to the memory FastMM manages in your app, the system sets aside room for the stack and heap. Each process gets a meg of stack space when it starts up, as room to put variables. This stack (and the heap) can grow dynamically as needed.
When your application exits, all of the memory it's allocated is released back to the OS. (It may not appear so immediately in Task Manager, but it is.)
Is it possible for example to get a report, mid-execution, similar to what FastMM generates upon closing the application?
Not as far as I can tell. Because FastMM stores it somewhere doesn't necessarily mean there's a way to access it during runtime from outside the memory manager. You can look at the source for FastMMUsageTracker to see how the information is retrieved (using GetMemoryManagerState and GetMemoryMap, in the RefreshSnapshot method). The source to FastMM4 is also available; you can look and see what public methods are available.
FastMM's own documentation (in the form of the readme files, FastMM4Options.inc comments, and the FastMM4_FAQ.txt file) is useful to some extent in explaining how it works and what debugging options (and information) is available.
For a detailed map of what memory a process is using, try VMMAP from www.sysinternals.com (also co-authored by Mark Russinovich, mentioned in David's answer). This also allows you to see what is stored in some of the locations (type control-T when a detail line is selected).
Warning: there is much more memory in use by your process than you might think. You may need to read the book first.
I have a MDI program. When It starts it takes 2-3MB of RAM. Then, in this program I create about 260 MDI child windows (each has a TStringGrid, a bitmap and some other controls) and display some data. The application needs about 500MB to load all those windows. If I close each MDI child manually, the application still uses 160MB of RAM. Why it doesn't return to few MB of RAM? Should I worry about this? 160MB it is A LOT for a system that has only 1GB or RAM!!
Note: I use the WORKING SET column in Task Manager to see RAM statistics. Maybe I need a better tool to read the RAM utilization. (Private Working Set is just a bit smaller than Working Set).
This is not a leak!
FastMM (set on aggressive) indicates no memory leak when I close the program. See my Answer post for additional evidence that it isn't a leak.
I release stuff
Many people told me that closing a child window only hides it. I know that. I use "Action:= caFree" to actually release the forms. Each form is responsible for releasing the controls it holds.
Answer
I have found that FastMM is responsible for this. See the answer I posted below.
Delphi 7, Win 7 32 bit
Similar posts:
Can memory be cleaned up?
When to call SetProcessWorkingSetSize? (Convincing the memory manager to release the memory)
Task Manager is not the right tool to detect memory leaks. Delphi allocates large blocks of memory and keeps them later for future use, so certain increase in allocated memory is expected even after you release all resources. Any other results (and more detailed answers) can be obtained only by using specialize memory analysis tools. AQTime is the first that comes to mind, or if you can find old but useful MemProof, it would help you a lot (MemProof is free and for memory analysis it's more handy than AQTime).
It is very well possible that FastMM does not show memory leaks upon application termination (for instance because all objects are TComponents that are owned, and the ownser frees them).
But in the mean time, while running those components can still be around, and not freed soon enough.
Did you use the FastMM unit that shows a form with the current memory usage?
< Edit >
This is the FastMMUsageTracker.pas in the directory ...\FastMM\Demos\Usage Tracker.
Use that unit, then call the ShowFastMMUsageTracker function in it.
You can refresh that form every once in a while to see how your memory consumption grows.
I have put a FastMMUsageTrackerProject sample on-line, including an update of FastMM4 that makes it easier to check and debug memory leaks:
the form in the FastMMUsageTracker unit is now resizable, and the controls in it anchor in the right way
there is a new FastMmBootstrapUnit unit making debugging specific memory leaks easier
Something I had at hand last week, was a 3rd party DLL, which was not written in Delphi.
The DLL had a memory leak using Windows GlobalAlloc calls, which are not tracked by FastMM.
NB: I'm about to post an update to FastMM on
--jeroen
Answer:
I just removed FastMM from my project and the program returned to few MB after freeing all those child windows. Many may argue that this is not a misbehavior and that FastMM is doing this in order to do some kind of kinky memory optimizations. They may be true. However, it may be good for MY application but it may not be good of other running applications.
So, at least we know who causes this. I worried to a whole day that may program is leaking RAM like an old bucket. I am relieved now.
UPDATE:
To confirm that this behavior is generated by FastMM (as suggested by Barry Kelly) I created a second program that allocated A LOT of RAM. As soon as Windows ran out of RAM, my program memory utilization returned to its original value.
(Note: I am not saying there is a bug in FastMM!)
My program is not leaking. Problem solved.
The main limitation of FastMM's memory leak tracing is that it can only run when you shut down the program. It could be that you're still holding references to objects or other data that gets cleaned up when you shut down the program, but stays around until then.
For example, when you close the MDI child windows, do you call Free or Release on them, or just make them disappear? If they're hidden but not freed, they'll still be in memory.
If you close an MDI form it is not freed automatically. Use Action = caFree (google for that) to make sure the form is also freed.
There are software applications, such as ArtMoney, that edit the memory of other applications.
Is there a way to detect when some other application is editing the memory of my application?
The basic idea to protect from basic memory modification is to encrypt the parts of memory you care about, and have redundant checks to ensure against modification.
None of which will stop a determined hacker, but it's sufficient to keep the script kiddies out of your address space.
One method, used by many virus checkers, is to perform a checksum of your executable or memory and save it. When running, occasionally calculate a new checksum and compare with the original. Most programs don't intentionally modify their executables.
The short answer is no, it's not possible in the general case. Even if you implement some of the suggestions that have been given, there's nothing stopping someone from patching the code that performs the checks.
I don't know the specifics of how ArtMonkey works, but if it functions as a debugger you could try checking regularly to see if DebugHook <> 0, and reacting appropriately if it is. (Just make sure to put that code in a {$IFNDEF DEBUG} block so it doesn't cause trouble for you!)
You might want to ask yourself why you want to prevent people from patchimg your memory, though. Unless there's a genuine security issue, you probably shouldn't even try. Remember that the user's computer, that your program will be running on, is their property, not yours, and if you interfere too much with the user's choices as to what to do with their property, your program is morally indistinguishable from malware.
I do not know how it works, I think it can be done in 3 ways:
ReadProcessMemory and WriteProcessMemory Windows API
using a debugger (check for debughook, but that's almost too easy so it won't use that)
injects a dll so it can acces all memory (because it is in the same process)
The last one is easier (check for injected dll or something like that). The first one is trickier, but I found some articles about it:
Memory breakpoints: http://www.codeproject.com/KB/security/AntiReverseEngineering.aspx?fid=1529949&fr=51&df=90&mpp=25&noise=3&sort=Position&view=Quick#BpMem
Hook "WriteProcessMemory" api: http://www.codeproject.com/KB/system/hooksys.aspx
I asked a similar question, and the conclusion was basically that you cannot stop this.
How can I increase memory security in Delphi