I'm using Delphi 6 and I've got an application which when being shut down produces access violation errors. We use EurekaLog so I'm getting stack traces for debugging, but the errors seem to occur randomly in a different unit each time, but always when something is being freed in the finalization section.
How can I go about debugging this to see what's causing the problem? I'm not sure how to start debugging things that happen when the application is being finalised.
[Edit:] Sorry if I was unclear, perhaps a better question would be: What's the best place to start debugging with breakpoints if I only want to walk through the finalisation sections? The errors seem to arise in third party components we use (the devexpress dx/cxgrid library) so I'd like to start debugging in my code at pretty much the last point before Delphi will start calling finalise routines in other units.
This isn't much to go on, but if I had to guess, based on past experience... are you using packages or COM libraries? If you've got a global variable that's an interface, or an object whose class is declared in a BPL, and you unload the DLL/BPL before cleaning up the object/interface, you'll get access violations because your code is trying to do a VMT lookup in address space that is no longer mapped into the application.
Check for that and make sure you clean up all such variables before finalization begins.
When the application is shutting down, do not free things in the finalization section.
1) When the application shuts down, Windows frees all the application memory. You don't have to do that.
2) When the application shuts down, the memory is released, and the infrastructure is unloaded. You can't call code to close or free objects, because that code may have been already unloaded. You can't access pointers to memory, because those pointers may have already been released.
3) When you try to free things in the finalization section while the application is shutting down, you may get failures that prevent your code from finalizing, thus preventing the application from shutting down, causing a hung application and memory loss. Which is what you were trying to prevent in the first place. Don't do it.
Ok, when you are running on Win95/98, or using external processes, you may in some circumstances have to free shared resources and notify those external processes that you are shutting down. Apart from that, it all happens automagically now.
Related
I am trying to debug memory usage in a large application using Delphi 7. I was able to installed fastmm debug full dll and with it solve some leak problems.
I also installed the memory usage tracker, allowing me to see which blocks were allocated and of what size they are.
My question is, is there a way to find out where the blocks were allocated? I know it is possible because if the memory wasn't freed a stack trace gets printed. Is there a way to 'poke' at fastmm to get it to print the stack trace for a given allocation?
Side question: if the start address of an allocation is known, is there a way to find out which class the object is? (assuming that the allocation was for a object.
You can either:
try to use LogAllocatedBlocksToFile procedure. If its ALastAllocationGroupToLog param is less than AFirstAllocationGroupToLog or is zero, then all blocks along with their allocation call stacks are logged. However, if your app has many memory allocations, prepare to long waiting. I experienced about 4 hrs wait time and 1.5Gb resulting file. (Side-note: use glogg to view such large files)
modify FastMM4.pas so implementation's LogCallStack will be visible in interface. Or you can try to use it directly from FastMM_FullDebugMode.dll
On the side question: try to use DetectClassInstance function.
If you use FullDebugMode and enable the conditionals that will write the data to a file, then you should get exactly what you're asking for. It will write out a stack trace for every leaked allocation when the program shuts down. (If you're debugging a program with a lot of memory leaks, this can take a while. I've seem it make shutdowns last for 10 minutes or more if the leaking item is a container that holds lots of other objects.)
Considering you said in a comment that the app's memory gets cleaned up nicely on app's closing for me sounds you're looking for logical memory leaks - in other words: objects that are alive more time than needed but when the time arrive for finish the app they are cleaned because exist code to clean them.
Example:
Create an TForm using the Application as owner and the variable that references it is the global one that Delphi creates when create the form's unit.
Config the Form's CloseAction to caHide (in the OnClose event)
Show the form, operate on it
Close the form and never more use it until the app closes
Close the app, which makes Application clear all the objects it owns
So you have an logical memory leak but not an physical memory leak - which is the kind that FastMM can easily detect. Since you not intended that our hypothetical TForm live until the application finish, it semantically leaked but since it is referenced and there is code that destroys it at the end of application, to FastMM is a normal allocation.
That said appears you need not the memory dump of the memory manager, but an memory profiler.
I use a lot of components in my Delphi 7 Service application, Indy, Synapse, Zeolibs, etc.
My application is generally stable, I use Eurekalog 6 to capture exceptions, but in rare situations, some threads hang because a 3rd party function it calls has hung, e.g. Indy gets stuck when trying to send email.
In many cases, the application that hung are my customer place, I've no access to their computer, so it is not possible for me to do a live debug. My application requires high availability so even if it hangs once a year, that is not acceptable to my users.
I am now looking for the best way to deal with such a situation where debugging is not feasible but I will still need the application recover by itself. Is it possible for a thread to terminate if a function it calls hangs? Alternatively, I can also restart the entire service when that happens. How about a Watchdog and what is the best way to implement it? Thanks.
I think you are being rather defeatist. Find and fix the bugs. It might be tricky, but it's the right solution.
Killing threads whose behviour you don't understand is never the solution. If you start killing threads you'll likely make things worse. That can lead to other runtime errors, deadlock and so on. Once you start killing threads you've lost control.
Now, it would be safe to kill the process (rather than a specific thread) and rely on a watchdog service to restart the process. But that's a really dire solution.
You should certainly use a tool like madExcept, EurekaLog etc. to debug unexpected exceptions. I see you are already using EurekaLog - that's good.
Deadlocks (it sounds like you have deadlock) can be more tricky to chase down. One good way to debug a deadlock is to get your client to produce a crash dump (e.g. from Process Explorer). Then debug it in WinDbg using map2dbg to produce symbolic stack traces. That will tell you which threads are blocking and that reveals the deadlock. And then fix the bugs.
For more details on this deadlock debugging technique see here: http://capnbry.net/blog/?p=18
I'm not familiar with EurekaLog since I use madExcept, but I would expect EurekaLog has a facility to allow generation of thread stack traces for a hung process. If so then that would most likely be the best approach for you.
Your question is rather too vague. If you don't know which of the various components you're using you wish to blame, then you have zero hope of fixing it. The most likely thing is you're doing something wrong, or that you don't understand how these components work. I very much doubt that it's purely a bug in the components themselves, but hey, either way it's all on you to find what's having a problem, and your job to fix it.
A deadlock that you've created, or a deep process corruption issue, that is happening, may prevent MadExcept from giving you any information, but it's worth trying.
To find out which one is freezing, if any at all, then the madexcept comment is the best suggestion yet. It will time-out (after a configurable # of seconds) and raise an artificial exception for you, interrupting your hung process. This works for user code, and for places where the thread is blocked in a Win32 or kernel function. For example, it's possible that you've set up Indy for infinite timeouts, as that's the default these days in Indy 10, and that what you're experiencing is a timeout related freeze, where network activity that you expected to complete but which never will complete, is causing your program to "hang". The cure here is to change your timeouts.
However, until you figure out WHERE the problem is, I doubt you'll be able to fix it. And so, for that, again, Marcus is right, you should be looking into madExcept. I can't live without it.
Secondly, you should really be adding trace logic to your program, so you know where it's going and what it was doing just before it had a problem. If you really need help doing that, you could try CodeSite, from Raize. Personally I find that OutputDebugString combined with the free Microsoft DebugView utility (formerly from SysInternals) tool is more than enough to debug such problems on a client computer.
Any program with background threads that does not have trace logging, is a badly designed program. Heck, any non-trivial single threaded application that might ever fail or have problems, needs trace logging.
Logging is always going to help, even when MadExcept or other exception tools don't. Trace-Logging is usually a roll-your-own solution, although CodeSite is also quite popular.
We have an application made up by a host (exe) and a lot of modules (dlls) containing gui etc.
Sometimes, the application freezes on shutdown. Mostly it happens during testing through TestComplete. We are not able to reproduce the behavior during debugging.
How can we find out why the application freezes?
I would guess that it is related to threads, but I do not know for sure.
Are there any tools or techniques we should try out?
I think that good old logging would help you. Add some logging to every unit finalization, add such logging to destructors of global data (database connection, global configuration etc). Of course do not destroy logger object.
If your application is multithreaded then add some logging to working threads such as writing '[date] thread [name of class] working' and write it every few seconds (you can use some debug mode). Also add logging when thread discovers that it should terminate.
Also use some system utilites such as ProcessMonitor, Handles, Process Explorer (all by Sysinternals/Microsoft). Monitor disk reads/writes, handle count, memory usage, network connections. Maybe your application dumps some big structures on disk at exit? Maybe it allocated a lot of memory and must release it?
Rig it with EurekaLog or MadExcept, and that may show you where the exception is, or where the memory leak is that is causing the exception. Both of those are excellent tools with fully-featured trial versions. Try 'em, buy 'em. Good stuff.
If the debugger's presence keeps the problem from occurring, then wait for the problem to occur, and then attach the debugger to it. Pause execution and you can inspect each thread's call stack.
If you use lots of GUI components in DLL's and/or do plenty of multi-threading then you'd first have to discover which DLL or thread is causing the problem. Or maybe it's a combination of both. Basically, you should create log events for every DLL and thread that gets loaded/started. Try to get a situation where you have a minimum of DLL's and threads loaded to generate the freeze. Then you've localized the problem to one of those.Also, create simple test-applications or use a unit-testing framework to test specific modules. For example, there is a Delphi version of NUnit available, which might help. (It's called DUnit...) Such a test framework is helpful to isolate the threads and DLL's to check each of them.
If your app is happenning under windows 7 and there are threads running in DLL, you must shutdown/terminate all running DLL threads before closing the main form of executable.
Good luck
Is there a way to access (read or free) memory chunks that are outside the memory that is allocated for the program without getting access violation exceptions.
Well what I actually would like to understand apart from this, is how a memory cleaner (system garbage collector) works. I've always wanted to write such a program. (The language isn't an issue)
Thanks in advance :)
No.
Any modern operating system will prevent one process from accessing memory that belongs to another process.
In fact, it you understood virtual memory, you'd understand that this is impossible. Each process has its own virtual address space.
The simple answer (less I'm mistaken), no. Generally it's not a good idea for 2 reasons. First is because it causes a trust problem between your program and other programs (not to mention us humans won't trust your application either). second is if you were able to access another applications memory and make a change without the application knowing about it, you will cause the application to crash (also viruses do this).
A garbage collector is called from a runtime. The runtime "owns" the memory space and allows other applications to "live" within that memory space. This is why the garbage collector can exist. You will have to create a runtime that the OS allocates memory to, have the runtime execute the application under it's authority and use the GC under it's authority as well. You will need to allow some instrumentation or API that allows the application developer to "request" memory from your runtime (not the OS) and your runtime have a way to not only response to such a request but also keep track of the memory space it's allocating to that application. You will probably need to have a framework (set of DLL's) that makes these calls available to the application (the developer would use them to form the request inside their application).
You have to be sure that your garbage collector does not remove memory other then the memory that is used by the application being executed, as you may have more then 1 application running within your runtime at the same time.
Hope this helps.
Actually the right answer is YES.. there are some programs that does it (and if they exists.. it means it is possible...)
maybe you need to write a kernel drive to accomplish this, but it is possible.
Oh - and I have another example... Debugger attach command... here is one program that interacts with another program memory even though both started as a different process....
of course - messing with another program memory.. if you don't know what you're doing will probably make it crush...
I have just ported several of our home-made Outlook COM-addins from Delphi 2007 to Delphi 2009 and am now experiencing some really weird errors (before you ask: none of which appear to have any obvious relationship to string-handling), for example modal dialogs that hang Outlook when one tries to invoke them a second time (the first time around everything appears to be fine) but only when they're invoked from one specific event handler and not when doing the same thing somewhere else. When I trace the error to a specific line of code and comment out that line or replace it with different code to the same effect (e.g. by copying code that would otherwise be called via a function directly to the calling site), the error will appear to go away - typically only to reoccur a couple of (equally inconspicuous looking) statements later.
When running this inside the Delphi debugger I can see that the freezes are often preceded by Access Violations in GetMem.inc . At least all of these issues are 100% reproducible...
Needless to say we had none of these issues when compiling these addins in Delphi 2007.
Now, I'm quite at a loss. I know I have just been lucky but even though I consider myself a fairly experienced programmer (though mostly in niche areas) I never really had to deal with this class of error before. As the title of this question says, I don't even really know where to start. I can step through the code as much as I like but the endless assembler statements mean nothing to me and neither am I proficient in effectively using the CPU view.
Furthermore, I don't even know for sure yet whether this is an issue with my own code to begin with (I actually tend to doubt it in this case). We are makign massive use of a number of third-party libraries (e.g. JCL, ADX, Redemption). ADX in particular still labels its Delphi 2009 support "beta".
I also tried using FastMM's FullDebugMode and indeed I did uncover a number of errors in ADX that way (e.g. blocks that were modified after having been freed) but all of these also occur when I compile with Delphi 2007 so it doesn't yet seem imperative that these are ultimately the cause for the observed regression.
So, how do I deal with this? - or better yet: Where can I find some good resources on learning how to deal with this? e.g. tutorials on using the CPU view or effectively interpreting and acting upon the reports put out by FastMM? Are these the correct tools at all? Where else should I look?
Addendum:
What types of code should I be suspicious of in this context? What kind of code even has the potential to wreak such havoc in memory? The only places I can think of where my code performs anything remotely approaching explicit memory manipulation is when reserving some buffer space in preparation of a WinAPI call. Also keep in mind that all of my code is identical between the Delphi 2007 and Delphi 2009 versions and the Delphi 2007 version exhibits no such problems.
Update:
With some probability the issue that prompted me to post this question has now been solved. See my own answer below.
The best tool for getting to a solution is probably memory breakpoints.
Debugging memory corruption is painful, so try to make your life as simple as possible first: find an exact, guaranteed-reproducible set of steps that work every time. If necessary, mock up the Outlook host so that you don't need to rely on Outlook timing issues or address space layout issues etc.
It's imperative that you get a reliably reproducible set of steps that results in an AV or other error at a predictable address.
What you then do is restart the process, create a memory breakpoint set for whatever referred to that address, and get familiar with the lifecycle of that chunk of memory. Minmizing and rationalizing your reproduction steps helps here. It may help to add other breakpoints and only enable the memory breakpoint later in the application; or use the logging features of D2009 breakpoints to log memory values / call stacks etc., rather than actually breaking into the debuggee.
Not exactly an answer to the question which was more general, but very probably the solution to the specific problem that prompted it:
I am 95% sure to have identified the problem now! :)
Here's what I did:
I enabled RangeChecking and OverflowChecking in the compiler
I tracked down and fixed all problems that caused ERangeError or EIntOverflow exceptions
(there was one of each)
I ran the program again with FastMM and FullDebugMode enabled
I was finally able to identify the cause of the problem in all cases to be a call to the JCL function GetWindowCaption
It seems that GetWindowCaption has obviously not yet been checked for Unicode-compatibility: It was using the value returned from the API function GetWindowTextLength (which returns the number of characters) as input for ReallocMem (which expects the number of bytes) to allocate the buffer for GetWindowText (which in Delphi 2009 returns a buffer of WideChars). Boom! The function was allocating too little memory for the buffer but GetWindowText simply overwrote the following memory thus corrupting the block footer.
I have now filed this in the JCL bug tracker as item #4648
The bottom line I took out of this is: Always be sure to fix all reported errors! Including (seemingly) non-critical ones like range and overflow errors. If nothing else, it will make debugging that much more predictable.
The fact that you catch double-free bugs in D2007, even though it does appear to work fine in this version, means that you NEED to fix those because you are merely lucky that the D2007 version does not need to recycle the memory as aggressively as the D2009 version and the bugs do not show up due to "shadow persistence" in memory.
I would use FastMM fulldebugmode to find the bad code and fix it as much as possible, then follow Barry's advice to trouble shoot memory usage.
For how to use the features of the Integrated Debugger, and how to log info from non breaking breakpoints, you may want to look at this CodeRage 3 session: Delphi Debugging for Dummies
I'd look in direction of full pageheap support built into the system.
Look in this post for how to configure it. Provided your memory usage is not too extensive, this is the easiest to do to find the problem.
It gets tricky when memory consumption is heavier - but like I said, try full peagheap first.