I am working on a Delphi application using the TService functionalities.
It is a large project that was started by someone else.
The application uses several separate threads for processing, communicating with clients, database access, etc. The application’s main job is to poll regularly (every 2-300ms) certain devices and, a couple times a day, execute specific actions.
Now somewhere there is an unhandled exception for which I cannot seem to find the cause:
According to the debugger, the faulting method is System.Classes.StdWndProc.
This also seems to be confirmed by analyzing the crash dump file with WinDbg.
After numerous tests and debugging, I noticed that the crash happened on my dev computer every day at almost the same time.
I looked at the windows log and found this event:
This, coupled with the fact that the stack trace from Delphi indicates that a message with ID=26 (WM_WININICHANGE) was processed, made me believe that there might be something wrong with my usage of FormatDateTime() or DateTimeToStr() when regional settings are reloaded.
I checked every call and made sure to be using the thread-safe overload with a local instance of TFormatSettings.
However today the service crashed again.
A few points that I think are worth mentioning:
The application is also installed on a Windows 2008 server and has
been running OK for over a month.
On 2012r2 I tried forcing DEP off, but it didn’t change anything.
The service’s OnExecute() method is not implemented. I create a base thread in TService. ServiceStart() which then in turn creates the main data module and all the other threads.
The service is not marked as interactive and is executed with the Local System account.
All data modules are created with AOwner=nil.
With a special parameter, the application can be started a normal windowed application with a main form (which is created only in this case). The exception does not seem to happen when running in GUI mode.
Almost all threads have a message pump and use PostThreadMessage() to exchange information. There are no window handle allocations anywhere.
I have checked the whole project and there are no timers or message dialogs or other graphical components anywhere.
I have activated range as well as overflow checking and found no issues.
I am a loss for what to do here. I have checked and re-checked the code several times without finding anything that could explain the error.
Looking online I found several reports that seem to be pertinent to my situation, but none that actually explains what is going on:
https://answers.microsoft.com/en-us/windows/forum/windows_10-other_settings/windows-10-group-policy-application-hang/72016ea4-ba89-4770-b1de-6ddf14b0a51f
https://www.experts-exchange.com/questions/20720591/Prevent-regional-settings-from-changing-inside-a-TService-class.html
https://forums.embarcadero.com/thread.jspa?messageID=832265
Before taking everything apart I would like to know if anyone experienced anything similar or has any tips.
Thanks
Edit: looking at the call stack and the first method executed I am thinking of the TApplication instance that is used in TService.
LPARAM is always 648680.
LPARAM is a pointer to a string (you can cast it to PChar).
You could try to catch WM_SETTINGS_CHANGE and log anytime it's processing.
I think you could use TApplicationEvents component. The component has OnSettingsChange event. The event provides a setting area name (section name) and a flag that points to changed parameter.
Have a look at doc-wiki.
There is a full list of possible parameters.
I did not test it, but maybe you can use Abort procedure in OnSettingsChange event handler to stop the message distribution. Of course it's not a solution, but it may work.
Related
We had to use CoFreeUnusedLibrariesEx for fixing a bug with heap not being cleared after using a MSXML library
Refer this link:
http://blogs.msdn.com/b/marcelolr/archive/2008/11/13/msxml-heaps-not-being-released.aspx
But this caused another issue with TTimers which takes while to show up and disappears when Delphi app is bounced and again shows after a while.
This app uses TTimers to schedule it's job like running a XML transform .
Here is the issue:
When TTimer.Enable is called it throws an error not enough Timer available.
I know this is a masked error and I would have to figure out how to get to the actual error.
This is a single threaded application with only one timer.
Here are the links I looked into
Most Common reason seems to be invalid windows handle
https://groups.google.com/forum/#!topic/borland.public.delphi.winapi/UrIskaFZggU
There are other threads that suggested that OS ran out of resources for TIMERS I'm not sure if that is relevant to me.
I'm Just trying to understand what is Interaction between CoFreeUnusedLibrariesEx and TTimers that it gradually some how robs it of resources and makes us bounce the app to get it working.
How do I go about solving this issue , I'm looking for some directions?
CoFreeUnusedLibrariesEx should not affect TTimers. But if loading and unloading a (buggy) dll leaks any user objects (this includes timers, window handles, ...) then I could imagine that you run out of user objects.
Use Windows Task Manager and configure it so it will show the "USER Objects" in the "Processes" tab. Then compare the number of user objects when you call CoFreeUnusedLibrariesEx and when you don't call CoFreeUnusedLibrariesEx.
my application uses two self-written components that perform actions during the "loaded" procedure.
The component created first is an single-instance control that should exit the application when another instance is found.
The component created secondly is my database access that establishes the database connection in the "loaded" procedure.
I try to end the application before anything else happens, if a first instance has been found.
Tryout 1:
In my instance control, i call "Application.Terminate" to stop my application. That does not work - my database connection still gets established. Seems that "Application.Terminate" uses a "PostMessage" call to itself, which won't get read until the message gets received (which will happen after all components are loaded...). So obviously, this solution does not work for me.
Tryout 2:
In my instance control, i call "Halt" to stop my application. This immediately stops my application, but seems to create a load of memory leaks - at least that's what delphi's "ReportMemoryLeaksOnShutdown" tells me. Afterwards, i get this annoying "This application has stopped working" windows dialogue. This solution does not work for me.
Question A:
Is there another way to end an Application?
Question B:
Does Delphi's memory leak report work correctly when using "Halt"? Are there really memory leaks after exiting the Application with "Halt"?
If Application.Terminate terminates too late, and you want to terminate post haste, then Halt seems to me to be the appropriate course of action. I don't particularly see why Halt should lead to This application has stopped working exceptions. That suggests that your program's unit finalization code is raising exceptions which are not handled. You could perhaps try to work out (using the debugger, say) why these exceptions are raised and protect against it. Or you could avoid unit finalization with a call to the even more brutal ExitProcess.
There will indeed be memory leaks when you call Halt. However, these can safely be ignored because the operating system reclaims all allocated memory when a process terminates. In other words, the memory is leaked in the sense that is not freed by the application, but the operating system frees it immediately in any case.
However, whether or not a brutal Halt is the correct course of action for you, I cannot say. That's for you to decide. FWIW, a call to Halt from inside a component's Loaded method sounds exceptionally dubious to me.
If all you want to do is ensure that just a single instance of your program runs, then you can make simple modifications to the .dpr file to achieve that. That subject has been covered over and over again here on Stack Overflow. You'll find good coverage and advice in Rob Kennedy's answer here: How can I tell if another instance of my program is already running?
I am getting this AV message about 3 to 5 seconds after the applications close as expected:
Exception EAccessViolation in module rtl160.bpl at 00073225. Access violation at address 500A3225 in module 'rtl160.bpl'. Read of address 00000004.
These (20) applications are very similar in that they are IBX business applications. About half of them did not cause the AV to occur.
These applications were ported from Delphi-xe and they worked flawlessly for a long time. No changes were made to the projects in the port. Both 32 and 64 bit builds gave the same results.
Is this a bug in some library's finalization section freeing a resource or something?
I am using Delphi-XE2 Update 3.
Would appreciate the help.
Try using madExcept / EurekaLog etc. - they give you detailed stack trace on AV. This is not always a panacea, but can point you to the problem.
Access Violations are by their nature already very troublesome beasts since they deal with invalid pointers in memory. One that occurs a while after an application shuts down is even worse because that's when your app is in "cleanup" mode. You're could be dealing with something that went wrong much earlier in the application, but is only exposing itself at shutdown.
General Tips:
Try to always undo things in the reverse order you did them. E.g.
Create A, Create B ... Destroy B, Destroy A
Connect to Database, Open Dataset ... Close Dataset, Disconnect from Database
Even making sure you've done all the above before shutting down can help tremendously.
Any threads that are still running while your application is running can cause problems.
Preferably ensure all your child threads are properly terminated before final shutdown.
Refer back to Closing datasets above. Depending on what you're doing, some database components will create their own threads.
If you're using COM, try ensure ComObj is high up in the initialization sequence (I.e. place it as high as possible in your DPR).
Delphi finalizes units in the reverse order that they were initialized.
And you don't want ComObj to finalize before other things that are dependent on ComObj have also done so.
If you're using interface references, make sure you resolve circular reference issues.
Some of these problems can be tricky to find, but you can do the following:
Setup a source-code "sandbox" environment (you're going to chuck all your changes as soon as you've found the problem).
Figure out the simplest set of steps required to guarantee the error. (Start app and immediately shutdown would be ideal.)
Then you're going to comment-out delete wipe out chunks of code between tests and basically follow a divide and conquer approach to:
rip out code
test
if the problem persists, repeat. Else roll-back and rip out a different chunk of code.
eventually your code base will be small enough to pinpoint likely problems which can be tackled with targeted testing.
I've had this kind of access violation problem on occasion with old Delphi or C++Builder projects. Today I had it with C++Builder. At the time of the crash, by looking in the Debug -> Call Stack window, I can see that it's happening inside a call to fflush, called by __exit_streams and _exit.
I'm not sure what is causing it, since it's so deep in the Borland library code, but it seems to come and go at random when the code changes. And it seems to be more common with multi-form applications.
This time the error went away when I just added a new button on the main form. A button which is just there, has no event handlers and does not do anything. I think that any random change to the code, classes, variables etc rearranges the memory layout when you relink the application, and that either triggers or untriggers the error.
For now, I just leave the new button on the form, set it to "not visible" so that there's no visible change. As it seems to work, it's good enough solution for me at this time.
I've started using SmartInspect in my Delphi applications because my users were running into bugs/problems I couldn't reproduce on my machine. When I have a general idea of the problem I'll monitor the application in a few specific places to confirm what is or is not working.
When the bug doesn't have an obvious cause, I feel lost. I don't know where to start logging in order to narrow down the problem. Are there common techniques or best practices for using a logger?
SmartInspect seems to be quite powerful, but I don't know quite what to log or how to organise my logs so the data is meaningful and useful for catching bugs.
NOTE: I'm using SmartInspect but I assume the answers should be suitable for any logging package.
Here are some guidelines I tried to implement in my own OpenSource logging unit, but it's fairly generic, and as you state, it should be suitable for any logging package:
Make several levels (we use sets) of logging, to tune the logging information needed;
Log all exceptions, even the handled one with a try...except block - and add an exception classes list not worth logging (e.g. EConvertError) - e.g. our unit is able to log all exceptions via a global exception "hook" (no try..except to add in your code), and handle a list of exception classes to be ignored;
Log all "fatal" errors, like database connection issues, or wrong SQL syntax - should be done though "log all exceptions" previous item;
For such exceptions, log the stack trace to know about the calling context;
Be able to log all SQL statements, or database access;
Add a generic User Interface logging, to know which main functions of the software the User did trigger (e.g. for every toolbar button or menu items): it's very common that the user said 'I have this on my screen/report, but I didn't do anything'... and when you see the log, you will discover that the "anything" was done. ;)
Monitor the main methods of your application, and associated parameters;
Logging is an evolving feature: use those general rules above, then tune your logging from experiment, according to your debugging needs.
For UI-driven applications here are the main things I instrument first:
ActionManager or ActionList's events when an action executes (gives me a user clicked here then here then here list).
Unhandled Exceptions with tracebacks using JCL debug go right in my main log, whereas if I was using MadExcept or EurekaLog, exceptions have their own log.
Background thread starts, stops and significant history events
Warnings, errors, API function failures, file access failures, handled (caught) exceptions.
Current memory usage can be useful for long running processes to see if there are memory leaks (which could lead to an out of memory error some day).
When we created our program, it closed properly under XP, but under Vista, even though the program is closed, the process is often still running until I open task manager and manually kill the process.
Any suggestions on how to make sure this is closed when we close the program? What a pain.
Mark Gundy
www.magchat.com
The debugger will be your friend here. Step through the shutdown until you get stuck. That'll be the best bet.
But... assuming for some reason the debugger is affecting the behaviour, or can't be used for some other reason:
A few earlier replies suggest using FastMM. This has been the default memory manager for delphi since D2006. You can try flipping on the options to report memory leaks... but that'll only work after you've finished shutting down the process, which is what isn't working. :) So I doubt it will help you much in this case. The full version of FastMM, as opposed to the standard one packaged with delphi, might have some other helpful features... But I can't recall any that would help here.
To inspect your app while it's stuck, you could try the sysinternals tools, like Process Explorer & Process Monitor. Process Monitor may show you if it's choking on any ACCESS_DENIED errors, etc, and the stack trace at the time of the error. Process Explorer could be especially useful, by listing all the handles your process is still holding open, and allowing you to view the stack of all its living threads. A familiar function or two, in the stack traces of any hung threads, may send you hunting in the right area.
Are you using multiple threads? If one of them can't terminate for some reason, it'll hang the cleanup process.
The short answer is that you have a bug in your application. So, have you tried debugging it?
If you have the Delphi IDE installed on Vista, run the app from the IDE and break it when it 'hangs'. Chances are that you'll have a thread that hasn't terminated itself, and the VCL is waiting for it to finish.
If you don't have the IDE installed on vista, you can probably use the remote debugger, but I'm not familiar with this.
What do you call to close your program? Try using
Application.Terminate;
To force all forms to close and the process to exit.
It probably means you have some memory leak - some resource is not released.
If your application instantiates COM objects, check that you properly close them.
If you use older version of Delphi, you can try to locate possible memory leaks with FastMM
Edit, examples as requested:
I had two situations where application sometimes would stay resident in memory after exit. Both applications would disappear from taskbar, but they would be still active - visible in task manager. Both applications were created with Delphi 7.
First application has one main window with panel in which ather forms are embedded. Each form is standard TForm. Each form is created first time user requests it. Everything worked without problems in test environment, but users reported that from time to time, application would remain in memory, usually after longer usage - when user displayed almost all of the forms in application. I could not replicate this behavior while testing. That was first time I found out about FastMM. When I first run application with FastMM, it reported that some of forms embedded in main form were not released. This version was tested at user site and it turned out that 2 forms that have lots of data aware components would hang up and prevent application from exiting. When I added code that makes sure that all created forms are released before main form, application never hung up on exit.
Second example is application that uses COM to activate GIS editor (Geomedia) to manipulate some map data. I imported type libraries and created Delphi wrapper object. Some of created objects were not freed when application ends. Application icon disappeared from task bar, but application and GIS editor were still active and visible in task manager. Again, it did not happen every time application run. I included FastMM in application and it reported that some objects were not freed. I made sure that every created object gets freed and after that application did not hang on exit any more.
... and if you need to close single form, then use:
Form.Close;
GUI AND HANDLER DELETED. Reference and some parts of memory is left to provide access to variables, constants, etc.
If you need to hide ( something like Minimize, only GUI is cleared ) form, then:
Form.Hide;
GUI is cleared, leaving internal resources untoauched ( reference, handler, memory )
If form is dynamic ( created at runetime ), then use:
Form.Destroy;
ALL RESOURCES WILL BE CLEARED FROM MEMORY, leaving reference and handlers attached so you could access its location in memory.
If form is dynamic and you will not use it for the same interface, then:
Form.Free;
ALL RESOURCES, REFERENCES, HANDLERS ARE DELETED. I recommend to use this to VCL TComponent class, not for TForm class.
Also, there is Form.FreeAndNill, but my guess that it deleted all memory and loaded handlers, only allowing to use same memory space in the same interface again... (I might be wrong though).
P.S. I hope I am not writing wrong things as the last time I read the theory was a long, long time ago... and it was about Destructor DESTROY in general ...
P.P.S ALSO PLEASE BE CAREFUL if you're writing a Vista-ready app - that it includes UAC Handling with manifests / runtime code and SuperBar compatibility requirement. Also Aero requires some additional megabytes to output file due its Aero feature ... :P