I have a problem I cannot solve. Of course here I just expect to have a suggestion that can help me to find a solution.
Basically my application is full of runtime generated TPopupMenuItems (while all the TPopupMenus are hardcoded). In some cases what I do is simply hide/show or enable/disable items, in other cases I create items at runtime.
In some machines only, after leaving the application running for days (2 or more) the popupmenus don't work anymore correctly.
The behaviour is:
all the TPopupmenu items look are the same, and execute the same action.
The action is the one performed by the first TPopupMenuItem of the application (the first generated at runtime when the application starts, this is the only hint I have).
Imagine in correct scenario I have (in a 3-items-TPopupMenu):
Item23
Item24
Item25
after the problem I see:
Item1
Item1
Item1
(where Item1 is the TPopupMenuItem belonging to another TPopupMenu).
Does this tell you something?
Thanks.
Update:
I tried to look at the code of my popupmenus and I found what could be a common cause, and this explains also why FastMM4 didn't find this:
while mnuItem.Count > 0 do
mnuItem.Delete(0);
Delete (I just read in the documentation) doesn't free the item, I should call free instead. Anyway when closing the application the main popupmenus are freed correctly, and FastMM4 doesn't complain. So this is probably the solution, now I don't know why Delete was used, I didn't write that code.
Further update:
I tried to make a sample application, I couldn't reproduce the problem, but for sure I noticed that using this is much more performant (I tried a loop with 10000 recursions):
while mnuItem.Count > 0 do
mnuItem.Items[0].Free;
I will try for this in my app (but I need to let some days pass to really know if I got the problem, ayway for sure this is a major improvement anyway).
I confirm that the problem was linked to Delete instead of Free. Popupmenu wsa refreshed every minute on the machines that had the problem (so it was not OS or HW specific, just config specific). Then according to user settings the menu could have 10 to 100 items, so leaving it idle for days made it possible to hit the handle limit.
By the way it also makes no sense to refresh the popupmenu in that way, so I found also an optimizaion removed a bug.
Did you check for memory leakage and handles that aren't freed?
Related
Updates
2016-02-18: Added process information
I have a Delphi program compiled using XE4. It is being used by a few hundred customers. A couple of weeks ago one of these customers reported that some areas of the executable was being erased (images bellow) randomly during the day. This client has 35 sites using this exe and the problem occurs on no more than 10 of these sites.
Investigation
1 - My first suspicion was an infinite loop. The exe keeps responding while the components are erased, nothing changed on the code so radically from the time this problem did'n happen and the logs don't show any loop (this exe has logs everywhere).
2 - Misbehaving threads. I have a separate thread that syncs data between this exe and our server in the cloud. Again, logs don't show that the thread is running when the problem occur and again, nothing was changed here.
3 - Some other program (antivirus?) is affecting my exe. Couldn't investigate this hipotesis properly yet, but until now couldn't find any installed program that raised my attention.
My question is: What could be causing this issue? How can I investigate further? I know this may be a wide question but this is all information I could gather and I can't imagine many more places to look at.
Images
1 - In the image bellow the red-stroked area should be a TToolBar
2 - In this second image there are three areas, from the top to the bottom the first one should be a TToolBar, the second one should be the title of the child form and the third one should be a TwwDBGrid
3 - The third example shows on the top the erased area where should be a TEdit, just bellow it there's what should be a line on a TwwDBGrid and on the side we can see an erased scrollbar from the TwwDBGrid
4 - This last example shows 5 erased areas: The title of the application, the main TToolBar, The title of the Form, a TButton and two TwwDBGrid
5 - This is an interesting example beacause beyond the erased components there are 4 TSpeedButtons that are not erased but they are without the images they have originally (the first red stroked areas). The other 3 red stroked areas are, in order, 2 TEdits, a TwwDBGrd and a TButton
Process Information
I got a screenshot by the momment the problem occurs. scgolr is my software.
There is really not enough detailed information to give you a definite answer. However, I can answer with some directions on your question:
How can I investigate further?
Because of what you have stated:
The program is in use by a few hundred customers
One (only) customer experiences the problem
First occurance of the problem was some weeks ago
the first thing to do, is get in contact with the customer, and get the information you say that you have asked for but not got. The questions that need to be answered are:
What has changed in the customers environment at the time the problem
started with respect to hardware, network, server, OS, other software
running in the PCs?
Has anything changed in the way your customer is using your software?
What do the customer have to do to get rid of the problem, once it occurs? Close the program? Restart the PC? Or maybe just minimize - restore the erroneous window?
With the above I do not suggest that the fault is with this one customer and their equipment or their way of using the software. It may just be that the combination at the site which is different from all your other customers, triggers the problem to show up.
Some specific things to check in your software and at the site when problem occurs and if the problem goes away with a minimize - restore of the application (which would suggest a painting interrupted problem:
Do you call Application.ProcessMessages at any time?
Does the background thread access same data as the GUI? If yes, are the data protection properly in place (locking, synchronisation).
Does the background thread access any GUI components without Synchronize?
Finally I suggest that you visit the customer onsite. You get much better and faster answers in a direct discussion.
Edit after process information received.
There is nothing alarming concerning GDI or User objects. But it is alarming when you say in the comments that you call Application.ProcessMessages in many places, obviously to 'fix' a non-responsive UI. For example, what happens if the user double clicks a button, but does it slowly enough that Windows detects it as to separate clicks? First click may start your long lasting procedure within which you call A.P. The second click is read from the message que which starts the same procedure. Now the second call to the procedure runs (with its own calls to A.P.) and eventually ends and execution returns to the first call. Depending on what you do in this procedure, you may well be messing up handles and device contexts etc. A strong recommendation said with a friendly intent: Get rid of those calls to A.P.
the problem is with the security plugin (Warsaw - Gas Tecnologia) bank's website that your client is accessing , update it and it will be solved , the problem happens in Brazil
As #SebastianZ and #AlekseyK pointed out you may experiment exaustin of some GDI resource (handles?).
If the system coukd be accessed some tools like Process explerer or process hacker could give you some hints. This utility may help too GDIView
I don't know if this may apply to your case, but sometimes database data corruption can lead to strange effect in running programs (i remember 'Data Bombs' causing out of memory exceptions ...
So if something cause a GDI allocation loop, the graphics of your app cauld be affected in 'strange' ways
We had to use CoFreeUnusedLibrariesEx for fixing a bug with heap not being cleared after using a MSXML library
Refer this link:
http://blogs.msdn.com/b/marcelolr/archive/2008/11/13/msxml-heaps-not-being-released.aspx
But this caused another issue with TTimers which takes while to show up and disappears when Delphi app is bounced and again shows after a while.
This app uses TTimers to schedule it's job like running a XML transform .
Here is the issue:
When TTimer.Enable is called it throws an error not enough Timer available.
I know this is a masked error and I would have to figure out how to get to the actual error.
This is a single threaded application with only one timer.
Here are the links I looked into
Most Common reason seems to be invalid windows handle
https://groups.google.com/forum/#!topic/borland.public.delphi.winapi/UrIskaFZggU
There are other threads that suggested that OS ran out of resources for TIMERS I'm not sure if that is relevant to me.
I'm Just trying to understand what is Interaction between CoFreeUnusedLibrariesEx and TTimers that it gradually some how robs it of resources and makes us bounce the app to get it working.
How do I go about solving this issue , I'm looking for some directions?
CoFreeUnusedLibrariesEx should not affect TTimers. But if loading and unloading a (buggy) dll leaks any user objects (this includes timers, window handles, ...) then I could imagine that you run out of user objects.
Use Windows Task Manager and configure it so it will show the "USER Objects" in the "Processes" tab. Then compare the number of user objects when you call CoFreeUnusedLibrariesEx and when you don't call CoFreeUnusedLibrariesEx.
I have an application written in Delphi 5, which runs fine on most (windows) computers.
However, occasionally the program begins to load (you can see it in task manager, uses about 2.5-3 MB of memory), but then stalls for a number of minutes, sometimes hours.
If you leave it long enough, the formshow event will eventually occur and the application window will pop up, but it seems like some other application or windows setting is preventing it from initially using all the memory it needs to run (approx. 35-40 MB).
Also, on some of my client's workstations, if they have MS Outlook running, they can close it and my application will pop up. Does anyone know what is going on here, and/or how to fix it?
Since nobody has given a better answer I'll take a stab at how to solve this:
There's something in your initialization that is locking it up somehow. Without seeing your code I do not know what it is so I'll only address how to go about finding it:
You need to log what you accomplish during startup. If you have any kind of screen showing I find the window title useful for this but it sounds like you don't--that means you need to write the log to a file. Let it get lost, kill the task and see where it got.
Note that this means you need to cleanly write your data despite an abnormal program termination. How to go about this:
A) Append, write your line, close.
B) Write your line, then flush the file handle.
C) Initially write your file to consist of a large number of blanks--ensure this is larger than the actual log will be. Write your line. In case of abnormal termination it will retain the original larger file size.
I would write a timestamp on every log item so you can see if it's just processing something too slowly.
If examining the log shows you where the problem is, fine. If, as usually happens, it's not enough you put a bunch more logging between the last item that did get logged and the next one that didn't--I've been known to log every line when hunting a cryptic problem that only happened on someone else's system.
If finding the line isn't enough to pinpoint the problem also dump the value of relevant variables.
Finally, if such intense scrutiny makes the bug go away start looking for an uninitialized variable. (While a memory stomp is also an option I doubt it's the culprit here.)
I am getting this AV message about 3 to 5 seconds after the applications close as expected:
Exception EAccessViolation in module rtl160.bpl at 00073225. Access violation at address 500A3225 in module 'rtl160.bpl'. Read of address 00000004.
These (20) applications are very similar in that they are IBX business applications. About half of them did not cause the AV to occur.
These applications were ported from Delphi-xe and they worked flawlessly for a long time. No changes were made to the projects in the port. Both 32 and 64 bit builds gave the same results.
Is this a bug in some library's finalization section freeing a resource or something?
I am using Delphi-XE2 Update 3.
Would appreciate the help.
Try using madExcept / EurekaLog etc. - they give you detailed stack trace on AV. This is not always a panacea, but can point you to the problem.
Access Violations are by their nature already very troublesome beasts since they deal with invalid pointers in memory. One that occurs a while after an application shuts down is even worse because that's when your app is in "cleanup" mode. You're could be dealing with something that went wrong much earlier in the application, but is only exposing itself at shutdown.
General Tips:
Try to always undo things in the reverse order you did them. E.g.
Create A, Create B ... Destroy B, Destroy A
Connect to Database, Open Dataset ... Close Dataset, Disconnect from Database
Even making sure you've done all the above before shutting down can help tremendously.
Any threads that are still running while your application is running can cause problems.
Preferably ensure all your child threads are properly terminated before final shutdown.
Refer back to Closing datasets above. Depending on what you're doing, some database components will create their own threads.
If you're using COM, try ensure ComObj is high up in the initialization sequence (I.e. place it as high as possible in your DPR).
Delphi finalizes units in the reverse order that they were initialized.
And you don't want ComObj to finalize before other things that are dependent on ComObj have also done so.
If you're using interface references, make sure you resolve circular reference issues.
Some of these problems can be tricky to find, but you can do the following:
Setup a source-code "sandbox" environment (you're going to chuck all your changes as soon as you've found the problem).
Figure out the simplest set of steps required to guarantee the error. (Start app and immediately shutdown would be ideal.)
Then you're going to comment-out delete wipe out chunks of code between tests and basically follow a divide and conquer approach to:
rip out code
test
if the problem persists, repeat. Else roll-back and rip out a different chunk of code.
eventually your code base will be small enough to pinpoint likely problems which can be tackled with targeted testing.
I've had this kind of access violation problem on occasion with old Delphi or C++Builder projects. Today I had it with C++Builder. At the time of the crash, by looking in the Debug -> Call Stack window, I can see that it's happening inside a call to fflush, called by __exit_streams and _exit.
I'm not sure what is causing it, since it's so deep in the Borland library code, but it seems to come and go at random when the code changes. And it seems to be more common with multi-form applications.
This time the error went away when I just added a new button on the main form. A button which is just there, has no event handlers and does not do anything. I think that any random change to the code, classes, variables etc rearranges the memory layout when you relink the application, and that either triggers or untriggers the error.
For now, I just leave the new button on the form, set it to "not visible" so that there's no visible change. As it seems to work, it's good enough solution for me at this time.
This has happened to me on more than one occasion and has led to many lost hours chasing a ghost. As typical, when I am debugging some really difficult timing-related code I start adding tons of OutputDebugString() calls, so I can get a good picture of the sequence of related operations. The problem is, the Delphi 6 IDE seems to be able to only handle that situation for so long. I'll use a concrete example I just went through to avoid generalities (as much as possible).
I spent several days debugging my inter-thread semaphore locking code along with my DirectShow timestamp calculation code that was causing some deeply frustrating problems. After having eliminated every bug I could think of, I still was having a problem with Skype, which my application sends audio to.
After about 10 seconds the delay between my talking and hearing my voice come out of Skype on the second PC that I was using for testing, the far end of the call, started to grow. At around 20 - 30 seconds the delay started to grow exponentially and at that point triggered code I have that checks to see if a critical section was being held too long.
Fortunately it wasn't too late at night and having been through this before, I decided to stop relentlessly tracing and turned off the majority of the OutputDebugString(). Thankfully I had most of them wrapped in a conditional compiler define so it was easy to do. The instant I did this the problems went away, and it turned out my code was working fine.
So it looks like the Delphi 6 IDE starts to really bog down when the amount of OutputDebugstring() traffic is above some threshold. Perhaps it's just the task of adding strings to the Event Log debugger pane, which holds all the OutputDebugString() reports. I don't know, but I have seen similar problems in my applications when a TMemo or similar control starts to contain too many strings.
What have those of you out there done to prevent this? Is there a way of clearing the Event Log via some method call or at least a way of limiting its size? Also, what techniques do you use via conditional defines, IDE plug-ins, or whatever, to cope with this situation?
A similar problem happened to me before with Delphi 2007. Disable event viewing in the IDE and instead use DebugView from Sysinternals.
I hardly ever use OutputDebugString. I find it hard to analyze the output in the IDE and it takes extra effort to keep several sets of multiple runs.
I really prefer a good logging component suite (CodeSite, SmartInspect) and usually log to various files. Standard files for example are "General", "Debug" (standard debug info that I want to collect from a client installation as well), "Configuration", "Services", "Clients". These are all set up to "overflow" to a set of numbered files, which allows you to keep the logs of several runs by simply allowing more numbered files. Comparing log info from different runs becomes a whole lot easier that way.
In the situation you describe I would add debug statements that log to a separate logfile. For example "Trace". The code to make "Trace" available is between conditional defines. That makes turning it on pretty simple.
To avoid leaving in these extra debug statements, I tend to make the changes to turn on the "Trace" log without checking it out from source control. That way, the compiler of the build server will throw out "identifier not defined" errors on any statements unintentionally left in. If I want to keep these extra statements I either change them to go to the "Debug" log, or put them between conditional defines.
The first thing I would do is make certain that the problem is what you think it is. It has been a long time since I've used Delphi, so I'm not sure about the IDE limitations, but I'm a bit skeptical that the event log will start bogging down exponentially over time with the same number of debug strings being written in a period of 20-30 seconds. It seems more likely that the number of debug strings being written is increasing over time for some reason, which could indicate a bug in your application control flow that is just not as obvious with the logging disabled.
To be sure I would try writing a simple application that just runs in a loop writing out debug strings in chunks of 100 or so, and start recording the time it takes for each chunk, and see if the time starts to increase as significantly over a 20-30 second timespan.
If you do verify that this is the problem - or even if it's not - then I would recommend using some type of logging library instead. OutputDebugString really loses it's effectiveness when you use it for massive log dumps like that. Even if you do find a way to reset or limit the output window, you'd be losing all of that logging data.
IDE Fix Pack has an optimisation to improve performance of OutputDebugString
The IDE’s Debug Log View also got an optimization. The debugger now
updates the Log View only when the IDE is idle. This allows the IDE to
stay responsive when hundreds of OutputDebugString messages or other
debug messages are written to the Debug Log View.
Note that this only runs on Delphi 2007 and above.