I've inherited a big #$$ Delphi Application that by itself are so full of problems, that I'm not sure how I will be able to track down a problem that I'm currently getting.
The application crashes and terminates abnormally, and I'm unable to establish a pattern. I've added madExcept, and that helped me a lot getting some other problems, but when the application dies, not even madExcept is able to stop it so that I can get an exception report. I've downloaded Eurekalog as well to see if this can help me, but no luck.
Does anybody have a solution on what I can try. Any tool that will be able to help me out in tracking down the problem. SmartInspect does have some kind of solution, but I will have to change a lot of code for it to work, and unfortunately the main unit of this application sit with over 53k lines of code. (Just nasty).
Any help on what I can use to track this error will be appreciated. I need to but this "tracker" on a user machine because this application is running live at almost 2000 users.
If anybody also know why a random RICHEDIT20.DLL access violation will pop up now and again, it will be very usefull, because why it happens just baffles me completely.
Thanks
Jaques
Last time I had to do such a task (when Eureka and mad didn't suffice), I included CodeSite (Raize) logging, so you can log initially at key points in your application. In several cycles you'll need to add more to narrow down the problem. Advantage of CodeSite is that it enables you to log to another machine, so if your application crashes the system, the logging will be secured.
Related
I understand that my problem is multidimensional and I will probably get some independent advices, but thank you very much for all this and I am sorry if this is a newbie question.
I use a CEF (TChromium) browser in Delphi RAD 10. I regularly receive an error at the customer's production which I cannot replicate while working on the debugger. After displaying basic Win error information, the system (Win7) kills program, which ends with unsaved previous work. I checked each element of the code step by step, asked fellow programmers for analysis and it seems that this error applies only to the CEF browser. Every time, regardless of what i can manage to note in the program's work, the error is the same:
Problem signature:
Problem Event Name: APPCRASH
Application Name: MyApp.exe
Application version: 0.0.0.0
Application timestamp: 5e36d888
Module name with error: libcef.dll
Module version with error: 3.2454.1344.0
Module timestamp with error: 562d8f27
Exception Code: c0000005
Exception Offset: 001dea9b
OS Version: 6.1.7601.2.1.0.256.1
Locale ID: 1045
Additional information 1: 0a9e
Additional information 2: 0a9e372d3b4ad19135b953a78882e789
Additional information 3: 0a9e
Additional information 4: 0a9e372d3b4ad19135b953a78882e789
Yes, I guess the problem may arise from a large number of different things, but I assume that since this only happens when using the browser (otherwise the program works perfectly), and every time the same problem is displayed, it may be a TChromium component.
Unfortunately, I was not able to understand what exactly causes this problem (libcef.dll c0000005 / 001dea9b exeption) and all threads found on various forums are terminated and/or unresolved.
I tried to program each page load by displaying a larger message instead of closing the program:
procedure LoadUrl(url: String);
begin
try
Form1.Chromium1.Load(url);
except
on E : Exception do
ShowMessage('CEF: '+E.ClassName+' error raised, with message : '+E.Message);
end;
end;
But when working on the debugger I do not get any error (again), and on production system just kills the application without any error message.
At first glance, I think I need an explanation:
TChromium component actually changes only when i call it with "Load (url)", so do I understand correctly that this place in code is where I should focus?
Can I program errors/exceptions of external libraries this way? Or maybe there is some other way to use them safely, so that the error will not be the reason for killing the app, but will be controlled on production?
Will this mentioned procedure calling TChromium component give me more information than the system that kills my app? (Of course, if this is error place, because it seems to be the surest shot)
I use the EurekaLog7 tool - but I can't understand how I should use it to track the browser library errors trace and where to call it, or even how to take it in code. Actually, I have absolutely no idea where to start using it on external library, I will gladly accept some documentation or a hint about what to read and where I will find an understandable example.
Thank you in advance and I apologize if this is too easy or a problem is stupid. Of course, I also know that since I do not provide the full code, it will be difficult to analyze the problem, but I want to learn such error analysis myself, so maybe you will forgive me. :)
~~Additional info
program got x32 structure, is runned in win 7 xs64;
program is a simple crawler whose task is to save selected elements of searched pages to a text file form;
optional: it is enough for me, if this error manages to trigger my own shutdown procedure allowing to save the results simply, the app can be killed after because the sheduler will resurrect it;
Thanks to #J... i think i figured it out and here's solution.
The indicated error results only from the "work" of the libcef.dll library which (unfortunately) is probably no longer supported. Although this is not a problem solved anywhere - most indications of this error in various forums appear when there is a problem of referring to the wrong address in memory, and then following this lead is basically about out of memory error in various versions.
The libcef library has some fatal way of allocating memory and in itself leads to constant memory leaks. These leaks and misallocation quickly lead to the use of virtually all available memory... and it's easy to get a similar problem. First of all, adding a directive
{$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE}
after
uses WinApi.Windows
to the dpr form solves the problem for a while, because the program has more memory available and it takes longer to exhaust resources. This, of course, does not end the matter, because the suspension of the program will simply take longer.
But! It is enough when every few dozen scans (crawls) one will relocate memory for the form, which will automatically allocate memory, kills leaks - also in terms of the libcef library. I found and used such code that is called from time to time. It was enough for me to do what every 20 turns of the browser, but it may look different for everyone. You need to try trigger.
procedure TrimAppMemorySize;
var
MainHandle : THandle;
begin
try
MainHandle := OpenProcess(PROCESS_ALL_ACCESS, false, GetCurrentProcessID) ;
SetProcessWorkingSetSize(MainHandle, $FFFFFFFF, $FFFFFFFF) ;
CloseHandle(MainHandle) ;
except
on E : Exception do
// inform me about problem
end;
Application.ProcessMessages;
end;
This is probably not the best solution, but it makes the unstable form with CEF3 stable and ready to work. For several days, working on several copies of the same form, no error appeared at all, so it can be a good solution maybe for everyone who has memory leaks with CEF3, has unknown errors similar to mine, or out of memory error caused by CEF3.
However - my question was how to debug libcef.dll and how to get more information about bugs of CEF3 library - and J... has finished the topic, thank you very much.
As for SalvadorDíazFau's offer - I really appreciate your involvement in the project, people like you are the foundation of our community; however, while CEF3 is still working it is less work for me than discovering new elements and possibilities of CEF4 which would require me to rebuild the form. There will be time for that. :)
Thank You Guys! Case closed already!
I am working on a Delphi application using the TService functionalities.
It is a large project that was started by someone else.
The application uses several separate threads for processing, communicating with clients, database access, etc. The application’s main job is to poll regularly (every 2-300ms) certain devices and, a couple times a day, execute specific actions.
Now somewhere there is an unhandled exception for which I cannot seem to find the cause:
According to the debugger, the faulting method is System.Classes.StdWndProc.
This also seems to be confirmed by analyzing the crash dump file with WinDbg.
After numerous tests and debugging, I noticed that the crash happened on my dev computer every day at almost the same time.
I looked at the windows log and found this event:
This, coupled with the fact that the stack trace from Delphi indicates that a message with ID=26 (WM_WININICHANGE) was processed, made me believe that there might be something wrong with my usage of FormatDateTime() or DateTimeToStr() when regional settings are reloaded.
I checked every call and made sure to be using the thread-safe overload with a local instance of TFormatSettings.
However today the service crashed again.
A few points that I think are worth mentioning:
The application is also installed on a Windows 2008 server and has
been running OK for over a month.
On 2012r2 I tried forcing DEP off, but it didn’t change anything.
The service’s OnExecute() method is not implemented. I create a base thread in TService. ServiceStart() which then in turn creates the main data module and all the other threads.
The service is not marked as interactive and is executed with the Local System account.
All data modules are created with AOwner=nil.
With a special parameter, the application can be started a normal windowed application with a main form (which is created only in this case). The exception does not seem to happen when running in GUI mode.
Almost all threads have a message pump and use PostThreadMessage() to exchange information. There are no window handle allocations anywhere.
I have checked the whole project and there are no timers or message dialogs or other graphical components anywhere.
I have activated range as well as overflow checking and found no issues.
I am a loss for what to do here. I have checked and re-checked the code several times without finding anything that could explain the error.
Looking online I found several reports that seem to be pertinent to my situation, but none that actually explains what is going on:
https://answers.microsoft.com/en-us/windows/forum/windows_10-other_settings/windows-10-group-policy-application-hang/72016ea4-ba89-4770-b1de-6ddf14b0a51f
https://www.experts-exchange.com/questions/20720591/Prevent-regional-settings-from-changing-inside-a-TService-class.html
https://forums.embarcadero.com/thread.jspa?messageID=832265
Before taking everything apart I would like to know if anyone experienced anything similar or has any tips.
Thanks
Edit: looking at the call stack and the first method executed I am thinking of the TApplication instance that is used in TService.
LPARAM is always 648680.
LPARAM is a pointer to a string (you can cast it to PChar).
You could try to catch WM_SETTINGS_CHANGE and log anytime it's processing.
I think you could use TApplicationEvents component. The component has OnSettingsChange event. The event provides a setting area name (section name) and a flag that points to changed parameter.
Have a look at doc-wiki.
There is a full list of possible parameters.
I did not test it, but maybe you can use Abort procedure in OnSettingsChange event handler to stop the message distribution. Of course it's not a solution, but it may work.
I am getting this AV message about 3 to 5 seconds after the applications close as expected:
Exception EAccessViolation in module rtl160.bpl at 00073225. Access violation at address 500A3225 in module 'rtl160.bpl'. Read of address 00000004.
These (20) applications are very similar in that they are IBX business applications. About half of them did not cause the AV to occur.
These applications were ported from Delphi-xe and they worked flawlessly for a long time. No changes were made to the projects in the port. Both 32 and 64 bit builds gave the same results.
Is this a bug in some library's finalization section freeing a resource or something?
I am using Delphi-XE2 Update 3.
Would appreciate the help.
Try using madExcept / EurekaLog etc. - they give you detailed stack trace on AV. This is not always a panacea, but can point you to the problem.
Access Violations are by their nature already very troublesome beasts since they deal with invalid pointers in memory. One that occurs a while after an application shuts down is even worse because that's when your app is in "cleanup" mode. You're could be dealing with something that went wrong much earlier in the application, but is only exposing itself at shutdown.
General Tips:
Try to always undo things in the reverse order you did them. E.g.
Create A, Create B ... Destroy B, Destroy A
Connect to Database, Open Dataset ... Close Dataset, Disconnect from Database
Even making sure you've done all the above before shutting down can help tremendously.
Any threads that are still running while your application is running can cause problems.
Preferably ensure all your child threads are properly terminated before final shutdown.
Refer back to Closing datasets above. Depending on what you're doing, some database components will create their own threads.
If you're using COM, try ensure ComObj is high up in the initialization sequence (I.e. place it as high as possible in your DPR).
Delphi finalizes units in the reverse order that they were initialized.
And you don't want ComObj to finalize before other things that are dependent on ComObj have also done so.
If you're using interface references, make sure you resolve circular reference issues.
Some of these problems can be tricky to find, but you can do the following:
Setup a source-code "sandbox" environment (you're going to chuck all your changes as soon as you've found the problem).
Figure out the simplest set of steps required to guarantee the error. (Start app and immediately shutdown would be ideal.)
Then you're going to comment-out delete wipe out chunks of code between tests and basically follow a divide and conquer approach to:
rip out code
test
if the problem persists, repeat. Else roll-back and rip out a different chunk of code.
eventually your code base will be small enough to pinpoint likely problems which can be tackled with targeted testing.
I've had this kind of access violation problem on occasion with old Delphi or C++Builder projects. Today I had it with C++Builder. At the time of the crash, by looking in the Debug -> Call Stack window, I can see that it's happening inside a call to fflush, called by __exit_streams and _exit.
I'm not sure what is causing it, since it's so deep in the Borland library code, but it seems to come and go at random when the code changes. And it seems to be more common with multi-form applications.
This time the error went away when I just added a new button on the main form. A button which is just there, has no event handlers and does not do anything. I think that any random change to the code, classes, variables etc rearranges the memory layout when you relink the application, and that either triggers or untriggers the error.
For now, I just leave the new button on the form, set it to "not visible" so that there's no visible change. As it seems to work, it's good enough solution for me at this time.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions concerning problems with code you've written must describe the specific problem — and include valid code to reproduce it — in the question itself. See SSCCE.org for guidance.
Closed 9 years ago.
Improve this question
I have a piece of code that is trying to write to disk many files in one second. However, it fails wince I have installed Kaspersky Anrivirus 2011.
Stream:= TFileStream.Create(sName, fmCreate);
The code totally worked with Kaspersky 2010 and also works with Kaspersky 2011 if I disable its scanners (it cannot be totally unloaded from memory - unless it is uninstalled). The code also works if (Kaspersky 2011 is running and) I write to disk slooooowly. So it obviously is not fast enough to handle my disk requests.
The error I get is EFCreateError ('Cannot create xxx file blablabla'). Error is random. Most of the files are written to disk. About 10% fail.
I have tried to get support but is impossible to find a real person at Kaspersky to speak with. Their so called 'support' is actually a FAQ data base. Of course it speaks about how to install the product and related stuff. There is nothing about programing-related issues. Any ideas?
PS: this has repercussions for the entire Delphi community! All our customers will fail to use Delphi software if they are using KIS 2011 as antivirus. For the moment I recommend to my users to disable their antivirus but I need a real solution.
It will be nice if a person with KIS 2011 can confirm the problem. Just create a tiny program that write 200 small files to disk using TFileStream.
UPDATE:
The problem appears ONLY when the file does not exist and it is created (created as opposed to overwritten).
Similar report: https://forums.embarcadero.com/thread.jspa?threadID=32751&tstart=15
Similar report: http://forum.kaspersky.com/index.php?showtopic=120561
A possible solution that popped in my mind is to detect if KIS is running and if it is, to put a delay after each writing to disk. Or at leat, let the user know there may be problems. Anybody knows how to detect if a service is running?
I added a delay of 650ms (after each file creation) and the bug is still there). So is not about how fast you write to disk but about how many files you write.
Just uninstalled KIS 2011. The problem does not appear anymore.
Just reinstalled the good old KIS 2010. The bug is still there but it appear rarely (about every 300 files instead of about 30 as in KIS 2011).
The problem was confirmed on a second computer.
NEWS: The crash appears in TFileStream.Create however it may be caused by a function called earlier: TestWriteAccess. If I disable this function, the TFileStream.Create doesn't fail anymore. Well, this doesn't change things too much. No matter which line of code generates the error, the program still fails (randomly) to write files to disk while Kaspersky is running.
Still waiting a response from a real person from Kaspersky...
More automated responses received from Kaspersky support (I sent emails to support in several countries). All pointing to a FAQ database.
I change my status from Kaspersky fan (and customer) to Kaspersky hater because I finally receive an answer from a real person from Kaspersky support and it was plain and simple obnoxious.
To test the code, try to use the code in a loop, to create 1000 files. The program creates a bunch of files (random number) then it fails at StreamFile:= TFileStream.Create.
Update: The issue can be fixed by entering a small delay after creating each file.
https://docs.google.com/forms/d/1H3_O1z1iEqfh9ZT9u3B0R1tGEj-Hc9o7rAE0LKPr33Y
2013 Update
Starting with this afternoon (after an update) KIS conflicts with Delphi.
Every time I compile a project KIS spikes to 100% CPU utilization. I will have to uninstall it.
2017 Update
All false positive alarms disappeared magically for all my Delphi programs starting with 2017. It seems that it was enough for a program like Kaspersky remove Delphi-generated executables from its virus list; all other smaller antivirus programs followed.
Delphi 7, Win 7 (32), KIS 2011
You need to instruct your users, i.e. Kaspersky's customers, that Kaspersky is interfering with the operation of your software, and that THEY should report it. Express your frustration that you, as a developer, don't have access to a real human being. This is the only way that the anti-malware companies will ever react - bad PR with their paying customers.
Kaspersky = pirate company? Maybe yes, maybe no. Maybe just yet another company with a bad product and nonexistent support. Their "support" consists in a FAQ database and an automatic email answering program. Phones are hooked to answering machines also. Their automated answer keep explaining me how to add my program in KIS "exception" database. I keep replying to those stupid emails that I cannot personally go to all my customers at home and put my program in the "exception" database and that it will be better if they will fix the bug.
When I finally got a non-automatic answer (the only one), the support guy fella is as rude as possible.
Possible solutions for Delphi programmers:
* Don't check if the user has write permission to a file (in order not to trigger Kaspersky bug)
* Check if the user has write permission. If the bug appears inform the user that Kaspersky creates problem and it should be temporary disable (while the program is running). Use a TRY EXCEPT block to do this.
Advice (based on my past experience):
Don't always blame your code if you ever received strange bug reports from your users when your program was trying to write to disk. Check also external factors (like existence of Kaspersky antivirus).
UPDATE:
I just applied for a refund. I will go for a chargeback if they won't refund the money (I strongly feel they won't).
Conclusion
When I posted this on StackOverflow I didn't realized the magnitude of the problem and I didn't realized it will deviate so much from initial course. Still I think it is well within the purpose of StackOverflow. We have all learned that sometimes the problems in out programs may not be caused by our faulty code and neither we can control the source of these problems (21 persons voted this question up - which means a lot of other people encountered issues with KIS).
We can just hope that poor designed programs that interacts with user's system at a very low level (such as KIS antivirus program) will be soon fixed so our sales won't suffer (much).
It is just frustrating when your program is labeled "buggy" and you can't do much about it!
Not an answer to solve your problem, but you should inform Kaspersky, probably they don't know there is a virus signature associated with a Delphi library.
And if your program isn't too complex, you might want to try Lazarus/FPC. It's not as good as Delphi, but I've been using it for several years now, and have got good results in Windows/MacOS/Linux.
i had similar problems with kaspersky 2011 when i was trying to add my prog to windows startup using d2010's new TFile.Copy() as well as raw api function:
CopyFile(PChar('C:\chellenger.exe'), PChar('C:\Documents and Settings\Omair\Start Menu\Programs\Startup\chellenger.exe'), False);
my solution was to put my delphi app in vb.net app as a resource, the vb.net app extracted it and put it to startup without false positives . Mixing two languages for your problem might solve your problem too(1 possible solution but a very ugly and nonprofessional solution i admit)
When you create file, any antivirus checks it. There is probably some kind of collision between your application and KAV. Have you tried to combine fmCreate with share modes. You can see in help for TFileStream.Create for available modes.
If the problem is just with kapersky, then just have your program detect if it is running. If so, scale back your file creation / writes to whatever passes their detection. Make sure you have some little status message somewhere that tells the user why things are slow. Incidentally, virus writers already know this which is why those heuristics simply don't work.
After doing that, contact Kapersky and work with them directly to get this resolved.
This gets past your immediate issue and will give you and kapersky time to figure out a long term solution.
Alternatively, you could simply shut kapersky down.. Just make sure you grab all of their watch dogs in the process.. But that tends to be a little more combative.
Creating a huge amount of files sounds like something that isn't necessarily A Good Thing, but you probably have your reasons :)
When you get the error code in Delphi, does KAV pop up any heuristic warnings, or is it completely silent? It wouldn't be weird to get a heuristic "omg, that app is doing something bad!" from creating a ton of new file, but if KAV is silent I'd say it's a bug.
Can you post a delphi executable with the tiniest amount of code that reproduces the bug? And a version that does the same step but only creates one file, it might be interesting to trace with SysInternals' ProcMon.
First, do you really need to test for write permissions by creating a file? Can't you just check the permission directly? I feel that creating a file for that purpose only is a lame way of doing it in any case.
Second, like noted above, it's likely that after you create and then delete a file, there is some intervention by Kaspersky's security mechanisms. Probably a driver tries to check the contents of the file you deleted, and keeps it alive for a while. Like this:
You create the file and open it, incrementing the refcount.
Kaspersky driver notices that and opens the file too. Even if you set share mode deny, as a driver it probably has the power to open it anyway (if Kaspersky could not circumvent sharing denials, any virus could have used the same trick to hide its data!).
You close the file and delete it. When you delete the file, the system just marks it "FILE_FLAG_DELETE_ON_CLOSE", but the file is still there until all the handles to it are closed.
Kaspersky continues to scan file, still haven't released the handle.
Therefore the file is still there.
You try to create a new file and the call fails because the old file is still not deleted.
The reason for all this mess is, of course, partly Kaspersky's checking mechanics, but they did nothing especially wrong here. Kaspersky needs to scan the file anyway, hardly anything can be done about that - it's antivirus, for crying out loud. On the other hand, checking permissions by creating and then deleting a file is (probably) very, very wrong. So I guess, you're the one at fault here.
I had the same problem. KIS made all kind of troubles. Until I reinstalled it. So, it was just a faulty installation.
Got a slightly odd one here.
We have an MVC site that is experiencing some instability. Once a week for the last couple of weeks the site has gone down. We have found that all we need to do to bring the site back up again is 'touch' the web config file (i.e. open it and save it, with no changes). This brings the site back to life immediately and keeps it alive for another week or so.
The site has custom error pages set up so we can't actually see the error that is being thrown, and there is nothing appearing in the IIS logs.
It looks like some kind of memory leak problem, however .NET garbage collector should manage this, right?
Any ideas?
Thanks,
Pat
You should have some kind of logging so that you can see what error you get. I usually use elmah. That will give you the full error message and stack trace. It's pretty hard to say what the problem is without knowing the error. But it can be a memory leak of some kind. Do you use a lot of unmanaged code? The garbage collector can't handle everything.
Touching the web.config forces recompilation and refreshes the app pool. My guess is something is knocking out your app pool, whatever it is will be in the event logs, i suggest you check there.