We have been trying to solve this problem which is causing our program to crash. However, we haven't been able to reproduce the crash in house.
The call stack that is coming from the client's machine is on the link here:
Click to enlarge:
It doesn't seem to have any reference to any of the file in our project, so we're a bit lost as to where to look for a solution.
Could this be an environmental issue? The clients' that are getting this problem is using Windows 7 SP1 and Windows Server 2003. Sometimes, just prior to crash, the customer has been reporting that they have been getting 'A call to an OS function failed' error messages. Can this be related? Based on the call stack, can anyone make sense of what it is trying to do?
[Update] The call stack came from EurekaLog. Also I attach below the call stack from the 'A call to an OS function failed' error, that the customer is experiencing as well. This seems to be related to the AV error that the customer is getting but we are not sure. http://postimage.org/image/jku5dlnuf/
Based on the portion of the stack trace in your image, it's impossible to tell. The stack trace is mostly showing Windows API internal functions from the kernel DLLs.
Exceptions with an address of all zeros is a nil pointer (an object being used before it's created), but there's no way to tell where it's happening from that stack trace.
You should look at adding an exception handling product like MadExcept or EurekaLog to your application, which would give you a usable stack trace and more error information. Both are relatively inexpensive, especially when compared to the time spent trying to track down this type of error without them. (My own experience is with MadExcept, but I'm not affiliated with either of them.)
Related
I'm using Delphi 11. Sometimes an exception happens, and instead of Delphi going to the unit that threw the exception, it goes to this CPU screen, and I don't know what to do with it.
What can I do in such situations ?
This kind of screen typically means there is no code located at the memory address where the CPU tried to execute code from. Which most likely means that a class method was called on an object via an invalid but not nil pointer, or a function was called via a similarly invalid function pointer.
You won't be able to debug the location where the exception was raised from, since it is an invalid memory address to begin with. But you should still have access to the Call Stack trace of function calls that led up to the exception. So, just walk back through the Call Stack until you reenter valid code, and then debug from there to find the invalid pointer.
There might be better answers but I think you might look at EurekaLog or madExcept. Either works great. Never quite understood why this is not built-in to Delphi.
I understand that my problem is multidimensional and I will probably get some independent advices, but thank you very much for all this and I am sorry if this is a newbie question.
I use a CEF (TChromium) browser in Delphi RAD 10. I regularly receive an error at the customer's production which I cannot replicate while working on the debugger. After displaying basic Win error information, the system (Win7) kills program, which ends with unsaved previous work. I checked each element of the code step by step, asked fellow programmers for analysis and it seems that this error applies only to the CEF browser. Every time, regardless of what i can manage to note in the program's work, the error is the same:
Problem signature:
Problem Event Name: APPCRASH
Application Name: MyApp.exe
Application version: 0.0.0.0
Application timestamp: 5e36d888
Module name with error: libcef.dll
Module version with error: 3.2454.1344.0
Module timestamp with error: 562d8f27
Exception Code: c0000005
Exception Offset: 001dea9b
OS Version: 6.1.7601.2.1.0.256.1
Locale ID: 1045
Additional information 1: 0a9e
Additional information 2: 0a9e372d3b4ad19135b953a78882e789
Additional information 3: 0a9e
Additional information 4: 0a9e372d3b4ad19135b953a78882e789
Yes, I guess the problem may arise from a large number of different things, but I assume that since this only happens when using the browser (otherwise the program works perfectly), and every time the same problem is displayed, it may be a TChromium component.
Unfortunately, I was not able to understand what exactly causes this problem (libcef.dll c0000005 / 001dea9b exeption) and all threads found on various forums are terminated and/or unresolved.
I tried to program each page load by displaying a larger message instead of closing the program:
procedure LoadUrl(url: String);
begin
try
Form1.Chromium1.Load(url);
except
on E : Exception do
ShowMessage('CEF: '+E.ClassName+' error raised, with message : '+E.Message);
end;
end;
But when working on the debugger I do not get any error (again), and on production system just kills the application without any error message.
At first glance, I think I need an explanation:
TChromium component actually changes only when i call it with "Load (url)", so do I understand correctly that this place in code is where I should focus?
Can I program errors/exceptions of external libraries this way? Or maybe there is some other way to use them safely, so that the error will not be the reason for killing the app, but will be controlled on production?
Will this mentioned procedure calling TChromium component give me more information than the system that kills my app? (Of course, if this is error place, because it seems to be the surest shot)
I use the EurekaLog7 tool - but I can't understand how I should use it to track the browser library errors trace and where to call it, or even how to take it in code. Actually, I have absolutely no idea where to start using it on external library, I will gladly accept some documentation or a hint about what to read and where I will find an understandable example.
Thank you in advance and I apologize if this is too easy or a problem is stupid. Of course, I also know that since I do not provide the full code, it will be difficult to analyze the problem, but I want to learn such error analysis myself, so maybe you will forgive me. :)
~~Additional info
program got x32 structure, is runned in win 7 xs64;
program is a simple crawler whose task is to save selected elements of searched pages to a text file form;
optional: it is enough for me, if this error manages to trigger my own shutdown procedure allowing to save the results simply, the app can be killed after because the sheduler will resurrect it;
Thanks to #J... i think i figured it out and here's solution.
The indicated error results only from the "work" of the libcef.dll library which (unfortunately) is probably no longer supported. Although this is not a problem solved anywhere - most indications of this error in various forums appear when there is a problem of referring to the wrong address in memory, and then following this lead is basically about out of memory error in various versions.
The libcef library has some fatal way of allocating memory and in itself leads to constant memory leaks. These leaks and misallocation quickly lead to the use of virtually all available memory... and it's easy to get a similar problem. First of all, adding a directive
{$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE}
after
uses WinApi.Windows
to the dpr form solves the problem for a while, because the program has more memory available and it takes longer to exhaust resources. This, of course, does not end the matter, because the suspension of the program will simply take longer.
But! It is enough when every few dozen scans (crawls) one will relocate memory for the form, which will automatically allocate memory, kills leaks - also in terms of the libcef library. I found and used such code that is called from time to time. It was enough for me to do what every 20 turns of the browser, but it may look different for everyone. You need to try trigger.
procedure TrimAppMemorySize;
var
MainHandle : THandle;
begin
try
MainHandle := OpenProcess(PROCESS_ALL_ACCESS, false, GetCurrentProcessID) ;
SetProcessWorkingSetSize(MainHandle, $FFFFFFFF, $FFFFFFFF) ;
CloseHandle(MainHandle) ;
except
on E : Exception do
// inform me about problem
end;
Application.ProcessMessages;
end;
This is probably not the best solution, but it makes the unstable form with CEF3 stable and ready to work. For several days, working on several copies of the same form, no error appeared at all, so it can be a good solution maybe for everyone who has memory leaks with CEF3, has unknown errors similar to mine, or out of memory error caused by CEF3.
However - my question was how to debug libcef.dll and how to get more information about bugs of CEF3 library - and J... has finished the topic, thank you very much.
As for SalvadorDÃazFau's offer - I really appreciate your involvement in the project, people like you are the foundation of our community; however, while CEF3 is still working it is less work for me than discovering new elements and possibilities of CEF4 which would require me to rebuild the form. There will be time for that. :)
Thank You Guys! Case closed already!
I've inherited a big #$$ Delphi Application that by itself are so full of problems, that I'm not sure how I will be able to track down a problem that I'm currently getting.
The application crashes and terminates abnormally, and I'm unable to establish a pattern. I've added madExcept, and that helped me a lot getting some other problems, but when the application dies, not even madExcept is able to stop it so that I can get an exception report. I've downloaded Eurekalog as well to see if this can help me, but no luck.
Does anybody have a solution on what I can try. Any tool that will be able to help me out in tracking down the problem. SmartInspect does have some kind of solution, but I will have to change a lot of code for it to work, and unfortunately the main unit of this application sit with over 53k lines of code. (Just nasty).
Any help on what I can use to track this error will be appreciated. I need to but this "tracker" on a user machine because this application is running live at almost 2000 users.
If anybody also know why a random RICHEDIT20.DLL access violation will pop up now and again, it will be very usefull, because why it happens just baffles me completely.
Thanks
Jaques
Last time I had to do such a task (when Eureka and mad didn't suffice), I included CodeSite (Raize) logging, so you can log initially at key points in your application. In several cycles you'll need to add more to narrow down the problem. Advantage of CodeSite is that it enables you to log to another machine, so if your application crashes the system, the logging will be secured.
An application in the field is getting this message intermittently:
I am not able to reproduce this on my machine. I have also traced what I believe is the relevant code and can't find any access to uninitialized objects.
I've never had to deal with this kind of problem.
I did a build with madExcept and unfortunately the program does not crash once it is bundled.
Any opinions on madExcept vs EurekaLog for finding this kind of thing? I've never used FastMM. Would it be useful in his situation? (Delphi 2010) Any suggested flags to set in FastMM? Any other recommendations?
Note the very low address you are attempting to read. This sort of error almost certainly means you attempted to dereference a nil pointer even if you can't find one.
Given your description of the behavior I would suspect you've got a memory stomp going on--something is blasting a zero on top of the pointer to an object. When you change things you move things around and the stomp moves to someplace harmless.
Turn on both range checking and overflow checking.
Note the offending object must be at least 3C0 bytes in size--this should help narrow it down, most objects will be smaller than this.
What I have done in the past with such errors that only show in the field is put logging checkpoints in--a bunch of lines that display something in an out of the way place--a simple sequence of numbers is fine. Find out what number is showing when it crashes and you know which of you checkpoints was the last to execute. If that doesn't narrow it down enough you can repeat the process now that you've narrowed it down.
With a full map file you can identify the exact point in the code where this occurs. I hope you have a full map file for this image! Subtract $00401000 from the address at which the exception is raised ($007ADE8B in your case) and that corresponds to the values in the map file.
Having done that you know which object is nil and from there it is usually not too hard to work out what is going on.
One of the most common ways for this to occur is when a constructor raises an exception. When this occurs the destructor runs. If you access, in a destructor, a field that has not been initialised, and do anything other than call Free on it, then you will get an exception like this.
Looks like a memory overwrite where changing memory layout (your machine vs field machine or adding madExcept) makes the overwrite change something harmless.
FastMM is great at of making this kind of problems happen more consistently (and finding their source). Download the full version of FastMM, add it as the first unit of your project, and turn on FullDebugMode on its settings.
It might cause the problem to be reproduceable in your machine right away. If not, don't forget to deploy FastMM_FullDebugMode.dll with your application for testing. Keep madExcept on and let it embed the .map file for call stacks.
I have some really complicated legacy code I've been working on that crashes when collecting big chunks of data. I've been unable to find the exact reason for the crashes and am trying different ways to solve it or at least recover nicely. The last thing I did was enclose the crashing code in a
try
...
except
cleanup();
end;
just to make it behave. But the cleanup never gets done. Under what circumstances does an exception not get caught? This might be due to some memory overflow or something since the app is collecting quite a bit of data.
Oh and the exception I got before adding the try was "Access violation" (what else?) and the CPU window points to very low addresses. Any ideas or pointers would be much appreciated!
"Very low address" probably means that somebody tried to call a virtual method on an object that was not really there (i.e. was 'nil'). For example:
TStringList(nil).Clear;
The first part is very mysterious, though. I have no idea how that can happen.
I think you should try to catch that exception with madExcept. It has never failed me yet. (Disclaimer: I am not using D7.)
A trashed stack or a stack overflow can both cause irreparable harm to the structures on the stack that structured exception handling (SEH) in Windows uses to find the actual exception handlers.
If you have a buffer overflow in a buffer on the stack (e.g. a static array as a local variable but written beyond its end), and overwrite an exception record, then you can overwrite the "next" pointer, which points at the next exception record on the stack. If the pointer gets clobbered, there's nothing the OS can do to find the next exception handler and eventually reach your catch-all one.
Stack overflows are different: they can prevent calling functions at all, since every function call requires at least one dword of stack space for the return address.
you have a number of good answers. the wildest problems i've had to chase come from stack corruption issues like barry mentioned. i've seen stuff happen with the project's "Memory sizes" section on the linker page. i might be superstitious but it seemed like larger wasn't necessarily better. you might consider using the enhanced memory manager FastMM4--it's free & very helpful.
http://sourceforge.net/projects/fastmm/
i've used it with d7 and found some access to stale pointers and other evil things.
you may also wish to create a way to track valid objects and or instrument the code in other ways to have the code checking itself as it works.
when i'm seeing access to addresses like 0x00001000 or less, i think of access to a nil pointer. myStringList:=nil; myStringList.Clear;
when i'm seeing access to other addresses with much larger numbers, i think of stale pointers.
when things are strangely unstable & stack traces are proving to be nonsense and/or wildly varying, i know i have stack issues. one time it's in Controls.pas; next time it's in mmsys.pas, etc.
using the wrong calling convention to a DLL can really mess up your stack as well. this is because of the parameter passing/releasing when calling/returning from the DLL.
MadExcept will be helpful in finding the source of this, even if it shows nonsense...you'll win either way because you'll know where the problem is occurring or you'll know you have a stack issue.
is there any testing framework you can put on it to exercise it? i've found that to be very powerful because it makes it entirely repeatable.
i've fixed some pretty ugly problems this way.
I'll leave the reasons why the except might not work to Barry...
But I strongly suggest a simple strategy to narrow down the area where it happens.
Cut the big chunk in smaller parts surrounded by
try
OutputDebugString('entering part abc');
... // part abc code here
except
OutputDebugString('horror in part abc');
raise;
end;
...
try
OutputDebugString('entering in part xyz');
... // part xyz code here
except
OutputDebugString('horror in part xyz');
raise;
end;
and run your code with DebugView on the side... (works for apps without GUI as well like services).
You'll see which part is executed and if the exceptions are caught there.
I used to get this strange behabiour when calling some COM object that used a safecall calling convention. This object/method may raise an EOleException, not trapped by the usual try/except on the client code.
You should trap an EOleException and the handle it properly.
try
...
except
on E: EOleException do
...
end;
I don't know if it is the problem you are facing. But if it is, i recommend you to take a look at Implement error handling correctly, a very clarifiyng post about exception handling in delphi.
You can also enable your IDE Debug Options to stop on delhi exceptions e monitor the stack trace.
Is this perhaps a DLL or a COM object? If so, it is possible that the FPUExcpetion mask is being set by the host application to something different than Delphi is used to. An overflow, by default in Delphi produces an exception, but the FPUExcpetionmask can be set so that it doesn't, and the value is set to NAN. See the math.pas unit for more information on FPUExceptionmask
I've gotten exceptions in the initialization and finalization blocks of my code which madExcept doesn't even seem to even catch. This might occur if you're referencing external DLL's inside of that try block. I'm not certain of the reason.
Actually (and thanks to #Gung for informing me of the worthlessness of my ancient answer), I read this recently in the ancient O'Reilly Delphi Tome. You should put SysUtils as the first (or second after your non-standard memory manager unit) in your main form's DPR so that it's resident in memory with all it's Exception Catching goodness. Otherwise, if it's loaded from some other unit, it will be unloaded with that unit too and you can kiss built in exception handling goodbye.