I wanted to better understand how debug symbols work in relation with the PE.
As I understand the pdb file holds symbol information on its PE. But how exactly can a debugger take a PE and its pdb file and display code?
For instance:
What information can be stored in the pdb file that will link to the symbols of the PE
How does a debugger know to transform an optimized for loop in binary to a for loop in code?
Thanks!
Found the following very helpful resource:
http://www.wintellect.com/blogs/jrobbins/pdb-files-what-every-developer-must-know
Related
I'd like to add the map file as resource, then extract the file every time that execute the system, to ensure that the map file is always right, and make the life of support easier. In Delphi 7 I can compile the project, with the map file as resource, but the map is wrong, because when an exception is raised the call stack is wrong. In Delphi Seattle, I can't even compile, because it tries to add the resource before generate the map. I know that I can add the file in a post-compile thask, but is there any way to do it in compiling/building time?
Sorry if I'm not very specific, that's my first question here. Thanks.
UPDATE
After I read some answers, I did some research. I was already using JEDI exception unit in my project, but I did not know the JEDI Debug expert. It does exactly what I want and more.
JEDI Debug expert convert a .map file to .jdbg file, wich is an encrypted file of map (map is just a text file). Also, the jdbg file is about 12% smaller then the map. It also has the options to insert the jdbg into binary (exe). To do that, I enabled the options:
Project -> JCL Debug expert -> Generate .jdbg files -> Enabled for this project
Project -> JCL Debug expert -> Insert jdbg data into binary -> Enabled for this project
Project -> JCL Debug expert -> Delete map files after conversion -> Enabled for this project (if you want to delete the file, of course)
To use this tool outside the IDE, with Jenkins for example, I had to build the project available in JEDI\jcl\examples\windows\debug\tools\MakeJclDbg.dpr. After build, it will generate the exe file in the bin directory of jcl. How to use:
MakeJclDbg -J -E -M map_filename
J - Create .JDBG files
E - Insert debug data into executable files
M - Delete MAP file after conversion
Executable files must be in the same directory as the MAP files. This will create the jdbg file (based in the map file), insert into the exe and delete the map. With this (and with the exception unit of JEDI), when an exception is raised, It's available to me the stack trace, the versions of all dll's used by the system, operation system info, and more, and also send all this to an email.
I realised that Embarcadero also have jdbg files of theirs bpl, so I think they use JCL tool as well.
No, the map file is generated after the program output is linked. It is impossible to incorporate a, not-yet generated, map file as a resource into the project.
You're missing the point.
The MAP files are generated as a separate file to avoid increasing the size of your executable. Trying to embed that file back into the executable as a resource simply defeats the purpose.
You haven't mentioned what debug framework you're using. But there are other ways to provide debug information, and I suggest you refer to the documentation of the debug framework you using for the specifics. I'll just offer some general concepts applicable to most of the frameworks I've tried.
If you're happy with increased EXE size and want debug information included within your executable: Don't use the map file option. Simply enable the linking option to include debug information. (And ensure your debug framework will use it.)
Most debug frameworks recommend compiling with stack frames turned on. This is very important because it makes it easier for the debug framework to deduce the call stack.
Some debugging frameworks have a feature that allows guessing missing call stack information. If enabled, you will need to manually ignore any stack entries that don't actually make sense.
Of course, don't forget that any units compiled without debug information won't have debug information to include in the final executable.
Im trying to compile/build my project but i receive this error below:
[dcc32 Fatal Error] F2438 UCS-4 text encoding not supported. Convert to UCS-2 or UTF-8
The IDE do not show me what file/line i need to convert. I even dont know if is this i need to do really.
As i see, i need to convert the file for utf-8 correct?
This project as originally made with delphi 2010, now im trying to convert for XE6.
How i resolve this error?
Im using Delphi XE6, win 7 64 bit ultimate.
Open a source file, right-click on the Code Editor, and select "File Format | UTF8". Do that for all of your source files.
Or, narrow down the specific offending file. Try using SysInternals Process Monitor to see which file is being accessed at the time of the failure.
I had the same error(F2438) in a Delphi\XE5 project which had many .pas files. Error showed up when I had rebuilt all files in the project after deleting DCU's. I was able to uncover the one ".pas" file that somehow got saved as UTF-32LE format. The tool I found to locate the problem file was "File Encoding checker" which located the problem file in my project folder. URL for the tool is https://encodingchecker.codeplex.com/. I then had to convert back the problem "pas" file to be saved as UTF8\ascii text and on compiling the F2438 error was cleared up.
I have a map file created by delphi compiler and I need to use it to analyze DMP file.
I converted the map file to dbg file, using Map2dbg.
The problem is that the process has a different signeture then the symbols.
I tried to use chkmatch to match the signeture, but I'm getting:
Error: Debug information not found in the executable
I guess this is because it is a delphi process.
Does anyone know how to match the signatures?
I made a minidumpreader some time ago:
https://asmprofiler.googlecode.com/svn/trunk/MiniDumpReader/ViewMinidump.exe (old link)
https://github.com/andremussche/asmprofiler/tree/master/Releases/ViewMinidump.exe (new link)
It is written in Delphi with jclDebug.pas so it supports all delphi debug symbols.
And because of this it has line number support (which map2dbg or tds2pdb don't have).
Note: I haven't made map2dbg or tds2pdb myself, just hosting it so other can easily find it
The Question
There are plenty of manual ways to make WinDBG find mscordacwks.dll without a symbol store (putting the file in the path somewhere, putting it in the same folder as windbg.exe, putting it in my Framework\v folder, specifying the path in WinDBG using .cordll -lp c:\dacFolder, etc.), but they all only fix it for me. I need to fix it more generally for everyone who uses my symbol store.
The possible solutions I can imagine are:
WinDBG be made to check the symbol store using mscordacwks.dll's subfolder name instead of mscorwks.dll's folder name.
SymStore.exe be made to add mscordacwks.dll under mscorwks.dll's subfolder name so WinDBG finds it when it looks there.
Q: Are either of these things possible, or is there another way that I'm not thinking of to solve the problem?
The Background
When analyzing a .NET process, I encountered the (apparently common) problem that psscor2 (and sosex) could not find the appropriate mscordacwks.dll on my machine. The error in WinDBG is:
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.
There are plenty of SO questions on this and plenty of good answers, practically all of which ultimately reference Doug Stewart's outstanding blog post, What is mscordacwks.dll?.
Thanks to that, I got my situation all straightened out by obtaining the correct mscordacwks.dll and placing it here:
"C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545829a3000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll"
where I knew WinDBG would look because I had previously tried it with !sym noisy.
So I'm all set now, but I had to put it in that path physically rather than adding it to my symbol server through the normal symstore.exe mechanism. Since my symbol store is used by more than just me, I need to do it the right way for everyone else using the store.
And that's the problem. When I add using symstore.exe instead of going into the above path, it goes into:
"C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545CB1bd000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll"
The only difference being that the subfolder name is 4E1545CB1bd000 here instead of the 4E1545829a3000 that WinDBG is looking for.
The reason for this is that when adding a binary to the symbol store, symstore.exe uses the PE of the binary to get the image timestamp and the image size. In the case of this particular .dll, dumpbin.exe /headers mscordacwks.dll reveals these to be:
image timestamp: 0x4E1545CB (Thu Jul 07 01:36:11 2011)
image size: 0x1BD000
Hence, the subfolder name 4E1545CB1BD000.
What WinDBG is looking for, on the other hand, is a subfolder based on the image timestamp and image size of mscorwks.dll, not mscordacwks.dll, because the former is loaded into the process, not the latter. WinDBG can't know the timestamp and size of the DAC module because that module is not in the process dump.
As further verification of this explanation, dumpbin.exe /headers mscorwks.dll reveals:
image timestamp: 0x4E154582 (Thu Jul 07 01:34:58 2011)
image size: 0x9A3000
which you can see add up to subfolder name 4E1545829A3000.
Knowing this, now it makes a lot more sense why all these many versions of the mscordacwks.dll that people keep running into seem to be missing from Microsoft's symbol servers. I'm sure they're there, it's just that WinDBG and psscor2 can't find them because they're picking the wrong subfolder name. Why it even bothers searching the symbols path is beyond me since it's guaranteed never to find it!
So that's my challenge. Can I somehow force symstore.exe to add mscordacwks.dll using the PE info of mscorwks.dll? If not, am I missing something about WinDBG and psscor2, might there be a way for them to know the correct timestamp and size of mscordacwks.dll even though it's not loaded (and a way for them to actually use those instead of mscorwks.dll)?
Since no other solution has appeared and my workaround seems to handle everything nicely, I'm just going to keep on with that, and I would recommend anyone else who never wants to see the annoying Failed to load data access DLL, 0x80004005 error again do the same.
So to make this work for you (and everyone who uses your symbol store, so I really wish Microsoft would do this to save us all a lot of trouble) simply place the compressed DAC file (mscordacwks.dl_) by hand into the correct path within your local SYM store.
Here are the steps I follow to accomplish this:
In WinDBG do a !sym noisy
In WinDBG do a .cordll -ve -u -l
In WinDBG do another !CLRStack or other psscor2 command if necessary to force it to load symbols again
The symbol search logging will reveal the dll it’s looking for and where it’s looking in your symbol store by showing lines like this: C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545829a3000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll which indicates two things:
you need the 64-bit mscordacwks.dll of version 2.0.50727.4216; see https://stackoverflow.com/a/12024171/1910619 for ways to get it
it needs to go in a subfolder called 4E1545829a3000 under a folder called mscordacwks_AMD64_AMD64_2.0.50727.4216.dll in your symbol store
Once you obtain the file, rename it according to the name WinDBG is looking for, e.g. "mscordacwks_AMD64_AMD64_2.0.50727.4216.dll"
Manually compress this file using makecab.exe like this: makecab /D CompressionType=LZX /D CompressionMemory=21 mscordacwks_AMD64_AMD64_2.0.50727.4216.dll mscordacwks_AMD64_AMD64_2.0.50727.4216.dl_
Copy that compressed file to the expected place in the symbol store. (That you found in step 4 above, so C:\Symbols\mscordacwks_AMD64_AMD64_2.0.50727.4216.dll\4E1545829a3000\mscordacwks_AMD64_AMD64_2.0.50727.4216.dl_ in our running example here.)
You have to put the CLR dll and the associated mscordacwks dll in the same folder and register the CLR dll with symstore.
In that case symstore will add the clr and the mscordacwks dll on the symbol store.
More importantly, it will use the timestamp and file size of the clr dll to create the mscordacwks subfolder, so windbg can find the mscordacwks dll when debugging a dump.
The mscordacwks name must follow the mscordacwks_ARCH_ARCH_fileversion pattern, otherwise symstore won't add it to the symbol store.
I didn't find any documentation on that feature so it may be removed in the future.
Here is the command and the symstore output:
symstore.exe add /o /f 4.6.1076.00\clr.dll /t clr.dll /s \\mystore\microsoft
SYMSTORE MESSAGE: 0 alternate indexers registered
SYMSTORE MESSAGE: LastId.txt reported id 0
SYMSTORE MESSAGE: History.txt reported id 58228
SYMSTORE MESSAGE: Final id is 0000058228
SYMSTORE MESSAGE: Copying C:\Users\build.robot\symstore\4.6.1076.00\clr.dll to \\mystore\microsoft\clr.dll\56D79ED4990000\clr.dll [Force: T, Compress: F]
SYMSTORE MESSAGE: Copying 4.6.1076.00\mscordacwks_AMD64_AMD64_4.6.1076.00.dll to \\mystore\microsoft\mscordacwks_AMD64_AMD64_4.6.1076.00.dll\56D79ED4990000\mscordacwks_AMD64_AMD64_4.6.1076.00.dll [Force: T, Compress: F]
SYMSTORE: Number of files stored = 2
SYMSTORE: Number of errors = 0
SYMSTORE: Number of files ignored = 0
I have private pdb file and I have to convert it to a public one. Is there tool for it?
Use PDBCopy.
pdbcopy is part of the Debugging Tools for Windows which is available through the Windows SDK
You can create a stripped pdb file that doesn't contain; Type information, Line number information, Per-object file CodeView symbols such as those for functions, locals, and static data. See the /PDBSTRIPPED compiler option.
EDIT: There does appear to be a utility that is part of the DDK that can convert a full symbol file to a stripped symbol file it is called BinPlace the forum I found the information on suggested there might be problems with certain versions of the utility so be warned (forum article).