Can I use map2dbg with 64 bit Delphi executables? - delphi

I am currently using map2dbg to create a .dbg file from my Delphi .map files. This works beautifully for 32 bit executables. For 64 bit executables the call to map2dbg.exe appears to succeed, but the resulting .dbg file does not appear to be useful. When I view stack traces in Process Explorer, they have no symbol names.
Should I even expect map2dbg to work in 64 bit? And if not, is there an alternative that I can use?

I've made a small research and it seems that map2dbg can in fact be used for 64bit executables made in Delphi XE2. The only point is you should modify WORD in the produced DBG file at offset 4 from $8664 to $014C.
Yes, this looks like a nonsense, because this means to change Machine field in DBG header from AMD64 to X86, but this really results in a DBG file correctly loading in both WinDbg and Process Explorer.
I've made a patched version of map2dbg version 1.3, so it automatically writes $14c into DBG. Here is the archive: http://yadi.sk/d/kbVFCGyI2gQzM
UPDATE:
DBG files made with the patched version of map2dbg are accepted by both Process Explorer and WinDbg, and the symbols from these DBGs are correctly linked with the corresponding addresses in the executable, but wrong stack frames are displayed.
The reason is in DBGHELP library. As can be seen from its disassembly, it only loads the DBG files made for X86 or Alpha processors (Machine field values $14c and $184). But if we manually change the Machine field in a DBG file from AMD64 to X86, then DBGHELP will treat the executable as a 32-bit module (so PDATA segment from the executable won't be used during the stack unwind), and incorrect stack frames will be shown by the debuggers.
I've patched both x86 and x64 versions of DBGHELP installed with WinSDK for Win8. The patched versions allow for loading DBG files with AMD64 Machine field ($8664), so the stack frames as displayed as expected. These versions are available in this archive: http://yadi.sk/d/7ZDLv2ed2gRGo
So, we now have two different approaches to use the symbols from 64-bit executables compiled with Delphi XE2:
Simple way: use the patched map2dbg to produce "fake-x86" DBGs, which can be loaded into WinDbg and Process Explorer, so the symbol addresses will be shown, but the debuggers won't be able to display the stack frames.
"Hardcore" way: use the patched dbghelp.dll, with the support of AMD64 DBG files. With this version of DBGHELP, WinDbg and Process Explorer can unwind the stack frames.
ONE MORE UPDATE:
cv2pdb tool can now convert DBG files created with map2dbg into PDBs. Both 32-bit and 64-bit executables are supported.
Here's a compiled version of the latest sources of cv2pdb.

Unfortunately, *.dbg support is deprecated (note: not even used or loaded!) in newer versions of Microsoft products (windbg, process explorer, visual studio etc).
So even if it creates a valid .dbg file, it will never be used... :-(
My biggest wish is to be able to create a .pdb file! So if anyone can get the specs for it?!
(it is a closed MS format?)
Because, to be even worse, the newest Intel VTune/Threading profiler also does not use .dbg files anymore, so I REALLY WANT A DELPHI TO PDB CONVERTER! (sorry for shouting)
I have tried several things, but no success yet.
That's why I created my own stack viewer and minidump viewer, which uses Delphi debug symbols (.map, .jdbg etc):
http://code.google.com/p/asmprofiler/wiki/ProcessStackViewer
http://andremussche.blogspot.com/2011/03/minidump-reader-for-delphi.html
Note: I haven't tested my stuff on 64bit Delphi apps yet... So it probably won't work, but you can try it anyway...

Just for your information: I found a PDB writer
https://github.com/jbevain/cecil/blob/master/symbols/pdb/Mono.Cecil.Pdb/PdbWriter.cs
It's part of the Mono Cecil library (open source .net implementation).
I hope it can modified to read Delphi .map files too...
(not tested yet)

Just for your information, I made a Proof of Concept dll for dbghelp.dll, so
it can also read Delphi .map files.
It is some kind of proxy dll: it has the same exports of the real dll, but
they are all forwarded to the real/original dll. 3 symbol functions are
implemented with a Delphi (jclDebug.pas) lookup:
https://plus.google.com/u/0/110131086673878874356/posts/4rmyQM5kVW7
https://plus.google.com/u/0/110131086673878874356/posts/TSJRqFJR3WZ
Only 32bit for now. ProcesExplorer runs only in 64bit in a 64bit Windows, but
ProcesHacker also has a 32bit version. When I have some more time I can maybe
improve it further... or try it yourself in the mean time! In 64bit mode you
cannot use "ASM JMP PToProc" but something like "ASM JMP qword ptr [rel p]".

I made some modifications (actually commented the exceptions :-) ) to tds2pdb.
Now it also works for Delphi .tds files, both 32bit and 64bit!
See my G+ post:
https://plus.google.com/u/0/110131086673878874356/posts/eJBKC16e5f6
Note: only ProcesExlorer did not show the full stack of my 64bit test program, ProcesHacker and WinDbg show the full stack though.

Related

Inspect shows ??? in Rad Studio 10.3.2

Rad Studio 10.3.2, create a new Windows VCL app C++. Put a breakpoint at the Form1 constructor. If I inspect "this" and the debugger shows ???? for any variable.
I am having really a lot of problems with 10.3.2, not only this one (std::variant not work, CLANG stops, ...). I would say that it has a lot of bugs but now I would ask if somebody has a similar problem.
It is a known bug in 10.3.2 and is currently being fixed by Embarcadero.
The bug is tracked as RSP-25527 | RSP-21126 (currently these are marked as duplicates but a request has been made to separate them).
They report that you can work with 64 bit windows ... provided of course you are not trying to link to external libraries or COM objects that are only available as 32 bit.
I can confirm that I have tested a Win64 VCL project and I can confirm that I do have the variables visible in the Inspect Window. You may have to manually add the 64 bit Windows target.
(I am having some problems with my 64 bit debug setup at the moment - I am loading packages which include debug info (as per the Events window it states the library has debug info) but I am unable to step into some of them which I normally can do with Win32. I do have the variables visible to me though.)

How to make exe file version readable to all editions of windows system

I have 32 bit program written in c++ builder xe2, which have dynamicly linked bpl files.
My program update system is based on exe file version. But in some of clients witch windows 2008 serwer 32 bit update system failed, because program see 1.0.0.0 file version instead of 2.3.0.94. When I check properities of file in this windows it shows 1.0.0.0 also.
How to compile exe file to be sure that version will be readable to all editions of windows system?
Your project likely has multiple Build Configurations and you did not define version info in all of the configurations that you compile for. With the introduction of Build Configurations and Option Sets, defining version info has become harder to manage, because users expect to define version info once and have it automatically carry through the various configurations, but that is simply not the case. Users typically have to either duplicate their version info multiple times as needed, or use third-party tools or IDE addons to handle it for them. This is a known shortcoming in Embarcadero IDE versions of recent years, and there are countless discussions about it in the Embarcadero forums.

port code from XP to Win 7

I have developed a Delphi application (XE4) on a Windows XP machine.
When I copy all the project files to a Win 7 machine (also Delphi XE4) it will not compile.
The source has uses Vcl.Grids and the compiler complains it can't find vcl.grids.dcu.
Changing to uses grids works but I don't want to edit all the source.
I've checked the Embarcadero website for information on Namespaces but couldn't find anything useful.
I know it's possible to say uses vcl.grids under Win 7 so there must be some setting somewhere in the project that is preventing the resolution.
I've tried deleting the dproj files but that had no effect.
How do I get the source to compile with minimal changes?
The error has nothing to do with OS. It means your IDE/Projects's search paths are not configured correctly, or your project is missing references to the relevant packages, so double check that.
Also, you can use uses Grids in the code, and then make sure Vcl is listed in the Unit scope names field in the Project Options.
The information that you describe seems to be erroneous. The compiler is not affected by the operating system on which it runs. Running the same compiler on the same source code on a different operating system does not result in compiler errors.
Here are the reasonable explanations for your problem:
You are compiling the code on different versions of the compiler. Your error message matches what happens when you compile modern namespace aware code on XE or earlier.
Your are not compiling the same source code on both machines.
It is extremely hard to see beyond these two explanations.
Ok, red face time. Turns out I was running an earlier version of Delphi on the Win 7 machine. Delphi XE4 was installed along with an earlier version and I was invoking the earlier version.
Once I actually brought up XE4 on the Win 7 machine the issue vanished.
So I will don a hair shirt and crawl under my rock.
Thanks everyone who contributed.

Solving Delphi BPL Package problems where BPLs won't load but you've already recompiled (Windows VirtualStore filesystem issue)

My general question is how do you troubleshoot "My BPL won't load due to a dependency that just won't go away, no matter how much I clean up and recompile". Update You may think you have a clean recompiled system, but thanks to the inverse-miracle that is Windows and its file system virtualization mis-features, you haven't.
When I try to load my designtime package (in this case named dclFsTee.bpl) into my Delphi IDE (it's the fast report 4 teechart wrapper component package), it complains:
The program can't start because tee7100.bpl is missing from your computer. Try reinstalling ...
That tee7100.bpl is not referenced on any DCP or DCU file on my system THAT I KNOW OF. But clearly, something is wrong, and I can't find the problem.
All Delphi users face a hundred "won't compile or won't load" problems with BPLs. The universal refrain when asked what to do is to clean up your computer.
However, I've now spent hours cleaning up my computer, and while everything compiles file, clearly there must be something out of date hiding somewhere, because the resulting BPL file that I'm trying to load still wants to load a version of a TeeChart BPL that I removed from this system days ago, along with every trace I could find.
The traces of TeeChart stuff in Delphi 2007 that I removed include everything in the $(BDS)\Lib and $(BDS)\Lib\debug folder, and all DCP and BPL folders on the system. Also every TeeChart-unit-named dcu file is gone.
Once you've gotten to the end of the road, what do you try next? (Format the hard drive, buy new computer.) Seriously. I think I'm a smart guy, but I have a 1 tb hard drive, a library path that runs to 80+ folders, and a source code repository that seems to be well organized, but clearly something is hiding where I can't find it.
I have TeeChart Standard 2012, with full source code, and as far as I know, my development machine no longer contains any old TeeChart BPLs or DCP files from the "tee chart tee7100.bpl" version that ships with delphi.
I have run the "recompile.exe" wizard that comes with teechart, which appears to just run MSBuild and build the packages, after writing a {$DEFINE x} declaration to the tee.inc files (there are two of them in the source distribution).
However, somehow, silently it seems like one of the implicit imports into one of the packages is drawing in some stale file which has not been rebuilt, and which therefore tries to load the tee7100.bpl. The new bpl name is tee911.bpl.
Rather than ask the pretty-specific-to-fastreport question, I'm only mentioning it as a specific instance of a general world of hurt that I have faced dozens of times while developing in Delphi.
I'm only giving the fast-report details so you can see that this is in fact a specific instance of a general problem that one faces sometimes inside Delphi IDE when dealing with a component source code or package, or set of packages, with dependencies. Cleaning up your computer so that your code even builds can be tricky.
So here is my Delphi package-to-package-dependency-resolution question:
What is the most effective way to find or trace implicit-load-of-some-no-longer-wanted BPL-problems so that my code (which builds and compiles just fine!) will actually load into the Delphi IDE. The BPL file that results from running Recompile seems to be linking properly to the right DCP files, and no old/stale DCP or DCU files are present. The new DCP file name is tee911.dcp, for instance.
Can you get somehow, any idea of what package is actually stale, and what is being read and linked and imported statically when the .bpl links? (I'm thinking maybe like a special MAP-like file for BPL files?)
Update After many hours of fighting with this, and using every trick I know, I realized I hadn't checked for some VirtualStore related issues caused by file virtualization in Windows 7. That means that Windows 7 lies to the programs that run on top of it. It gives you another version of the file, that isn't the one you want. This can be deadly in several ways; One; You recompile a BPL but that's not the one that loads. The BPL that was killing me was in the SysWow64 folder that was part of the VirtualStore. Note that the virtualstore basically makes phantom files appear that are only there if you're a certain "low privelege" program, which Delphi 2007 on Win7/64 bit, apparently is. To remove BPL files in your SysWow64 VIRTUALSTORE folder for your current user account:
del %HOMEPATH%\AppData\Local\VirtualStore\Windows\SysWow64\*.bpl
... Some days I just hate Windows architecture. Anyways, I'm not going to put the above as the answer, because I'd like to know if anyone has a better way or any tip or suggestion that might help next time.
Okay nobody else answered so I'll put this here to be helpful for future people:
-- Remember Windows VirtualStore when cleaning up broken systems which have old versions of DLLs on them including TeeChart, FastReport, Indy and so on which tend to be involved in messes because they can exist both as "out of box packages that ship with delphi" as well as frequently installed as upgraded versions if you purchased and installed them from the vendors directly, or third, you may have your own compiled copy in your company's mega-component-pack-directory.
-- When searching for duplicate or out of date BPLs, doing a file search in windows doesn't look in the virtualstores, you'll have to locate and zap the whole virtualstore area for your process or user, or program, manually.
The second level of this issue is this:
The dependency graph for FastReports is complex:
It depends on Indy and you might have your own version of Indy, and Delphi itself has one, and other things on your hard drive might have their own copy of Indy.
It supports various editions of TeeChart, including the binaries that come with Delphi, and perhaps the Standard version or other purchased version of TeeChart that you might have bought from Steema.
It uses a precompiled header include file to do the compilation and not just ONE but TWO different copies of an identically named include (.inc) file.
When you use their own compiler tool (recompile FastReport) it works pretty reliably but isn't the best for when you want to build everything in your project from a single build script, thus the source of my problem.
The key is to learn everything there is to know about the dependencies of all the components in your giant pile of packages, and to organize your system cleanly so that you don't have old stuff (like Indy and TeeChart bpls, dcp, or dcu files) lying around. Cleaning that up is quite a complex job if you don't know what you're doing.
A utility to really remove all traces of the version of Indy and TeeChart that ship with your system, and the "Embarcadero edition" of FastReports is key to getting this situation resolved. A general tip is that "if a version of X ships with Delphi and you are going to install a new version, prepare to suffer until your system is really cleaned up".
A really amazing technique to avoid all this crap is to just not install Indy, FastReport or TeeChart (uncheck them or skip them) during your initial Delphi IDE install, then install them yourself, one by one, from sources. Just because a version comes pre-installed in Delphi doesn't make that a good thing. (Update: You can no longer unselect Indy during install, it's part of the base Delphi product since at least Delphi XE8. A clean-up utility to remove the built-in Indy from Delphi's own lib dirs is necessary for anyone who builds their own.)
Another really amazing technique is to run the Installers for commercial components on a virtual machine, then just collect up the pascal source code and transfer that onto your clean development machine, and build it yourself. That way you can avoid the terrible things that happen when you've got BPLs and stuff scattered around your system, and even installed into C:\Windows\System32 (on 32 bit systems) and C:\Windows\SysWow64 (the equivalent path on 64 bit systems).
put that BPL (tee7100.bpl) under $(BDSCOMMONDIR)\Bpl
for XE: $(BDSCOMMONDIR)= "C:\Users\Public\Documents\RAD Studio\8.0"
for XE5: $(BDSCOMMONDIR)= "C:\Users\Public\Documents\RAD Studio\12.0"
The other issue that can cause this, is not having the folder where you've stored your .bpl files in your system path.
This happens because Delphi attempts to call the WinAPI function LoadLibrary with a file name, instead of an absolute path. So if Windows can't find the file, Delphi can't load it.
See this forum post for more information.
This seems to be an issue in Windows 7, though not in Windows 10.

Why can't my program find its DLLs on Vista 64?

I recently got a new laptop. Unfortunately, it came with Vista. It's been one big hassle getting it to work, and the comp has hardware components for which there are no XP drivers, so I can't "upgrade" to an OS that actually works. I've mostly gotten things working, but one particularly odd problem has me stumped.
I installed Delphi and tried to build a project. It compiled, but wouldn't run. "This application failed to start because sdl.dll was not found." Fair enough. So I grabbed SDL.dll and put it in the C:\windows\system32 folder. (Using Vista 64-bit Home Premium. This is a 32-bit dll, though, so I put it in the 32 folder instead of the 64 one.)
Hit Run again. Same problem. But why? That's where it goes, right? And C:\windows\system32 is in the system path. Anyone know why it can't link to the DLL?
(And yes, I know that I can work around the problem by putting the DLL in the same folder as the .exe. I'm currently doing that as a workaround. It's a bad idea in the long term, though, because I have a handful of different projects that all require SDL.)
That is not a Vista problem, but a 64 bit Windows problem: The name System32 is really confusing, but this is actually the folder where the systems (64 bit) DLLs reside.
So on any 64 bit version of Windows...
... all 64 bit system DLLs are located in C:\Windows\System32.
... all 32 bit system DLLs are located in C:\Windows\SysWOW64.
The name comes from Windows on Windows 64 (WOW64) which is the name of the translation layer allowing 32 bit applications to use the native 64 bit system resources.
Raymond Chen recently addressed the basic reason behind why 32-bit system directories are weird on 64-bit Windows. The first paragraph of the entry is really the key to understanding the reason behind segregated 32-bit directories:
On 64-bit Windows, 32-bit programs run
in an emulation layer...If a 32-bit
program tries to look at the system,
it will see a 32-bit system.
I think you'd have to have separate directories to keep these things all separate and working. The seemingly counter-intuitive name of SysWOW64 for the directory where the files reside is makes more sense when you consider that WOW64 means Windows On Windows 64-bit, which is what the emulater that's mentioned above is called.

Resources