We are reversing Delphi internal structures, does anyone know of any good resource or other details about how Delphi executables are compiled and linked together and what is the layout of the various parts inside the final exe.
I am not looking for high level details like it has n sections.
I am looking for something (the following is 'made up') like section '.text' has 3 parts:
data (a)
pointers table (b)
code interleaved with data which can be identified by using mechanism (c)
etc,etc.
Ok, I am looking for references or knowledge about the structures. I already have the tools mentioned below, and a ton more to aid in reverse engineering this from scratch.
I should have mentioned that I know the resource layout as that is well documented. It's the internal compiled structures embedded in code and data sections I am after. (e.g. how RTTI info is compiled in, where is the mapping between event handlers and form resources, etc).
Start with using existing tools like IDR (Interactive Delphi Reconstructor) and IDA (the Interactive Disassembler) and the already mentioned PE Explorer.
I would also suggest compiling some simple executables and studying the disassembly of that.
The most identifiable parts of a Delphi or C++ Builder executable are the resources.
They will contain a resource section named RCDATA. This section contains the following:
A section named DVCLAL, which identifies the compiler's SKU, such as Personal, Professional or Enterprise.
A section named PACKAGEINFO, which contains a list of contained units and a flag for Delphi or C++ Builder
Individual resources for each DFM.
On some settings, compressors like UPX may hide these resources, so you won't be able to see them unless you decompress the executable.
It could be worthwhile to have a look at Jedi JCL. IIRC their traceback tools (jcldebug unit?) open binaries to recover debug info. It would at least teach the global structure of the file.
Some bits might be delphi version specific btw.
Downloading the evaluation version of PE-Explorer might be a good start too.
You can find all types in a Delphi module (exe or dll) because they are always prepended with a pointer to the structure itself. You can create a small app and scan the exe to see for yourself how to get that information. With type information comes good information where units are / where methods are etc etc.
Combine that information with the resources available (a good resource viewer can decode the binary resources in text for all forms /frames / datamodule ).
Now if you disassemble the application you can use the information from the resources / typeinfo to make sense of the assembly code found.
I have a quite old Windows application (1998) which is no longer maintained by the author and I don't have its source code. This application in one of its windows has a dropdown list widget where the user can choose one of the 4 predefined values (numbers). I need to add new predefined values there or change the widget to something that lets me introduce any value (some edit field or editable combobox).
Some other data about the application:
Probably written using some Borland tools, I guess it uses BDE and Paradox as its database.
There is a file with .mme extension in the directory. I unpacked it and it contains 5 files: .data, .rdata, .rsrc, .sdata, .text. I viewed them in a hex editor and they contain some text data but I don't know how to look for those predefined values. Since they are numbers it's probable that I will find just some other numbers.
I tried using RedEdit and XN Resource Editor but they show only the icon and version info.
I know my question is kind of vague, but if I don't find any solution to this I will have to rewrite the whole program, so I'm asking just in case there is a solution.
It might be possible to write another app that looks for this program, gets a handle to the window owning the drop-down box (if t's a native windows component), and then gets a handle to the drop-down box and use the Windows SendMessage API to manipulate the contents of the drop-down box.
http://msdn.microsoft.com/en-us/library/ms632595(v=VS.85).aspx
http://msdn.microsoft.com/en-us/library/ms644950(v=vs.85).aspx
It might require so much effort that it may be better to just rewrite the app!
If it is written with Borland tools, you can probably find out which one by searching the EXE for the string "Borland" - there should be a copyright string somewhere.
Once you know, find someone with Borland tools experience and have her find out if those numbers in the drop-down list happen to come out of a database table. If they do, try modifying that table, if not, rewrite the app :)
I release a single executable (.EXE) for a desktop program using Delphi 2009. I have no external DLLs or resources that I need for the program to run.
I use two components: LMD Innovative's ELPack and Sergey Tkachenko's TRichView that are compiled into my executable.
When I build my production version, using the "Release" build configuration, the executable file produced is 13,533 KB.
Prior to using Delphi 2009, I was using Delphi 4. The executable it produced was only 2,671 KB while incorporating the same two components and basically having the same code as my current version.
I do understand that Delphi 2009 is completely Unicode (which is the main reason why I upgraded), and being Unicode can cause up to a doubling of size. But this is about 5 times larger.
Is there a reason why my executable has to remain 5 times larger? Or are there some simple ways to cut down a significant chunk of the executable size?
Please note. Some people are answering with ways to compress the Delphi EXE. That is not what I am trying to do. I am trying to simply see why so much space is being used to remove what might not be necessary. If that is done, compression can still be done afterwards if so desired.
It really doesn't matter how big or small the executable is once it is installed. It is for downloading purposes and to minimize server load and download times that you want to compress it. I prefer to use Inno Setup and compress the program inside the install routine itself. Then when it is installed, it is expanded to full size. That both prevents possible detection as a virus and eliminates the extra startup time needed to uncompress the program in memory. Also I code sign both my executable and my install routine and some compression techniques are incompatible with that.
For more info about compressing, see the StackOverflow question: Delphi EXE compressor?
ldsandon asked me to provide exactly what options I'm using, so here they are:
(source: beholdgenealogy.com)
(source: beholdgenealogy.com)
When moving from Delphi 7 to Delphi 2010., our .exe's grew for example from 16 megs to 35 megs.
I asked a question similar to yours on the Embarcadero forum a few weeks ago. (link) In my OP, I listed a series of links on this subject that you might find helpful.
We tried using UPX to compress our .exe's. Letting it work for hours significantly reduced our .exe, but we probably won't use it in production for these reasons:
We have quite a few .exe's and don't want to wait 1/2-day on each build. (It's possible that we could find a non-brute force set of parameters to UPX that would reduce this...)
Although the size of the .exe is reduced, our shippable was not, because our installer (not surprisingly) is unable squeeze much more compression out of the already compressed file... whereas it was able to reduce the original 16 meg .exe down to 8 megs.
I've read some reports that at some time (rarely, but not never), UPX exe's triggered various anti-virus programs to report the application contained a virus. (I don't recall the date, site, or details of where I saw this, so it's a bit unfair of me to report it here.) But, we are so adverse to taking a risk of that even possibility happening, that UPX is off the table...
The link on the Embarcadero forum also includes a link to another SO thread on this topic.
I continue to be surprised and disappointed at the code bloat we found when moving to Delphi 2010. As Nick notes, 2X for Unicode is quite excessive.
However, the bloat is a relatively minor trade-off when moving to D2010, because, IMO, D2010 is such a terrific upgrade in so many other ways. But, it does mean that we'll probably have to move to shipping 2 CDs rather than one. I'm not looking forward to seeing the reaction to this from our organization...
Without seeing the actual settings that your "Release" build configuration uses explaining this increase in size requires a great deal of speculation.
Beyond some perhaps unlikely factors resulting in a vast increase in the amount of code being "dragged in" even though it isn't used, that magnitude of increase would most easily be explained by the inclusion of debug information.
I would check your compiler and linker settings for:
Debug Information (compiler setting)
TD32 info (linker)
Remote debug info (linker)
Compare these settings in your Delphi 2009 project with the equivalents in Delphi 4.
Factor out the expected 2X increase from Unicode and you end up with a 2.5X increase unaccounted for. This makes sense considering how many versions you've skipped. A lot's been added to the VCL and RTL since Delphi 4, and not all of it is stuff that can be easily smartlinked out, even if you never use it. Depending on how many units you're using, you could be hauling in quite a bit of extra baggage.
Allen Bauer and the compiler team added a new feature into D2010 to help reduce this, but apparently they're treading cautiously and didn't use it in as many places as they could have. Hopefully we'll see more cruft reduction in 2011 and subsequent releases.
I will add my few words.
Linker can remove unused procedures and functions only if it can follow the code hierarchy. The nightmare list for linker listed below:
Message-driven code, the sad news is that this code can't be removed whatsoever, that's why Delphi blank project size continues growing from version to version. Every new windows message (WM_TOUCH for example as long as I know introduced recently) creates procedure call hierarchy that can't be removed (even if you don't have plan to use Touch API at all). This is because every case WM_: fragment is something linker can't decide whether it will be used or not.
Code and data structures accessed from the begin end, initialization, finalization secions of the units. Here you have some control, remove unnecessary calls or object creation. Even if you create objects on demand and only free them in finalization section, make it carefully
Use "upx - compress or expand executable files" # http://upx.sourceforge.net
If you go to tools/configure tools, and set it up like this, you can compress the executable that you're working on easily via a menu item in the IDE.
Another way is to have a look to 'what unit increase the size ?'.
To do this, I use the JCL 'Project Analyser IDE', integrated in the IDE with the JCL/JVCL installation, it show you all the units with their respective size. You can export it in text file.
If you do it with the 2 environnements (D4 & D2009) you will have a lot of pertinent informations.
I've done some tests to see the difference between D2007 and D2010, because we are upgrading to D2010. I've tested a medium sized management GUI application, with about 60 forms (grids with detail forms, frames, etc). We're using TMS components + Remobjects.
D2007:
"normal" compilation: 18.8mb
with debug dcu's: 18.8mb (same size!)
D2010
normal: 23.9
debug dcu's: 48.8mb (!)
So using debug dcu's doubles our exe size...
Test with our business service (no big dfm's):
D2007: 12.3mb
D2010: 17.1mb
So yes, D2010 increases the exe (a bit), but this is not a problem for my customer.
Edit: some information about compiled size:
D2007:
D2010:
So an increase of code size, but a more than doubling of the data!
If you don't want to use an exe compressor then you should give StripReloc a try.
Check format of your dfm-s. They must be in binary format if you want to make your exe smaller.
1) You are generating a detailed map file, and because you've set "used debug dcus" it will also contains symbols for the RTL/VCL units. If it is used by an exception handling systems to generate call stacks and the like, it could be added to the executable. And if not compressed somehow, it could make your .exe size pretty large.
2) Using debug dcus will also make your .exe somewhat larger because usually they are compiled without optimization and debug options set, and they will make also your code slower. They shouldn't be used in a release version.
3) Debug information should add debig info only to the unit and not to the executable, although it is required IIRC to generate the map file.
Since D2010 adds extended RTTI, and RTTI is a notorious factor in increasing exe size, it would be interesting to see how big D2009 binaries are for that application.
If D2009 binaries are significantly smaller, it is not Unicode etc. For my own binaries, I only have a 30% increase or so going from D7 to D2009.
It has been stated earlier that using an executable compresser reduces the size of the exe but not of the install package. However, if you want a good compressor then try ASPack.
#Tom1952: ASPack is pretty fast, just a few seconds to compress a file
Also you can change the Icon. Icon in newest delphi IDE (ie XE3) is Vista/7 compatible and contains all sizes (up to 256x256 as far as I know). So you can reduce exe file size with changing the Icon.
The standard units in you newer delphi may contain more strings and constants such as error strings, that is included even if you disable debug information. Check your uses.
Don't have much of a somution besides not using a specific unit, or removing unneeded data from it.
(My experiences are with Delphi 5)
For Delphi 10.3 Rio with default setiings:
Step 1: Switch from Debug to Release in "Projects" window. This reduced my exe file from 22 MB to 5 MB !
Step 2: Use an exe compresor like ASPack. It further reduced my exe file to 1.3 MB. Unbelievable, isn't it ? :)
Uncheck debug information in project options.
If embarcadero can't provide any solution or explanation!!! I think the solution is simple: don't stuck only with Delphi there is a lot of programming languages, every one is limited only by programmer imagination.
In Delphi 2010 a new Find bar has been added beneath the source editor.
Though I sounds like a good idea I find it prettry anoying when F3 should be pressed multiple times for finding next matches. It works from time to time for me.
How I can switch it off and use the old search fnctionality?
It is not possible to switch back to the old modal behavior. That code is gone. If you’d like to see all the results in a list, you can use “Find in Files” and choose either “open files” or “files in project” to keep the number of files being searched to a minimum
Depending on how reliant you are on the latest versions of the VCL components, you may be able to run a slightly older version of the IDE as your editor, and use command-line compilation to compile your code using Delphi 2010. I have in the past mixed and matched IDEs in this way, but admittedly this is more difficult if your software is component-heavy, so to speak.
Edit: Moved my comment.
I have developed about 300 Applications which I would like to provide with multi-language capabilities independent from the Operating System. I have written a just-in-time translator, but that is too slow in applications with many components. What would you suggest I do?
We are using TsiLang and are very happy with it.
One of the best points is that you can pretranslate the project with a dictionary (which you filled from existing translations).
I've heard that the TsiLang components are nice, but your looking at an inplace solution...
I've used GNU gettext for Delphi which does exactly the thing you want, it loads the translations from a text file and replaces the text in your components. It even has a pas/dfm scanner to automatically generate the English translation file.
It's also possible to automatically change your pascal source code to inject the gettext procedure inplace of your static strings. If I'm not mistaken it just adds a underscore as function to it, as below.
ShowMessage('Hello'); // before
ShowMessage(_('Hello')); // after
I must say it has been 2 years since I last used this method.
One thing will remain problematic, the Delphi components are not unicode enabled (D2009 fixes this), so when you don't change the components you'll still have limited support for other languages.
A good free solution would be GNU gettext for Delphi. It has some capabilities not present in TsiLang - for example, you can put the knowledge on how to count things (different endings for one, two, four, hundred and two, many ...) into the translation file so that you don't have to teach each program to know this stuff.
License for the Delphi part is very permissive but I'm not sure how much the included GNU stuff will affect your application.
Get Multilizer. It is made in Delphi and it can handle Delphi programs like no other with special support for VCL. You can even redo your screens easy for every language. With Multilizer you can use different techniques to translate and run your program.
Delphi 2009 has added an Integrated Translation Environment/External Translation Manager
ITE and ETM are now available for both Delphi and C++Builder.
In Codegear's article: What's New in Delphi and C++Builder 2009, they state:
The Integrated Translation Environment
(ITE) is a part of the IDE that
simplifies localizing your projects.
ITE can create a new localized project
from an existing project. ITE does not
automatically translate text, but
provides a dialog listing all text
that needs to be localized and fields
in which to enter the corresponding
translated text. Once you have entered
the translated text and built the
localized project, you can set another
language active and display a form in
the localized text; you don't have to
switch locales and reboot your system.
This allows you to perform
localization without requiring a
localized system.
The External Translation Manager (ETM)
is a standalone application that works
with DFM files and text strings in the
source code. Although ETM does not
allow you to create a new localized
project, it does provide a dialog
listing localized text and the
translated text, similarly to ITE.
This is what I plan to try first once I am at the point that I want to Internationalize my product.
However, to me the easy part is to translate the program. The hard part is to translate the help file.
I would say GNU gettext for Delphi in combination with TMS Unicode Component Pack (previously free under TntWare) to get unicode support in the components.
To work with, or have translators work with, the gettext files I recommend looking at the free cross-platform Poedit that may edit the .po files.
Just to mention cxLocalizer if you own DexExpress components.