In Delphi the Image Base linker option defaults to 00400000.
Per the help:
Specifies the preferred load address
of the compiled image. This value is
typically only changed when compiling
DLLs. Default = 400000
Is there no effect for changing it on EXE's? What would the effect be? Is the address relative to each process?
Executable images (EXEs and DLLs, and other things that are DLLs in disguise, like BPLs and OCXs) are loaded by the OS loader at their preferred load address (Image Base) if possible; if that area of the virtual address space is reserved for some other purpose (another image, a thread stack, heap allocation), then the OS loader will relocate the image. Relocating the image involves putting it somewhere else in the address space, then taking the difference between the new load address and the preferred load address and adding this difference to every relocation fixup inside the image. Relocation fixups point to all the places in the executable image where the code or data refers to itself, such as code loading values from global variables, or making absolute jumps to other routines.
Because relocation involves the OS modifying the in-memory version of the image data, it takes longer, it takes up more I/O and commits more pages (the entire image with relocations needs to be paged in), and the OS virtual memory subsystem won't be able to share the loaded image with other processes that have loaded the same executable image (since it will be different in-memory). Thus, it's desirable to avoid relocation upon loading.
The preferred address for executable images is $00400000 by convention on 32-bit Windows, and other DLLs (including OS DLLs) rely on this convention by not having default load addresses that are likely to coincide with the main executable. Thus they avoid relocation. In fact, relocating an EXE image is so infrequently done that the relocation data can often be stripped from EXE images without harm.
Changing it for DLLs makes sense to avoid conflicting with any of the default OS DLLs and any other DLLs that normally ship with the DLL / EXE. Since changing it for an EXE increases the chances that the OS will need to relocate a DLL, it's not recommended that the EXE load address be changed.
Executable image compactors like UPX are not recommended for DLLs in particular, and for executables that may have many instances running, because the in-memory decompression acts like relocation in preventing the in-memory image from being shared between multiple processes.
Change EXE's image base is almost useless unless you're doing some very low-level dirty hack.
Is the address relative to each process?
Yes, each process has its own address space.
Related
My Delphi win32 VCL Application is deployed as a single big exe. The application is a Client Server application with a Fat Client that connects to SQL Server.
This is convenient (it is enough to update one file in a shared folder and the application is updated- of course the DB schema gets updated too).
Since I use many "fat" third party components like ReportBuilder and DevExpress at every release the exe size grows mostly because those components become bigger and bigger.
In the dpr I set these flags to ensure when exe is launched from shared folder or from removable device it is always loaded in memory (and avoid odd connections errors):
{$SetPEFlags IMAGE_FILE_NET_RUN_FROM_SWAP}
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
If the exe is built with the Release Build Configuration a normal exe usually reduces the size of about 50% (compared to Debug Build Configuration). But since I use EurekaLog, building with Build instead than Debug just slightly reduces the size.
Currently my exe is 115MB (built with "Build" Build configuration and EurekaLog).
Is it ok in anyone experience? Is it there a known limit that is advisable not to exceed?
Here's how the size has grown in recent years:
2014: 76MB
2015: 82MB
2016: 90MB
2017: 97MB
2018: 115MB
Since I recently exceeded the 100MB limit I started to worry.
I know that it is possible to build with runtime packages but my question focuses on keeping the single big exe approach if possible.
I remember cnPack has a Uses Cleaner feature I tried in the past. This could likely help me remove some non used unit and therefore reduce the exe size, but anyway this cannot stop the trend that brought me over 100MB.
Thanks.
I asked EurekaLog and they replied:
The size of your EXE after EurekaLog processing is mostly determined
by the size of your MAP file. Other things that affect the size of the
EXE include the EurekaLog options you select, such as memory checks,
Senders, JCL support, Etc.
You can reduce the size of the EXE somewhat by selecting the EL
Compression option. Keep in mind that compression can affect startup
time, since the debug information needs to be decompressed when you
launch the EXE.
You can further reduce the MAP file size by turning off debug
information in parts of your program that you don't need to stack
trace after a crash. Large component libraries like DevExpress would
be a good start.
Last, you may have other compiler options like range checking turned
on/off in tandem with EurekaLog. Some compiler options can contribute
to EXE size.
You can use our EurekaLog PE Analyzer to view details on debug
information size, Compression, Etc.
So somehow user Ville Krumlinde is right when he says that debug info is the cause. So to answer the question there are no problems in having a big exe, to reduce exe size the Release Build Configuration must be used, to use Eureka Log one should try to reduce he the map file size, in the above quote most hints are there.
You can use "Store all names externally" option to offload some debug information to external file, but your executables will no longer be self-contained.
When i install a new version of my Delphi 7.0 application on the network and i restart the application on the client desktop it sometimes executes the previous version, but the modified date is the new one...
After some googleing i found out that Windows Offline Files sometimes causes this, and therefore i added the folowing PE Flags to my .dpr
const
IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE = $8000;
// Following are already defined in D7's Windows.pas
IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP = $0400;
IMAGE_FILE_NET_RUN_FROM_SWAP = $0800;
{$SetPEOptFlags IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE}
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP
or IMAGE_FILE_NET_RUN_FROM_SWAP}
Even though i added these flags i (sometimes - it is not really reproducable) still have the same issues.
Can someone explain what these PE flags do and how to prevent windows to "cache" my exefile when running from a network path.
The PE Flags
When a process is created, the executable files are mapped into memory. These executable files are held in virtual memory pages like other virtual memory such as stack or heap memory.
The virtual memory manager may decide to unload pages from physical memory if the memory is needed elsewhere. And that could happen to your executable's pages.
Another scenario is that the executable file was never in physical memory at all. The process starts and mapped pages are loaded on demand. If some parts of the executable file are not touched then they will not get fetched from disk.
If the file resides on removable media, or on network volumes this can cause problems. Running process may be unable to access their own executable's memory. This results in runtime failure.
These PE flags force the executable to be copied to the swap file and mapped from there. The copy happens during the process creation and is potentially expensive.
These PE flags won't help you at all. If you start the process and the network is not available, all you have to work with is the offline copy. Include these PE flags and you'll just get the loader to copy the offline copy to the swap file. With the same end result.
Preventing the executable file from being cached
Before you can get an answer to that you would need to get a clean diagnosis of what causes the behaviour you observe. As it stands you suspect offline files, but I see no evidence for that being the cause. Perhaps something else is behind this. Perhaps your file server is mis-configured. Really, there could be all sorts of explanations for this.
Instead of adding PE flags to your executable, you really need to step back and accurately diagnose the problem. Once you know what the problem is then you will be in a position to attempt to solve it.
Using Delphi (or in general any tools, if exist of course), is it possible to reduce size of an exe file, removing all code that not is used but that is present there?
(for example, a function or procedure that is present but is never called).
I remember that with Pascal, using unit (without objects) the compiler includes only procedures and functions that are really used and will strip out non-used routines in a unit from the final exe.
With Object-pascal, I remember that in Delphi 1 all members of a object are included in the exe;
Has something has changed since than till Delphi-XE2?
If you aren't using RTTI you can add this to the top of your .dpr file (immediately after program) to remove the extra RTTI information:
{$IFOPT D-}{$WEAKLINKRTTI ON}{$ENDIF}
{$RTTI EXPLICIT METHODS([]) PROPERTIES([]) FIELDS([])}
If you want to strip it out of all the RTL/VCL units then you'd need to include those in your project file too so that the settings above could take effect. I don't think I would recommend doing that since I don't believe the reduction in executable size is worth the complications of compiling your own RTL/VCL.
You can also add the following, again somewhere in your .dpr file:
{$SetPEFlags IMAGE_FILE_RELOCS_STRIPPED}
This will strip the relocation information which is not needed in a .exe. Don't add this to a DLL or package!
reduce the application EXE size - excellent article
(Taken from the link above)
Generally, EXE files created with Delphi are larger than EXE files
created with another programming language. The reason is the VCL.
(Sure, VCL has many advantages...)
There are several ways to reduce a EXE's size:
01) Use a EXE-Packer (UPX, ASPack,...)-UPX
02) Use KOL.
03) Write your application without VCL
04) Use the ACL (API Controls Library)
05) Use StripReloc.
06) Deactivate remote debugging information and TD32.
07) You might want to put code in a dll.
08) Don't put the same images several times on a form. Load them at runtime.
09) Use compressed images (JPG and not BMP)
10) Store less properties in DFM files
If your aim is to reduce the size of your executable, you can use a tool which compress it and allow to execute it as it was not compress.. (=not a zip)
For instance, you can check UPX which works nicely with delphi programs.
Delphi has a smart linking option that is ON by default if I remember correctly. It does exactly what you describe. Only used functions and data are linked to your exe. If you need further size compression you can try one of the many "exe compressor" programs out there.
Just to supplement what other's have written...
The smart linker will eliminate unused routines except under the following conditions:
The unit has an initialization section - unfortunately many of the largest units in the RTL/VCL (Classes, SysUtils, Windows, Forms... to name a few) have initialization sections so if they are in your uses clause you get the whole enchilada linked into your code. Much of the code in the RTL/VCL could be refactored to reduce or eliminate this but it would break backward compatibility.
The routine is part of a class and is marked as virtual or dynamic - If you instantiate and object in your code that has virtual or dynamic methods, those methods are linked into your executable whether you call them or not.
Additional steps to can take to reduce exe file size:
Take advantage of form inheritance - If you have many similar forms, create a base form for them to inherit from. These child forms will only store properties that differ from the base form in their dfms. This can drastically reduce the size of your executable by eliminating redundant information in your dfms.
Store any large graphics in external files and load them on demand - Splash screens, "skins", icon sets, etc. can really bloat the size of an exe. Delphi doesn't always store these files in the most efficient format when they're embedded in the exe.
You can shave 10% - 30% off an exe's size by stripping certain unused data from the exe after its been compiled. There are third party tools that can do this but you can eliminate some of the cruft by setting appropriate PE header flags. Make sure you understand the implications of each flag before using it as some flags could make your exe unusable.
Copy portions of the RTL/VLC into your own units - instead of including Classes or SysUtils in your uses clause, create an new unit and copy just the classes and functions you need into the unit. Then use it instead.
Break code up into loadable modules - If you have multiple exes that reuse the same units you can make them smaller by using dlls or bpls rather than statically linking everything into the exes. A dll/bpl will only be loaded into memory once no matter how many exes need it.
Yet another option:
Use WinRar to create a Setup file (yes, WinRar can do that).
WinRar can automatically execute an EXE file once the unpacking is done.
WinRar has a superior compression ratio. One of the reasons is that it will merge all your files as a single file and then will start the compression, while ZIP (and other similar not-so-sophisticated compressors) will compress each file individually and merge them in a large file after the compression.
This give RAR's algorithm a better chance to find duplicate data among your files.
Plus, WinRar is less complicated than other installers (minus: it does not offer an uninstaller also, but that it is easy to fix with your own program).
Disclaimer: I am not affiliated with WinRAR.
I always compile my delphi apps without 'build with runtime packages', but for my latest Project i had to use it (as it started swelling day by day ) . I use some third party VCL (TMS component pack with source code , source code directory is in search path also ),
when i compile with build with runtime packages whole bpl package is used by app in runtime (otherwise it complies only the needed vcls inside the package into the app)so i think it consumes much ram memory (normally my app uses 38 Mb ram but now 62 Mb (not only tms i have used many other vcl too )according to windows task manager).
Is there any ways to make my app consume low memory like it was compiled as single exe.
(I know to recompile the VCL packages with only needed vcl (i have the source) but it is too hard for dig the source and find out the needed vcls and sub programs)
I think you're measuring the wrong thing. Although the package files are bigger, that doesn't necessarily mean your program is occupying more space in RAM. The compiler has to include code for all functions and units in a package, no matter which parts any given program uses, but that doesn't mean that all that code is loaded into memory. The OS will load the pages it needs; the rest will continue to reside on disk, in the BPL file it came from. The whole BPL file will occupy address space, but it won't be loaded into physical RAM, so there's not much to worry about unless you're really in danger of using the entire 2 GB of address space the OS grants your process.
Packages are DLLs, they need to be load into memory to work. And each package will contains the code for all the units it is built from. Thereby they can use more memory than an exe built withoyt run-time packages - although your increase looks a bit too large. On the other side, if more than your application use the same packages and they are properly installed, their code will be loaded once into memory.
You could build ad-hoc packages, but you should be very careful to use different names from the standard ones or you could break some other applications, especially if you put your packages in a shared location or in a directory that comes first in the path.
Before trying them, I will check that your application is not linking unused packages. Delphi will put in the options more or less all the packages it knows. You can check after a compile which packages are really used, and add them only to the package list to be used.
I've wrote some program in Delphi and when I am running it from a disk on key. At some point I'm required to unplug the disk on key while the application is running. If I do this on a computer with at least 1gb of ram everything is okay. When I do this on a machine with 512mb I get an external exception C0000006. If I'm not mistaken, this is because the OS is trying to read the next line of code but cannot find the resource for it (meaning, the application wasn't loaded to the ram) which is absurd because it's a 500kb application.
How can I solve this? or at least handle this exception in a more elegant way? (Since I can't catch it since it's an external exception).
Oh, and my Delphi application is a console application under windows xp.
What you need to do is tell windows to load your whole program into memory, rather than allowing it to demand load pages when it needs to. I have done this successfully for applications running off a CD. I don't have the code with me right now, but I recall that I found hints on how to do it in source for the fantastic open source install program Inno Setup.
Edit: Actually, after doing a little research, you can use a Delphi compiler directive to tell windows to load the full executable. This works if you have Delphi > 2006. This will have the effect that you will never get the external exception.
Put this line in your applications project file:
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
This tells windows that the executable is going to be used from removable media, so load the the executable into memory (or the swap file). Then you don't need to worry about things like copying the file to the machine first, etc.
Edit 2: I currently have access to Delphi 7 and I can confirm, as noted by others, that this also works with Delphi 7 (and likely Delphi 6) with the following code:
const
IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP = $0400;
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
For Delphi < 6, you can go down the path of forcing the executable to be paged in. There is an example of how to do it in C++ here (unless you find a way to modify the PE header flags after link time, which looks to be complicated)
N#
That's EXCEPTION_IN_PAGE_ERROR. It means that the OS loader failed to page in some data required for the application to run, probably due to an I/O error or other error. Since you're removing the disk, that would make sense.
Perhaps the working set (the set of often-used memory pages) for the application was allowed to grow large enough on 1GB machines such that recourse to the disk to reload pages wasn't necessary, but that wasn't the case on 512MB machines?
I would suggest trying to copy the executable to a temporary location and starting it from there, possibly with delayed deletion; or use some other mechanism to guarantee on-disk backing for all memory pages touched by the application in normal use, and prevent this error in cases of memory pressure, where the OS will trim the working set of running processes.
Like #Barry, I would recommend checking the drive type of the volume that your executable is running from; if it is a removeable drive (and missing a "already in temp" command line parameter) copy the executable (and any of its dependencies) to the user's %TEMP% folder and then re-launch it from there with an extra command line parameter to indicate "already in temp".
Create each temporary file using File.Create(targetPath, bufferSize, FileOptions.DeleteOnClose) (or one of the FileStream constructors that takes a FileOptions parameter), but make sure to hang onto the returned File instance until after the second copy is launched (e.g. in a List<File>).
Copy the contents of each file.
Launch the executable from the temp folder.
Call Close() on each of the File instances saved above.
Exit the original executable.
This way the files get closed regardless of which process finishes first, and the source volume can be removed earlier.
The solution
uses Windows;
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP}
is available for Delphi since version 6.
There's a Delphi working version of RunImageLocally from MSJ which forces the executable/dll to be paged in. This can prevent C0000006 errors when running from network or removable media...
Check it out at https://github.com/jrsoftware/issrc/blob/master/Projects/SetupLdr.dpr
This exception C0000006 also often occurs if your software is run from a network drive. To prevent that problem you can combine the flag
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
with the following flag:
IMAGE_FILE_NET_RUN_FROM_SWAP = $0800;
{$SetPEFlags $0C00}