When i install a new version of my Delphi 7.0 application on the network and i restart the application on the client desktop it sometimes executes the previous version, but the modified date is the new one...
After some googleing i found out that Windows Offline Files sometimes causes this, and therefore i added the folowing PE Flags to my .dpr
const
IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE = $8000;
// Following are already defined in D7's Windows.pas
IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP = $0400;
IMAGE_FILE_NET_RUN_FROM_SWAP = $0800;
{$SetPEOptFlags IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE}
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP
or IMAGE_FILE_NET_RUN_FROM_SWAP}
Even though i added these flags i (sometimes - it is not really reproducable) still have the same issues.
Can someone explain what these PE flags do and how to prevent windows to "cache" my exefile when running from a network path.
The PE Flags
When a process is created, the executable files are mapped into memory. These executable files are held in virtual memory pages like other virtual memory such as stack or heap memory.
The virtual memory manager may decide to unload pages from physical memory if the memory is needed elsewhere. And that could happen to your executable's pages.
Another scenario is that the executable file was never in physical memory at all. The process starts and mapped pages are loaded on demand. If some parts of the executable file are not touched then they will not get fetched from disk.
If the file resides on removable media, or on network volumes this can cause problems. Running process may be unable to access their own executable's memory. This results in runtime failure.
These PE flags force the executable to be copied to the swap file and mapped from there. The copy happens during the process creation and is potentially expensive.
These PE flags won't help you at all. If you start the process and the network is not available, all you have to work with is the offline copy. Include these PE flags and you'll just get the loader to copy the offline copy to the swap file. With the same end result.
Preventing the executable file from being cached
Before you can get an answer to that you would need to get a clean diagnosis of what causes the behaviour you observe. As it stands you suspect offline files, but I see no evidence for that being the cause. Perhaps something else is behind this. Perhaps your file server is mis-configured. Really, there could be all sorts of explanations for this.
Instead of adding PE flags to your executable, you really need to step back and accurately diagnose the problem. Once you know what the problem is then you will be in a position to attempt to solve it.
Related
My Delphi win32 VCL Application is deployed as a single big exe. The application is a Client Server application with a Fat Client that connects to SQL Server.
This is convenient (it is enough to update one file in a shared folder and the application is updated- of course the DB schema gets updated too).
Since I use many "fat" third party components like ReportBuilder and DevExpress at every release the exe size grows mostly because those components become bigger and bigger.
In the dpr I set these flags to ensure when exe is launched from shared folder or from removable device it is always loaded in memory (and avoid odd connections errors):
{$SetPEFlags IMAGE_FILE_NET_RUN_FROM_SWAP}
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
If the exe is built with the Release Build Configuration a normal exe usually reduces the size of about 50% (compared to Debug Build Configuration). But since I use EurekaLog, building with Build instead than Debug just slightly reduces the size.
Currently my exe is 115MB (built with "Build" Build configuration and EurekaLog).
Is it ok in anyone experience? Is it there a known limit that is advisable not to exceed?
Here's how the size has grown in recent years:
2014: 76MB
2015: 82MB
2016: 90MB
2017: 97MB
2018: 115MB
Since I recently exceeded the 100MB limit I started to worry.
I know that it is possible to build with runtime packages but my question focuses on keeping the single big exe approach if possible.
I remember cnPack has a Uses Cleaner feature I tried in the past. This could likely help me remove some non used unit and therefore reduce the exe size, but anyway this cannot stop the trend that brought me over 100MB.
Thanks.
I asked EurekaLog and they replied:
The size of your EXE after EurekaLog processing is mostly determined
by the size of your MAP file. Other things that affect the size of the
EXE include the EurekaLog options you select, such as memory checks,
Senders, JCL support, Etc.
You can reduce the size of the EXE somewhat by selecting the EL
Compression option. Keep in mind that compression can affect startup
time, since the debug information needs to be decompressed when you
launch the EXE.
You can further reduce the MAP file size by turning off debug
information in parts of your program that you don't need to stack
trace after a crash. Large component libraries like DevExpress would
be a good start.
Last, you may have other compiler options like range checking turned
on/off in tandem with EurekaLog. Some compiler options can contribute
to EXE size.
You can use our EurekaLog PE Analyzer to view details on debug
information size, Compression, Etc.
So somehow user Ville Krumlinde is right when he says that debug info is the cause. So to answer the question there are no problems in having a big exe, to reduce exe size the Release Build Configuration must be used, to use Eureka Log one should try to reduce he the map file size, in the above quote most hints are there.
You can use "Store all names externally" option to offload some debug information to external file, but your executables will no longer be self-contained.
Delphi 6 Prof.
We have many applications. The programs have 8-12 MB size.
In this period we many times got reports about "Invalid stream format" errors.
We use shared Windows (or Linux) folders to store the applications, and users running them from these directories with links.
This meaning that OS is paging the files, and loading the needed parts only.
Formerly we got C000006 exceptions.
As I know this meaning that the file paging (loading) failed on any network problem (timeout, etc).
Now we face with "Invalid stream format" errors, and "invalid property xxxx" errors.
If I know well, both error caused by "paging problem", but C06 happens in code, and stream error in the data area of the Exe.
But maybe I know wrong...
Anyway the problem is strange. Sometimes we got it, sometimes we not.
How to avoid it? These errors prevents the users to create new dialogs, to use the programs...
(In other place the user used wifi - then we got same side effects.)
Maybe you have any idea how to prevent, avoid this problem.
UPX (vs. Antiviruses)?
Copy the exe-s to local place?
The system administrators of this customer are "our enemies", because they said: "everything is ok". The source of the problem isn't identifiable...
Thanks for every idea: dd
Assuming your analysis is correct, and the problem is that the executable is located on a network drive with a flaky connection, then there is a solution. You need to add PE flags to your executable that forces Windows to copy the file from the network to the local machine before running it.
Make sure that your .dpr file's uses clause includes the Windows unit. And then add this line:
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP}
just before the begin in your .dpr file. We added the Windows unit so that the two constants would be recognised.
Another possibility could be to pack the exe with upx tool for instance.
http://upx.sourceforge.net/
It will expand the whole file in memory before run.
And it will save bandwidth.
I always compile my delphi apps without 'build with runtime packages', but for my latest Project i had to use it (as it started swelling day by day ) . I use some third party VCL (TMS component pack with source code , source code directory is in search path also ),
when i compile with build with runtime packages whole bpl package is used by app in runtime (otherwise it complies only the needed vcls inside the package into the app)so i think it consumes much ram memory (normally my app uses 38 Mb ram but now 62 Mb (not only tms i have used many other vcl too )according to windows task manager).
Is there any ways to make my app consume low memory like it was compiled as single exe.
(I know to recompile the VCL packages with only needed vcl (i have the source) but it is too hard for dig the source and find out the needed vcls and sub programs)
I think you're measuring the wrong thing. Although the package files are bigger, that doesn't necessarily mean your program is occupying more space in RAM. The compiler has to include code for all functions and units in a package, no matter which parts any given program uses, but that doesn't mean that all that code is loaded into memory. The OS will load the pages it needs; the rest will continue to reside on disk, in the BPL file it came from. The whole BPL file will occupy address space, but it won't be loaded into physical RAM, so there's not much to worry about unless you're really in danger of using the entire 2 GB of address space the OS grants your process.
Packages are DLLs, they need to be load into memory to work. And each package will contains the code for all the units it is built from. Thereby they can use more memory than an exe built withoyt run-time packages - although your increase looks a bit too large. On the other side, if more than your application use the same packages and they are properly installed, their code will be loaded once into memory.
You could build ad-hoc packages, but you should be very careful to use different names from the standard ones or you could break some other applications, especially if you put your packages in a shared location or in a directory that comes first in the path.
Before trying them, I will check that your application is not linking unused packages. Delphi will put in the options more or less all the packages it knows. You can check after a compile which packages are really used, and add them only to the package list to be used.
I've wrote some program in Delphi and when I am running it from a disk on key. At some point I'm required to unplug the disk on key while the application is running. If I do this on a computer with at least 1gb of ram everything is okay. When I do this on a machine with 512mb I get an external exception C0000006. If I'm not mistaken, this is because the OS is trying to read the next line of code but cannot find the resource for it (meaning, the application wasn't loaded to the ram) which is absurd because it's a 500kb application.
How can I solve this? or at least handle this exception in a more elegant way? (Since I can't catch it since it's an external exception).
Oh, and my Delphi application is a console application under windows xp.
What you need to do is tell windows to load your whole program into memory, rather than allowing it to demand load pages when it needs to. I have done this successfully for applications running off a CD. I don't have the code with me right now, but I recall that I found hints on how to do it in source for the fantastic open source install program Inno Setup.
Edit: Actually, after doing a little research, you can use a Delphi compiler directive to tell windows to load the full executable. This works if you have Delphi > 2006. This will have the effect that you will never get the external exception.
Put this line in your applications project file:
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
This tells windows that the executable is going to be used from removable media, so load the the executable into memory (or the swap file). Then you don't need to worry about things like copying the file to the machine first, etc.
Edit 2: I currently have access to Delphi 7 and I can confirm, as noted by others, that this also works with Delphi 7 (and likely Delphi 6) with the following code:
const
IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP = $0400;
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
For Delphi < 6, you can go down the path of forcing the executable to be paged in. There is an example of how to do it in C++ here (unless you find a way to modify the PE header flags after link time, which looks to be complicated)
N#
That's EXCEPTION_IN_PAGE_ERROR. It means that the OS loader failed to page in some data required for the application to run, probably due to an I/O error or other error. Since you're removing the disk, that would make sense.
Perhaps the working set (the set of often-used memory pages) for the application was allowed to grow large enough on 1GB machines such that recourse to the disk to reload pages wasn't necessary, but that wasn't the case on 512MB machines?
I would suggest trying to copy the executable to a temporary location and starting it from there, possibly with delayed deletion; or use some other mechanism to guarantee on-disk backing for all memory pages touched by the application in normal use, and prevent this error in cases of memory pressure, where the OS will trim the working set of running processes.
Like #Barry, I would recommend checking the drive type of the volume that your executable is running from; if it is a removeable drive (and missing a "already in temp" command line parameter) copy the executable (and any of its dependencies) to the user's %TEMP% folder and then re-launch it from there with an extra command line parameter to indicate "already in temp".
Create each temporary file using File.Create(targetPath, bufferSize, FileOptions.DeleteOnClose) (or one of the FileStream constructors that takes a FileOptions parameter), but make sure to hang onto the returned File instance until after the second copy is launched (e.g. in a List<File>).
Copy the contents of each file.
Launch the executable from the temp folder.
Call Close() on each of the File instances saved above.
Exit the original executable.
This way the files get closed regardless of which process finishes first, and the source volume can be removed earlier.
The solution
uses Windows;
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP}
is available for Delphi since version 6.
There's a Delphi working version of RunImageLocally from MSJ which forces the executable/dll to be paged in. This can prevent C0000006 errors when running from network or removable media...
Check it out at https://github.com/jrsoftware/issrc/blob/master/Projects/SetupLdr.dpr
This exception C0000006 also often occurs if your software is run from a network drive. To prevent that problem you can combine the flag
{$SetPEFlags IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP}
with the following flag:
IMAGE_FILE_NET_RUN_FROM_SWAP = $0800;
{$SetPEFlags $0C00}
In Delphi the Image Base linker option defaults to 00400000.
Per the help:
Specifies the preferred load address
of the compiled image. This value is
typically only changed when compiling
DLLs. Default = 400000
Is there no effect for changing it on EXE's? What would the effect be? Is the address relative to each process?
Executable images (EXEs and DLLs, and other things that are DLLs in disguise, like BPLs and OCXs) are loaded by the OS loader at their preferred load address (Image Base) if possible; if that area of the virtual address space is reserved for some other purpose (another image, a thread stack, heap allocation), then the OS loader will relocate the image. Relocating the image involves putting it somewhere else in the address space, then taking the difference between the new load address and the preferred load address and adding this difference to every relocation fixup inside the image. Relocation fixups point to all the places in the executable image where the code or data refers to itself, such as code loading values from global variables, or making absolute jumps to other routines.
Because relocation involves the OS modifying the in-memory version of the image data, it takes longer, it takes up more I/O and commits more pages (the entire image with relocations needs to be paged in), and the OS virtual memory subsystem won't be able to share the loaded image with other processes that have loaded the same executable image (since it will be different in-memory). Thus, it's desirable to avoid relocation upon loading.
The preferred address for executable images is $00400000 by convention on 32-bit Windows, and other DLLs (including OS DLLs) rely on this convention by not having default load addresses that are likely to coincide with the main executable. Thus they avoid relocation. In fact, relocating an EXE image is so infrequently done that the relocation data can often be stripped from EXE images without harm.
Changing it for DLLs makes sense to avoid conflicting with any of the default OS DLLs and any other DLLs that normally ship with the DLL / EXE. Since changing it for an EXE increases the chances that the OS will need to relocate a DLL, it's not recommended that the EXE load address be changed.
Executable image compactors like UPX are not recommended for DLLs in particular, and for executables that may have many instances running, because the in-memory decompression acts like relocation in preventing the in-memory image from being shared between multiple processes.
Change EXE's image base is almost useless unless you're doing some very low-level dirty hack.
Is the address relative to each process?
Yes, each process has its own address space.