If I compile my entire Delphi application to a single exe, that file will grow to 5MB, 10MB, maybe more. When is that too big? What are the issues with this? This is a commercial application, currently on Delphi XE.
I'm aware of the option to Build with Runtime Packages. That sounded like a good idea, but I see comments here noting that there are some issues and disadvantages.
A Delphi application is never really too big.
However the larger the exe is, the harder it will be to redistribute the file.
Also if the executable is located on a network-disk start-up time may suffer.
A number of factors make the exe grow:
enabling debug info (will more or less double the exe size). Disable the inclusion of debug info in the final exe (see screenshot above).
including bitmaps (in an imagelist or likewise component) will also grow the exe substantially.
including resources (using a custom *.res) file will grow the size.
I would advise against putting resources in a separate dll.
This will complicate your application, whilst not reducing the loading time and distribution issues.
Turning off debug info in production code is a must.
If you have a Delphi-2010 or newer you can choose to include images in the png format.
This will take up much less space than old-skool bitmaps.
As long as your app is below 30 MB I would not really worry overmuch about the file size though.
Strip RTTI info
David suggests stripping RTTI info (this will disable live-bindings and some other advanced stuff), see: Reduce exe file
According to David it saves about 30% in exe size.
Exe-size will only increase loading time
Far more important is the amount of data your application allocates as storage.
The amount of space you use (or waste) here will have a far greater impact on the performance of your application than the raw exe size.
Strategy or tools to find "non-leak" memory usage problems in Delphi?
A better way to optimize is to make sure you don't leak resources
How to activate ReportMemoryLeaksOnShutdown only in debug mode?
Windows API calls memory leak detection
Use smart datastructures and algorithms
It gets too general to really narrow it down here, but use algorithms with O(slowly increasing) over O(wasteful increase).
Big-O for Eight Year Olds?
And try and limit memory usage by only fetching the data that you need instead of all the data you might need but probably never will.
Delphi data structures
Etc etc.
I don't know any issues with the exe-size of an application. I'm currently working at an application where the exe is around 60MB and there is no problem.
The only limitation I know are the limitation of the available memory. And an application with use of runtime-packages will consume more working memory because all runtime packages are load on application start. And the packages contains a lot of code wich is problably not used in your application.
I really like the idea of runtime-packages but I don't like the implementation in Delphi. One main disadvantage is that you have to ship your app with a bunch of packages wich makes it hard to maintain.
use RELEASE Build for reduce execute size , increase performance.also use runtime package for reduce exe file but use runtime package cause increase package(setup) file size.
Related
Using the cloud compiling website I created a custom NodeMCU firmware that has a lot of modules. So many that the firmware itself is almost 700KB in size. I usually only use up to 5 modules for a single project, so I'm wondering if the inclusion of all the other modules in the firmware have a noticeable negative impact on the RAM usage.
There's an excellent explanation of the ESP8266 memory map (and other interesting bits) at https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-beautifully-easy-iot/posts/1501224. Furthermore, you got a great answer as a comment.
Every module baked into the binary consumes memory just by "being there". If you wanted to measure the impact a single module has on the available heap you'd have to build two binaries, one with and one without that module. You'd flash both and calculate the delta of running node.heap() right after start.
Does compiling NodeMCU with lots of modules have an impact on the memory usage?
Yes, it definitely does as you noticed.
I usually only use up to 5 modules for a single project
That's why we recommend to use a different set (read "minimal set") of modules for every project. The beauty of the NodeMCU firmware is that you only have to do this once, contrary to e.g. Arduino, after which swapping scripts or even individual functions in'n out is super quick.
I suggest you also take a look at https://nodemcu.readthedocs.io/en/dev/en/lua-developer-faq/#techniques-for-reducing-ram-and-spiffs-footprint. A major overhaul is in the making at https://github.com/nodemcu/nodemcu-firmware/pull/1899.
I understand from other posts here that "IMAGE_FILE_LARGE_ADDRESS_AWARE" may work to effectively expand memory availability in e.g. Delphi 2007.
I don't get this to work in Delphi6, is this indeed the case, or should it work? Or is there an alternative command that does the same thing?
If not, I may need to migrate to a later version of Delphi. Then, does anyone know what the most recent version of Delphi is that would easily allow me to migrate my existing code (ideally, my existing code, which is fairly simple Turbo Pascal-type code, would just work as is) AND would support the "IMAGE_FILE_LARGE_ADDRESS_AWARE" 'trick' to expand memory?
Many thanks!
Remco
You can apply the IMAGE_FILE_LARGE_ADDRESS_AWARE PE flag to a Delphi 6 application, but you must beware of the following issues:
The default memory manager for Delphi 6, the Borland memory manager, does not support memory allocations with addresses above 2GB. You must replace the memory manager with one that supports large addresses. For instance FastMM.
Your code may well contain pointer truncation bugs that will need to be found and fixed.
The same goes for any third party software that you use. This includes the Borland RTL and VCL libraries. I did not encounter many problems with these libraries, but it may be that your program uses different parts of the runtime libraries that have pointer truncation bugs.
In order to stress test your program under large address conditions you should turn on top down memory allocation. Do not be surprised if your anti-malware software (or indeed other system level software) has to be disabled whilst you operate in top down memory allocation mode. This type of software is notoriously poor at operating in top down memory allocation mode.
Finally, it is worth pointing out that large address aware cannot solve all out of memory problems. All it does is open up the top half of the 32 bit address space. Your program might require even more address space than that. In which case you'd need to either re-design your program, or move to a 64 bit compiler.
Since Neo4j works primarily in memory, I was wondering if it would be advantageous to enable hugepages (https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt) in my Linux Kernel, and then XX:+UseLargePages or maybe -XX:+UseHugeTLBFS in the (OpenJDK 8) JVM ?
If so, what rule of thumb should I use to decide how many hugepages to configured?
The Neo4j Performance Guide (http://neo4j.com/docs/stable/performance-guide.html) does not mention this, and Google didn't turn up anyone else discussing it (in the first couple of search pages anyway), so I thought I'd ask.
I'm wrestling to get acceptable performance from my new Neo4j instance (2.3.2-community). Any little bit will help. I want to know if this is worth trying before I bring down the database to change JVM flags... I'm hoping someone else has done some experiments along these lines already.
Since Neo4j does its own file paging and doesn't rely on the OS to do this, it should be advantageous or at least not hurt. Huge pages will reduce the probability of TLB cache misses when you use a large amount of memory, which Neo4j often would like to do when there's a lot of data stored in it.
However, Neo4j does not directly use hugepages even though it could and it would be a nice addition. This means you have to rely on transparent huge pages and whatever features the JVM provides. The transparent huge pages can cause more-or-less short stalls when smaller pages are merged.
If you have a representative staging environment then I advise you to make the changes there first, and measure their effect.
Transparent huge pages are mostly a problem for programs that use mmap because I think it can lead to changing the size of the unit of IO, which will make the hard-pagefault latency much higher. I'm not entirely sure about this, though, so please correct me if I'm wrong.
The JVM actually does use mmap for telemetry and tooling, through a file in /tmp so make sure this directory is mounted on tmpfs to avoid gnarly IO stalls, for instance during safe-points (!!!). Always do this even if you don't use huge pages.
Also make sure you are using the latest Linux kernel and the latest Java version.
You may be able to squeeze some percentage points out of it with tuning G1, but this is a bit of a black art.
And if so, how. I'm talking about this 4GB Patch.
On the face of it, it seems like a pretty nifty idea: on Windows, each 32-bit application normally only has access to 2GB of address space, but if you have 64-bit Windows, you can enable a little flag to allow a 32-bit application to access the full 4GB. The page gives some examples of applications that might benefit from it.
HOWEVER, most applications seem to assume that memory allocation is always successful. Some applications do check if allocations are successful, but even then can at best quit gracefully on failure. I've never in my (short) life come across an application that could fail a memory allocation and still keep going with no loss of functionality or impact on correctness, and I have a feeling that such applications are from extremely rare to essentially non-existent in the realm of desktop computers. With this in mind, it would seem reasonable to assume that any such application would be programmed to not exceed 2GB memory usage under normal conditions, and those few that do would have been built with this magic flag already enabled for the benefit of 64-bit users.
So, have I made some incorrect assumptions? If not, how does this tool help in practice? I don't see how it could, yet I see quite a few people around the internet claiming it works (for some definition of works).
Your troublesome assumptions are these ones:
Some applications do check if allocations are successful, but even then can at best quit gracefully on failure. I've never in my (short) life come across an application that could fail a memory allocation and still keep going with no loss of functionality or impact on correctness, and I have a feeling that such applications are from extremely rare to essentially non-existent in the realm of desktop computers.
There do exist applications that do better than "quit gracefully" on failure. Yes, functionality will be impacted (after all, there wasn't enough memory to continue with the requested operation), but many apps will at least be able to stay running - so, for example, you may not be able to add any more text to your enormous document, but you can at least save the document in its current state (or make it smaller, etc.)
With this in mind, it would seem reasonable to assume that any such application would be programmed to not exceed 2GB memory usage under normal conditions, and those few that do would have been built with this magic flag already enabled for the benefit of 64-bit users.
The trouble with this assumption is that, in general, an application's memory usage is determined by what you do with it. So, as over the past years storage sizes have grown, and memory sizes have grown, the sizes of files that people want to operate on have also grown - so an application that worked fine when 1GB files were unheard of may struggle now that (for example) high definition video can be taken by many consumer cameras.
Putting that another way: applications that used to fit comfortably within 2GB of memory no longer do, because people want do do more with them now.
I do think the following extract from your link of 4 GB Patch pretty much explains the reason of how and why it works.
Why things are this way on x64 is easy to explain. On x86 applications have 2GB of virtual memory out of 4GB (the other 2GB are reserved for the system). On x64 these two other GB can now be accessed by 32bit applications. In order to achieve this, a flag has to be set in the file's internal format. This is, of course, very easy for insiders who do it every day with the CFF Explorer. This tool was written because not everybody is an insider, and most probably a lot of people don't even know that this can be achieved. Even I wouldn't have written this tool if someone didn't explicitly ask me to.
And to expand on CFF,
The CFF Explorer was designed to make PE editing as easy as possible,
but without losing sight on the portable executable's internal
structure. This application includes a series of tools which might
help not only reverse engineers but also programmers. It offers a
multi-file environment and a switchable interface.
And to quote a Microsoft insider, Larry Miller of Microsoft MCSA on a blog post about patching games using the tool,
Under 32 bit windows an application has access to 2GB of VIRTUAL
memory space. 64 bit Windows makes 4GB available to applications.
Without the change mentioned an application will only be able to
access 2GB.
This was not an arbitrary restriction. Most 32 bit applications simply
can not cope with a larger than 2GB address space. The switch
mentioned indicates to the system that it is able to cope. If this
switch is manually set most 32 bit applications will crash in 64 bit
environment.
In some cases the switch may be useful. But don't be surprised if it
crashes.
And finally to add from MSDN - Migrating 32-bit Managed Code to 64-bit,
There is also information in the PE that tells the Windows loader if
the assembly is targeted for a specific architecture. This additional
information ensures that assemblies targeted for a particular
architecture are not loaded in a different one. The C#, Visual Basic
.NET, and C++ Whidbey compilers let you set the appropriate flags in
the PE header. For example, C# and THIRD have a /platform:{anycpu,
x86, Itanium, x64} compiler option.
Note: While it is technically possible to modify the flags in the PE header of an assembly after it has been compiled, Microsoft does not recommend doing this.
Finally to answer your question - how does this tool help in practice?
Since you have malloc in your tags, I believe you are working on unmanaged memory. This patch would mostly result in invalid pointers as they become twice the size now, and almost all other primitive datatypes would be scaled by a factor of 2X.
But for managed code since all these are handled by the CLR in .NET, this would mean really helpful and would not have much problems unless you are dealing with any of the following :
Invoking platform APIs via p/invoke
Invoking COM objects
Making use of unsafe code
Using marshaling as a mechanism for sharing information
Using serialization as a way of persisting state
To summarize, being a programmer I would not use the tool to convert my application and rather would migrate it myself by changing build targets. being said that if I have a exe that can do well like games with more RAM, then this is worth a try.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
At one point I had a nice little compression utility that smashed my Delphi compiled EXE's to a smaller download size, but now I can't find it. Any recommendations?
Also, are there any downsides to using these kinds of utilities? (I mainly use them to shorten download times for rural / dial-up users).
Related question: Are there any downsides to using UPX to compress a Windows executable?
Years ago I looked into compressing my executable to make the download smaller.
What I ended up doing, and what I recommend for you, is to use an installer program like Inno Setup instead. Not only does it create a single EXE that will install/uninstall your program, but it also compresses that EXE practically as much as a separate compressor would do to your executable alone.
When the program is installed it gets decompressed, so it never appears to be a virus and does not increase load times.
So I get the benefits of smaller download size, and a professional-looking installation script at the same time.
p.s. Inno Setup is free.
The recommendation is that you not:
EXE compressors can make your application seem like a virus (self-modifying)
gzip/zip are just as effective at compressing and don't tinker with your app
EXE compressors make the load times of your app increase (unless you're just talking about the setup program which is a different matter
This crazy looking site brings up an argument I had heard in the distant past (whether it's true or not still today, I'm not sure, modern packers probably have a different strategy today) This article references Win32! :)
http://topic.csdn.net/t/20000408/08/6785.html
Modern multitasking OSes such as
Windows 95/98 and NT use what is
called a "virtual memory" system. When
programs start, all of their code is
not loaded into memory immediately
upon startup, as was the case with DOS
programs. Instead, only portions of
the code being actively executed are
stored into memory. For example, say
your program has a Print option on its
menu, and code behind it that handles
the printing. This code will only be
loaded into memory once the Print
feature is first selected by the user.
And if after the code is loaded into
memory the Print feature is not used
for a while the system will "discard"
the code, freeing the memory it
occupied, if another application
desperately needs memory. This is part
of a process called "paging" and is
completely transparent to the program.
Another way paging under Win32
conserves memory is it causes multiple
instances of a program (or DLL) to
share the same memory for code. In
other words, under normal
circumstances there is no real
difference in the amount of physical
memory allocated for code between
starting 100 instances of a program
and starting one instance.
If all Win32 programs behaved like
DOS programs, loading everything into
memory and keeping it there until the
program terminated and also not
sharing any memory between multiple
instances, you can probably imagine
how quickly physical memory could run
out on systems with a limited amount,
causing disk swapping to start.
Yet this is precisely what current
Win32 EXE compressors do to your
EXE's/DLL's! They completely go
against the OS's paging system by
decompressing all code into memory and
keeping it there until termination.
And because code isn't stored in a
"raw" format in the EXE file (i.e. the
same way it is stored in memory), the
OS is unable to share code between
multiple instances.
I don't know of any that are specifically for Delphi, but UPX is very popular for this sort of thing. The only downside is that the executable has to be decompressed when it's launched, and that can eat some time. It seems to be very fast for sanely sized executables, though.
The one you are probably thinking of is ASPack - it is a EXE compressor written in Delphi, but will compress any EXE. It might do extra well on Delphi EXE's though. I would agree with the other answers that you should not use an EXE compressor just to save on download times. There may be specific situations where an EXE compress is a good idea, but generally it is not.
Instead use a good installation builder, especially if you can find one that uses 7zip compression. I know InstallAware uses 7zip internally for maximum compression. Depending on which versions of Delphi you own you may have an InstallAware license too.
If nothing else you can build a self extracting archive with basic install behavior with 7zip for free. It is a separate download for SFXs for installers.
Use UPX with lzma option for max compression.
upx --lzma yourfile.exe
The main inconvenience of a compressed EXE or DLL is that the OS cannot share the code amongst multiple instances.
So you're wasting memory, have to decompress each time you start an instance, exhibit a virus-like behavior without even an download advantage over a compressed install.
Only positive case is when launching directly from a network drive.
I believe Terminal servers (Like Citrix) will use the same memory for you're application binary if it is uncompressed. Meaning a compressed exe could smell a small disaster in a Citrix environment.
UPX should work, although it's not Delphi specific.
I use PEtite: http://un4seen.com/petite/
I would also vote for upx. Beside the downsides which were mentioned it also protects from basic reverse engineering and those lame "resource hacker" tools. Which by the way are plenty, and most of them fail to open a compressed executable.
I asked a question about the con's of using UPX on Delphi executables here on SO a while back, and I got some great responses.
Are there any downsides to using UPX to compress a Windows executable?
You can use PECompact since people can't decrypt it easily, and as test showed (showed on main page, just scroll down a bit) it's better than ASPack or UPX, i've using it on my previous Delphi projects