FastMM and Dynamically loaded DLLs - delphi

I have a host application, that loads a dozen of libraries at start up. I want to switch from Delphi 7s default memory manager to the full version of FastMM4 for better mem leak reporting.
Should I include FastMM4 in the uses section of both the host application and the libraries?
What about shared runtime packages?
Some added information:
We have one exe and >20 dlls. Everyone is sharing a single runtime package.
We do not use sharemem today. Not that I know about. ShareMem is not included in eighter the exe or the dlls that I had a quick look at to day.
Additional questions:
Can I use the same options inside fastmems inc file in all projects, or do the exe and the dll need different settings?

Excerpt from FastMM4options.inc file.
To me it means that if all your packages, dll's and executables are compiled with ShareMM, it should be possible to replace the Delphi 7s default memory manager.
{-----------------------Memory Manager
Sharing
Options------------------------}
{Allow sharing of the memory manager
between a main application and DLLs
that were also compiled with FastMM.
This allows you to pass dynamic arrays
and long strings to DLL functions
provided both are compiled to use
FastMM. Sharing will only work if the
library that is supposed to share the
memory manager was compiled with the
"AttemptToUseSharedMM" option set.
Note that if the main application is
single threaded and the DLL is
multi-threaded that you have to set
the IsMultiThread variable in the main
application to true or it will crash
when a thread contention occurs. Note
that statically linked DLL files are
initialized before the main
application, so the main application
may well end up sharing a statically
loaded DLL's memory manager and not
the other way around. }
{.$define ShareMM}
{Allow sharing of the memory manager
by a DLL with other DLLs (or the main
application if this is a statically
loaded DLL) that were also compiled
with FastMM. Set this option with
care in dynamically loaded DLLs,
because if the DLL that is sharing
its MM is unloaded and any other DLL
is still sharing the MM then the
application will crash. This setting
is only relevant for DLL libraries
and requires ShareMM to also be set to
have any effect. Sharing will only
work if the library that is supposed
to share the memory manager was
compiled with the
"AttemptToUseSharedMM" option set.
Note that if DLLs are statically
linked then they will be initialized
before the main application and
then the DLL will in fact share its MM
with the main application. This
option has no effect unless ShareMM is
also set.}
{.$define ShareMMIfLibrary}
{Define this to attempt to share the
MM of the main application or other
loaded DLLs in the same process that
were compiled with ShareMM set. When
sharing a memory manager, memory
leaks caused by the sharer will not be
freed automatically. Take into
account that statically linked DLLs
are initialized before the main
application, so set the sharing
options accordingly.}
{.$define AttemptToUseSharedMM}
{Define this to enable backward
compatibility for the memory manager
sharing mechanism used by Delphi 2006
and 2007, as well as older FastMM
versions.}
{$define
EnableBackwardCompatibleMMSharing}

What you're looking for is called SimpleShareMem. It's included in the FastMM package. Make sure both your app and your DLLs are using it as well as FastMM4 at the top of their uses clause. That makes sure they all share the same heap instead of using separate ones.
Of course, shared memory is only necessary if you're going to be passing dynamic memory between the app and the libraries, such as strings or objects. If not, then you don't need SimpleShareMem, but I'd still recommend to switch the libraries to FastMM as the memory manager for increased performance and improved stability.

since "Everyone is sharing a single runtime package." I'd recommend the method I'm using. Simply, add FastMM4 to your shared runtime package (Of course, you'll need to put FastMM4 in "using" declaration in each library and host application anyway).
This way has some advantages like:
simple method to switch FullDebugMode on/off [and other options of course]. No project recompilation is needed, recompiling the shared package only is enough to switch the FullDebugMode in the whole application (including dll's) as the FastMM is instanced only once.
no versioning problem. When you update FastMM (which, once used, becomes crucial component of the whole app and plugins) you don't have to ship the whole build (app & plugins) again. Once again, updating the shared package is enough.
no memory manager issues when unloading plugins. I've had some problems with FastMM being uninstalled once the first plugin library was unloaded, thus causing tons of bugs on application shutdown.
no tracking problems - in case of any memory leak or other memory issue you'll be provided with valid call stack, even if the bug occurred in the DLL code.
Here are the (non-standard) options I'm using in my FastMM4Options.inc in order to get the whole thing working as described above.
{$define NeverUninstall}
{$define UseRuntimePackages}
{.$define ShareMM}
well I believe the rest was left unchanged, but if something is wrong, here is the full file:
http://pastebin.4programmers.net/693

Afaik there is sharemm, a version of fastmm to share over DLL bounderies.

If your application is compiled with runtime packages, no additional actions are needed because runtime packages use only one memory manager, always. You just need to specify the memory manager of your choice once, preferably in the main application code as the first unit in the uses list. All the other runtime packages do not require modifications, they'll use the correct memory manager automatically, whatever that memory manager is.
If your app is compiled without runtime packages then every dll has its own memory manager. By default they all use their default memory managers, and if I'm not mistaken, it's a sharing MM, that is, you can transfer strings for example to and from dlls. If you want to replace the default MM with another sharing one, you'll need to include new memory manager as the first unit in the uses list in EVERY dll or exe you want to work together.
The difference is that with runtime packages sharing occurs at the package level. Even non-sharing MM will be shared. Without runtime packages every DLL uses a separate MM, and it's only when those separate memory managers cooperate that you get sharing.
My opinion is that unless your app interacts with dlls really closely (read: has forms and components in dlls), the best approach is not to rely on having sharing memory manager at all, but to manage your memory properly, keeping track of who creates what and deleting objects in the same library where you instantiated them. It's not hard. Just return interfaces instead of objects, return shortstrings (fixed-length) or copy the data into the provided buffer instead of returning strings, and so on. Do not return things that will be freed by the caller, don't accept things that the callee will have to free, and you'll be basically safe. Not only this solves all the possible memory manager problems, it also lets you write dlls in any language.

Ypu have two choices: keep on using the ShareMem unit and the BorldMM.dll replacement you find in the FastMM distribution, or modify both the executable and DLLs to include the FastMM memory manager, by placing the unit as the first one. You should also tailor some of the $DEFINEs you find in FastMM4Options.inc to your needs, they are well explained there. AFAIK run-time packages use the installed memory manager.

Related

External modules implementation

What is the best way to implement an external module system for a DELPHI application?
What I need is very basic really:
The main APP detects if a module is present and loads it(Run time)
Modules can store form
Modules can store DataModules
Modules can Store code
I need the use the store forms inside other forms, and only as a standAlone
I use something like this
if Assigned(pNewClass) then begin
Application.CreateForm(pNewClass, _lFrm);
_lFrm.Hide;
_lFrm.BorderStyle := bsNone;
_lFrm.Parent := pBasePNL //(TPanel);
_lFrm.Align := alClient;
end;
So I create a TForm, but place it inside a TPanel.
As for DataModules I usally store ImageLists so the ideia is to change the app ICOs just bit changing the external module.
So what is the best way to achieve this?
Looked at runtime BPLs but don’t seem to understand how to do it.
Thanks.
UPDATE : .....................................
After reading some related question and answers i think I found my answer and solution.
http://edn.embarcadero.com/article/27178
The article is old stuff but amazingly simple.
Well the logic is there I just don’t seem to get it to Show the forms 
I am just testing example 2
It loads the BPL, but doesn’t get the Form:
AClass := GetClass('TForm2');
Always retrievex ‘nil’
But the BPL as it registered:
RegisterClass(TForm2);
Can anyone help with this one.
Packages are an easy solution but they have one huge drawback. Using packages forces plugin authors to use not only Delphi, but the same version of the compiler as you do.
I personally would prefer to expose the functionality of the app through a number of interfaces. This allows accessibility from languages other than Delphi.
Typically the plugin would be implemented in a DLL and would export a function that the app would call to pass in the root interface representing the app. The plugin would then call methods of that interface thus establishing two-way interaction.
I made you a demo, it's so easy to get started! However... Started is not finished.
Every time I started using plugins, I later regretted it. However, as you say, you want a binary plugin system. So BPLs are the correct solution. David suggests using interfaces (with plain DLLs instead of the full runtime package BPL) and this would solve some of the commonly encountered BPL-instability problems due to not versioning your classes, and thus your Application-and-package-binary-compatibility dependencies, properly. If you do not need to share memory and do not need to use borlandmm.dll (shared memory management) then a straight DLL with interfaces will work fine.
If you can do everything you need to do with just scripting, and make your plugin system with just scripts, then do it that way. If you can get away with DLLs, interfaces, and no memory sharing, then use plain DLLs. If you must share Memory and class types, then yes, use BPLs.
Beware that using BPLs (Runtime Packages) comes with a lot of side effects you might not have been expecting. The largest BPL-based applications I have worked on have been more of a mess, and less stable, than any monolithic apps I have worked on. It seems that it is only possible to pine for packages until you try them, and then I find, I pine for monoliths again.
If you use BPL packages correctly, and you version your plugins properly, everything is fine. BPLs are great. But in the real world, it seems that sane versioning and ABI compatibility and interoperability, and stability are at a premium.
Update: I made you a demo it's here (plugin_r2.zip). It was done in Delphi XE, but if you are using an older version of delphi, you just delete the .dproj files and open the .dpr file for the main app, and the .dpk for the package.

Which are the differences between a bpl delphi expert and a dll delphi expert

I am working in a delphi IDE expert , now to avoid dependencies problems, i was thinking in rebuild this expert as a dll expert as was suggested in one of these answers, now my expert (compiled as bpl) access the Screen and Application global variables (instances of the Delphi IDE), so i was wondering if i compile my expert as a dll i can still accesing these variables and also i want to know which are the main differences between a bpl delphi expert and a dll delphi expert?
Should I compile my wizard as a DLL or a Package? Packages are easier to
load and unload without restarting the
IDE (and hence easier to debug), but
they can create unit naming conflicts
in the IDE. Conflicts happen when the
name a wizard's unit matches the name
of a unit in another loaded
design-time package. In this case,
both packages can not be loaded at the
same time. The recommended workaround
is to prefix all of your unit names
with a "unique" prefix. GExperts, for
example, uses "GX_" as the name prefix
for its units.
From this very good source about OTA: GExperts
When you access a global variable those would be global variables that are global to your DLL, not global to the main BDS.exe. I am not sure but I think your DLL would have its own Screen and Application global variable, if you linked in Forms, and the core of the VCL.
Those things which belong to the IDE itself are accessed through the Open Tools Api (OTA). I believe that you do not normally share any objects between the IDE your expert anyways, and if you were to try to do so, it would be problematic. Anything at all that you do that bypasses the OTA is going to be vulnerable to breaking in strange ways, especially in future versions of the IDE.
Dependency problems are of course a big reason to not use BPL based packages, but I think an even bigger reason is to maintain a complete separation between your tool's internals, and the internals of the IDE.
Remember that a DLL target, like an executable target, is statically linked. That is the core of the difference. If your expert provides functionality that uses the legal public documented OTA interfaces only, then moving to a DLL should be problem free. If you use some back door hacks that are possible with BPLs, then I can't advise you further.

How do I make my program use less memory when built with runtime packages?

I always compile my delphi apps without 'build with runtime packages', but for my latest Project i had to use it (as it started swelling day by day ) . I use some third party VCL (TMS component pack with source code , source code directory is in search path also ),
when i compile with build with runtime packages whole bpl package is used by app in runtime (otherwise it complies only the needed vcls inside the package into the app)so i think it consumes much ram memory (normally my app uses 38 Mb ram but now 62 Mb (not only tms i have used many other vcl too )according to windows task manager).
Is there any ways to make my app consume low memory like it was compiled as single exe.
(I know to recompile the VCL packages with only needed vcl (i have the source) but it is too hard for dig the source and find out the needed vcls and sub programs)
I think you're measuring the wrong thing. Although the package files are bigger, that doesn't necessarily mean your program is occupying more space in RAM. The compiler has to include code for all functions and units in a package, no matter which parts any given program uses, but that doesn't mean that all that code is loaded into memory. The OS will load the pages it needs; the rest will continue to reside on disk, in the BPL file it came from. The whole BPL file will occupy address space, but it won't be loaded into physical RAM, so there's not much to worry about unless you're really in danger of using the entire 2 GB of address space the OS grants your process.
Packages are DLLs, they need to be load into memory to work. And each package will contains the code for all the units it is built from. Thereby they can use more memory than an exe built withoyt run-time packages - although your increase looks a bit too large. On the other side, if more than your application use the same packages and they are properly installed, their code will be loaded once into memory.
You could build ad-hoc packages, but you should be very careful to use different names from the standard ones or you could break some other applications, especially if you put your packages in a shared location or in a directory that comes first in the path.
Before trying them, I will check that your application is not linking unused packages. Delphi will put in the options more or less all the packages it knows. You can check after a compile which packages are really used, and add them only to the package list to be used.

FastShareMem still necessary in Delphi-2010?

Up until now I am developing using Delphi 7. In order to pass f.e. TStringLists to my DLL's I use the FastShareMem unit as first unit in every program and dll I develop.
If I should migrate to Delphi-2010, Does FastShareMem still necessary ?
Thanks for any insight you may provide.
Short answer: No, SimpleShareMem comes with Delphi 2010
Long answer: Yes, Delphi still has its own memory manager and memory claimed from one memory manager (exe) can not be returned to another (dll). But since Delphi 2006 Delphi comes with a new memory manager called FastMM which can do the same as FastShareMem and also does not require any extra dlls to be distributed. You need to use a unit called SimpleShareMem. FastMM also has other nice features you might want to check out. FastMM is also available for Delphi 7 BTW.
You don't need to use any of those tricks if you compile with runtime packages, since the memory manager is then shared. It also comes with the advantage of sharing the same types. No more TFont can not be assigned to TFont problems. Of course this does mean you have to distribute the runtime packages.
No, use SimpleShareMem instead as first unit in your Application and DLL.
Delphi 2007 and above include now FastMM as default memory manager, which used by SimpleShareMem and no need to distribute any DLL with your application.

Plugins system for Delphi application - bpl vs dll?

I'm writing delphi app which should have capability of loading plugins. I'm using JvPluginManager as plugin system/manager ;) Now, in the new plugin wizard they say it's better to use .bpl type plugins instead of .dll plugins ... What are the pros of this solution versus dll type plugins?
So far I've found only cons of this solution:
I have to put all the common interface units in separate package so that while loading plugins it won't throw any error about the other package containing common unit
if, let's say, one of plugin developers decides to uses some well-known unit (like synapse), which doesn't have runtime package by default, and the second plugin developer does the same, than bump... it's crash here ...
So, what are actually the pros of using bpls instead of dlls compiled with runtime packages?
Thanks in advance
Another disadvantage to BPL's. When you switch Delphi versions you will have to re-distribute new plugins. After many attempts at attempting to find the perfect plugin system, I ended up with COM and I have never regretted that decision. In a commercial application which has had the plugin requirement for over 8 years, the application has continued to move forward and yet some of the plugins that were released with the first iteration still exist in their ORIGINAL form.
If you choose this method, do yourself a favor and start with a simple interface and then add new interfaces upon it. You don't want to ever change your base interface, so keep it simple and sweet.
As Alexander said, a BPL is basically a DLL. But there are some conditions (from a not-so-short summary I made: http://wiki.freepascal.org/packages ):
A unit can only exist once in BPL's +Exe. This avoids duplication of state (twice the heapmanager and other global vars of system etc, VMT tables etc)
.BPL's can only USE other .BPLs.
This means that dynamic types like ansistring and IS/AS work properly over BPL interfaces.
Initialization/finalization are separate procedure and their initialization order is strictly controlled. For static dynamic loading this is simpler, for dynamic loading (plugin-like) all dependancies on units are checked .
Everything is essentially one big program, which means that the BPL's must be compiled with the same compiler version and RTL and depends on the versions other dependancies. It might be harder to make .BPL's to plugin to an existing EXE, since the Delphi version must match.
This also means that you must deliver .dcp's for (non Delphi) .BPLs the plugin .BPLs depend on
In short: if the plugin architecture is open, make it a DLL. Otherwise people have to have the exact same Delphi version to write plugins.
Hybrid is also possible. An higher level .BPL interface for functionality you factor out into .BPL yourself and selected devels, and a rock bottom procedure DLL interface for the rest.
A third option is using DLLs, but ordain Sharemem. Strings will work, multiple Delphi versions will work. Objects can work but are unsafe (e.g. I guess e.g. D2009 with an earlier version wouldn't work). Even other language users might be able to allocate over COM, not entirely excluding non Delphi.
Your first con is also a pro. If you replicate shared code in each dll the dlls get bigger and bigger. Even when using dlls you can prevent this by moving shared code in a separate dll.
Pros:
Types are shared. No TFont is not a TFont problem
Memory manager is shared. Strings and classes can be used as parameter between plugins without problems.
Cons:
Plugins can be built using Delphi or BCB only.
Plugins should use the same Delphi or BCB version.
Have you considerd using COM? COM makes it possible to share types, strings and classes and the plugins can be written in many programming languages.
I'm not familiar of JvPluginManager, but it depends on how you're going to use BPLs.
Basically, BPL - is just a usual DLL, but its initialization/finalization work is stripped from DllMain to separate functions: 'Initialize'/'Finalize'.
So, if you're going to use BPL like usual DLL, there are no cons that I'm aware of, only pros: there will be no more troubles with DllMain. That's all. The only difference.
But BPL in Delphi also provide a convient way to share code. This means great advantages (common memory manager, no duplicated code, etc, etc). So usual BPL does a lot more than "being just a DLL".
But this also means, that now your plugin system is limited to Delphi only (well, may be C++ Builder too). I.e. both plugins and exe MUST be compiled in the very same compiler to run smoothly.
If this is acceptable for you (i.e. no MS Visual Studio, no, sir, never) - then go ahead, you can use all power of BPLs.
P.S. But upgrading such BPLs plugins can be also a nightmare, if you do not design interface side carefully. In certain worst cases, you may need to recompile everything.
P.P.S. Like I said: I have no idea, how it is applied to plugins, created by JvPluginManager.
Avoid blp approach as you will have to ship a big bag of bpl with you software and thus, distribution will become bulky.
why do we use Delphi to compile small stand alone programs that just RUN anywhere without any runtime dependency. Using bpls means defeating this very purpose.
I don't know as to how comfortable you are with DLLs, but I would suggest you to use DLLs.
This will give other developers (who
may get interested in your software)
a chance to use any development
language (as long as that language
can spit out dll) to write their own
plugins that can be used in your
developed software.
Another thing is that you will be
saved from Delphi's vcl version
dependency tyranny. A major weak
point of Delphi till date.

Resources