What is the best way to implement an external module system for a DELPHI application?
What I need is very basic really:
The main APP detects if a module is present and loads it(Run time)
Modules can store form
Modules can store DataModules
Modules can Store code
I need the use the store forms inside other forms, and only as a standAlone
I use something like this
if Assigned(pNewClass) then begin
Application.CreateForm(pNewClass, _lFrm);
_lFrm.Hide;
_lFrm.BorderStyle := bsNone;
_lFrm.Parent := pBasePNL //(TPanel);
_lFrm.Align := alClient;
end;
So I create a TForm, but place it inside a TPanel.
As for DataModules I usally store ImageLists so the ideia is to change the app ICOs just bit changing the external module.
So what is the best way to achieve this?
Looked at runtime BPLs but don’t seem to understand how to do it.
Thanks.
UPDATE : .....................................
After reading some related question and answers i think I found my answer and solution.
http://edn.embarcadero.com/article/27178
The article is old stuff but amazingly simple.
Well the logic is there I just don’t seem to get it to Show the forms
I am just testing example 2
It loads the BPL, but doesn’t get the Form:
AClass := GetClass('TForm2');
Always retrievex ‘nil’
But the BPL as it registered:
RegisterClass(TForm2);
Can anyone help with this one.
Packages are an easy solution but they have one huge drawback. Using packages forces plugin authors to use not only Delphi, but the same version of the compiler as you do.
I personally would prefer to expose the functionality of the app through a number of interfaces. This allows accessibility from languages other than Delphi.
Typically the plugin would be implemented in a DLL and would export a function that the app would call to pass in the root interface representing the app. The plugin would then call methods of that interface thus establishing two-way interaction.
I made you a demo, it's so easy to get started! However... Started is not finished.
Every time I started using plugins, I later regretted it. However, as you say, you want a binary plugin system. So BPLs are the correct solution. David suggests using interfaces (with plain DLLs instead of the full runtime package BPL) and this would solve some of the commonly encountered BPL-instability problems due to not versioning your classes, and thus your Application-and-package-binary-compatibility dependencies, properly. If you do not need to share memory and do not need to use borlandmm.dll (shared memory management) then a straight DLL with interfaces will work fine.
If you can do everything you need to do with just scripting, and make your plugin system with just scripts, then do it that way. If you can get away with DLLs, interfaces, and no memory sharing, then use plain DLLs. If you must share Memory and class types, then yes, use BPLs.
Beware that using BPLs (Runtime Packages) comes with a lot of side effects you might not have been expecting. The largest BPL-based applications I have worked on have been more of a mess, and less stable, than any monolithic apps I have worked on. It seems that it is only possible to pine for packages until you try them, and then I find, I pine for monoliths again.
If you use BPL packages correctly, and you version your plugins properly, everything is fine. BPLs are great. But in the real world, it seems that sane versioning and ABI compatibility and interoperability, and stability are at a premium.
Update: I made you a demo it's here (plugin_r2.zip). It was done in Delphi XE, but if you are using an older version of delphi, you just delete the .dproj files and open the .dpr file for the main app, and the .dpk for the package.
Related
So, I have a myself written run-time package. If a package is statically linked, the project that uses has full access to exported data because the compiler has full knowledge of what is imported from it, am I right? But it's also possible to load a package dynamically via LoadPackage(). But, how to work with imported complex data structures like classes then? I couldn't find a feasible way other than constructing complex expressions like using FindClass('TSomeClass') and invoking RTTI to operate on an instance of the imported class.
The compiler has full knowledge of what's in the package because the DCU and DCP files tell it what's there.
The IDE knows what's in the package because it knows how to find the Register procedure in all the units, and that procedure tells the IDE about the available classes.
In most cases, a program knows what's in a package because the program used units from that package, and the compiler assured that mentioning names of things in those units would resolve to corresponding things in the BPL file at run time. This includes mentioning the BPL file in the program's import table, so the OS loads the BPL automatically.
If the list of BPLs you wish to load can only be determined at run time, then you cannot use any units from those packages. You have to load the package dynamically.
There's still the matter of how to use what's in those packages. You could try to discover the entire contents with RTTI. That's no picnic, though. Instead, define an intermediary package that all involved modules will use.
Define an interface or a common base class for all your packages' classes to have. Put the definition of that class in a unit that's in its own package, which we'll call Shared.bpl. Include that package in the "requires" list of all your other packages and your EXE. Now, everything can refer to the shared unit and the common base class.
This is exactly what Delphi itself does. The shared packages are called RTL and VCL. There are several common base classes already defined there, including TComponent. In your case, it sounds like you need some common definitions beyond what TComponent has.
Short answer:
What you need is the compiler/linker to set you up and use DCPs to do all the linking to types and such.
Then loading of BPLs should be delayed/done by you in custom code.
Unfortunately Delphi won't allow this, probably because of political reasons, you could try and hack it though, see below for more thorough answer.
Long answer:
Apperently DCP describe to the compiler/linker/Delphi yadayadayada what's in those packages/DLLs/BPLs type-wise.
These DCPs can be considered the interface to these DLLs/BPLs, these DCPs are probably somehow compiled into the executable.
Then the BPL probably contains the "implementation".
Now here is where the problem begins. These BPLs are "auto-loaded" as somebody else already mentioned by mentioning "import table".
What you could try is "nuking" / altering the import table so that these BPLs are not loaded automatically anymore.
Then you could try loading these BPLs manually.
I am not sure if it's just as simple as doing a "load package" operation.
Perhaps the loading involves more like acquiring pointers to routines/methods/classes, not sure how that works.
However perhaps that code can be re-used, so all you need to do is "hack" into the import table and hack into the "load bpl" and disable it.
Then you should be able to replace that with your own custom loading code and perhaps finally "re-patch" into other importing routines... like routines calling getprocaddress and such if so required, not sure about this last part.
Anyway this is very sloppy from Delphi developers that there is no functionality to do this little "import table"/"load step" yourself.
This is politics at play though, there is no technical reason why this could not be delayed, done custom by you and then call the rest if necessary.
For some reason they do not want you loading these BPLs manually. They probably want you to keep using the IDE this way.
If you were able to load these things manually, maybe the IDE would then no longer be necessary.
Currently the IDE is necessary to specify where to load this stuff from, without the IDE it will get difficult. Though there are probably also some compiler options somewhere to specify the same, though few would use that.
So to me it seems some very weird "tie-in" into this product, which I honestly must admit is pretty fucking retarded.
They did the same with winsock for example which has two different versions, very easy to load this yourself manually so you can choose which version to use instead of always v1 or v2. For easy for them to delay these getprocs and such, yet they do not...
If they did, Delphi would be much more backwards compatible with windows 95/98, currently Delphi exes no longer works for those older operating systems, though there is no real technical reason why it could not work, it basically has to do with which DLLs are loaded, just a few lines of code could very easily make it work.
This is probably again politics as play to make Delphi only support the latest Windows versions a deal struck between Delphi creators and Microsoft.
Same thing with Windows 10, it apperently detects older processors and then refuses to run on them, removing/hacking these few lines of code/instructions will make Windows 10 work on older processors. Surprise ! =D
I am working in a delphi IDE expert , now to avoid dependencies problems, i was thinking in rebuild this expert as a dll expert as was suggested in one of these answers, now my expert (compiled as bpl) access the Screen and Application global variables (instances of the Delphi IDE), so i was wondering if i compile my expert as a dll i can still accesing these variables and also i want to know which are the main differences between a bpl delphi expert and a dll delphi expert?
Should I compile my wizard as a DLL or a Package? Packages are easier to
load and unload without restarting the
IDE (and hence easier to debug), but
they can create unit naming conflicts
in the IDE. Conflicts happen when the
name a wizard's unit matches the name
of a unit in another loaded
design-time package. In this case,
both packages can not be loaded at the
same time. The recommended workaround
is to prefix all of your unit names
with a "unique" prefix. GExperts, for
example, uses "GX_" as the name prefix
for its units.
From this very good source about OTA: GExperts
When you access a global variable those would be global variables that are global to your DLL, not global to the main BDS.exe. I am not sure but I think your DLL would have its own Screen and Application global variable, if you linked in Forms, and the core of the VCL.
Those things which belong to the IDE itself are accessed through the Open Tools Api (OTA). I believe that you do not normally share any objects between the IDE your expert anyways, and if you were to try to do so, it would be problematic. Anything at all that you do that bypasses the OTA is going to be vulnerable to breaking in strange ways, especially in future versions of the IDE.
Dependency problems are of course a big reason to not use BPL based packages, but I think an even bigger reason is to maintain a complete separation between your tool's internals, and the internals of the IDE.
Remember that a DLL target, like an executable target, is statically linked. That is the core of the difference. If your expert provides functionality that uses the legal public documented OTA interfaces only, then moving to a DLL should be problem free. If you use some back door hacks that are possible with BPLs, then I can't advise you further.
Is it good idea to put Forms that have complete functionality in dll.
And main app will invoke dll function that returns form object.
The accepted way to do this in Delphi is to use packages rather than DLLs.
Packages are essentially DLLs but with Delphi specific capabilities that allow VCL objects to be used across package boundaries.
Trying to do this with DLLs will lead to a variety of problems that packages deal with. One downside of packages is that all modules must be compiled with the same version of Delphi. But if you are wanting to share objects across module boundaries then you would face the same restriction if you used DLLs.
The Delphi documentation has extensive coverage of packages.
Having said all that, I would add that if you can put all your code into a single module (.exe or .dll) then it does make life a lot simpler.
Adding to the answers about using packages:
Packages can only be used if both, the main app and all dlls (plugins) are written in Delphi and are written using the same version of Delphi
DLLs can be written in any programming language that can create them and can be used by any program regardless of the programming language
So, using dlls rather than packages does make sense.
Regarding the actual question: Yes, it is possible to put forms into dlls and they usually work fine. Just make sure that you do not pass them around because they are only valid objects within the context of the dll. You will experience the odd problem with forms losing focus or coming up behind other forms. This can usually be fixed by passing a window handle from the main executable to the dll which is then used as the parent for the form.
Also note: TObject of your dll is different from TObject of your application. The same applies to other commonly used classes and variables like (Forms.)Application.
I have done it and it was a pain in the lower back but it was not impossible. The main program was written in Visual Basic 6, some modules were written in Delphi 6, others were written in Delphi 7 and Delphi 2007.
Conclusion: If you are sure you will never use something different than Delphi for your app and for your dlls (plugins) and are willing to always recompile everything when you switch Delphi versions, you should use packages. Otherwise it might be better to use regular dlls. (And are you sure you will always be the only person writing these dlls? Maybe at some time there will be a 3rd party developer for one of the dlls who does not own the Delphi version he needs.)
IMO this is sometimes a very good idea and the only way to go - for the reasons others have mentioned, I'm not a fan of packages, and am very comfortable with DLL's. I am currently adding functionality to an app written in Delphi 5 using Delphi XE - it was either use DLL's or write in D5 - of course I opted for the former: D5 app calls DLL's written in XE that contain all the latest and greatest features. (The first projects I did in Delphi were done via the old Borland Paradox - Paradox app invoked DLL's written in Delphi 1!)
But, I don't send the form or module from the DLL back to the main app - I just send the DLL module a structure containing what it needs to know to do its work, and when it's done and the DLL's form closes, it cleans up and then and returns a numerical code or structure back to caller indicating success, failure etc ( old fashioned but very effective).
Passing the form instance from the DLL back to your main app across the DLL threshhold can be problematic - note #dummzeuch's excellent answer above with some good tips on how to negotiate some of those problems should you decide that is your only solution.
+1 for everything that David Heffernan says.
Strategically, you really only need to implement forms (or other functionality) in external files if you're implementing a plug in system.
If you're going to allow plugins to be authored in any language, then DLL's are the only way to go.
If your plugin system will be restricted to developers with the same version of Delphi (same team perhaps?) then go with BPL's. The additional drawback of Delphi packages, from my perspective, is the need to deploy the VCL BPL's with your app, which are always more Mb than a single compiled module.
If on the other hand you want to write a modular system, you can still do that by implementing loose coupling & "plugin" techniques within your code and still compile to a single module.
If you put a form in a normal dll the form won't be able to intercept the TAB or arrow keys. I have been told that this is due to the OnKeyDown not be passed through.
I'm writing delphi app which should have capability of loading plugins. I'm using JvPluginManager as plugin system/manager ;) Now, in the new plugin wizard they say it's better to use .bpl type plugins instead of .dll plugins ... What are the pros of this solution versus dll type plugins?
So far I've found only cons of this solution:
I have to put all the common interface units in separate package so that while loading plugins it won't throw any error about the other package containing common unit
if, let's say, one of plugin developers decides to uses some well-known unit (like synapse), which doesn't have runtime package by default, and the second plugin developer does the same, than bump... it's crash here ...
So, what are actually the pros of using bpls instead of dlls compiled with runtime packages?
Thanks in advance
Another disadvantage to BPL's. When you switch Delphi versions you will have to re-distribute new plugins. After many attempts at attempting to find the perfect plugin system, I ended up with COM and I have never regretted that decision. In a commercial application which has had the plugin requirement for over 8 years, the application has continued to move forward and yet some of the plugins that were released with the first iteration still exist in their ORIGINAL form.
If you choose this method, do yourself a favor and start with a simple interface and then add new interfaces upon it. You don't want to ever change your base interface, so keep it simple and sweet.
As Alexander said, a BPL is basically a DLL. But there are some conditions (from a not-so-short summary I made: http://wiki.freepascal.org/packages ):
A unit can only exist once in BPL's +Exe. This avoids duplication of state (twice the heapmanager and other global vars of system etc, VMT tables etc)
.BPL's can only USE other .BPLs.
This means that dynamic types like ansistring and IS/AS work properly over BPL interfaces.
Initialization/finalization are separate procedure and their initialization order is strictly controlled. For static dynamic loading this is simpler, for dynamic loading (plugin-like) all dependancies on units are checked .
Everything is essentially one big program, which means that the BPL's must be compiled with the same compiler version and RTL and depends on the versions other dependancies. It might be harder to make .BPL's to plugin to an existing EXE, since the Delphi version must match.
This also means that you must deliver .dcp's for (non Delphi) .BPLs the plugin .BPLs depend on
In short: if the plugin architecture is open, make it a DLL. Otherwise people have to have the exact same Delphi version to write plugins.
Hybrid is also possible. An higher level .BPL interface for functionality you factor out into .BPL yourself and selected devels, and a rock bottom procedure DLL interface for the rest.
A third option is using DLLs, but ordain Sharemem. Strings will work, multiple Delphi versions will work. Objects can work but are unsafe (e.g. I guess e.g. D2009 with an earlier version wouldn't work). Even other language users might be able to allocate over COM, not entirely excluding non Delphi.
Your first con is also a pro. If you replicate shared code in each dll the dlls get bigger and bigger. Even when using dlls you can prevent this by moving shared code in a separate dll.
Pros:
Types are shared. No TFont is not a TFont problem
Memory manager is shared. Strings and classes can be used as parameter between plugins without problems.
Cons:
Plugins can be built using Delphi or BCB only.
Plugins should use the same Delphi or BCB version.
Have you considerd using COM? COM makes it possible to share types, strings and classes and the plugins can be written in many programming languages.
I'm not familiar of JvPluginManager, but it depends on how you're going to use BPLs.
Basically, BPL - is just a usual DLL, but its initialization/finalization work is stripped from DllMain to separate functions: 'Initialize'/'Finalize'.
So, if you're going to use BPL like usual DLL, there are no cons that I'm aware of, only pros: there will be no more troubles with DllMain. That's all. The only difference.
But BPL in Delphi also provide a convient way to share code. This means great advantages (common memory manager, no duplicated code, etc, etc). So usual BPL does a lot more than "being just a DLL".
But this also means, that now your plugin system is limited to Delphi only (well, may be C++ Builder too). I.e. both plugins and exe MUST be compiled in the very same compiler to run smoothly.
If this is acceptable for you (i.e. no MS Visual Studio, no, sir, never) - then go ahead, you can use all power of BPLs.
P.S. But upgrading such BPLs plugins can be also a nightmare, if you do not design interface side carefully. In certain worst cases, you may need to recompile everything.
P.P.S. Like I said: I have no idea, how it is applied to plugins, created by JvPluginManager.
Avoid blp approach as you will have to ship a big bag of bpl with you software and thus, distribution will become bulky.
why do we use Delphi to compile small stand alone programs that just RUN anywhere without any runtime dependency. Using bpls means defeating this very purpose.
I don't know as to how comfortable you are with DLLs, but I would suggest you to use DLLs.
This will give other developers (who
may get interested in your software)
a chance to use any development
language (as long as that language
can spit out dll) to write their own
plugins that can be used in your
developed software.
Another thing is that you will be
saved from Delphi's vcl version
dependency tyranny. A major weak
point of Delphi till date.
I am looking to add the capability for users to write plugins to the program I have developed in Delphi. The program is a single executable with no DLLs used.
This would allow the user community to write extensions to my program to access the internal data and add capabilities that they may find useful.
I've seen the post at: Suggestions for Adding Plugin Capability? but its answers don't seem transferrable to a Delphi program.
I would like, if possible, to add this capability and keep my application as a single executable without any DLLs or additional modules required.
Do you know of any resources, components or articles that would suggest how to best do this in Delphi, or do you have your own recommendation?
The first question I would ask is, do you need the plugins to access the UI of your host application, or add any UI elements of their own? Or will the plugins be limited to querying and/or supplying data to your host app?
The latter is much easier and opens up two possibilities. Others have already mentioned DLLs, which is the first way to go. Certain caveats apply - in general you should be interfacing with a dll using only the data types that are used in Windows API. THat way you can be sure that the plugin DLLs will understand your data types, no matter what language/compiler they were created in. (So for example, use PChars, not strings. Do not as a rule pass Delphi classes such as TStream to a DLL. This will appear to work in some cases, but is unsafe in general, because even if the DLL was compiled in Delphi, it may have been a different version of the compiler, with a slightly different idea of what TStream is). Google for using DLLs in Delphi, and you'll find plenty more tips.
Another way that hasn't been mentioned yet is to enable scripting in your application itself. There are several very capable 3rd-party scripting engines, both commercial and free, and most of them allow you to exchange Delphi objects with the script. Some of these libraries support only Pascal as the scripting language, others will let you use Basic (perhaps better for beginner users) or other languages. See for example RemObjects Pascal Script (free) at http://www.remobjects.com/free.aspx .
My favorite scripting solution at the moment is Python for Delphi (P4D, also free) from http://mmm-experts.com/Products.aspx?ProductID=3 It can interface beautifully with your classes via RTTI, and allows you to execute Python code in your Delphi app, as well as use Delphi classes in Python scripts. Given the popularity of Python, this may be a viable solution if you want to attract developers to your project. However, every user will need to have a Python distribution installed.
It seems to me that the barrier to entry, from the point of view of potential plugin writers, is lower if you use scripting than if you choose DLLs.
Now, back to my initial question: things get much more complicated if you need the plugins to interact with your user interface, e.g. by placing controls on it. In general, DLLs cannot be used to do this. The Borland/CodeGear-sanctioned way is to use packages (BPLs). With BPLs, you can access and instantiate classes offered by a plugin as if they were declared in your host application. The catch is, all BPLs must be compiled with the same exact version and build of Delphi that your main application is. In my opinion this makes packages completely impractical, since it's hard to expect that all the potential plugin writers around the world will be using the same version of Delphi you are. A major pitfall.
To get around this, I have experimented with a different approach: keep using DLLs, and develop a syntax for the plugin to describe the UI it needs, then create the UI yourself in the host app. (XML is a convenient way to express the UI, since you get the concept of parenting / nesting for free.) The UI description syntax can include callbacks to the DLL triggered when the contents or state of the control changes. This method will restrict plugins to the set of VCL controls your application already uses or has registered, though. And it's not a one-nighter job, while BPLs certainly are.
Google for "Delphi plugin framework", too. There are some ready-made solutions, but as far as I know they usually use BPLs, with their limited usefulness.
Actually, the accepted answer to the question you cite is quite appropriate for Delphi as well. Your plug-ins will be DLLs, and you can dictate that they should export a function with a certain name and signature. Then, your program will load the DLL (with LoadLibrary) and get the address of the function (with GetProcAddress). If the DLL doesn't load, or the function isn't there, then the DLL is not a plug-in for your application.
After you have the address for the DLL, you can call it. You can pass the function an interface to something that represents the parts of your application that you wish to expose. You can also require that the function return an interface with methods that your application will call at various times.
When testing your plug-in system, it will be wise to write a plug-in yourself with a language other than Delphi. That way, you can be more confident that you haven't inadvertently required everyone to use Delphi.
At first I went for BPL and DLL base plugins. And found them hard to mantain.
If you use BPL system, then you need to match BPL version with EXE version. This includes Delphi updates which can break something. I found out (the hard way) that if I need to include all my plugins with every release, there is no point in having plugins at all.
Then I switched to plain DLL plugins. But that system just complicated code base. And that's not a good thing.
While crusing the net I found out Lua embedded script language, and delivered with it. Lua is 150K DLL, embedding bytecode compiler, interpreter and very simple and smart dynamic programing language.
My plugins are simple lua scripts. Easily mantaind and made. There are premade Delphi examples, so you can export any class or procedure as Lua table or function. GUI or not. For example I had TurboPower Abbrevia in my app for zipping. I exported my zipping class to lua, and all plugins now can call zip('.', 'dir.zip') and unzip(). Then I switched to 7zip and only implemented old class to use 7zip. All plugins work as they did, with support for new zip('.', 'dir.7z').
I made TLuaAction which calls Execute(), Update(), Hint() procedure from its script.
Lua allso have it's own plugin system that makes it easy to add funcionality to it. Eg luacom make is easy to use COM automation, luainterface allows calling .net from lua.
See luaforge for more. There is Lua IDE made in Delphi, with source.
I tried to make an overview of all such options some time ago. Together with my readers/commenters we built this list:
DLL/BPL loaded from the program.
DLL/BPL loaded from a sandbox (which could be another copy of the program or a specialized "server" application and which communicates with the main program via messages/sockets/msmq/named pipes/mailslots/memory mapped files).
COM (in any of its variants).
DDE (please not).
External program that communicates via stanard input/output.
External program that communicates via files (file name is a parameter to the program).
External program that works via drop folder (useful for batch processing).
External program that communicates with windows messages/windows sockets/msmq/named pipes/mailslots/memory mapped files/database publish-subscribe.
The most universal method of adding plug-in capability is to use COM. A good book to get you started on the road is Delphi Com Programming by Eric Harmon. While it was originally written for Delphi versions 3 through 5, the books contents are still valid with the latest versions of Delphi.
Personally, I have used this technique along with active scripting to allow end user customization.
I'm using plug-ins to implement most of the functionality in the game engine I'm building. The main EXE is made up of a script engine, a plug-in manager, some basic graphics routines and not much else. I'm using TJvPluginManager from the JEDI VCL library. It's a very good manager, and it can add just about anything you want to your program. Check out the demos included with the package to see how it works. The only downside is that it adds a lot of JCL/JVCL code to your program, but that's not really an issue if you're already using other JVCL components.
If plugins will be developed in Delphi or C++ builder, use packages + interfaces. Delphi OTA is a good example for that.
If plugins will be language independent, COM is a good way to go.
Addition: If you won't use COM, u may need to provide SDKs for each language. And datatype handling between different languages can be pain(for example delphi string type). Delphi COM support is excellent, u don't need to worry about kind of details. Those are mostly impilicit with Delphi COM support. Don't try to invent the wheel again. I'm surprised why people doesn't tend to mention about it.
I found an article by Tim Sullivan:
Implementing Plug-Ins for Your Delphi Applications
You could have a look at Hydra from Remobjects. Not only will this allow you to add plugins, but also to mix win32 and .net.