For over a decade, I have been stuck (lovingly) on Delphi 6 and have developed hundred of thousands of lines of code in a logical (to me) unit structure, where a project tends to be a few hundred lines of code referencing high-level work routines in my "library". In trying to migrate to XE5, I just can't find a way to have all my library units compiled in one place on the search path, and then just used by the project (and recompiled if necessary), but that the dcus are off with the library source rather than in with each individual project.
I'm just starting to accept that "hello world" takes 2.5Mb in XE5, and I can't stand the idea that each library unit has to be separately compiled into dcus at the project level. In the "old" days, these unit dcus would sit next to the pas files and not be recompiled if nothing changed in the source file.
The obvious place to look is the Project options, but I can't find the right setting to make the project stop keeping copies of each dcu.
I am vaguely aware that multi-platform development will cause restructuring, but I can't help feeling that there is some compromise position.
There must be something big I am missing.
Starting in Delphi XE2, Delphi supports compilation for multiple platforms, as well as different build configurations. Because of this, Delphi needs to create DCU files for each combination. For example, Win32, Win64, and OS-X DCU files are saved in separate folders by default. Otherwise, if it weren't like this, the DCU files would overwrite each other, which you should avoid (if you use different configurations/platforms).
These settings can be changed in the Project Options in the very first section Delphi Compiler by modifying the Unit output directory. This is by default .\$(Platform)\$(Config) which creates a subfolder for the platform, then another subfolder for the config, for example \Win32\Debug\. Careful for the Target at the very top, which by default is set to your current platform/config. You would typically want to first change it to All Configurations. If you clear this field completely from the options, it will produce your default behavior from older versions.
It sounds like you should create a Package. This would allow you to group all your "library" units together in one place (BPL). This package can then be installed into your IDE, and if you have any components, those components can then be installed into your component pallet.
Or you can do without a package too. All the units from all these different projects should be moved to this central place though - a single folder containing all your "library" units. This way it's less maintenance, and you can just add that one folder to your global library path.
If you put your files in a central folder, and use those files from a project, the DCU files for both the project and this "library" will be saved for that project. Delphi doesn't know that these files are a "library", it just knows that you're using them, and since it can't find an already compiled version of those units, it creates one in your project. If you want the DCU files to be saved only once and in this central place, then you would need a package.
First, let me thank all the respondents to this question - all provided useful insight. I experimented with the various suggestions (including breaking XE5 so badly that I had to reinstall -- at least I learned some areas not to mess with.)
Important to me, but a known bad coding practice, is having individual projects edit shared library units (only my own units - I do not mess with code belonging to Delphi or 3-rd party). This is critical to having multiple applications working on the same data, but in bite-sized pieces. the shared code lets me make high-level pieces of an app available to other projects. There might be better ways (I would love to hear about them), but this has worked for me for a long time.
The multi-platform model really requires the dcu structure used by default, so I will adapt to it. Share the source code, but accept multiple compilations to individual projects. A good suggestion by JensG is just to clean up the dcus when the project is not actively being worked on. Should be straight-forward utility program.
The D6 -> XE5 migration (which will take months for some of the less used areas) requires me to know which units compile successfully, so I will maintain one project whose function is to include all units and recompile them all. This will make it practical to map library unit pas files to dcu files.
The AnsiString/AnsiChar <-> String/Char problem is the major migration problem area. Simply making edit-level changes may get the code past the compiler, but there is no guarantee that the code still works the same way. Especially troubling is at interface points to Windows calls and such. My answer will be to make the units compilable first, but then write test code for trouble areas. This is what will take the months - I need to get on with new stuff, as well as fixing the old. I REALLY don't know yet if I will be able to substitute the XE5 compatible code back into Delphi 6 without another layer of testing. I THINK it should work, but it will take careful checking.
A second major migration problem is 3rd party code such as VCLZip. XE5 has its own zip support, but I have a lot of places where I use VCLzip and the conversion will not be trivial. For this specific library, it may be possible to find XE5-level source and simply work it in. There are other pieces of code gotten from the internet that I used, but never needed to truly understand which will cause significant hassles.
Again - thank you to all. This has been an interesting 24 hours. Howard
Related
I have this craving to do some experiments with modifying the underbelly of the Delphi run time library (RTL), system.pas and the likes... It is possible or not?
I'm very fond of challenges like "yes, but you'll have to provide custom .obj files for some assembler wizardry because they were never distributed with the official Delphi source". Fine with me, I just want to know.
I want to do this experiment with Delphi 7, but inside information on any other version is fine. It is one of the perks of being with a company that worked with Delphi since the Stone Age.
(I always figured this to be one of those RTFM questions, with the answer being a resounding "NO!", but for some reason google won't confirm it.)
You can recompile the RTL like any other unit.
For System.pas you must use the command line compiler.
For instance, here is a working batch file content (there is some not well documented command line switches):
del *.dcu /s
"c:\program files\borland\delphi7\bin\dcc32.exe" -O+ -Q -M -Y -Z -$D+ System.pas
This will recompile System.pas and SysInit.pas (both lowest level RTL files).
But in order to use your recreated dcu files, you'll have to put the folder containing the updated dcu files into the first position of your IDE: for instance, in Delphi 7 it's Option / Environment Options / Library, then put your folder FIRST in both "Libary path" and "Browsing path" field.
And it's perhaps worth deleting the original .dcu files in your Delphi installation directory.
But be sure you won't change the "interface" part of the unit, or you'll have troubles with compiling with other not modified units of the RTL (or third-party components). You can change the "implementation" part, apply fixes or rewrite some part for speed or such, but don't change the "interface" part to avoid any linking error.
Always make a backup of the original .pas and .dcu files which you are changing. And it's a good idea to make some automated compilation test, so that you could be sure that your modifications of the RTL won't add any regression.
We made such a RTL recompilation for our Enhanced Run Time Library for better speed of low-level RTL functions (mostly System.pas and SysUtils.pas). Designed for Delphi 7 and 2007. For more recent Delphi version, you still can use the same principle.
You can only recompile the RTL from the command-line. There should be a makefile in the RTL source directory of your installation. It is designed to be used with the make.exe command-line utility which should be in the "bin" folder of your installation. I would recommend you copy the relevant sources to a separate location for experimentation. I must caution you that the System unit is tightly coupled with the compiler which expects many functions to have a specific name and have particular parameter lists, if any are even declared. Many RTL "helper" functions don't have any formally declared parameters, yet expect parameters to be passed in a certain fashion.
Another bit of caution is changing the interface declarations of certain classes, functions or types. Doing so may cause serious incompatibilities with existing DCU files and components. For this reason you must be very careful when intermixing DCU files from the included RTL or third-party components with your custom modified versions. I would suggest you start by only making implementation section changes only before venturing into the mine-field of interface breaking changes.
I look after a suite of over 100 delphi (XE2) projects which are currently all built without packages. Several executables and DLLs include some large third-party component sets which add several megabytes to their size. Our installer has gone up to over 100MB, with a few executables over 10MB, several around 4MB, and a lot between 1 and 2 MB. I am considering using packages for some of the third-party components and the VCL. In general, visual components are only used from the executables, not the DLLs. The DLLs must remain DLLs, because some are called from other languages.
What are the main considerations and dangers to be aware of if we switch to use packages? What might break? Will it affect the convenience of debugging, testing, building (with FinalBuilder), or maintenance? Will run-time memory utilization increase unacceptably, because we will be loading whole BPLs at run-time instead of just the units needed by the linker?
I am aware that the only true answer will come from trying it with our specific applications, but I would greatly appreciate some advice from anyone who has either been down this path, or turned away from it.
I avoid Runtime BPLs like the plague. Here are the reasons:
A. You need to think about versioning. What happens when someone updates ONE BPL or EXE and not the others.
B. You need to think about DLL hell. What is your path, and are all the pieces of your application in the same folder, or will they be in directories on the path?
C. You need to think about the possibility that one or more file will be missing at runtime. This usually results in a broken application. Some people actually use this to good effect (some customers who have not paid for feature X do not even receive that BPL) but mostly, this is just another source of trouble.
D. In the unlikely event that you need to fix a bug in the VCL sources, do you really want to add rebuilding the VCL BPLs to your list of Fun Maintenance Programming Activities?
E. In one large BPL-based application that I worked in, I found that the main EXE and all its plugin BPL sub-parts, opened in one project group resulted in Delphi (XE at the time) being substantially less stable (crashing more) than when I loaded similarly large, but single-executable projects. IDE issues are not unheard of, especially where dependencies between multiple items in a project group happen.
Whenever the notion of using packages comes up, I find that a good stiff drink, and an 8 hour break from programming usually suffices.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Reduce exe file
What are some tools that, given an EXE file, remove all unused code and make a new EXE file with code really used by the application? I think that something like that should exist. Just for curiosity; but I think that it can be really good as tools for producing a smaller EXE file without unused code.
P.S.: Delphi produces standalone EXE files that contain code of all object used in an application, but not all elements of this object are really used. This makes big files.
I remember the first version of Pascal that include only code really used and did not insert unused code, and EXE files were smaller.
Current Delphi also excludes unused code. You can easily see this in Delphi. If you compile, you get 'blue dots' in the gutter of your code. Functions that are not used don't have blue dots, meaning they are excluded. If you check the Optimization checkbox in project options, code is rewritten to make more use of registers thus eliminating certain variables.
Nevertheless, Delphi executables grow, especiall when using certain units. I think the smallest .exe you can create in Delphi 7 is about 10Kb. In later versions this will probably be a bit larger.
Important causes of file size is
resources. Large or lots of images consume a lot of space. If you use icons on multiple forms, make sure to put them in a central image list (on a shared datasource). Use one of the available PngImageList implementations for smaller image size against better quality.
rtti. The runtime type information causes class definitions to consume extra space. Thise space is partly due to the meta information about the class, but mainly because all extra code. Any methods that may be called using RTTI could be bound in a kind of 'late binding' fashion. Therefor, the compiler cannot know whether the methods can be eliminated, so it needs to include them in the executable.
registered classes. Similar to 2. If a class is registered, it can be fetched and instantiated using its name as a string. Those classes must be included in the project, even if they are never used, just because the compiler cannot know if they are needed.
It's a fact that RTTI is expanded in recent Delphi versions. I think this also causes the RTTI meta information about classes to consume more space. There's only so much you can do about that.
In general, I think the Delphi compiler still does a lot of optimizing. You shouldn't have to worry about exe file size. If you do, you could try a packer like upx. I've got good esperiences with upx. It cuts down the executables to about a fifth or less of their original size, while retaining all functionality.
If you application is huge, check if you don't have a lot of debug info compiled in.
Try a release build and check if it's any smaller.
What i sometimes do is configure the project to compile all .dcu files into a single folder. That way you quickly see all of the useless units that get compiled into your .exe. It often happens that you include a single unit for a single function, but in turn you get a whole tree of dependent units. You'll just have to search your uses clauses and try to get rid of these dependencies somehow.
I think both gexperts and cnpack contain tools to show dependencies, or to scan for unused units in your project. They can be helpful with this.
After you've removed useless dependencies, you can always compress your compiled exe with upx. There are supposed disadvantages to that (barry kelly wrote about it some time ago), but i've got good experiences with it. It sometimes makes the file 4x as small which can be a big deal.
The company I work for develops a system in Delphi, that contains dozens of exe modules, and each of them is identical to a certain degree if it comes to source code. Sadly, nobody has ever cared about using libraries to put the shared code in. This means that each time there is a bug fix to do in the code all these modules share, a programmer has to make corrections in all of them separately! It always takes so much time...
I decided to find a method to put the shared code into libraries. I considered DLLs and BPLs. In this case BPLs seemed much more programmer-friendly and much less troublesome, especially that the code is used only in our software and only in Delphi.
I put all the code shared by all the exe modules into BPLs and everything seems fine, but there are certain things I don't understand and would be grateful if you explained them to me.
What I expected after dividing the code into BPLs was that it would be enough to deploy exe files with the BPLs I created. But it turned out that they need an rtl100.bpl and vcl100.bpl as well. Why is it so? I want to deploy exes and my BPLs only. I don't want to provide end users with a whole bunch of libraries supplied by Borland and third party companies :). I want them to be compiled within exes as they used to be compiled before. Is it possible to do that?
What I did so far was:
I put all shared pas units to BPLs. Each BPL contains units belonging to the same category so it is clear for programmers what code to expect in a given BPL.
Each BPL is a "runtime and designtime" library.
Each BPL is "rebuilt explicitly".
The two latter are default project settings for BPLs.
And if it comes to the exe projects:
I deleted all units that I had earlier put to BPLs.
I installed my BPLs from the Tools->Install package menu in BDS 2006.
In my exe project settings I checked the option "build with runtime packages" and I listed all my BPL packages in the edit box below (only my packages, as I cleared all other ones that appeared there).
This is all I did. The exe projects compile properly, but I have no access to the source code of BPLs (I can't navigate into that code from my exe projects), even though all BPLs are stored together with their source code files. Why? It seems strange to me.
I always tend to write lengthy descriptions - sorry for that :). I will appreciate your help. I just need a few words of explanation to the points I mentioned: deploying exe with my BPLs only, the correctness of what I did as a whole, and the inability to navigate into BPL source codes. Thank you very much in advance!
Thank you all for the discussion. Some said the approach I chose was not a good idea. Our software consists of more than 100 modules (most of them being something like drivers for different devices). Most of them share the same code - in most cases classes. The problem is that those classes are not always put into separate, standalone pas units. I mean that the shared code is often put into units containing code specific to a module. This means that when you fix a bug in a shared class, it is not enough to copy the pas unit it is defined in into all software modules and recompile them. Unfortunately, you have to copy and paste the fixed pieces of code into each module, one by one, into a proper unit and class. This takes a lot of time and this is what I would like to eliminate, choosing a correct approach - please help me.
I thought that using BPLs would be a good solution, but it has some downsides, as some of you mentioned. The worst problem is that if each EXE needs several BPLs, our technical support people will have to know which EXE needs which BPLs and then provide end users with proper files. As long as we don't have a software updater, this will be a great deal for both our technicians and end user. They will certainly get lost and angry :-/.
Also compatibility issues may happen - if one BPL is shared by many EXEs, a modification of one BPL can bee good for one EXE and bad for some other ones - #Warren P.
What should I do then to make bug fixes quicker to make in so many projects? I think of one of the following approaches. If you have better ideas, please let me know.
Put shared code into separate and standalone pas units, so when there is a bug fix in one of them, it is enough to copy it to all projects (overwrite the old files) and recompile all of them.
This solution seems to be OK as far as a rearly modified code is concrened. But we also have pas units with general use functions and procedures, which often undrego modifications - we add new functions there whenever necessary, but in single projects. So imagine that you write a new function in one of the 100 modules and put it into its general use unit. After a month or two you modify a different module and you think you need the same function you wrote 2 months ago. You have to find the module (it's difficult if you don't remember which one it was) and copy the function to your code. And obviously - the general use units become completely different in each module as long as they are stored in each project separately. And then, if there is a bug fix to do... the whole story repeats.
Create BPLs for all the shared code, but link them into EXEs, so that EXEs are standalone.
For me it seems the best solution now, but there are several cons. If I do a bug fix in a BPL, each programmer will have to update the BPLs on their computer. What if they forget? But still, I think it is a minor problem. If we take care of informing each other about changes, everything should be fine.
#CodeInChaos: I don't know if I understood you properly. Do you mean sharing pas files between projects? How to do that? We store source codes in SVN. This means that we would have to store shared code in a separate folder and make all projects search for that code there, right? And download from the SVN a project and all folders it is dependent on...
Please, help me choose a good solution. I just don't want the company to lose much more time and money than necessary on bugfixes just because of a stupid approach to software development.
Thank you very much.
Even though this question has an accepted answer I'm going to take a stab at it.
The title asks how to divide a project into bpls but the real question appears to be:
"What's the best way to share code between projects?"
There are a few ways to do this:
Shared units
Dlls
BPLs
Regardless of which direction you go you will likely need to restructure your projects. From your description it sounds like each project is developed in relative isolation. Code is shared using copy/paste, which quickly gets out of sync and result in a lot of duplicated effort. So lets examine each of the techniques for sharing code.
Shared units
This is the most straightforward approach. You create a shared location and place code you would like to reuse among your projects into this location. The units are statically linked into your projects so you don't need to worry about deploying extra dependencies along with the main executables. Statically linked units are by far the easiest to troubleshoot and debug.
The compiler needs to be able to find your shared units. There are 4 ways to tell the compiler where to look.
Add them to the project - SHIFT+F11 - Adds a reference to the unit into the project files (dpr, dproj). The IDE will normally use relative paths if the unit is located under the same directory tree as the project files, otherwise it will use absolute paths, which can be problematic if developer machines aren't configured identically.
The project's Search Path - CTRL+SHIFT+F11 Delphi Compiler > Search path - Add a directory and the compiler will look there to find units mentioned in the uses clause of any unit in the project. Its best to use relative paths if you can. You can also use environment variables: $(MyPath)
Global Search Path - Tools > Options > Environment Options > Delphi Options > Library - Win32 > Library Path - Any paths listed here are available to all projects on a machine. This is machine dependant
Command line - If you build from a script or build automation tool you can set the search path using the dcc32's -U switch or msbuild's /property:UnitSearchPath= switch.
Options 1 and 2 will be the most useful.
As far as your SVN repository goes you have a few options for organizing the projects and shared units. The simplest would be to place all projects under single trunk along with the shared units:
Projects
trunk
ProjectA
ProjectB
ProjectC
Library (shared units)
If for some reason the above structure isn't possible you could try this alternative:
ProjectA
trunk
Library (branch of main library)
ProjectB
trunk
Library (branch of main library)
ProjectC
trunk
Library (branch of main library)
Library
trunk (main library)
In this configuration changes made to each project's library folder would not be immediately available to the other projects. Each project would need to synchronize changes with the main Library project on a regular basis. A side effect of this is that changes that break other projects will be delayed until the other projects are synchronized. Whether you consider this a good or bad thing depends. On the one hand bugs are easier and cheaper to fix when the code they involve is still fresh in the developer's mind. On the other hand if you don't practice unit testing (which I highly recommend you do) or the code is very fragile or you just have developers prone to making reckless changes you may want to control how frequently those changes get pushed into other projects.
Dlls
Dlls allow you to share code by linking to it at runtime. They expose functions that can be called from a main executable or another dll.
While dlls are always linked at runtime you decide whether they are loaded at application startup or only when needed. Loading at startup is called static loading and in Delphi is accomplished using the external directive. The vast majority of the rtl/vcl classes that wrap system api calls use static loading. Dynamic loading lets to delay the loading of a dll until it is required. This uses the WinAPI functions LoadLibrary and GetProcAddress. A corresponding call to FreeLibrary will unload a dll.
Unfortunately standard dlls limit what kind of datatypes can be passed. If you need to access a dll from non-Delphi projects you will need to limit yourself to using c style data types. If you will only be using a dll with Delphi projects you can safely use Delphi strings and dynamic arrays as well if you use the SharedMem unit in the dll and any projects that use it.
You can safely use object's within the dll without problems but if you want to pass objects between the dll and the application you'll need to extract the object's data and pass it as primitive types and reassemble it into an object on the other end. This is called (de)serialization or marshalling and there are much easier ways to do this than rolling your own.
COM (Component Object Model) is well supported in Delphi but it has a bit of a learning curve. Consuming COM objects is pretty straightforward but designing one will take time if you're not familiar with COM. COM has the advantage that it is language neutral and is supported in the majority of languages targeting the Windows platform (including languages targeting the .NET framework).
Bpls
Bpls (also called simply "packages") are specially formatted dlls that make working with objects a lot easier. Like standard dlls they are linked at runtime and can be statically or dynamically loaded. They are easier to learn and use than COM dlls and provide more seamles integration into your projects than COM. Packages are composed of two parts: the bpl and the dcp. The dcp is like the dcu files generated when you compile a normal unit file except it contains a whole bunch of units in it. Using a class that is compiled in a bpl is as simple as adding the dcp to the project's package list then adding a unit to a uses clause of one of the project's units.
When you deploy the app you'll need to install the bpl as well. As other's have noted you have to include the rtl package at a minimum and most likely the vcl package if you use any forms. There is a way around deploying Borland supplied bpls with your projects. You can create a "mini" rtl package that contains only the units your project need. The difficultly is in determining which units to include.
Summary
From the description you've given creating a library of shared unit files to statically link against may be the most expedient route. I would also suggest trying out a program called Simian. It will help you track down duplicate code in your code base for inclusion in your shared library. It doesn't directly support pascal but it does a decent enough job using the plain text parser with a little tweaking of its configuration.
Also I can't stress enough the value of unit testing. Especially if you're moving toward shared libraries. A suite of well written unit tests run on a frequent basis will give you instant feedback when a developer changes a class and it breaks an unrelated project.
Imagine you have a project with an EXE and two different BPL modules, and somewhere in that codebase, there's a line that says if MyObject is TStringList then DoSomething;. The is operator works by examining the object's class metadata, stored in the VMT, and then following a chain of VMTs through the ClassParent pointer, to see if any of them match the class reference (also a VMT pointer) for TStringList. In order to make sure that this will work correctly, there needs to be one single VMT for TStringList that's the same throughout your entire program, no matter how many BPLs it's divided up into, which means it has to be in its own package. That's why system runtimes like rtl*.bpl and vcl*.bpl are necessary, and there's not much you can do about that. It's part of the price of using BPLs.
As for not being able to debug, you need to make sure that the BPLs are built with debug info enabled and that the debugger knows how to find the folder where the DCP (the file containing the debug info for the BPL) is located. And you won't be able to trace into system BPLs, because debug-enabled DCPs weren't shipped with your version. They were added pretty recently, I think in XE but it might have been in D2010.
Why can't I browse my source code? Is there a way to fix this?
You can not browse the source code of the units included in the packages because they are neither in your project, your library or search path.
The way I solve this is adding the directories to the project search path. This way the compiler does not know about those files (and does not try to recompile them) but the IDE let's you browse their content and debug into them.
"In my exe project settings I checked the option "build with runtime packages"
That is why you cannot deploy without the BPL's etc - this option is confusing for a lot of developers -"build with runtime packages" means that you will need the bpl's present at runtime. Uncheck that option and the packages will be linked into your exe at compileTime. (Your exe will g-r-o-w in size.) The idea behind the "build with runtime packages" is to keep the size of exe's down and allow several apps to share common bpl's because they are NOT linked into the exe # compileTime - that's the upside. The downside you are now experiencing - you must distribute your bpl's with your exe.
I am trying to start making my own libraries avaialble as packages prior to compiling my Apps with these packages hence modularising my code. For years I've 'sort of' understood packages, breathing a sigh of relief when I load a component package and click on 'Install' and it does. I understand that the process of installing a component (or components) is via the creation of a BPL which is then registered with the IDE.
Where I begin to get lost is how to make files available so that I can compile with EITHER a package OR pre-compiled dcu's (like the third party vendors do) and without pointing my project at the source code all the time. I can create a package with the following settings:
where I've specified that all my output will go into 'c:\scratch\wow'. After a build I find TEST.BPL, TEST.DCP and lots of DUC's. Now, when I point another project at this folder to use the DCU's, I get a missing DFM error (one of the units is a form). Should I be manually copying needed DFM's into this output folder? The DPK knows about this form, so why do I not get the DFM copied for me? I presume that using TEST.BPL, that file contains everything, but I wish to work in the two modes. Of course I can get around this by including the source folder in my project search path to find the DFM but third party libraries seem to already have the DFM's in their output folder. Did they install them there using the installer?
Thanks
instead
As others say you could use post-build events to copy your DFM files into place. Other people use a one-time external batch file that copies the DFMs to the DCU folder.
Personally I see very little benefit to making packages for things which are not developed also as reusable components. I also see very little benefit in partitioning an existing application into packages, when you don't reasonably need to use the same subsection or package more than once, or at designtime.
Things I would put into packages:
Delphi visual and non-visual components.
Things which absolutely must be plugged in at runtime, or left out. For example, supposing I sell MetaWare Light and MetaWare Pro, and instead of using compiler IFDEFs to build a differnt binary, I preferred for some reason to simply not ship the ADVANCEDFEATURE.BPL with my systems.
Things to beware of with packages:
I have run into a lot of compiler bugs when combining packages with generics. I have also run into IDE crashes and lockups, in Delphi 2009, 2010, XE and XE2. (I believe XE3 is better)
You should learn a bit about BorlandMM.dll and shared memory management in the BPL world before you move to it. There are some subtleties.
Packages limits the ability of the linker to decide what to remove. In fact, it pretty much destroys it. Packages contain everything that is linked into them, and nothing publically accessible can be removed.
Once you've created a binary package and shipped it to even one customer, you have a pretty difficult to modify contract (this BPL contains a particular signature or application binary interface) you have to be careful in the future to never change them, or mix and match them. Beware of DLL hell, even among your own customers, and be prepared to use versioning on your packages. Just as delphi packages have a version suffix, I recommend you use version suffixes in your own packages right off the bat, and bump them whenever binary compatibility has changed.
Delphi handles build dependencies between packages about as well as could be hoped, which is less well than a single monolithic application. In applications that I have that make heavy use of packages, I find project groups that contain a bunch of packages that depend on each other are very difficult to manage and build quickly. In fact, I have experienced that both compiles and builds are slower and more frustrating than they would be in a singular 750Kline megaproject.
I really wonder if you're not that into the package area of Delphi (you breath a sigh of relief whenever a delphi component actually builds and installs without issue?) if you really want to move into the Packages World totally. By all means, you should experiment. But I wouldn't bet the farm on it yet. Learn some more first.
Yes, you should copy the .dfm to the directory with the compiled units (.dcus), if that is the only directory you want in your search path. The BPL will of course contain the .dfms, and you need a .dcp to be able to link a BPL with your app.
Third party tools must have put the .dfms together with the .dcus in the directory using their installer, indeed.
Instead of copying *.DFM manually you can use Post-Build Event (Project/Options/Build Event), ex:
copy “$(PROJECTDIR)\Unit1.DFM” “c:\Scratch\wow\Unit1.DFM”
I found a way to do this without moving .dfm files to the directory of .dcu files, so you can have a directory for .dcu files only one for .dcp files only and another for .bpl files only.
All you need to do is to create another directory on your good structure, as I do. The directory is called RES and in it should be placed all the resource files (.res files, not .dcr files) that are used by applications compiled using your packages (components). In the Delphi Library Path, you must include in addition to the DCU directory (you should already have) a directory named RES.
On your component (design time) do everything you want with the form (design it, put other components, etc). In the source code of the unit you replace {$R *.dfm} with {$R UnitName.dfm}. In doing so, save all and close the DPK. Now move the .dfm file (do not copy, move!) to the RES folder (the .dfm file is a resource file to the Delphi. The {$R} directive is proof!) and after that open the DPK again to understand what has changed.
First realize that you may not open the form (F12) from his unit, though no error was issued by Delphi about "DFM missing".
Now, do a Build on your package and then install it. Realized again? No errors displayed! This happened because you have indicated the location of .dfm file in the Delphi library search path (RES directory).
Done! You can use your component and dfm will be found when your component is included in an application.
Many of you can now say that this way I will not longer be able to visually edit a form in the component design time. Yes this is true, but if you think about it, why would I want to change so often a form into a component that, in practice, should only be used and slightly edited? Draw your own conclusions ;)