Can I recompile the .PAS files used by the Delphi IDE? - delphi

I am familiar with Jeff Atwood's article about how errors are always the programmer's fault, but I believe I have really and truly found a bug in a Delphi .pas file.
Specifically, I'm using Delphi 2007, and the error is on line 955 of the DBCommon.pas file, which on my machine is located here:
C:\program files\codegear\rad studio\5.0\source\Win32\db\DBCommon.pas
And the code is this:
...
FieldIndex := StrToInt(Token);
if DataSet.FieldCount >= FieldIndex then
LastField := DataSet.Fields[FieldIndex-1].FieldName else
...
If "Token" has a value of zero, then we try to access index -1 of DataSet.Fields, resulting in a list index out of bounds error.
This error is not raised to the user, because it is handled before it gets that high up, but it is enormously irritating to have the debugger break in every time this happens.
I could "Ignore this exception type" but Index out of bounds errors are common enough that I don't want to universally ignore them.
The situation that causes FieldIndex to be zero is when you have a SELECT statement whose ORDER BY contains a function, as in:
ORDER BY
CASE WHEN FIELD1 = FIELD3 THEN 1 ELSE 2 END
,CASE WHEN FIELD2 = FIELD4 THEN 1 ELSE 2 END
I can fix the bug in DBCommon.pas, but Delphi will not recompile itself, and my change does not take effect. If I rename the .DCU file, then it just complains that "DBCommon.dcu" cannot be found.
So (finally) my question is: Can I recompile DBCommon.pas with my fix, and if so, how?

You can probably put dbcommon.pas in youre project directory. It will then be compiled along with the rest of the project.

See previous answers for how to create the situation where you can recompile modified VCL source. I would however add that you seriously consider managing your changes in your change control system, using the "Vendor Branch" SCM pattern.
In simple terms (using SVN as a reference):
Create a "vendor source" copy of the original vendor supplied files. This is your "pristine" reference copy.
Create a branch representing that pristine copy (e.g. "2009" for the Delphi 2009 version of the VCL)
create a further branch into a separate "vendor library" folder. THIS is the copy of the library that you should reference in your projects
any modifications to the vendor source are made in the "vendor library" branch.
when the vendor provides a new version of the library you check the new version in to the "vendor source" project and create a new branch for the new version.
you can then easily diff the vendor revisions. But more importantly (with SubVersion, and possibly othe SCM systems) you should also be able to simply merge (i.e. automatically) the new vendor source with your "vendor library" branch to easily incorporate vendor changes with your own modifications.
This is all described far better than I just did in the excellent O'Reilly book: "Version Control with SubVersion"
NOTE however that the "loaddirs" utility mentioned in that book is no longer supported due to copyright issues, so updating "vendor drops" is currently a manual exercise, but this occurs only infrequently and is not a major burden.
We are using this pattern ourselves, although in the case of the VCL we do not maintain a complete copy of the entire VCL source tree in our "vendor source" or "vendor library", but instead only track changed and dependent units. For other libraries managed under a vendor branch we typically do maintain complete copies but decided this wasn't necessary for the VCL.
We've only just implemented this pattern however, so we may yet decide that we need to take a more comprehensive approach with the VCL too.
ymmv

You can, but often you don't have to. Recompiling a VCL unit sometime means recompiling all the rest of the VCL units either because you've changed the interface of a unit or because the compiler gets confused and thinks you've changed the interface. Recompiling a VCL unit also rules out the possibility of using most run-time packages because you can't recompile Delphi's run-time packages.
Instead of recompiling, you can use run-time patching. I've used the method in the TNT Unicode controls, but Madshi also provides a way to replace a function with your own implementation. If you copy the implementation of DBCommon.GetIndexForOrderBy into your own unit and make your fixes, you can use this command to patch the VCL version with your own:
var
Old_GetIndexForOrderBy: Pointer;
HookCode(#DBCommon.GetIndexForOrderBy,
#Fixed_GetIndexForOrderBy,
Old_GetIndexForOrderBy,
0);
With the Tnt Unicode library, find the OverwriteProcedure routine in the TntSystem unit. It's not public, so you'll need to either declare it in the unit interface or copy it into your own unit. Then you can call it much like the Madshi code above:
var
Old_GetIndexForOrderBy_Data: TOverwrittenData;
OverwriteProcedure(#DBCommon.GetIndexForOrderBy,
#Fixed_GetIndexForOrderBy,
#Old_GetIndexForOrderBy_Data);

We have a folder under our project source tree called VCL, into which we place copies of the VCL source that we wish to modify slightly.
Your example is a good candidate for doing the same thing.
You will need to modify the search path for your project so that "your" VCL folder is earlier on your path than the "Source" folders under your Delphi installation.
You may also find that if you copy one VCL source unit out and modify it, you will have to also copy other VCL source units out into "your" folder which may be dependencies.
Our reason for doing this is that we want our builds to have zero compiler hints and warnings. There are some parts of the VCL source that are not hint/warning free.

"I am familiar with Jeff Atwood's article about how errors are always the programmer's fault, but I believe I have really and truly found a bug in a Delphi .pas file"
Are you joking me? With Delphi you always blame Borland first :)
Something is weird, go Google it and see if it is a Delphi bug. Only if cannot find any similar reports you site down and check your code line by line.
After I reinstall Delphi I have to patch the original PAS files in 6 (six) places. There are tons of bugs that appear on a fresh Delphi installation and can be easily reproduced. Delphi (the one that we all love) is full of bugs. There is an entire history created around this.
There are so many people releasing external patches (such as http://andy.jgknet.de/blog/?page_id=288) and Borland/Imprise/GoGear/Embarcadero keep ignoring them. It is a true wonder they included FastMM in their release.
Anyway, I have recompiled those PAS files and now I replace the original DCUs with the patched ones.

Simply - Yes. Using one of the above answers [By Tom or Connor]. Copy DBCommon.pas to your project folder rather than edit the original. This leaves other projects and compilations unaffected because it won't be on the path.

You can set:
DataSetProvider.Option.poRetainServerOrder = True

Related

how to build custom system.pas in delphi to create system.dcu? [duplicate]

I have this craving to do some experiments with modifying the underbelly of the Delphi run time library (RTL), system.pas and the likes... It is possible or not?
I'm very fond of challenges like "yes, but you'll have to provide custom .obj files for some assembler wizardry because they were never distributed with the official Delphi source". Fine with me, I just want to know.
I want to do this experiment with Delphi 7, but inside information on any other version is fine. It is one of the perks of being with a company that worked with Delphi since the Stone Age.
(I always figured this to be one of those RTFM questions, with the answer being a resounding "NO!", but for some reason google won't confirm it.)
You can recompile the RTL like any other unit.
For System.pas you must use the command line compiler.
For instance, here is a working batch file content (there is some not well documented command line switches):
del *.dcu /s
"c:\program files\borland\delphi7\bin\dcc32.exe" -O+ -Q -M -Y -Z -$D+ System.pas
This will recompile System.pas and SysInit.pas (both lowest level RTL files).
But in order to use your recreated dcu files, you'll have to put the folder containing the updated dcu files into the first position of your IDE: for instance, in Delphi 7 it's Option / Environment Options / Library, then put your folder FIRST in both "Libary path" and "Browsing path" field.
And it's perhaps worth deleting the original .dcu files in your Delphi installation directory.
But be sure you won't change the "interface" part of the unit, or you'll have troubles with compiling with other not modified units of the RTL (or third-party components). You can change the "implementation" part, apply fixes or rewrite some part for speed or such, but don't change the "interface" part to avoid any linking error.
Always make a backup of the original .pas and .dcu files which you are changing. And it's a good idea to make some automated compilation test, so that you could be sure that your modifications of the RTL won't add any regression.
We made such a RTL recompilation for our Enhanced Run Time Library for better speed of low-level RTL functions (mostly System.pas and SysUtils.pas). Designed for Delphi 7 and 2007. For more recent Delphi version, you still can use the same principle.
You can only recompile the RTL from the command-line. There should be a makefile in the RTL source directory of your installation. It is designed to be used with the make.exe command-line utility which should be in the "bin" folder of your installation. I would recommend you copy the relevant sources to a separate location for experimentation. I must caution you that the System unit is tightly coupled with the compiler which expects many functions to have a specific name and have particular parameter lists, if any are even declared. Many RTL "helper" functions don't have any formally declared parameters, yet expect parameters to be passed in a certain fashion.
Another bit of caution is changing the interface declarations of certain classes, functions or types. Doing so may cause serious incompatibilities with existing DCU files and components. For this reason you must be very careful when intermixing DCU files from the included RTL or third-party components with your custom modified versions. I would suggest you start by only making implementation section changes only before venturing into the mine-field of interface breaking changes.

Decompiling Delphi .dcu to .pas

I'm running RAD Studio Delphi XE2. Something strange appears to have been done to one of my .pas files. Many of the lines from my unit1.pas have been replaced with simply 'º' and others are missing. I am left with a small section of code from somewhere in the middle of my original file. Can I get the original unit1.pas back by somehow decompiling unit1.dcu?
Also, why would this happen? Have you heard of this before? The code runs but, obviously, I can't edit anything I can't see.
Even if you could decompile it, it would likely be an unmaintainable mess compared to your original.
Instead, try looking in the __History folder for your project. You should find older versions of your code. Pick the newest one ;-)
why would this happen?
Any number of reasons. Bug in the IDE? Bug in an IDE plugin or a custom component? Bug in your code that overwrites a source file, maybe?
You cannot decompile a .dcu file to the original source. Information is lost in the translation from source code to executable code. Type declarations, variable names, method names, code layout and so on are simply not present in the compiler's output. Compilation is a one way process.
If you use revision control, you'll just be able to pull the latest version from the repository. If you don't use revision control, you've just learnt a lesson the hard way and you will never again attempt to perform software development without revision control.

Package (BPL) Automatic Naming Suffix

I write a lot of components and libraries for Delphi, most of which require the use of BPL Packaging so that they may be installed into the IDE.
This is simple enough and works well, right up until you want to maintain a single set of Package Project Files (in a single Project Group), but also want to compile and distribute those same packages for different Delphi versions.
Up to now I've been creating a different Package Project for each version of Delphi, and explicitly defining a Delphi Version Identifier as a suffix (e.g. Kinect_XE.bpl and Kinect_XE2.bpl).
I am aware that, in the Project Options for a Package Project, under Description, there are the fields LIB prefix and (more importantly for my needs) LIB suffix.
I am further aware that if I place a value in LIB suffix, it'll be appended to the end of the compiled BPL's filename.
My question, however, is first whether it is possible to have the IDE automatically populate the LIB suffix field with the IDE/RTL Version number, and if so... how?
I'm fairly certain this is possible, as it would appear that vcl120.bpl (and its counterparts for each respective version of Delphi) can be Referenced (as requirements) of your own packages using just vcl rather than having to type the full vcl120. It is, in fact, this same behaviour I'm hoping to achieve... where my packages can intra-reference eachother (as neccessary) without having to provide version-specific references to accommodate the suffixes.
Equally important is that resolving this will enable me to maintain a single set of Project Files in a single Project Group (with the obvious exception of XE2 where its Project Files don't necessarily behave very well with previous versions of Delphi due to the Platforms addition).
I suspect that I may need to put a value like $(VER) (or something similar) in the LIB suffix field, but this appears not to work and I've scoured Google looking for the correct solution.
Hope you can help!
UPDATE 1
I am now writing an IDE plugin to be used with (in the very least) Delphi 2007 to XE2, which gives DLL and BPL projects a new option called AutoSuffix. When toggled On, any IDE with the AutoSuffix plugin installed will immediately apply the correct IDE version suffix to the project.
The AutoSuffix plugin will be made available (freely) for everyone within the next 24 hours, and this question updated accordingly.
UPDATE 2
Okay... Delphi 2007 is being a pain! I've made AutoSuffix work with 2009 to XE2, so far, but 2007 requires a little more time (patience appreciated).
UPDATE 3
It would appear as though Embarcadero have heard our collective cry for simplier package unification between versions.
Mark is going to push this through to see if future versions of Delphi can accommodate a {$LIBSUFFIX AUTO} feature. I hope to hear back very soon whether this will be the case. If so, it certainly affects the way AutoSuffix will need to work on XE2 and older versions (as presently it doesn't provide the simple AUTO switch.. it has its own method).
My hope now is that EMB will take this request seriously, provide it as an integral feature going forward, so that it becomes a simple case of using AutoSuffix on existing versions to unify the process accross all versions!
AFAIK for Delphi up to XE2 there is no automatism for doing this.
Concerning the requires clause: when you require another package you are actually using the dcp, which doesn't inherit the LIBSUFFIX. Thus it is sufficient to require VCL.dcp during compile time, while VCL160.bpl is actually used during runtime. The DCP includes the complete BPL name to resolve that.
This makes the LIBSUFFIX approach superior to the simple "rename the package for each version of Delphi" one.
A solution like that suggested in QC83229 would make it easier to port a package to a newer Delphi version, but then you are still stuck with dproj files that are not backwards compatible.
I normally use different folders for each Delphi version, where only the project files are stored. For a new Delphi version I only have to copy a folder and change the LIBSUFFIX.
LIBSUFFIX directive is in *.dpk file, and you can edit *.dpk file manually.
You can write, for example
{$IFDEF CONDITIONALEXPRESSIONS}
{$IF CompilerVersion = 20.0}
{$LIBSUFFIX '120'}
{$IFEND}
{$IF CompilerVersion = 21.0}
{$LIBSUFFIX '140'}
{$IFEND}
{$ENDIF}
The bad thing is that IDE does not respect your manual edits of *.dpk file and deletes them then you make changes in a package. That is why some component vendors that use conditional defines in *.dpk file say in installation instruction if asked to save changes say 'NO'.
My suggestion would be to add this as a configurable option to project option sets (see QC #86491.) Rather than updating all the packages it would be sufficient to update a single option set file.
In fact, it seems the DllSuffix tag is recognized by the option set files in Delphi XE/XE2. Adding <DllSuffix>160</DllSuffix> to the <PropertyGroup> section of an option set file will cause the suffix to be appended to the package in the project manager. However, you still have to open the project options and click OK in order for it to be saved to the .dpk file.
I agree that it would be extremely useful with this feature (I would think also for the packages in the RTL.)
{$LIBSUFFIX AUTO} feature was added in Delphi 10.4.1:
In previous versions (version 10.3 and earlier) of C++Builder and
Delphi developers building packages needed to manually set their
package’s library suffix setting. The DocWiki “What’s new in version
10.4.1” mentions a new IDE projects option for setting the library suffix:
“Package AUTO libsuffix: packages can now have an automatic version
suffix, instead of manually updating and specifying the right version
suffix with each new release. (The compiler quietly supported this in
10.4, but full support for the feature in the IDE and package project settings is introduced in 10.4.1.)”

Moving to use of Delphi packages - best practice please?

I am trying to start making my own libraries avaialble as packages prior to compiling my Apps with these packages hence modularising my code. For years I've 'sort of' understood packages, breathing a sigh of relief when I load a component package and click on 'Install' and it does. I understand that the process of installing a component (or components) is via the creation of a BPL which is then registered with the IDE.
Where I begin to get lost is how to make files available so that I can compile with EITHER a package OR pre-compiled dcu's (like the third party vendors do) and without pointing my project at the source code all the time. I can create a package with the following settings:
where I've specified that all my output will go into 'c:\scratch\wow'. After a build I find TEST.BPL, TEST.DCP and lots of DUC's. Now, when I point another project at this folder to use the DCU's, I get a missing DFM error (one of the units is a form). Should I be manually copying needed DFM's into this output folder? The DPK knows about this form, so why do I not get the DFM copied for me? I presume that using TEST.BPL, that file contains everything, but I wish to work in the two modes. Of course I can get around this by including the source folder in my project search path to find the DFM but third party libraries seem to already have the DFM's in their output folder. Did they install them there using the installer?
Thanks
instead
As others say you could use post-build events to copy your DFM files into place. Other people use a one-time external batch file that copies the DFMs to the DCU folder.
Personally I see very little benefit to making packages for things which are not developed also as reusable components. I also see very little benefit in partitioning an existing application into packages, when you don't reasonably need to use the same subsection or package more than once, or at designtime.
Things I would put into packages:
Delphi visual and non-visual components.
Things which absolutely must be plugged in at runtime, or left out. For example, supposing I sell MetaWare Light and MetaWare Pro, and instead of using compiler IFDEFs to build a differnt binary, I preferred for some reason to simply not ship the ADVANCEDFEATURE.BPL with my systems.
Things to beware of with packages:
I have run into a lot of compiler bugs when combining packages with generics. I have also run into IDE crashes and lockups, in Delphi 2009, 2010, XE and XE2. (I believe XE3 is better)
You should learn a bit about BorlandMM.dll and shared memory management in the BPL world before you move to it. There are some subtleties.
Packages limits the ability of the linker to decide what to remove. In fact, it pretty much destroys it. Packages contain everything that is linked into them, and nothing publically accessible can be removed.
Once you've created a binary package and shipped it to even one customer, you have a pretty difficult to modify contract (this BPL contains a particular signature or application binary interface) you have to be careful in the future to never change them, or mix and match them. Beware of DLL hell, even among your own customers, and be prepared to use versioning on your packages. Just as delphi packages have a version suffix, I recommend you use version suffixes in your own packages right off the bat, and bump them whenever binary compatibility has changed.
Delphi handles build dependencies between packages about as well as could be hoped, which is less well than a single monolithic application. In applications that I have that make heavy use of packages, I find project groups that contain a bunch of packages that depend on each other are very difficult to manage and build quickly. In fact, I have experienced that both compiles and builds are slower and more frustrating than they would be in a singular 750Kline megaproject.
I really wonder if you're not that into the package area of Delphi (you breath a sigh of relief whenever a delphi component actually builds and installs without issue?) if you really want to move into the Packages World totally. By all means, you should experiment. But I wouldn't bet the farm on it yet. Learn some more first.
Yes, you should copy the .dfm to the directory with the compiled units (.dcus), if that is the only directory you want in your search path. The BPL will of course contain the .dfms, and you need a .dcp to be able to link a BPL with your app.
Third party tools must have put the .dfms together with the .dcus in the directory using their installer, indeed.
Instead of copying *.DFM manually you can use Post-Build Event (Project/Options/Build Event), ex:
copy “$(PROJECTDIR)\Unit1.DFM” “c:\Scratch\wow\Unit1.DFM”
I found a way to do this without moving .dfm files to the directory of .dcu files, so you can have a directory for .dcu files only one for .dcp files only and another for .bpl files only.
All you need to do is to create another directory on your good structure, as I do. The directory is called RES and in it should be placed all the resource files (.res files, not .dcr files) that are used by applications compiled using your packages (components). In the Delphi Library Path, you must include in addition to the DCU directory (you should already have) a directory named RES.
On your component (design time) do everything you want with the form (design it, put other components, etc). In the source code of the unit you replace {$R *.dfm} with {$R UnitName.dfm}. In doing so, save all and close the DPK. Now move the .dfm file (do not copy, move!) to the RES folder (the .dfm file is a resource file to the Delphi. The {$R} directive is proof!) and after that open the DPK again to understand what has changed.
First realize that you may not open the form (F12) from his unit, though no error was issued by Delphi about "DFM missing".
Now, do a Build on your package and then install it. Realized again? No errors displayed! This happened because you have indicated the location of .dfm file in the Delphi library search path (RES directory).
Done! You can use your component and dfm will be found when your component is included in an application.
Many of you can now say that this way I will not longer be able to visually edit a form in the component design time. Yes this is true, but if you think about it, why would I want to change so often a form into a component that, in practice, should only be used and slightly edited? Draw your own conclusions ;)

How to recompile a specific unit from the VCL?

I want to apply a fix from QC to a Delphi 2009 unit (DBClient as it happens). I know I need to copy the unit to another directory and make the change to the copy. How do I then get Delphi to compile that unit and use it in favour of the DCU that already exists?
If you don't want to modify the original .Pas file, I do this by copy the .Pas file into my application folder, then choose built project, it will create new dcu file in my application folder, which will be used instead of the original one.
It's kind of a last resort (and not supported by CodeGear), but I do something similar to Mohammed when necessary. Except instead of putting any modified units into the application folder, I put them into their own folder with the rest of my library code and include this folder in my library path where it will be used by all of my projects. It also prevents me from having multiple (possibly slightly different) copies hanging around.
I also make a point of checking any updates to see what has changed so I can either remove the modified units or re-apply the changes to the newer (and presumably better) units from CodeGear.
I've never did this myself but there are projects in {RAD}\source\rtl along with batch build script. I believe this makes recompiling RTL functions easy. Other units should be recompiled easier.
If the changes you want to do are local and the units aren't widely used by other RTL/VCL units, the simplest way is to place copies of modified units separately from their standard place.
Another option is run-time patching aka detouring.

Resources