The businesswise calm of the summer has started so I picked up the migration to D2009. I roughly determined for every subsystem of the program if they should remain ascii, or can be unicode, and started porting.
It went pretty ok, all components were there in D2009 versions (some, like VSTView, slightly incompatible though) but I now have run into a problem, in some part that must remain ansistring, I extensively use TStringList, mostly as a basic map.
Is there already something easy to replace it with, or should I simply include a cut down ansistring tstringlist, based on old Delphi or FPC source?
I can't imagine I'm the first to run into this?
The changes must be relatively localised, so that the code remains compilable with BDS2006 while I go through the validation-trajectory. A few ifdefs here and there are no problem.
Of course string->ansistring and char ->ansichar etc don't count as modifications in my source, since I have to do that anyway, and it is fully backwards compat.
Edit: I've been able to work away some of the stuff in reader/writer classes. This makes going for Mason's solution easier than I originally thought. I'll holds Gabr's suggestion in mind as a fallback.
Generics is pretty much the reason I bought D2009. Pity that they made it FPC incompatible though
JCL implements TAnsiStrings and TAnsiStringList in the JclAnsiStrings unit.
If by "map" you mean "hash table", you can replace it with the generic TDictionary. Try declaring something like this:
uses
Generics.Collections;
type
TStringMap<T: class> = TDictionary<ansiString, T>;
Then just replace your StringLists with TStringMaps of the right object type. (Better type-safety gets thrown in free.) Also, if you'd like the dictionary to own the objects and free them when you're done, change it to a TObjectDictionary and when you call the constructor, pass [doOwnsValues] to the appropriate parameter.
(BTW if you're going to use TDictionary, make sure you download D2009 Update 3. The original release had some severe bugs in TDictionary that made it almost unusable.)
EDIT: If it still has to compile under D2006, then you'll have to tweak things a little. Try something like this:
type
TStringMap =
{$IFDEF UNICODE}
class TDictionary<ansiString, TObject>
(Add some basic wrapper functions here.)
end;
{$ELSE}
TStringList;
{$ENDIF}
The wrapper shouldn't take too much work if you were using it as a map in the first place. You lose the extra type safety in exchange for backwards compatibility, but you gain a real hash table that does its lookups in O(1) time.
TStringList.LoadFromFile/SaveToFile also take an optional parameter of type TEncoding, that allows you to use TStringList to store any type of string that you want.
procedure LoadFromFile(const FileName: string; Encoding: TEncoding); overload; virtual;
procedure SaveToFile(const FileName: string; Encoding: TEncoding); overload; virtual;
Also note that by default, TStringList uses ANSI as the codepage so that all existing code works as it has.
Do these subsystems need to remain ansistring, or just how they communicate with the outside world (RS232, text files, etc...)? Just like I do with C#, I treat strings in Delphi 2009 as just strings, and only worry about conversions when someone else needs them.
This will also help avoid unintentional implicit conversions in your code and when calling Windows API methods, improving performance.
You can modify Delphi 2007(or earlier)'s TStrings and TStringList classes and rename them to TAnsiStrings and TAnsiStringList. You should find that to be a very easy modification, and that will give you the classes you need.
Related
I´m migrating my Delphi 5 source code to Delphi 10 Berlin. I´ve got many DLLs in my project which export functions. These functions are called from other DLLs. There are two DLLs which I can not migrate to Delphi 10 but I still want to use them in my program.
Here an example:
function DoSomething( aList: TStringList ): Boolean; external 'Delphi5.dll';
I want to call "DoSomething" from my Delphi 10 Project. But the problem is, that TStringList in Delphi 5 is not compatible to TStringList in Delphi 10 Berlin (unicode). It would work, when DoSomething would have a parameter like "aString: AnsiString" because AnsiString is compatible to "string" in Delphi 5.
Is there a way to send a List between these two Delphi-Versions? Perhaps a TList or something else? Of course I could send a AnsiString with a separator between the strings to simulate a list, but I want a clean solution, because I´ve got many of these export-functions.
Thanks!
One should NEVER pass an object reference from an EXE to a DLL if it is meant to be used inside the DLL, or vice versa. An object reference can safely be passed to a DLL only if all the DLL does is pass the object back to the EXE (ro vice versa), such as through a callback function.
As you experienced, an object reference is not valid if the EXE and DLL aren't compiled with the same version of Delphi. Even if they are compiled with the same version, I suspect some compiler options could make them incompatible ({$Align} comes to mind, though I have never verified it). And even then, some incompatibilities might still occur (such as "Cannot assign TStringList to TStringList" errors due to RTTI mismatches).
Something that could fix your issue with minimal changes to your code would be to change the declaration of your functions to pass an interface to the DLL, and create a wrapper around TStringList that supports that interface. Said interface would need to support all the functionality you need from TStringList.
function DoSomething( aList: IStringList ): Boolean
Interfaces can be passed between DLL/EXE without most of the problems related to the object reference (as long as they use the exact same interface definition when they are compiled). (Edit: You still need to ensure the data passed to the interface's method are safe to pass to/from a DLL.)
That said, the interface should explicitly use AnsiString use a null-terminated PAnsiChar, or even a WideString (which can safely be sent to/from DLL - Reference).
function DoSomething( aListText: PAnsiChar ): Boolean
function DoSomething( aListText: WideString ): Boolean
Do not use String, which is AnsiString in Delphi 5 but is UnicodeString is Delphi 10. And don't use AnsiString, as it is not compatible between Delphi 5 and Delphi 10 due to internal structure differences.
I need to be able to pass the same set of structures (basically arrays of different records) over two different interfaces
The first (legacy) which is working requires a pointer to a record and the record size
The second, which I am attempting to develop, is type-safe and requires individual fields to be set using Get/Set methods for each field
Existing code uses records (probably around 100 or so) with memory management being handled in a 3rd party DLL (i.e. we pass the record pointer and size to it and it deals with memory management of new records).
My original thought was to bring the memory management into my app and then copy over the data on the API call. This would be easy enough with the old interface, as I just need to be able to access SizeOf() and the pointer to the record structure held in my internal TList. The problem comes when writing the adapter for the new type-safe interface
As these records are reliant on having a known size, there is heavy use of array 0..n of char static arrays, however as soon as I try to access these via 2010-flavour RTTI I get error messages stating 'Insufficient RTTI information available to support this operation'. Standard Delphi strings work, but old short-strings don't. Unfortunately, fixing string lengths is important for the old-style interface to work properly. I've had a look at 3rd party solutions such as SuperObject and the streaming in MorMot, though they can't do anything out of the box which doesn't give me too much hope of a solution not needing significant re-work.
What I want to be able to do is something like the following (don't have access to my Delphi VM at the moment, so not perfect code, but hopefully you get the gist):
type
RTestRec = record
a : array [0..5] of char;
b : integer;
end;
// hopefully this would be handled by generic <T = record> or passing instance as a pointer
procedure PassToAPI(TypeInfo: (old or new RTTI info); instance: TestRec)
var
Field: RTTIField;
begin
for Field in TypeInfo.Fields do
begin
case Field.FieldType of
ftArray: APICallArray(Field.FieldName, Field.Value);
ftInteger: APICallInteger(Field.FieldName, Field.Value.AsInteger);
...
end;
end;
Called as:
var
MyTestRec: RTestRec;
begin
MyTestRec.a := 'TEST';
MyTestRec.b := 5;
PassToAPI(TypeInfo(TestRec), MyTestRec);
end;
Can the lack of RTTI be forced by a Compiler flag or similar (wishful thinking I feel!)
Can a mixture of old-style and new-style RTTI help?
Can I declare the arrays differently to give RTTI but still having the size constraints needed for old-style streaming?
Would moving from Records to Classes help? (I think I'd need to write my own streaming to an ArrayOfByte to handle the old interface)
Could a hacky solution using Attributes help? Maybe storing some of the missing RTTI information there? Feels like a bit of a long-term maintenance issue, though.
The following code should compile and does compile with many other types.
However, the compiler reports a "Constant object cannot be passed as var parameter" error - despite the variable quite obviously being a variable.
program CurrencyConstant;
{$APPTYPE CONSOLE}
var
GVar: Currency;
begin
FillChar(GVar, SizeOf(GVar), 0);
end.
Similarly, the same problem occurs with a local variable in a procedure.
procedure TestCurrency;
var
LVar: Currency;
begin
FillChar(LVar, SizeOf(LVar), 0);
end;
I suspect it has something to do with the fact that FillChar is a compiler magic procedure, and that Dest is an untyped var parameter. FillChar is the only routine I've found with this problem.
What causes this problem?
Are any other types affected?
In response to the inevitable "Why would you do that comments": We have a code generator that uses FillChar to generically initialise record structures & primitive types. It works with everything else, but unexpectedly failed with Currency. We do have workarounds, but it would be nice to understand the root cause, and know whether anything else is likely to cause us trouble.
Edit
From Jeroen's answer it is reasonable to conclude that the issue exists in all vesions of Delphi. Furthermore array's of Currency apparently exhibit a similar problem.
David's answer provides some nice workarounds.
One final workaround to consider is, modifying the generator to deal with Currency as a special case and simply set the Value := 0.
What causes the problem?
A compiler bug. Please submit a QC report.
Are any other types affected?
Maybe. Try some to find out.
As for a work around I would write it like this:
FillChar(Pointer(#LVar)^, SizeOf(LVar), 0);
or perhaps like this:
ZeroMemory(#LVar, SizeOf(LVar));
or even like this:
LVar := Default(Currency);
Personally I regard ZeroMemory as being more descriptive than FillChar.
As requested by Craig Young:
It still occurs in Delphi XE4.
Report No: 118866 Status: Reported
Cannot perform FillChar on Currency variables
https://web.archive.org/web/20150322021442/http://qc.embarcadero.com/wc/qcmain.aspx?d=118866
It is similar to
http://qc.embarcadero.com/wc/qcmain.aspx?d=87168 (not archived)
The workaround for this compiler bug for Delphi < 2009: use ZeroMemory or FillMemory from the Windows unit which works just as well as FillChar.
On the Delphi side, ZeroMemory and FillMemory use FillChar underneath which might be inlined as of Delphi 2006.
On the C++ side both use compiler macros.
It might be that this issue only happens with Currency because that is the only numeric compiler type that is scaled.
The issue does not reproduce with ordinal types, regular floating point types, and Comp.
Edit: The issue has been fixed in XE5 Update 2
I have discovered (the hard way) that if a file has a valid UTF-8 BOM but contains any invalid UTF8 encodings, and is read by any of the Delphi (2009+) encoding-enabled methods such as LoadFromFile, then the result is a completely empty file with no error indication. In several of my applications, I would prefer to simply lose a few bad encodings, even if I get no error report in this case either.
Debugging reveals that MultiByteToWideChar is called twice, first to get the output buffer size, then to do the conversion. But TEncoding.UTF8 contains a private FMBToWCharFlags value for these calls, and this is initialized with a MB_ERR_INVALID_CHARS value. So the call to get the charcount returns 0 and the loaded file is completely empty. Calling this API without the flag would 'silently drop illegal code points'.
My question is how best to weave through the nest of classes in the Encoding area to work around the fact that this is a private value (and needs to be, because it is a class var for all threads). I think I could add a custom UTF8 encoding, using the guidance in Marco Cantu's Delphi 2009 book. And it could optionally raise an exception if MultiByteToWideChar has returned an encoding error, after calling it again without the flag. But that does not solve the problem of how to get my custom encoding used instead of Tencoding.UTF8.
If I could just set this up as a default for the application at initialization, perhaps by actually modifying the class var for Tencoding.UFT8, this would probably be sufficient.
Of course, I need a solution without waiting to lodge a QC report asking for a more robust design, getting it accepted, and seeing it changed.
Any ideas would be very welcome. And can someone confirm this is still an issue for XE4, which I have not yet installed?
I ran into the MB_ERR_INVALID_CHARS issue when I first updated Indy to support TEncoding, and ended up implementing a custom TEncoding-derived class for UTF-8 handling to avoid specifying MB_ERR_INVALID_CHARS. I didn't think to use a class helper.
However, this issue is not just limited to UTF-8. Any decoding failure of any of the TEncoding classes will result in a blank result, not an exception being raised. Why Embarcadero chose that route, when most of the RTL/VCL uses exceptions instead, is beyond me. Not raising an exception on error caused a fair amount of issues in Indy that had to be worked around.
This can be done pretty simply, at least in Delphi XE5 (have not checked earlier versions). Just instantiate your own TUTF8Encoding:
procedure LoadInvalidUTF8File(const Filename: string);
var
FEncoding: TUTF8Encoding;
begin
FEncoding := TUTF8Encoding.Create(CP_UTF8, 0, 0);
// Instead of CP_UTF8, MB_ERR_INVALID_CHARS, 0
try
with TStringList.Create do
try
LoadFromFile(Filename, FEncoding);
// ...
finally
Free;
end;
finally
FEncoding.Free;
end;
end;
The only issue here is that the IsSingleByte property for the newly instantiated TUTF8Encoding is then incorrectly set to False, but this property is not currently used anywhere in the Delphi sources.
A partial workaround is to force the UTF8 encoding to suppress MB_ERR_INVALID_CHARS globally. For me, this avoids the need for raising an exception, because I find it makes MultiByteToWideChar not quite 'silent': it actually inserts $fffd characters (Unicode 'replacement character') which I can then find in the cases where this is important. The following code does this:
unit fixutf8;
interface
uses System.Sysutils;
type
TUTF8fixer = class helper for Tmbcsencoding
public
procedure setflag0;
end;
implementation
procedure TUTF8fixer.setflag0;
{$if CompilerVersion = 31}
asm
XOR ECX,ECX
MOV Self.FMBToWCharFlags,ECX
end;
{$else}
begin
Self.FMBToWCharFlags := 0;
end;
{$endif}
procedure initencoding;
begin
(Tencoding.UTF8 as TmbcsEncoding).setflag0;
end;
initialization
initencoding;
end.
A more useful and principled fix would require changing the calls to MultiByteToWideChar not to use MB_ERR_INVALID_CHARS, and to make an initial call with this flag so that an exception could be raised after the load is complete, to indicate that characters will have been replaced.
There are relevant QC reports on this issue, including 76571, 79042 and 111980. The first one has been resolved 'as designed'.
(Edited to work with Delphi Berlin)
Your "global" approach is not really global - it relies upon the assumption that all the code would only use one and the same instance of TUTF8Encoding. The same instance where you hacked the flags field.
But it would not work if one obtain TUTF8Encoding object(s) by other means than TEncoding.GetUTF8, for example in XE2 another method - TEncoding.GetEncoding(CP_UTF8) - would create a new instance of TUTF8Encoding instead of re-using FUTF8 shared one. Or some function might run TUTF8Encode.Create directly.
So i'd suggest two more approaches.
Approach with patching the class implementation, somewhat hacky. You introduce your own class for the sake of obtaining new "fixes" constructor body.
type TMyUTF8Encoding = class(TUTF8Encoding)
public constructor Create; override;
end;
This constructor would be the copycat of TUTF8Encoding.Create() implementation, except for setting the flag as you want it ( in XE2 it is done by calling another, inherited Create(x,y,z) so u would not need an access to the private field ) instead.
Then you can patch the stock TUTF8Encoding VMT overriding its virtual constructor to that new constructor of yours.
You may read Delphi documentation about "internal formats" and so forth, to get the VMT layout. You would also need calling VirtualProtect (or other platform-specific function) to remove protection from VMT memory area before patching and then to restore it.
Examples to learn from
How to change the implementation (detour) of an externally declared function
https://stackoverflow.com/a/1482802/976391
Or you may try using Delphi Detours library, hopefully it can patch virtual constructors. Then... it might be an overkill here to use that rather complex lib for that single goal.
After you hacked the TUTF8Encoding class do call the TEncoding.FreeEncodings to remove the already created shared instances (if any) if any and thus trigger recreating the UTF8 instances with your modifications.
Then, if you compile your program as a single monolithic EXE , without using runtime BPL modules, you just can copy the SysUtils.pas sources to your application folder and then to include that local copy into your project explicitly.
How to patch a method in Classes.pas
There you would change the very TUTF8Encoding implementation as you see fit in the sources and Delphi would use it.
This brain-deadly simplistic (hence - equally reliable) approach would not work though if your projects would be built to reuse rtlNNN.bpl runtime package instead of being monolithic.
When I approach Java problems, I use the collection pattern. However, doing it in Delphi is quite a nightmare since there is no Integer object to handle things.
I need a data structure that holds numbers. I want to be able to add numbers, remove numbers, and check the contents of the collection, and each number must be unique.
I'm not interested in a solution I need to implement and test for bugs myself. Is there a ready object like Java's HashTable?
uses GpLists;
var
numberList: TGpIntegerList;
begin
numberList := TGpIntegerList.Create;
numberList.Duplicates := dupIgnore;
numberList.Sorted := true;
numberList.add(1);
numberList.add(2);
numberList.add(3);
numberList.add(1);
GpLists comes with a BSD license. It also contains a class holding 64-bit integers - TGpInt64List - and bunch of other stuff.
Dictionary<Integer,Boolean> or similar would do.
I know it's dirty, but you could misuse TStringList (or THashedStringList).
var
numberList: TStringList;
begin
numberList := TStringList.Create;
numberList.Duplicates := dupIgnore;
numberList.Sorted := true;
numberList.add(IntToStr(1));
numberList.add(IntToStr(2));
numberList.add(IntToStr(3));
numberList.add(IntToStr(1));
// numberList.CommaText = '1,2,3'
This is a simple solution for the Delphi version with generics:
TUniqueList<T> = class(TList<T>)
public
function Add(const Value: T): Integer;
end;
{ TUniqueList<T> }
function TUniqueList<T>.Add(const Value: T): Integer;
begin
if not Contains(Value) then
Result := inherited Add(Value);
end;
And if performance with lots of integers is important then you can keep the list sorted and use the binary serch
Delphi containers class in the "standard" VCL library are poor. This is a long standing issue only partially corrected in latest versions.
If you are using Delphi >= 2009 you have generics class that can handle integer data types as well, before you have to write your own class, use TList in a non standard way, or use a third party library.
If you have to store numbers, if they are at most 32 bit long you can store them in a TList, casting them to and from pointers. You have to override the Add() method to ensure uniqueness. You could also use TBits and set to true the corresponding "slot".
Otherwise you need to use third party libraries like the JCL (free) or DIContainers (commercial), for example.
You can use a TList to store a set of integers. It is supposed to store Pointers, but since Pointers are just integers it works perfectly when storing Integers.
Delphi has unit mxarrays (Decision Cube), there is a class TIntArray, set it's property Duplicates to dupIgnore. It's also can sort values. If you will use it, see Quality Central Report #:2703 to correct the bug in this unit.
Personally, i strongly recommend start using DeCAL for storing data. It has DMap container which can handle almost any data type, is self optimized because it uses internal Red-Black tree and it won't allow you to add duplicates (if you need to insert duplicates, you can use DMultiMap). Another great thing with DMap is that finding element in the list is very fast (much faster than in TStringList). Working with DeCal is a bit different than with other Delphi libraries but once you get comfortable with it, you won't use any StringList in your code.
Edit: older version of DeCAL is on SourceForge, but here you can find great pdf manual.
Yes, there is, it's called TDictionary