I need to be able to pass the same set of structures (basically arrays of different records) over two different interfaces
The first (legacy) which is working requires a pointer to a record and the record size
The second, which I am attempting to develop, is type-safe and requires individual fields to be set using Get/Set methods for each field
Existing code uses records (probably around 100 or so) with memory management being handled in a 3rd party DLL (i.e. we pass the record pointer and size to it and it deals with memory management of new records).
My original thought was to bring the memory management into my app and then copy over the data on the API call. This would be easy enough with the old interface, as I just need to be able to access SizeOf() and the pointer to the record structure held in my internal TList. The problem comes when writing the adapter for the new type-safe interface
As these records are reliant on having a known size, there is heavy use of array 0..n of char static arrays, however as soon as I try to access these via 2010-flavour RTTI I get error messages stating 'Insufficient RTTI information available to support this operation'. Standard Delphi strings work, but old short-strings don't. Unfortunately, fixing string lengths is important for the old-style interface to work properly. I've had a look at 3rd party solutions such as SuperObject and the streaming in MorMot, though they can't do anything out of the box which doesn't give me too much hope of a solution not needing significant re-work.
What I want to be able to do is something like the following (don't have access to my Delphi VM at the moment, so not perfect code, but hopefully you get the gist):
type
RTestRec = record
a : array [0..5] of char;
b : integer;
end;
// hopefully this would be handled by generic <T = record> or passing instance as a pointer
procedure PassToAPI(TypeInfo: (old or new RTTI info); instance: TestRec)
var
Field: RTTIField;
begin
for Field in TypeInfo.Fields do
begin
case Field.FieldType of
ftArray: APICallArray(Field.FieldName, Field.Value);
ftInteger: APICallInteger(Field.FieldName, Field.Value.AsInteger);
...
end;
end;
Called as:
var
MyTestRec: RTestRec;
begin
MyTestRec.a := 'TEST';
MyTestRec.b := 5;
PassToAPI(TypeInfo(TestRec), MyTestRec);
end;
Can the lack of RTTI be forced by a Compiler flag or similar (wishful thinking I feel!)
Can a mixture of old-style and new-style RTTI help?
Can I declare the arrays differently to give RTTI but still having the size constraints needed for old-style streaming?
Would moving from Records to Classes help? (I think I'd need to write my own streaming to an ArrayOfByte to handle the old interface)
Could a hacky solution using Attributes help? Maybe storing some of the missing RTTI information there? Feels like a bit of a long-term maintenance issue, though.
Related
tl;dr:
Trying to pass a list of objects to a function, which expects list of objects implementing an interface. Said objects implement that interface. Compiler will not allow it.
Looking for an alternative workaround or my mistake.
The setup
I did not use the actual code, and that should not matter IMO. Seems like a conceptional problem to me.
The class TObjectWithInterface implements the interface ISomeInterface and extends the class TCustomInterfacedObject, which simply works around the reference counting of IInterface as given in the documentation.
TObjectWithInterface = class(TCustomInterfacedObject, ISomeInterface)
If have a procedure which takes a list of objects implementing that interface:
procedure SomeFunction(List: TList<ISomeInterface>)
The issue
Inside of a function of TObjectWithInterface, I try to call that function, using a list of objects of TObjectWithInterface:
procedure TObjectWithInterface.DoTheStuff(ListOfObjects: TList<TObjectWithInterface>)
begin
// ...
SomeFunction(ListOfObjects); // <-- Compiler error: Incompatible types
// ...
end;
The compiler tells me the following:
E2010: Incompatible types: 'System.Generics.Collections.TList' and 'System.Generics.Collections.TList'
The dumb workaround
I really dislike my current workaround, which consists of creating a new list and type casting each TObjectWithInterface to ISomeInterface:
procedure TObjectWithInterface.DoTheStuff(ListOfObjects: TList<TObjectWithInterface>)
var
ListOfInterfaceObjects: TList<ISomeInterface>;
begin
// ...
ListOfInterfaceObjects := TList<ISomeInterface>.Create;
for var Object in ListOfObjects do
ListOfInterfaceObjects.Add(Objects as ISomeInterface);
SomeFunction(ListOfInterfaceObjects)
// ...
end;
This seems very hacky to me. I may have done something stupid, or do not understand something correctly, as this is the first time, that I am trying to use Interfaces in Delphi. Please don't be mad.
Either way, I hope someone can point out my mistake or, if this is a language limitation, has an alternative workaround.
Your "workaround" to copy the list of objects into a separate list of interfaces is actually the correct and proper solution. You can't use a TList<X> where a TList<Y> is expected, they are different and unrelated types. Just because you are disabling reference counting on the interfaces doesn't change the memory layout of the objects in relation to their interfaces, so you still have to pass around proper memory pointers, which means performing necessary conversions from one to the other.
I'm trying to create a generic method to get all the references to an object from an object.
For example:
TTest2 = class(TObject);
TTest = class(TObject)
Test2: TTest2;
end;
I want to create a method like:
var
Local: TTest;
LinkedObjects: TList;
begin
Local := TTest.Create;
LinkedObjects := Local.GetChildren;
//blah
end;
I'd like to create a method that says to me that on offset X, there is a reference for an object. The objective is to be able to list any object in any kind of field, so, published field's (that are listed on object header - vmtFieldTable) won't solve, Rtti (As it's not default for every classes) won't solve too.
It's probably not possible without some help from compiler (providing some information), but if you have some idea, please let me know.
I'm researching the possibility to develop a GC for Delphi. Everything on a GC is very mature, the technology is not a problem. But how to have access for some information is what make things complicated. At this point, I'm thinking a way to deal with the Mark step.
Some thoughts
Overload the assign operator of TObject ?
It's not possible just on NextGen compilers. Full answer.
Go through all the allocated memory and search for valid pointers on its space ?
Slow, but is it possible ? initialize and finalize all the object's memory clear, and then go through the memory looking for pointer with a valid object header ? or can I create a parity bit on objects to make it easier to identify?
Update: I found an interesting link! Talking about the same problem we discussed here.
I'll try do it.
I'm putting some information together here.
Thanks you,
Unlike the case with common objects, it is impossible to directly assign generics of different related types in Delphi as follows:
Possible (normal objects):
var
var_1 : TObject;
var_2 : MyTObjectSubClass;
var_1 := var_2; //Works
Not possible (generics):
var
var_1 : TList<TObject>;
var_2 : TList<MyTObjectSubClass>;
var_1 := var_2; //Does not compile
It is possible to use casts to accomplish this though, as follows:
var
var_1 : TList<TObject>;
var_2 : TList<MyTObjectSubClass>;
var_1 := TList<TObject>(var_2); //Works
This creates the need to be able to dynamically cast generics (i.e. to dynamically parameterize their generic type specification) somehow, but I have not been able to find a way to do this, so my question is: Is this in any way possible?
I'm indeed aware of the covariance/contravariance problems related to this, but in some cases it would indeed both be useful and "correct" to do such a thing.
One example of such a situation is the current code I'm writing for generic streaming of Delphi objects over a TStream, where the receiving end knows the exact type of the object that is incoming over the stream, e.g. TList<MyTObjectSubClass>. This type information is extracted by means of RTTI though (from a provided target variable to which the loaded object should be written), so I cannot explicitly mention the exact generics type in my stream-loading code in advance, but rather have to detect it by means of RTTI (which is possible, although somewhat hacky) and then write it to a target variable that I only at that run-time point will know the exact type of.
Thus, the load-object-from-stream code has to be fully generic, and thus, it would need to dynamically cast an existing TList<TObject> variable (which is defined explicitly in the code) to the exact type of TList<MyTObjectSubClass> (which I at that point have just learned about, through the use of RTTI), in order to be able to pass this object loaded from the stream to its final destination variable.
So again, is there ANY way whatsoever to accomplish this, or is it on the contrary actually completely impossible to assign to a not-in-advance-known generics collections using generic code (i.e. code that does not explicitly have some kind of "if [type of xxx is TList<TMyObject1>] then ... else if [type of xxx is TList<TMyObject2>] then ... else ..." test, containing explicit mentions of every single generics type that should be supported by it)?
PS.
The generics type of the stream-loaded object obviously already exists somewhere in the program (since it is concluded by means of RTTI on the target variable that the stream-loaded object should be written to), so I'm not asking about full run-time dynamic creation of generics types, but rather just about how to be able to dynamically pick the right one of those generics types already defined at compile-time in the program, and then cast a variable to that type.
EDIT:
By request from #RemyLebeau , here comes some more example code from my application, from its stream-loading function:
var
source_stream : TStream;
field_to_process : TRttiField;
field_type : TRttiType;
loaded_value : TValue;
temp_int : integer;
//...
//The fields of any object given to the streaming function are
//enumerated and sorted here
//...
//Then, for each field (provided in field_to_process),
//the following is done:
case field_to_process.FieldType.TypeKind of
//...
tkInteger:
begin
source_stream.ReadBufferData(temp_int);
loaded_value := TValue.From(temp_int);
end;
tkString,
tkLString,
tkWString,
tkUString:
begin
source_stream.ReadBufferData(noof_raw_bytes_in_string_data);
SetLength(raw_byte_buf, noof_raw_bytes_in_string_data + 4);
source_stream.ReadBuffer(raw_byte_buf[0], noof_raw_bytes_in_string_data);
temp_str := used_string_encoding.GetString(raw_byte_buf, 0, noof_raw_bytes_in_string_data);
loaded_value := TValue.From(temp_str);
end;
tkClass:
begin
is_generics_collection_containing_TObject_descendants := <does some hacky detection here>; //Thanks Remy :-)
if is_generics_collection_containing_TObject_descendants then
begin
<magic code goes here that loads data from the stream into the currently processed field, whose type has been detected to be of some specific generics collection type>
end;
end;
//...
end;
field_to_process.SetValue(self, loaded_value);
That should hopefully give a somewhat better overview of my problem. The superfluous code for strings and integers are just for context, by showing how some simple types are handled.
For more info about the (necessarily) "hacky detection" mentioned in the code, please see this question. After doing that, I will know the exact type of the generics collection and its subitems, for example TList<TSomeTObjectDescendant>.
So, as you hopefully can see now, the question is about the <magic code goes here that loads data from the stream into the currently processed field, whose type has been detected to be of some specific generics collection type> part. How can it be implemented?
NOTE: My problem is not to understand how to serialize/deserialize contents of an enumerable through a stream (which can of course be done by simply iterating over the items in the enumerable and then recursing the stream saving/loading code for each of them, where the number of items is given first of all in the stream). The problem is rather how to create generic code that will be able to recreate/populate any kind of generics collection of TObject descentants, whose type you only get to know at runtime, and then to finally get this into the object field that was originally enumerated by RTTI at the beginning of the stream-loading code. As an example, assume that the processed field has the type TList<TSomeTObjectDescendant>, and that you can easily load its subobjects from the stream using a call like function load_list_TSomeTObjectDescendant_subitems(input_stream : TStream) : array of TSomeTObjectDescendant. How could I then get these subitems into the TList<TSomeTObjectDescendant> field?
Type-casts and variable declarations are parsed at compile-time (though is and as casts are executed at runtime based on compiler-provided RTTI). The type being casted to, and the type of the variable being assigned to, must be known to the compiler. So what you are asking for is simply not possible with Generics. Not the way you have described it, anyway.
In the past, I have seen this work, but I never really understood how it should be done.
Assume we have a file of known data types, but unknown length, like a dynamic array of TSomething, where
type
TSomething = class
Name: String;
Var1: Integer;
Var2: boolean;
end;
The problem, though, is that this object type may be extended in the future, adding more variables (e.g. Var3: String).
Then, files saved with an older version will not contain the newest variables.
The File Read procedure should somehow recognize data in blocks, with an algorithm like:
procedure Read(Path: String)
begin
// Read Array Size
// Read TSomething --> where does this record end? May not contain Var3!
// --> how to know that the next data block I read is not a new object?
end;
I have seen this work with BlockRead and BlockWrite, and I assume each object should probably write its size before writing itself in the file, but I would appreciate an example (not necessarily code), to know that I am thinking towards the right direction.
Related readings I have found:
SO - Delphi 2010: How to save a whole record to a file?
Delphi Basics - BlockRead
SO - Reading/writing dynamic arrays of objects to a file - Delphi
SO - How Can I Save a Dynamic Array to a FileStream in Delphi?
In order to make this work, you need to write the element size to the file. Then when you read the file, you read that element length which allows you to read each entire element, even if your program does not know how to understand all of it.
In terms of matching up your record with the on-disk record that's easy enough if your record only contains simple types. In that scenario you can read from the file Min(ElementLength, YourRecordSize) bytes into your record.
But it does not look as though you actually have that scenario. Your record is in fact a class and so not suitable for memory copying. What's more, its first member is a string which is most definitely not a simple type.
Back in the day (say the 1970s), the techniques you described were how files were read. But these days programming has moved on. Saving structured data to files usually means using a more flexible and adaptable serialization format. You should be looking to using JSON, XML, YAML or similar for such tasks.
I'd say you need a method of versioning you file. That way you know what version of the record is contained in the file. Write it at the start of the file and then on reading, read in the version identifier first and then use the corresponding structure to read the rest.
If I understand you correctly your main issue is if TSomething changes. Most important thing is that you need to add version info into your file, this you really cannot avoid.
As for actual storage using Sqlite would most likely solve all your problems, but depending on your situation it might be an overkill.
Except for unexceptional circumstances I wouldn't really worry about extending the class too much.If you add add version number to the beginning of the file you can easily convert the file after the class have changed. All you need to do is implement your solution so that adding conversions would as simple as reasonable.
In order to read/write files I would prefer streams/XML/JSON (depending on situation) instead of blockread/blockwrite as you don't have to implement a hack to store version number.
In theory you could also have unused space for each record so I you could avoid recreating entire file if class changes upto a point (until you have enough unused space). It maybe helpful if TSomething changes often and files are big, but most likely I would not go that route.
This is how I would do it: Include a simple version number in the header. This can be any string, integer or whatever.
Reading and writing the file is very easy (I am using pseudocode):
Procedure Read (MyFile : TFile);
Var
reader : IMyFileReader;
begin
versionInfo = MyFile.ReadVersionInfo();
reader = ReaderFactory.CreateFromVersion(versionInfo);
reader.Read(MyFile);
end;
Type
ReaderFactory = Class
public
class function CreateFromVersion(VersionInfo : TVersionInfo) : IMyFileReader;
end;
function ReaderFactory.CreateFromVersion(VersionInfo : TVersionInfo) : IMyFileReader;
begin
if VersionInfo = '0.9-Alpha' then
result := TVersion_0_9_Alpha_Reader.Create()
else if VersionInfo = '1.0' then
result := TVersion1_0_Reader.Create()
else ....
end;
This can easily be maintained and extended forever. You will never have to touch the Read-routine, but only add a new reader and enhance the factory. With a simple registration method and a TDictionary<TVersionInfo,TMyFileReaderClass>, you can even avoid having to modify the factory.
When I approach Java problems, I use the collection pattern. However, doing it in Delphi is quite a nightmare since there is no Integer object to handle things.
I need a data structure that holds numbers. I want to be able to add numbers, remove numbers, and check the contents of the collection, and each number must be unique.
I'm not interested in a solution I need to implement and test for bugs myself. Is there a ready object like Java's HashTable?
uses GpLists;
var
numberList: TGpIntegerList;
begin
numberList := TGpIntegerList.Create;
numberList.Duplicates := dupIgnore;
numberList.Sorted := true;
numberList.add(1);
numberList.add(2);
numberList.add(3);
numberList.add(1);
GpLists comes with a BSD license. It also contains a class holding 64-bit integers - TGpInt64List - and bunch of other stuff.
Dictionary<Integer,Boolean> or similar would do.
I know it's dirty, but you could misuse TStringList (or THashedStringList).
var
numberList: TStringList;
begin
numberList := TStringList.Create;
numberList.Duplicates := dupIgnore;
numberList.Sorted := true;
numberList.add(IntToStr(1));
numberList.add(IntToStr(2));
numberList.add(IntToStr(3));
numberList.add(IntToStr(1));
// numberList.CommaText = '1,2,3'
This is a simple solution for the Delphi version with generics:
TUniqueList<T> = class(TList<T>)
public
function Add(const Value: T): Integer;
end;
{ TUniqueList<T> }
function TUniqueList<T>.Add(const Value: T): Integer;
begin
if not Contains(Value) then
Result := inherited Add(Value);
end;
And if performance with lots of integers is important then you can keep the list sorted and use the binary serch
Delphi containers class in the "standard" VCL library are poor. This is a long standing issue only partially corrected in latest versions.
If you are using Delphi >= 2009 you have generics class that can handle integer data types as well, before you have to write your own class, use TList in a non standard way, or use a third party library.
If you have to store numbers, if they are at most 32 bit long you can store them in a TList, casting them to and from pointers. You have to override the Add() method to ensure uniqueness. You could also use TBits and set to true the corresponding "slot".
Otherwise you need to use third party libraries like the JCL (free) or DIContainers (commercial), for example.
You can use a TList to store a set of integers. It is supposed to store Pointers, but since Pointers are just integers it works perfectly when storing Integers.
Delphi has unit mxarrays (Decision Cube), there is a class TIntArray, set it's property Duplicates to dupIgnore. It's also can sort values. If you will use it, see Quality Central Report #:2703 to correct the bug in this unit.
Personally, i strongly recommend start using DeCAL for storing data. It has DMap container which can handle almost any data type, is self optimized because it uses internal Red-Black tree and it won't allow you to add duplicates (if you need to insert duplicates, you can use DMultiMap). Another great thing with DMap is that finding element in the list is very fast (much faster than in TStringList). Working with DeCal is a bit different than with other Delphi libraries but once you get comfortable with it, you won't use any StringList in your code.
Edit: older version of DeCAL is on SourceForge, but here you can find great pdf manual.
Yes, there is, it's called TDictionary