I'm creating a component with many published properties for the IDE, and one of such properties is an enum set with 38 values...
type
TBigEnum = (beOne, beTwo, beThree, beFour, beFive, beSix, beSeven, beEight,
beNine, beTen, beEleven, beTwelve, beThirteen, beFourteen, beFifteen,
beSixteen, beSeventeen, beEighteen, beNineteen, beTwenty, beTwentyOne,
beTwentyTwo, beTwentyThree, beTwentyFour, beTwentyFive, beTwentySix,
beTwentySeven, beTwentyEight, beTwentyNine, beThirty, beThirtyOne,
beThirtyTwo, beThirtyThree, beThirtyFour, beThirtyFive, beThirtySix,
beThirtySeven, beThirtyEight);
TBigEnums = set of TBigEnum;
Now, I try to use this in a component as a published property...
type
TMyComponent = class(TComponent)
private
FBigEnums: TBigEnums;
published
property BigEnums: TBigEnums read FBigEnums write FBigEnums;
end;
But the compiler does not let me...
[DCC Error] MyUnit.pas(50): E2187 Size of published set 'BigEnums' is >4 bytes
I understand this limitation, but how can I get around this without splitting it into two different sets?
PS - Each of these values actually has a unique name and purpose, but for the sake of example I just used the number as their names.
I don't remember the exactly correct syntax but in principle:
1 If the "property" does not have to be easily editable in the property inspector then definining 38 different consts of type Long with their values set to 1 shl 0, 1 shl 1, 1 shl 2..
so that those consts can be combined like this PropOne or PropTwo or PropThree
2 if the property must be editable in the property inspector then the TMyPersistent class proposed in Jerry's answer seems ok to me
3 there might be a way built-into the language (or compiler directive) how to type-cast the set representation so that it uses 8 bytes for storage. Int32 and Int64 are both native data types well supported on new processors and both assembler, C++ and C# can deal with it.
Some Pascal flavor (Free Pascal?) either has it implemented or it was in the road map
EDIT option 3 seems to be misleading. What can Free Pascal compiler do regarding enums is listed here http://www.freepascal.org/docs-html/prog/prog.html, especially in chapter $PACKENUM. As of today enums are always backed by 32bit ordinals. So the possibility to increase number of bits used for "enums" possible in assembler, C++, C# is not likely to be available in Delphi.
I'm not even sure if bitwise operators and, or, not, shl, shr used in other languages to implement enums and sets are available for 8-byte integers either in Delphi or in Free Pascal so the option 1 might also be misleading and the winner is option 2
Related
All I want to do is to define a set of integers that may have values above 255, but I'm not seeing any good options. For instance:
with MyObject do Visible := Tag in [100, 155, 200..225, 240]; // Works just fine
but
with MyObject do Visible := Tag in [100, 201..212, 314, 820, 7006]; // Compiler error
I've gotten by with (often lengthy) conditional statements such as:
with MyObject do Visible := (Tag in [100, 202..212]) or (Tag = 314) or (Tag = 820) or (Tag = 7006);
but that seems ridiculous, and this is just a hard-coded example. What if I want to write a procedure and pass a set of integers whose values may be above 255? There HAS to be a better, more concise way of doing this.
The base type of a Delphi set must be an ordinal type with at most 256 distinct values. Under the hood, such a variable has one bit for each possible value, so a variable of type set of Byte has size 256 bits = 32 bytes.
Suppose it were possible to create a variable of type set of Integer. There would be 232 = 4294967296 distinct integer values, so this variable must have 4294967296 bits. Hence, it would be of size 512 MB. That's a HUGE variable. Maybe you can put such a value on the stack in 100 years.
Consequently, if you truly need to work with (mathematical) sets of integers, you need a custom data structure; the built-in set types won't do. For instance, you could implement it as an advanced record. Then you can even overload the in operator to make it look like a true Pascal set!
Implementing such a slow and inefficient type is trivial, and that might be good enough for small sets. Implementing a general-purpose integer set data structure with efficient operations (membership test, subset tests, intersection, union, etc.) is more work. There might be third-party code available on the WWW (but StackOverflow is not the place for library recommendations).
If your needs are more modest, you can use a simple array of integers instead (TArray<Integer>). Maybe you don't need O(1) membership tests, subset tests, intersections, and unions?
I would say, that such task already requires a database. Something small and simple like TFDMemTable + TFDLocalSQL should do.
While reviewing some code in our legacy Delphi 7 program, I noticed that everywhere there is a record it is marked with packed. This of course means that the record is stored byte-for-byte and not aligned to be faster for the CPU to access. The packing seems to have been done blindly as an attempt to outsmart the compiler or something -- basically valuing a few bytes of memory instead of faster access
An example record:
TFooTypeRec = packed record
RID : Integer;
Description : String;
CalcInTotalIncome : Boolean;
RequireAddress : Boolean;
end;
Should I fix this and make every record normal or "not" packed? Or with modern CPUs and memory is this negligible and probably a waste of time? Are there any problems that can result from unpacking?
There is no way to answer this question without a full understanding of how each of those packed records are used in your application code. It is the same as asking "Should I change this variable declaration from Int64 to Byte ?"
Without knowing what values that variable will be expected and required to maintain the answer could be yes. Or it could be no.
Similarly in your case. If a record needs to be packed then it should be left packed. If it does not need to be packed then there is no harm in not packing it. If you are not sure or cannot tell, then the safest course is to leave them as they are.
As a guide to making this determination (should you decide to proceed), situations where record packing is required or recommended include:
persistence of record values
sharing of record values with [potentially] differently compiled code
strict compatibility with externally defined structures
deliberately overlaying a type layout over differently structured memory
This isn't necessarily an exhaustive list, and what these all have in common is:
records comprising a series of values in adjacent bytes that must and can be relied upon by any potential producer or consumer of the record without possibility of interference from the compiler or other factors
What I would recommend is that (if possible and practical) you determine what purpose packing serves in each case and add documentation to that effect to the record declaration itself so that anyone in the future with the same question doesn't have to go through that discovery process, e.g.:
type
TSomeRecordType = packed record
// This record must be packed as it is used for persistence
..
end;
TSomeExternType = packed record
// This record must be packed as it is required to be compatible
// in memory with an externally defined struct (ref: extern code docs)
..
end;
The main idea of using packed records is not that you save a few bytes of memory! Instead, it is about guaranteeing that the variables are where you expect them to be in memory. Without such a guarantee, it would be impossible (or, at least, difficult) to manage memory manually on the heap and write to and read from files.
Hence, the program might malfunction if you 'unpack' the records!
If the record is stored/retrieved as packed or transfered in any way to a receiver that expects it to be packed, then do not change it.
Update :
There is a string type declared in your example. It looks suspicious, since storing the record in a binary file will not preserve the string content.
Packed record have length exactly size like members are.
No packed record are optimised (thay are aligned -> consequently higher) for better performance.
I can't find the implementation of AtomicCmpExchange (seems to be hidden), so I don't know what it does.
Is AtomicCmpExchange reliable on all platforms? How is it implemented internally? Does it use something like a critical section?
I have this scenario :
MainThread:
Target := 1;
Thread1:
x := AtomicCmpExchange(Target, 0, 0);
Thread2:
Target := 2;
Thread3:
Target := 3;
Will x always be an integer 1, 2 or 3, or could it be something else? I mean, even if the AtomicCmpExchange(Target, 0, 0) failed to exchange the value, does it return a "valid" integer (I mean, not a half-read integer, for exemple if another thread has already started to half write of the value)?
I want to avoid using a critical section, I need maximum speed.
AtomicCmpExchange is what is known as an intrinsic routine, or a standard function. It is intrinsically known to the compiler and may or may not have a visible implementation. For example, Writeln is a standard function, but you won't find a single implementation for it. The compiler breaks it up into multiple calls to lower-level functions in System.pas. Some standard functions, such as Inc() and Dec() don't have any implementation in System.pas. The compiler will generate machine instructions which amount to simple INC or DEC instructions.
Like Inc() or Dec(), AtomicCmpExchange() is implemented using whatever code is needed for a given platform. It will generate inline instructions. For x86/x64 it will generate a CMPXCHG instruction (along with whatever setup is necessary to get variables/values into the registers). For ARM it will generate a few more instructions around the LDREX and STREX instructions.
So the direct answer to your question is that even calling into assembly code, you cannot get much more efficient than using that standard function along with others such as AtomicIncrement, AtomicDecrement, and AtomicExchange.
I'm working on adapting a large Delphi code base to 64-bits. In many cases there are lines where pointers are casted to/from 32-bit values similar to this:
var
p1,p2 : pointer;
begin
inc(Integer(p1),10);
p2 := Pointer(Integer(p1) + 42);
Where I can find these casts I have replaced them with NativeInt-casts instead to make them correct in 64-bit mode.
However I'm not sure I have found them all. Sometimes the casts are more subtle so just text-searching for the string "integer(" is not sufficient either.
Since the "integer(" casts will fail in 64-bit if the pointer value is above the range of integer type I have an idea: what if I could force the memory manager to allocate memory above 4gb (so the pointer values are using more than 32-bits)? Then I would get runtime errors and can more easily find the casts that are wrong. Is this possible? Or can anyone recommend some other technique?
There's no magic trick to finding these casts beyond the sort of text search that you are using. It would be really nice if the compiler warned of such a cast. I find it very disappointing that it doesn't.
When you do find such a problem, don't change to NativeInt. Change the pointers to be typed pointers, and use pointer arithmetic.
var
p1, p2: PByte;
....
inc(p1, 10);
p2 := p2;
inc(p2, 42);
Then your code will be safe forever.
There are still some situations where you need to cast to integers. For example when passing addresses to SendMessage. But cast these to either WPARAM or LPARAM as appropriate.
Your idea of forcing runtime errors is sound and, thankfully for you, not original! You should use the full version of FastMM and define AlwaysAllocateTopDown. This forces the calls that FastMM makes to VirtualAlloc to pass the MEM_TOP_DOWN flag. This will flush out most of your erroneous casts as runtime pointer truncation errors.
However, that will only force top down allocation for memory allocated by your memory manager. Other modules in your process will use the default policy of bottom up. You can set a machine wide setting to change that default policy. Set HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference to REG_DWORD with value 0x100000 and reboot.
Note that this might cause your machine to have stability problems. Many applications cannot cope with this. In particular there are very few anti-virus products that can cope with this setting. MSE is the one that I found works with machine wide top down allocation. What's more the 64 bit debugger does not run under top down allocation! So you have to do this kind of testing without the debugger. My QC report is still open and this problem has not been addressed, even in XE3.
In a Delphi application we are working on we have a big structure of related objects. Some of the properties of these objects have values which are calculated at runtime and I am looking for a way to cache the results for the more intensive calculations. An approach which I use is saving the value in a private member the first time it is calculated. Here's a short example:
unit Unit1;
interface
type
TMyObject = class
private
FObject1, FObject2: TMyOtherObject;
FMyCalculatedValue: Integer;
function GetMyCalculatedValue: Integer;
public
property MyCalculatedValue: Integer read GetMyCalculatedValue;
end;
implementation
function TMyObject.GetMyCalculatedValue: Integer;
begin
if FMyCalculatedValue = 0 then
begin
FMyCalculatedValue :=
FObject1.OtherCalculatedValue + // This is also calculated
FObject2.OtherValue;
end;
Result := FMyCalculatedValue;
end;
end.
It is not uncommon that the objects used for the calculation change and the cached value should be reset and recalculated. So far we addressed this issue by using the observer pattern: objects implement an OnChange event so that others can subscribe, get notified when they change and reset cached values. This approach works but has some downsides:
It takes a lot of memory to manage subscriptions.
It doesn't scale well when a cached value depends on lots of objects (a list for example).
The dependency is not very specific (even if a cache value depends only on one property it will be reset also when other properties change).
Managing subscriptions impacts the overall performance and is hard to maintain (objects are deleted, moved, ...).
It is not clear how to deal with calculations depending on other calculated values.
And finally the question: can you suggest other approaches for implementing cached calculated values?
If you want to avoid the Observer Pattern, you might try to use a hashing approach.
The idea would be that you 'hash' the arguments, and check if this match the 'hash' for which the state is saved. If it does not, then you recompute (and thus save the new hash as key).
I know I make it sound like I just thought about it, but in fact it is used by well-known softwares.
For example, SCons (Makefile alternative) does it to check if the target needs to be re-built preferably to a timestamp approach.
We have used SCons for over a year now, and we never detected any problem of target that was not rebuilt, so their hash works well!
You could store local copies of the external object values which are required. The access routine then compares the local copy with the external value, and only does the recalculation on a change.
Accessing the external objects properties would likewise force a possible re-evaluation of those properties, so the system should keep itself up-to-date automatically, but only re-calculate when it needs to. I don't know if you need to take steps to avoid circular dependencies.
This increases the amount of space you need for each object, but removes the observer pattern. It also defers all calculations until they are needed, instead of performing the calculation every time a source parameter changes. I hope this is relevant for your system.
unit Unit1;
interface
type
TMyObject = class
private
FObject1, FObject2: TMyOtherObject;
FObject1Val, FObject2Val: Integer;
FMyCalculatedValue: Integer;
function GetMyCalculatedValue: Integer;
public
property MyCalculatedValue: Integer read GetMyCalculatedValue;
end;
implementation
function TMyObject.GetMyCalculatedValue: Integer;
begin
if (FObject1.OtherCalculatedValue <> FObjectVal1)
or (FObject2.OtherValue <> FObjectVal2) then
begin
FMyCalculatedValue :=
FObject1.OtherCalculatedValue + // This is also calculated
FObject2.OtherValue;
FObjectVal1 := FObject1.OtherCalculatedValue;
FObjectVal2 := Object2.OtherValue;
end;
Result := FMyCalculatedValue;
end;
end.
In my work I use Bold for Delphi that can manage unlimited complex structures of cached values depending on each other. Usually each variable only holds a small part of the problem. In this framework that is called derived attributes. Derived because the value is not saved in the database, It just depends on on other derived attributes or persistant attributes in the database.
The code behind such attribute is written in Delphi as a procedure or in OCL (Object Constraint Language) in the model. If you write it as Delphi code you have to subscribe to the depending variables. So if attribute C depends on A and B then whenever A or B changes the code for recalc C is called automatically when C is read. So the first time C is read A and B is also read (maybe from the database). As long as A and B is not changed you can read C and got very fast performance. For complex calculations this can save quite a lot of CPU-time.
The downside and bad news is that Bold is not offically supported anymore and you cannot buy it either. I suppose you can get if you ask enough people, but I don't know where you can download it. Around 2005-2006 it was downloadable for free from Borland but not anymore.
It is not ready for D2009 as someone have to port it to Unicode.
Another option is ECO with dot.net from Capable Objects. ECO is a plugin in Visual Studio. It is a supported framwork that have the same idea and author as Bold for Delphi. Many things are also improved, for example databinding is used for the GUI-components. Both Bold and ECO use a model as a central point with classes, attributes and links. Those can be persisted in a database or a xml-file. With the free version of ECO the model can have max 12 classes, but as I remember there is no other limits.
Bold and ECO contains lot more than derived attributes that makes you more productive and allow you to think on the problem instead of technical details of database or in your case how to cache values. You are welcome with more questions about those frameworks!
Edit:
There is actually a download link for Embarcadero registred users for Bold for Delphi for D7, quite old... I know there was updates for D2005, ad D2006.