I found that a SizeOf(set) i different in 32-bit and 64-bit, the example below shows 5 byte for 32-bit and 8 for 64-bit. But i found nothing information about changes in SizeOf(sets) for 64-bit. Is there any Embarcadero documentation about it or compiler directive to get a similar results on 32 and 64-bit.
program Project1;
{$APPTYPE CONSOLE}
{$R *.res}
uses System.SysUtils;
type
{ Enumeration of properties}
TProperty1 = (p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13, p14,
p15, p16, p17, p18, p19, p20, p21, p22, p23, p24, p25, p26, p27, p28,
p29, p30, p31, p32, p33, p34, p35, p36, p37);
TProperties1 = set of TProperty1;
begin
WriteLn(SizeOf(TProperties1));
ReadLn;
end.
To answer your question. I couldn't find anything on the Embarcadero site regarding the differences or a compiler directive to change the behavior. My research indicates the following:
Sets have the following sizes in bytes in 32 bit:
Up to 8 elements - 1 Byte
9 to 16 elements - 2 Bytes
17 to 32 elements - 4 Bytes
From this point onwards it adds adds bytes as needed, one at a time. So 33 to 40 elements uses 5 bytes and 41 to 48 elements uses 6 bytes.
In 64 bit mode, things are slightly different:
Up to 8 elements - 1 Byte
9 to 16 elements - 2 Bytes
17 to 32 elements - 4 Bytes
33 to 64 elements - 8 Bytes
From this point onwards it adds adds bytes as needed, one at a time. So 65 to 72 elements uses 9 bytes and 73 to 80 elements uses 10 bytes.
To get around this you are going to need to either use something like WriteSet in TWriter.WriteProperty and TReader.ReadSet or you can do something like this:
procedure SaveSetToStream(aStream: TStream; const aSet: TProperties1);
var
streamData: array[0..7] of byte;
begin
Assert(SizeOf(aSet) <= SizeOf(streamData), 'Set is too large to save. Increase the array length.');
FillChar(streamData, SizeOf(streamData), 0);
Move(aSet, streamData, SizeOf(aSet));
aStream.Write(streamData, SizeOf(streamData));
end;
function ReadFromStream(aStream: TStream): TProperties1;
var
streamData: array[0..7] of byte;
begin
Assert(SizeOf(Result) <= SizeOf(streamData), 'Set is too large to load. Increase the array length.');
aStream.Read(streamData, SizeOf(streamData));
Move(streamData, Result, SizeOf(Result));
end;
Another workaround, to make sure a 32 bit machine can read a stream from a 64 bit machine and vice-versa is to create a function
function SizeCheck( const p : integer ) : integer;
begin
if p in [5..8 ] then Result := 8 else Result := p; // adjust for 64 bit set sizes
end;
and then use
Stream.Write(set, SizeCheck(SizeOf(set)));
Obviously only use for sets.
Related
Given an enum TEnum with 33 items (Ceil (33 / 8) = 5 bytes), and a TEnumSet = Set of TEnum, the SizeOf (TEnumSet) gives a different result when running in 32 vs. 64-bit Windows:
32 bit: 5 bytes as per the calculation above
64 bit: 8 bytes
When increasing the number of elements in the enum the size will vary to, say, 6 bytes in 32-bit, while in 64-bit, it remains 8 bytes. As if the memory alignment in 64-bit is rounding up the size to the nearest multiple of XX? (not 8, smaller enums do yield a set size of 2, or 4). And a power of 2 is most likely not the case either?
In any case: this is causing a problem while reading a file to a packed record written as a buffer from a 32 bit program. Trying to read the same file back into a 64 bit program, since the packed record sizes don't match (the record contains this mismatching set, among other things), reading fails.
I tried looking in the compiler options for some options related to memory alignment: there is an option for record memory alignment but it does not impact sets, and is already the same in both configurations.
Any explanation on why the set is taking more memory in 64-bit, and any potential solutions to be able to read the file into my packed record on a 64-bit platform?
Note that I have no control over the writing of the file: it is written using a 32-bit program to which I don't have access (so altering the writing is not an option).
Here is my test program:
{$APPTYPE CONSOLE}
type
TEnumSet16 = set of 0..16-1;
TEnumSet17 = set of 0..17-1;
TEnumSet24 = set of 0..24-1;
TEnumSet25 = set of 0..25-1;
TEnumSet32 = set of 0..32-1;
TEnumSet33 = set of 0..33-1;
TEnumSet64 = set of 0..64-1;
TEnumSet65 = set of 0..65-1;
begin
Writeln(16, ':', SizeOf(TEnumSet16));
Writeln(17, ':', SizeOf(TEnumSet17));
Writeln(24, ':', SizeOf(TEnumSet24));
Writeln(25, ':', SizeOf(TEnumSet25));
Writeln(32, ':', SizeOf(TEnumSet32));
Writeln(33, ':', SizeOf(TEnumSet33));
Writeln(64, ':', SizeOf(TEnumSet64));
Writeln(65, ':', SizeOf(TEnumSet65));
end.
And the output (I am using XE7 but I expect that it is the same in all versions):
32 bit
64 bit
16:2
16:2
17:4
17:4
24:4
24:4
25:4
25:4
32:4
32:4
33:5
33:8
64:8
64:8
65:9
65:9
Leaving aside the 32 vs 64 but difference, notice that the 17 and 24 bit cases could theoretically fit in a 3 byte type, they are stored in a 4 byte type.
Why does the compiler choose to use a 4 byte type rather than a 3 byte type? It can only be that this allows for more efficient code. Operating on data that can be mapped directly onto CPU registers is more efficient than picking at the data byte by byte, or in this case by accessing two bytes in one operation, and then the third byte in another.
This then points to why anything between 33 and 64 bits is mapped to an 8 byte type under the 64 bit compiler. The 64 bit compiler has 64 bit registers, and the 32 bit compiler does not.
As for how to solve your problem, then I can see two main approaches:
In your 64 bit program, read and write the record field by field. For the fields which are afflicted by this 32 vs 64 bit issue, you will have to introduce special code to read and write just the first 5 bytes of the field.
Change your record definition to replace the set with array [0..4] of Byte, and then introduce a property that maps the set type onto that 5 byte array.
Working with the memory size of a set leads to process errors sooner or later. This becomes particularly clear when working with subtypes.
program Project1;
{$APPTYPE CONSOLE}
{$R *.res}
uses
System.SysUtils;
type
TBoolSet=set of boolean;
TByteSet=set of byte;
TSubEnum1=5..10;
TSubSet1=set of TSubEnum1;
TSubEnum2=205..210;
TSubSet2=set of TSubEnum2;
var
i, j: integer;
a, a1: TByteSet;
b, b1: TSubSet1;
begin
try
writeln('SizeOf(TBoolSet): ', SizeOf(TBoolSet)); //1
writeln('SizeOf(TByteSet): ', SizeOf(TByteSet)); //32
writeln('SizeOf(TSubSet1): ', SizeOf(TSubSet1)); //2
writeln('SizeOf(TSubSet2): ', SizeOf(TSubSet2)); //2
//Assignments are allowed.
a := [6, 9];
b := [6, 9];
writeln('a = b ?: ', BoolToStr(a = b, true)); //true
a1 := a + b; //OK
b1 := a + b; //OL
a := [7, 200];
b1 := a + b; //??? no exception, Value 200 was lost. !
i := 0;
for j in b1 do
i := succ(i);
writeln('b1 Count: ', i);
readln(i);
except
on E: Exception do
Writeln(E.ClassName, ': ', E.Message);
end;
end.
I have a Delphi 7 code that receives sensor values from an DSP TMS32F28069. The value received by Delphi is Hex file data. For example I send data:
F1;01;01;07;00;00;0A;00;00;00;00;F7
from DSP.
I use Comport 411f and actually when I use windows 10 64 bit english version everything is fine. But when I use windows chinese 64 bit, the data that received sometimes fine sometimes change. I have try on several notebook using windows 7 64 bit chinese version, and it has the same problem. The received files on windows 7 64 bit chinese version showing:
F1;01;01;01;00;00;00;F7;00;00;F7;00.or F1;01;07;01;00;0A;00;00;F7;F7;00;00
and always change.This is the code I wrote in Delphi 7:
procedure TForm1.ComPort1RxChar(Sender: TObject; Count: Integer);
var
p:integer;
r:array[1..12]of integer;
h:array[1..12]of String;
begin
comport1.Open;
for p:=1 to 12 do
begin
comport1.Read(r[p],1);
h[p]:= IntToHex((r[p]),2);
sMemo3.Text:= h[1]+';'+h[2]+';'+h[3]+';'+h[4]+';'+h[5]+';'+h[6]+';'+h[7]+';'+h[8]+';'+h[9]+';'+h[10]+';'+h[11]+';'+h[12];//Show data Receive on Memo4//
end;
end;
Please give me any suggestion why this happened on windows 7 64 bit chinese version? because when I use windows 7 64 bit english version, it was also work fine.
Thank you
Remove comport1.Open - it is undoubtedly opened if RxChar event occurs
Local integer array is filled with some crap. comport1.Read(r[p],1); fills only one byte. So use byte array
You output full data array after every byte - it is strange method.
When event fires, port buffer contains Count bytes - so read real number of bytes. Better approach - accumulate received info in global array (or ansistring) and treat it when 12 bytes are received.
Buffer: AnsiString;
...
procedure TForm1.ComPort1RxChar(Sender: TObject; Count: Integer);
var
sa: AnsiString;
ByteBuf: array[1..12] of Byte;
begin
SetLength(sa, Count);
comport1.Read(sa[1], Count);
Buffer := Buffer + sa;
while Length(Buffer) >= 12 do begin
Move(Buffer[1], ByteBuf, 12);
TreatData(ByteBuf);
Delete(Buffer, 1, 12);
end;
end;
procedure TreatData(bb: array of Byte);
//treat and output here
I know there a many ways to split a String, so that you'll get a StringList.
But my problem is that I want to split every character of the string.
That means the following String:
'That is my Example String'
should be converted to an array/Stringlist or what so ever:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
T h a t i s m y E x a m p l e S t r i n g
In Perl or Java the delimiter field of the split-function is just be let empty like
for example:
Perl: #string=split("",$string);
Java: String[] myStringArray=myString.split("");
What would be the best way for Delphi to manage this?
Usually there is no real need for such function in Delphi, because Delphi string behaves as char array, every element is accessible by index, and char is assignment compatible with string. Thereby you can use s[i] instead of splited[i] in almost all cases.
If you do need special function to fill a list, it may look like
procedure SplitStringEx(const s: string; Splitted: TStrings);
var
i: Integer;
begin
Splitted.Clear;
for i := 1 to Length(s) do
{possible variant for fresh Delph versions
to take zero-based strings into account:}
//for i := Low(s) to High(s) do
Splitted.Add(s[i])
end;
usage
SplitStringEx('abc', Memo1.Lines);
I have such file:
file of record
Str: string[250];
RecType: Cardinal;
end;
but after some time using of this file my customer found, that Str never be bigger than 100 chars and also he need additional fields.
In new version we have such file:
file of packed record
Str: string[200];
Reserved: array[1..47] of Byte;
NewFiled: Cardinal;
RecType: Cardinal;
end;
This record have the same size, in previous record between Str and RecType was one unused byte when aligned to 8 bytes.
Question: what happened, when this new file will be readed from old code? He need backward compatability.
Old code reading sample:
var
FS: TFileStream;
Rec: record
Str: string[250];
RecType: Cardinal;
end;
...
// reading record by record from file:
FS.Read(Rec, SizeOf(Rec));
The old school pascal string use the first byte of the string (index 0) to store the length of the string.
Let's look at the memory of this record:
byte 0 1 2 3 4 5 6 7 8 9 10 11 12 13 ........ 243..246 247..250
value 10 65 66 67 68 69 70 71 72 73 74 0 200 130 NewField RecType
From byte 11 to 242, the memory can contain garbage, it is simply ignored by the program (never shown) as this takes the value 10 at the byte 0 as the length of the string, so the string becomes 'ABCDEFGHIJ'
This ensures the old program reading a file created with the most recent version will never see garbage at the end of the strings, since the view of that strings will be limited to the actual size of the string and that memory positions are just ignored.
You have to double check if the old program does not change the values stored in case it writes the records back to the file. I think it is also safe, but I'm just not sure and have no Delphi at hand to test.
Here's a simple program to check memory allocation. Checking before and after values with Task Manager suggests that each dynamic array takes up 20 bytes of memory at size = 1. The element size is 4, which means 16 bytes of overhead for bookkeeping data.
From looking through system.pas, I can find an array length field at -4 bytes, and a reference count at -8 bytes, but I can't seem to find any references to the other 8. Anyone know what they do?
Sample program:
program Project1;
{$APPTYPE CONSOLE}
type
TDynArray = array of integer;
TLotsOfArrays = array[1..1000000] of TDynArray;
PLotsOfArrays = ^TLotsOfArrays;
procedure allocateArrays;
var
arrays: PLotsOfArrays;
i: integer;
begin
new(arrays);
for I := 1 to 1000000 do
setLength(arrays^[i], 1);
end;
begin
readln;
allocateArrays;
readln;
end.
I had a look into System.pas as well and noticed that the GetMem call in _DynArrayCopyRange supports your analyis:
allocated size = count * element size
+ 2 * Sizeof(Longint)
. So maybe the numbers you get from task manager aren't very accurate. You could try Pointer(someDynArray) := nil and check which memory leak size FastMM reports for more reliable numbers.
Edit: I did a little test program:
program DynArrayLeak;
{$APPTYPE CONSOLE}
uses
SysUtils;
procedure Test;
var
arr: array of Integer;
i: Integer;
begin
for i := 1 to 6 do
begin
SetLength(arr, i);
Pointer(arr) := nil;
end;
end;
begin
ReportMemoryLeaksOnShutdown := True;
Test;
end.
This yields
An unexpected memory leak has occurred. The unexpected small block leaks are:
1 - 12 bytes: Unknown x 1
13 - 20 bytes: Unknown x 2
21 - 28 bytes: Unknown x 2
29 - 36 bytes: Unknown x 1
which supports the 8 byte overhead theory.
Memory allocations have granularity to ensure all allocations are aligned. This is just the slop caused by this.
Updated...
I actually went to check the code (which I should've done before) and I came to the same conclusion as Ulrich, it's not storing any type information, just the 2 Longint overhead then NbElements*ElementSize.
And, Task manager is not accurate for this kind of measure.
With the oddity that if you measure the memory used by the dynarray, it increases non linearly with the size of the element: for a Record with 2 or 3 Integers it's the same size (20), with 4 or 5 it's 28... following the granularity of the blocksizes.
Memory measured with:
// Return the total Memory used as reported by the Memory Manager
function MemoryUsed: Cardinal;
var
MemMgrState: TMemoryManagerState;
SmallBlockState: TSmallBlockTypeState;
begin
GetMemoryManagerState(MemMgrState);
Result := MemMgrState.TotalAllocatedMediumBlockSize + MemMgrState.TotalAllocatedLargeBlockSize;
for SmallBlockState in MemMgrState.SmallBlockTypeStates do begin
Result := Result + SmallBlockState.UseableBlockSize * SmallBlockState.AllocatedBlockCount;
end;
end;