I need to read a large (2000x2000) matrix of binary data from a file into a dynamic array with Delphi 2010. I don't know the dimensions until run-time.
I've never read raw data like this, and don't know IEEE so I'm posting this to see if I'm on track.
I plan to use a TFileStream to read one row at a time.
I need to be able to read as many of these formats as possible:
16-bit two's complement binary integer
32-bit two's complement binary integer
64-bit two's complement binary integer
IEEE single precision floating-point
For 32-bit two's complement, I'm thinking something like the code below. Changing to Int64 and Int16 should be straight forward. How can I read the IEEE?
Am I on the right track? Any suggestions on this code, or how to elegantly extend it for all 4 data types above? Since my post-processing will be the same after reading this data, I guess I'll have to copy the matrix into a common format when done.
I have no problem just having four procedures (one for each data type) like the one below, but perhaps there's an elegant way to use RTTI or buffers and then move()'s so that the same code works for all 4 datatypes?
Thanks!
type
TRowData = array of Int32;
procedure ReadMatrix;
var
Matrix: array of TRowData;
NumberOfRows: Cardinal;
NumberOfCols: Cardinal;
CurRow: Integer;
begin
NumberOfRows := 20; // not known until run time
NumberOfCols := 100; // not known until run time
SetLength(Matrix, NumberOfRows);
for CurRow := 0 to NumberOfRows do
begin
SetLength(Matrix[CurRow], NumberOfCols);
FileStream.ReadBuffer(Matrix[CurRow], NumberOfCols * SizeOf(Int32)) );
end;
end;
No, AFAIK there's no way to use RTTI to set up multidimensional arrays. But if you're using Delphi 2010, you should be able to use generics, like so:
type
TRowData<T> = array of T;
procedure ReadMatrix<T>;
var
Matrix: array of TRowData<T>;
NumberOfRows: Cardinal;
NumberOfCols: Cardinal;
CurRow: Integer;
begin
NumberOfRows := 20; // not known until run time
NumberOfCols := 100; // not known until run time
SetLength(Matrix, NumberOfRows, NumberOfCols);
for CurRow := 0 to NumberOfRows do
FileStream.ReadBuffer(Matrix[CurRow][0], NumberOfCols * SizeOf(T)) );
end;
This will have to be in a class, though, as Delphi 2010 doesn't support standalone procedures with generic types. Once you've got this set up, you can call TWhateverClass.ReadMatrix<integer>, TWhateverClass.ReadMatrix<int64>, TWhateverClass.ReadMatrix<single>, and so on.
Also, if you have a multidimensional array with X dimensions, you can pass X length parameters to SetLength, not just one. So use one call to SetLength(Matrix, NumberOfRows, NumberOfCols) outside the loop, instead of initializing each row separately to the same width.
Related
I am trying to pass an array of bytes from VBscript to my windows Delphi Application and can't seem to find the correct syntax to interpret the passed data.
The requirement is fairly simple as the VBscript snippet below demonstrates
Dim inst,arr(5)
Sub Main
set inst=instruments.Find("EP1")
arr(0) = 0
arr(1) = 1
arr(2) = 2
arr(3) = 3
arr(4) = 4
inst.writebytes arr,5
end Sub
I can get the server to accept the olevariant passed by the script but the data seems garbled, my example server code is shown below and is based on the Stackoverflow question here How to use variant arrays in Delphi
procedure TInstrument.WriteBytes(Data: OleVariant; Length: Integer);
var i,n:integer; Pdat:Pbyte; Adata:PvarArray;
begin
if VarIsArray(data) then
begin
n:=TVarData(Data).VArray^.Bounds[0].ElementCount;
Adata:= VarArrayLock(Data);
Getmem(Pdat,length);
try
for i:=0 to length-1 do
Pdat[i]:=integerArray(Adata.data^)[i];
Finstrument.WriteBytes(Pdat,Length);
finally
freemem(Pdat)
end;
end;
end;
So the idea is to accept the integers passed by the script, convert it to the local data representation (array of byte) then pass it on to my function to use the data.
I have tried several different data types and methods to try and get some ungarbled data out of the variant all to no avail.
What is the correct method of extracting the array data from the passed variant?
Also, TVarData(Data).VArray^.Bounds[0].ElementCount has a value of zero, why would that be?
Arrays created in VBScript are
zero based
untyped
declared with upper bound (not size as you assumed; size of array declared as Dim arr(5) is 6)
include dimension info in them (so you don't need to pass it along with the array)
When used in COM, they are passed as variant arrays of type varVariant (as the Ondrej Kelle points out in his comment). To process such an array in your method you have to assert that:
the value is a single dimensional array
each element can be converted to byte
You can write helper routine for that:
function ToBytes(const Data: Variant): TBytes;
var
Index, LowBound, HighBound: Integer;
ArrayData: Pointer;
begin
if not VarIsArray(Data) then
raise EArgumentException.Create('Variant array expected.');
if VarArrayDimCount(Data) <> 1 then
raise EArgumentException.Create('Single dimensional variant array expected.');
LowBound := VarArrayLowBound(Data, 1);
HighBound := VarArrayHighBound(Data, 1);
SetLength(Result, HighBound - LowBound + 1);
if TVarData(Data).VType = varArray or varByte then
begin
ArrayData := VarArrayLock(Data);
try
Move(ArrayData^, Result[0], Length(Result));
finally
VarArrayUnlock(Data);
end;
end
else
begin
for Index := LowBound to HighBound do
Result[Index - LowBound] := Data[Index];
end;
end;
for loop in the routine will be horribly slow when processing large arrays, so there's optimization for special case (variant array of bytes) that uses Move to copy bytes to result. But this will never happen with VBScript array. You might consider using VB.Net or PowerShell.
Using such a routine has downside of keeping 2 instances of the array in memory - as variant array and as byte array. Use it as a guide when applying it to your use case.
In Delphi, I declared a 3x3 matrix table as an array of array of Single, like this:
m_Table: array [0..2] of array [0..2] of Single;
Now I want to memory compare the content with another table, or memory copy the table content from another table. I know that I can create a nested loop to do that, but I want to do the job without any loop, if possible.
My question is, it is correct to copy or compare the memory like this:
CompareMem(m_Table, other.m_Table, 9 * SizeOf(Single));
CopyMemory(m_Table, other.m_Table, 9 * SizeOf(Single));
If not, what is the correct way to do that?
And as a subsidiary question, is there a better way to get the length to copy instead of 9 * SizeOf(Single), like e.g. SizeOf(m_Table^)?
Regards
The code in the question works fine. Personally I would say that Move is the idiomatic way to copy memory. Further I would use SizeOf(m_Table) to obtain the size of the type.
I would point out that your comparison differs from the floating point equality operator. Perhaps that's what you want, but you should be aware of this. For instance zero and minus zero compare equal using floating point comparison but not with memory compare. And NaNs always compare not equal, even with identical bit patterns.
Let me also comment that it would make your code more extendible if you declared a type for these matrices. Without that you won't be able to write functions that accept such objects.
The correct and easiest way may be to define a type:
type
TMatrix3x3 = array [0..2,0..2] of Single;
Then you can directly write:
var
v1, v2: TMatrix3x3;
begin
fillchar(v1,sizeof(v1),0);
move(v1,v2,sizeof(v1));
if comparemem(#v1,#v2,sizeof(v1)) then
writeln('equals');
end;
Using sizeof() make your code safe and readable.
You may define a wrapper type with methods:
{ TMatrix3x3 }
type
TMatrix3x3 = record
v: array [0..2,0..2] of Single;
procedure Zero;
procedure Copy(var dest: TMatrix3x3);
procedure Fill(const source: TMatrix3x3);
function Equals(const other: TMatrix3x3): boolean;
end;
procedure TMatrix3x3.Copy(var dest: TMatrix3x3);
begin
move(v,dest,sizeof(v));
end;
function TMatrix3x3.Equals(const other: TMatrix3x3): boolean;
begin
result := CompareMem(#v,#other.v,sizeof(v));
end;
procedure TMatrix3x3.Fill(const source: TMatrix3x3);
begin
move(source,v,sizeof(v));
end;
procedure TMatrix3x3.Zero;
begin
fillchar(v,sizeof(v),0);
end;
Including then advanced features like implicit assignment, and operators, if needed.
But don't reinvent the wheel, if you really to work with matrix arithmetic. Use an already existing and fully tested library, which will save you a lot of trouble and debugging time.
You should use standard TMatrix type from System.Math.Vectors unit, then you can just compare it directly as if Matrix1 = Matrix2 then and assign as Matrix1 := Matrix2
How can I create a random 64-bit integer value in Delphi 2006? The built-in integer-based Random() function seems to return only values between 0 and 2^31.
You can use my GpRandomGen. It implements Marsaglia/Zaman/James algorithm, is extremely fast and supposedly very random. Released as a freeware.
Generate two 32 bit randoms and splice them together.
EDIT
Similar to #Andreas's answer I like the following (equivalent) implementation:
function Random64: UInt64;
var
Overlay: packed record
a, b: UInt32;
end absolute Result;
begin
Assert(SizeOf(Overlay)=SizeOf(Result));
Overlay.a := Random32;
Overlay.b := Random32;
end;
To answer my own question I came up with the following code:
function GetRandomInt64() : int64;
begin
Int64Rec(result).Words[0] := Random(High(Word));
Int64Rec(result).Words[1] := Random(High(Word));
Int64Rec(result).Words[2] := Random(High(Word));
Int64Rec(result).Words[3] := Random(High(Word));
end;
Not sure if this is a valid solution or it will always create the same follow-up number X+1 after a given result number X.
You can generate 64 random bits and interpret the result as an integer. (63 bits if you are working with signed integers and want the result to be non-negative.) Equivalently you can take two random integers in the range 0..2^31-1, plus two extra random bits, and concatenate them to get a random 64-bit integer.
EDIT: I was curious about the statistical properties of pseudo-random numbers generated by concatenating pseudo-random components and found that (apparently) this approach might not work well depending on your pseudo-random generator (of course for true random number generation, as from atmospheric noise, concatenating random bits is no problem). For recreational use, the loss of various statistical properties might be acceptable, but for more serious use you might end up needing a custom pseudo-random generator as #gabr suggested. Here is a related question: Best method of generating a number with 256 random bits?
Create a GUID (eg CoCreateGuid) and cast it to Int64.
Simple:
function Random64: UInt64;
begin
PCardinal(#result)^ := Random32;
PCardinal(cardinal(#result) + 4)^ := Random32;
end;
where Random32 is your favourite 32-bit unsigned integer random number function.
I would like to use the FFTW C library from Delphi 2009 and according to this documentation;
http://www.fftw.org/install/fftw_usage_from_delphi.txt
to increase the performance inside the FFTW library (such that it can use SIMD extensions) arrays passed in of either Single (float) or Double (double) need to be aligned either at 4 or 8 byte boundaries. I found documentation talking about alignment of record structures, but nothing specific about arrays. Is there a way to do this in Delphi 2009.
So the code (copied from the above documentation) would look like this;
var
in, out : Array of Single; // Array aligned at 4 byte boundary
plan : Pointer;
{$APPTYPE CONSOLE}
begin
...
SetLength(in, N);
SetLength(out, N);
plan := _fftwf_plan_dft_1d(dataLength, #in[0], #out[0],
FFTW_FORWARD, FFTW_ESTIMATE);
Also in the above documentation they talk about 8 and 16 byte boundaries but it looks to me it should be 4 and 8 byte boundaries, if any could clear that up to that would be great.
Thanks,
Bruce
Note that you can create data structures with any custom alignment you might need. For example to align your FFT data on 128 byte boundaries:
procedure TForm1.Button1Click(Sender: TObject);
type
TFFTData = array[0..63535] of double;
PFFTData = ^TFFTData;
var
Buffer: pointer;
FFTDataPtr: PFFTData;
i: integer;
const
Alignment = 128; // needs to be power of 2
begin
GetMem(Buffer, SizeOf(TFFTData) + Alignment);
try
FFTDataPtr := PFFTData((LongWord(Buffer) + Alignment - 1)
and not (Alignment - 1));
// use data...
for i := Low(TFFTData) to High(TFFTData) do
FFTDataPtr[i] := i * pi;
finally
FreeMem(Buffer);
end;
end;
Edit:
Regarding the comment about twice the memory being allocated: The stack variable FFTData is of type PFFTData, not of TFFTData, so it's a pointer. It's not that obvious because of the syntax enhancement allowing to omit the ^ for dereferencing the pointer. The memory is allocated with GetMem(), and to work with the proper type instead of the untyped memory block the typecast is employed. I should probably have called it FFTDataPtr.
Delphi provides no way to control the alignment of any memory it allocates. You're left to either rely on the documented behavior for the memory manager currently installed, or allocate memory with some slack space and then align it yourself, as Mghie demonstrates.
If you're concerned that Delphi's memory manager is not providing the desired alignment for dynamic arrays, then you can go ahead and use the memory functions provided by the DLL. The note you cite mentions _fftwf_malloc and _fftwf_free, but then it gives some kind of warning that memory allocated from _fftwf_malloc "may not be accessed directly from Delphi." That can't be what the authors meant to say, though, because that's not how memory works in Windows. The authors probably meant to say that memory allocated by _fftwf_malloc cannot be freed by Delphi's FreeMem, and memory allocated by Delphi's GetMem cannot be freed by _fftwf_free. That's nothing special, though; you always need to keep your memory-management functions paired together.
If you use _fftwf_malloc to get your array, then you can access it through an ordinary pointer type. For example:
var
dataIn, dataOut: PDouble;
begin
dataIn := _fftwf_malloc(...);
dataOut := _fftwf_malloc(...);
_fftwf_plan_dft_1d(dataLength, dataIn, dataOut,
FFTW_FORWARD, FFTW_ESTIMATE);
As of Delphi 2009, you can even use array syntax on those pointers:
dataIn[0] := 3.5;
dataIn[2] := 7.3;
In order to enable that, use the {$POINTERMATH ON} compiler directive; it's not enabled by default except for the character-pointer types.
The disadvantage to manually allocating arrays like this is that you lose range checking. If you index beyond the end of an array, you won't get an easy-to-recognize ERangeError exception anymore. You'll get corrupted memory, access violations, or mysteriously crashing programs instead.
Heap blocks are iirc always aligned to 16-byte bounderies by FastMM (the old D7 memmanager aligned to 8). I don't know about sharemem, since I don't use it.
And dynamic arrays are heap based structures. OTOH dyn arrays could maybe become unaligned (from 16 to 8) because there is a length and ref count prefixed. Easiest is to simply print
ptruint(#in[0]) in hex and see if the end is 0 or 8. (*)
Note that there are fftw headers in FPC. ( packages/fftw), afaik it was recently fixed for 64-bit even.
I'm not aware of Stack alignment directives in Delphi. Maybe they are automatically "naturally" aligned though.
(*) ptruint is FPC speak for an unsigned integer type that is sizeof(pointer) large. cardinal on 32-bit, qword on 64-bit.
This is another possible variant of mghie's solution:
procedure TForm1.Button1Click(Sender: TObject);
type
TFFTData = array [0..0] of Double;
PFFTData = ^TFFTData;
var
AllocatedBuffer: Pointer;
AlignedArray: PFFTData;
i: Integer;
const
cFFTDataSize=63536;
begin
GetMem(AllocatedBuffer, cFFTDataSize*SizeOf(Double) + 16); // e.g 16 Bytes boudaries alignement
try
AlignedArray := PFFTData((Integer(AllocatedBuffer) and $FFFFFFF0) + 16);
// use data...
for i := 0 to cFFTDataSize-1 do
AlignedArray[i] := i * Pi;
finally
FreeMem(AllocatedBuffer);
end;
end;
I've refactored the piece of code to make it more meaningfull and make use of a similar manual alignement fix up technique.
I originally had an array[1..1000] that was defined as a global variable.
But now I need that to be n, not 1000 and I don't find out n until later.
I know what n is before I fill the array up but I need it to be global therefore need a way to define the size of a global array at run time.
Context is filling an array with a linear transformation of the bytes in a file.
I don't know how big the file is until someone wants to open it and the files can be of any size.
As of Delphi 4, Delphi supports dynamic arrays. You can modify their sizes at run time and they will retain the data you stored in other elements at the old size. They can hold elements of any homogeneous type, including records and other arrays. You can declare a dynamic array the same as you declare normal, "static" arrays, but simply omit the array bounds:
var
ArthurArray: array of TForm;
Although static arrays allow you to specify both the lower and upper bound, the low index of a dynamic array is always zero. The high index is given by the High function, which always returns one less than the length of the array. For any dynamic array x, High(x) = Length(x)-1.
A global variable can be accessed by any code, including local procedures.
A global variable of dynamic-array type will be initialized to be an empty array. Its length will be zero and High called on that array will be -1. Low on that array will still return zero.
At any time, you may resize a dynamic array. Use the SetLength function, just as you can do with strings:
var
NumElements: Integer;
begin
NumElements := GetNumberOfArthurForms();
SetLength(ArthurArray, NumElements);
end;
If you have a multidimensional array, you can set their lengths in a loop:
var
matrix: array of array of Double;
i: Integer;
begin
SetLength(matrix, height);
for i := 0 to height - 1 do
SetLength(matrix[i], width);
end;
There's a shortcut for that to set the lengths of all the inner arrays at once:
begin
SetLength(matrix, height, width);
end;
Like I mentioned, dynamic arrays keep their old values when you resize them:
var
data: array of string;
begin
SetLength(data, 2);
data[1] := 'foo';
SetLength(data, 20);
Assert(data[1] = 'foo');
end;
But if you shorten the array, any elements that resided beyond the new last element are gone forever:
begin
SetLength(data, 20);
data[15] := 'foo';
SetLength(data, 2);
// data[15] does not exist anymore.
SetLength(data, 16);
writeln(data[15); // Should print an *empty* line.
end;
My demonstrations above used strings. Strings are special in Delphi; they're managed by the compiler through reference counts. Because of that, new dynamic-array elements of type string are initialized to be empty. But if I had used integers instead, there would be no guarantee of the values of new elements. They might be zero, but they might be anything else, too, just like the initial values of standalone local variables.
The Delphi 7 help files are very good, I'm told. Please read more about dynamic arrays there. You can find demonstrations of their use throughout the VCL and RTL source code provided in your Delphi installation, as well as in nearly any Delphi code example produced in the last 10 years.
First, here's a general answer to the first part of your question:
If your array is no longer static, you might want to consider using a TList, a TStringList or one of the many container classes in the Contnrs unit.
They may better represent what you are doing, provide additional capabilities you might need, e.g. sorting or name/value pairs, they dynamically grow as you need them, and have been very well optimized.
Then you said:
"Context is filling an array with a linear transformation of the bytes in a file. I don't know how big the file is until someone wants to open it and the files can be of any size."
For your specific problem, I would load the bytes in a file using:
MyFileStream := TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite);
Size := MyFileStream.Size - MyFileStream.Position;
SetLength(Buffer, Size);
MyFileStream.Read(Buffer[0], Size);
Then you can easily use a PChar pointer to go through each character or even each byte in the Buffer one by one and transform them the way you need to.