Given a buffer and its size in bytes, is there a way to convert this to TBytes without copying it?
Example:
procedure HandleBuffer(_Buffer: PByte; _BufSize: integer);
var
Arr: TBytes;
i: Integer;
begin
// some clever code here to get contents of the buffer into the Array
for i := 0 to Length(Arr)-1 do begin
HandleByte(Arr[i]);
end;
end;
I could of course copy the data:
procedure HandleBuffer(_Buffer: PByte; _BufSize: integer);
var
Arr: TBytes;
i: Integer;
begin
// this works but is very inefficient
SetLength(Arr, _BufSize);
Move(PByte(_Buffer)^, Arr[0], _BufSize);
//
for i := 0 to Length(Arr)-1 do begin
HandleByte(Arr[i]);
end;
end;
But for a large buffer (about a hundred megabytes) this would mean I have double the memory requirement and also spend a lot of time unnecessarily copying data.
I am aware that I could simply use a PByte to process each byte in the buffer, I'm only interested in a solution to use a TBytes instead.
I think it's not possible, but I have been wrong before.
No, this is not possible (without unreasonable hacks).
The problem is that TBytes = TArray<Byte> = array of Byte is a dynamic array and the heap object for a non-empty dynamic array has a header containing the array's reference count and length.
A function that accepts a TBytes parameter, when given a plain pointer to an array of bytes, might (rightfully) attempt to read the (non-existing) header, and then you are in serious trouble.
Also, dynamic arrays are managed types (as indicated by the reference count I mentioned), so you might have problems with that as well.
However, in your particular example code, you don't actually use the dynamic array nature of the data at all, so you can work directly with the buffer:
procedure HandleBuffer(_Buffer: PByte; _BufSize: integer);
var
i: Integer;
begin
for i := 0 to _BufSize - 1 do
HandleByte(_Buffer[i]);
end;
Related
I have this code:
procedure MyFunct(const aBin; aBinSize : Cardinal);
var bytes: Tbytes;
begin
bytes := Tbytes(#aBin);
for var I := 0 to aBinSize - 1 do
writeln(bytes[i]);
end;
var Memory: Pointer
...init the memory...
MyFunct(Memory^, sizeOfMemory);
this was working very well for several years with {$R-} (range check off). however today I decide to deactivate {$R-} and now the code below crash with range check error and when yes that normal because I do length(bytes) it's often equal to 0.
So I can reactivate the {$R-} but now I think it's a fundamental mistake because as far as I understand the length of a Tbyte is store at bytes[-32bit] and most important the reference count of the Tbytes is store at bytes[-64bit]. So now I m affraid that the code before was simply writen the reference count in bytes[-64bit] and destroying my memory (maybe not not sure).
so is it a good practice to do
bytes := Tbytes(#aBin);
If not why the compiler authorize it ? How without a Tbytes I can navigate through each byte of my memory (ie how to access myMemory[x])
You can't type-cast an arbitrary pointer to a TBytes like you are, they are completely different things. The code will fail if the memory being pointed at is not a valid dynamic array to begin with. Your code has been faulty for years, and you are just lucky it did anything at all.
The function needs to look more like this instead when using TBytes as you are:
procedure MyFunct(const aBin; aBinSize : Cardinal);
var bytes: TBytes;
begin
SetLength(bytes, aBinSize);
Move(aBin, bytes[0], aBinSize);
for var I := 0 to aBinSize - 1 do
WriteLn(bytes[i]);
end;
Otherwise, a simpler approach (which is likely what you were attempting to do) would be more like this instead:
procedure MyFunct(const aBin; aBinSize : Cardinal);
var bytes: PByte;
begin
bytes := PByte(#aBin);
for var I := 0 to aBinSize - 1 do
WriteLn(bytes[i]);
end;
I'm trying to read a PGM binary file (http://netpbm.sourceforge.net/doc/pgm.html) to fill a 0-based 2D matrix of integers (16-bit grayscale values).
The file may be 50 megs, so I'm trying to fill a buffer in one call.
I've never done anything with Streams before, but the Google results on Delphi streams going back 20 years and are a cluttered mess in which I couldn't find my way.
I've managed to lock up Delphi (first time in 15 years!) while running some code that uses pointers and buffers (and probably is based on my misunderstanding of an antiquated approach.)
Here's my pseudo code, doing it integer by integer. Is there a way to do the read and fill of the matrix with a single Stream call? (Assuming the file was created on the same machine, so byte-sex is the same.)
type
TMatrix: Array of Array of Integer;
procedure ReadMatrix( const AFileName: String;
const AStartingByte: Integer;
const AMaxRow: Integer;
const AMaxCol: Integer;
const AMatrix: TMatrix)
begin
SetLength(AMatrix, aMaxRow, aMaxCol);
Open(AFileName);
Seek(AStartingByte);
for Row := 0 to aMaxCol do
for Col := 0 to aMaxCol do
AMatrix[Row, Col] := ReadWord
end;
And, no, this isn't a homework assignment! :-)
As already stated, you cannot read 2D dynamic array in a single operation, because its memory is non continuous. But every 1D subarray can be filled.
I also changed array element type to 16-bit. If you really need matrix of Integer (it is 32 bit), then you have to read 16-bit data and assign elements to Integers one-by-one
type
TMatrix = Array of Array of Word;
procedure ReadMatrix( const AFileName: String;
const AStartingByte: Integer;
const AMaxRow: Integer;
const AMaxCol: Integer;
const AMatrix: TMatrix)
var
FS: TFileStream;
Row: Integer;
begin
SetLength(AMatrix, aMaxRow, aMaxCol);
FS := TFileStream.Create(AFileName, fmOpenRead);
try
FS.Position := AStartingByte;
for Row := 0 to aMaxRow - 1 do
FS.Read(AMatrix[Row, 0], SizeOf(Word) * aMaxCol);
finally
FS.Free;
end;
end;
Following on from this question (Dynamic arrays and memory management in Delphi), if I create a dynamic array in Delphi, how do I access the reference count?
SetLength(a1, 100);
a2 := a1;
// The reference count for the array pointed to by both
// a1 and a2 should be 2. How do I retrieve this?
Additionally, if the reference count can be accessed, can it also be modified manually? This latter question is mainly theoretical rather than for use practically (unlike the first question above).
You can see how the reference count is managed by inspecting the code in the System unit. Here are the pertinent parts from the XE3 source:
type
PDynArrayRec = ^TDynArrayRec;
TDynArrayRec = packed record
{$IFDEF CPUX64}
_Padding: LongInt; // Make 16 byte align for payload..
{$ENDIF}
RefCnt: LongInt;
Length: NativeInt;
end;
....
procedure _DynArrayAddRef(P: Pointer);
begin
if P <> nil then
AtomicIncrement(PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt);
end;
function _DynArrayRelease(P: Pointer): LongInt;
begin
Result := AtomicDecrement(PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt);
end;
A dynamic array variable holds a pointer. If the array is empty, then the pointer is nil. Otherwise the pointer contains the address of the first element of the array. Immediately before the first element of the array is stored the metadata for the array. The TDynArrayRec type describes that metadata.
So, if you wish to read the reference count you can use the exact same technique as does the RTL. For instance:
function DynArrayRefCount(P: Pointer): LongInt;
begin
if P <> nil then
Result := PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt
else
Result := 0;
end;
If you want to modify the reference count then you can do so by exposing the functions in System:
procedure DynArrayAddRef(P: Pointer);
asm
JMP System.#DynArrayAddRef
end;
function DynArrayRelease(P: Pointer): LongInt;
asm
JMP System.#DynArrayRelease
end;
Note that the name DynArrayRelease as chosen by the RTL designers is a little mis-leading because it merely reduces the reference count. It does not release memory when the count reaches zero.
I'm not sure why you would want to do this mind you. Bear in mind that once you start modifying the reference count, you have to take full responsibility for getting it right. For instance, this program leaks:
{$APPTYPE CONSOLE}
var
a, b: array of Integer;
type
PDynArrayRec = ^TDynArrayRec;
TDynArrayRec = packed record
{$IFDEF CPUX64}
_Padding: LongInt; // Make 16 byte align for payload..
{$ENDIF}
RefCnt: LongInt;
Length: NativeInt;
end;
function DynArrayRefCount(P: Pointer): LongInt;
begin
if P <> nil then
Result := PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt
else
Result := 0;
end;
procedure DynArrayAddRef(P: Pointer);
asm
JMP System.#DynArrayAddRef
end;
function DynArrayRelease(P: Pointer): LongInt;
asm
JMP System.#DynArrayRelease
end;
begin
ReportMemoryLeaksOnShutdown := True;
SetLength(a, 1);
Writeln(DynArrayRefCount(a));
b := a;
Writeln(DynArrayRefCount(a));
DynArrayAddRef(a);
Writeln(DynArrayRefCount(a));
a := nil;
Writeln(DynArrayRefCount(b));
b := nil;
Writeln(DynArrayRefCount(b));
end.
And if you make a call to DynArrayRelease that takes the reference count to zero then you would also need to dispose of the array, for reasons discussed above. I've never encountered a problem that would require manipulation of the reference count, and strongly suggest that you avoid doing so.
One final point. The RTL does not offer this functionality through its public interface. Which means that all of the above is private implementation detail. And so is subject to change in a future release. If you do attempt to read or modify the reference count then you must recognise that doing so relies on such implementation detail.
After some googling, I found an excellent article by Rudy Velthuis. I highly recommend to read it. Quoting dynamic arrays part from http://rvelthuis.de/articles/articles-pointers.html#dynarrays
At the memory location below the address to which the pointer points, there are two more fields, the number of elements allocated, and the reference count.
If, as in the diagram above, N is the address in the dynamic array variable, then the reference count is at address N-8, and the number of allocated elements (the length indicator) at N-4. The first element is at address N.
How to access these:
SetLength(a1, 100);
a2 := a1;
// Reference Count = 2
refCount := PInteger(NativeUInt(#a1[0]) - SizeOf(NativeInt) - SizeOf(Integer))^;
// Array Length = 100
arrLength := PNativeInt(NativeUInt(#a1[0]) - SizeOf(NativeInt))^;
The trick in computing proper offsets is to account for differences between 32bit and 64bit platforms code. Fields size in bytes is as follows:
32bit 64bit
RefCount 4 4
Length 4 8
As the topic indicates above, I'm wondering if there's a good example of a clean and efficient way to handle pointers as passed in function parms when processing the data sequentially. What I have is something like:
function myfunc(inptr: pointer; inptrsize: longint): boolean;
var
inproc: pointer;
i: integer;
begin
inproc := inptr;
for i := 1 to inptrsize do
begin
// do stuff against byte data here.
inc(longint(inproc), 1);
end;
end;
The idea is that instead of finite pieces of data, I want it to be able to process whatever is pushed its way, no matter the size.
Now when it comes to processing the data, I've figured out a couple of ways to do it successfully.
Assign the parm pointers to identical temporary pointers, then use those to access each piece of data, incrementing them to move on. This method is quickest, but not very clean looking with all the pointer increments spread all over the code. (this is what I'm talking about above)
Assign the parm pointers to a pointer representing a big array value and then incremently process that using standard table logic. Much cleaner, but about 500 ms slower than #1.
Is there another way to efficiently handle processing pointers in this way, or is there some method I'm missing that will both be clean and not time inefficient?
Your code here is basically fine. I would always choose to increment a pointer than cast to a fake array.
But you should not cast to an integer. That is semantically wrong and you'll pay the penalty anytime you compile on a platform that has pointer size different from your integer size. Always use a pointer to an element of the right size. In this case a pointer to byte.
function MyFunc(Data: PByte; Length: Integer): Boolean;
var
i: Integer;
begin
for i := 1 to Length do
begin
// do stuff against byte data here.
inc(Data);
end;
end;
Unless the compiler is having a really bad day, you won't find it easy to get better performing code than this. What's more, I think this style is actually rather clear and easy to understand. Most of the clarity gain comes in avoiding the need to cast. Always strive to remove casts from your code.
If you want to allow any pointer type to be passed then you can write it like this:
function MyFunc(P: Pointer; Length: Integer): Boolean;
var
i: Integer;
Data: PByte;
begin
Data := P;
for i := 1 to Length do
begin
// do stuff against byte data here.
inc(Data);
end;
end;
Or if you want to avoid pointers in the interface, then use an untyped const parameter.
function MyFunc(const Buffer; Length: Integer): Boolean;
var
i: Integer;
Data: PByte;
begin
Data := PByte(#Buffer);
for i := 1 to Length do
begin
// do stuff against byte data here.
inc(Data);
end;
end;
Use a var parameter if you need to modify the buffer.
I have a different opinion: For sake of readability I would use an array. Pascal was not designed to be able to access memory directly. Original pascal did not even have pointer arithmetic.
This is how I would use an array:
function MyFunc(P: Pointer; Length: Integer): Boolean;
var
ArrayPtr : PByteArray Absolute P;
I : Integer;
begin
For I := 0 to Length-1 do
// do stuff against ArrayPtr^[I]
end;
But if performance matters, I would write it like this
function MyFunc(P: Pointer; Length: Integer): Boolean;
var
EndOfMemoryBlock: PByte;
begin
EndOfMemoryBlock := PByte(Int_Ptr(Data)+Length);
While P<EndOfMemoryBlock Do begin
// do stuff against byte data here.
inc(P);
end;
end;
Question One
I have
var example : array[0..15] of char;
I want to assign the value from an input to that variable
example := inputbox('Enter Name', 'Name', '');
In the highscores unit I have record and array
type
points = record
var
_MemoryName : array[0..15] of char;
_MemoryScore : integer;
end;
var
rank : array[1..3] of points;
var s: string;
a: packed array[0..15] of char;
highscoresdata.position[1]._MemoryName := StrPLCopy(a, s, Length(a)) ;
returns -> (186): E2010 Incompatible types: 'array[0..15] of Char' and 'PWideChar'
var s: string;
a: packed array[0..15] of char;
s := InputBox('caption', 'Caption', 'Caption');
FillChar(a[0], length(a) * sizeof(char), #0);
Move(s[1], a[0], length(a) * sizeof(char));
scores.rank[1]._MemoryName := <<tried both s and a>> ;
returns (189): E2008 Incompatible types
Question One
There are many ways. One is:
procedure TForm1.FormCreate(Sender: TObject);
var
s: string;
a: packed array[0..15] of char;
begin
s := InputBox(Caption, Caption, Caption);
assert(length(s) <= 16);
FillChar(a[0], length(a) * sizeof(char), #0);
Move(s[1], a[0], length(s) * sizeof(char));
end;
But there might be a more elegant solution to your original problem, I suspect.
Question Two
Every time you wish a function/procedure didn't have a particular argument, you should realize that there might be a problem with the design of the project. Nevertheless, it isn't uncommon that Sender parameters are superfluous, because they are almost omnipresent because of the design of the VCL (in particular, the TNotifyEvent). If you know that the receiving procedure doesn't care about the Sender parameter, simply give it anything, like Self or nil.
Question Three
Consider this code:
procedure TForm4.FormCreate(Sender: TObject);
var
a: packed array[0..15] of char;
b: packed array[0..15] of char;
begin
a := b;
end;
This doesn't work. You cannot treat arrays like strings; in particular, you cannot assign static arrays like this (a := b).
Instead, you have to do something like...
Move(b[0], a[0], length(a) * sizeof(char));
...or simply loop and copy one value at a time. But the above simple assignment (a := b) does work if you declare a static array type:
type
TChrArr = packed array[0..15] of char;
procedure TForm4.FormCreate(Sender: TObject);
var
a: TChrArr;
b: TChrArr;
begin
b := a;
end;
Andreas has you covered for question 1.
Question 2
I would arrange that your event handler called another method:
procedure TForm5.Edit1KeyPress(Sender: TObject; var Key: Char);
begin
RespondToEditControlKeyPress;
end;
That way you can just call RespondToEditControlKeyPress directly.
I'd guess that you want to call it with no parameters because you want code to run when the edit control's text is modified. You could perhaps use the OnChange event instead. And it may be that OnChange is more appropriate because pressing a key is not the only way to get text into an edit control.
By the way, it's better to ask one question at a time here on Stack Overflow.
For a quick way to copy string-type values into array-of-character type values. I suggest a small helper function like this:
procedure StrToCharArray( inputStr:String; var output; maxlen:Integer);
type
ArrayChar = Array[0..1] of Char;
begin
StrLCopy( PChar(#ArrayChar(output)[0]),PChar(inputStr),maxlen);
end;
Each time you call it, pass in the maximum length to be copied. Remember that if the buffer length is 15, you should pass in 14 as the maxlen, so that you leave room for the terminating nul character, if you intend to always terminate your strings:
StrToCharArray( UserInputStr, MyRecord.MyField, 14 );
This function will ensure that the data you copy into the record is null terminated, assuming that's what you wanted. Remember that in a fixed length character array it's up to you to decide what the rules are. Null terminated? Fully padded with spaces or null characters.... Strings and arrays-of-characters are so different, that there exist multiple possible ways of converting between the two.
If you don't intend to terminate your strings with nul, then you should use the FillChar+Move combination shown in someone else's answer.
The obvious answer is of course.
Don't use a packed array of char.
Use a string instead.
If you use ansistring, 1 char will always take 1 byte.
If you use shortstring ditto.
Ansistring is compatible with Pchar which is a pointer to a packed array of char.
So you can write
function inputbox(a,b,c: ansistring): pchar;
begin
Result:= a+b+c;
end;
var s: ansistring;
begin
s:= inputbox('a','b','c');
end;
Some advice
It looks like your are translating code from c to Delphi.
a packed array of char is exactly the same as the old (1995) shortstring minus the length byte at the beginning of shortstring.
The only reason I can think of to use packed array of char is when you are reading data to and from disk, and you have legacy code that you don't want to change.
I would keep the legacy code to read and write from disk and then transfer the data into an ansistring and from there on only use ansistring.
It's soooooooo much easier, Delphi does everything for you.
And... ansistring is much faster, gets automatically created and destroyed, can have any length (up to 2GB), uses less memory --because identical strings only get stored once (which means stringa:= stringb where a string is 20 chars is at least 5x faster using ansistrings than array's of char).
And of course best of all, buffer overflow errors are impossible with ansistring.
What about unicodestring?
Unicodestring is fine to use, but sometimes translation of chars happens when converting between packed array of char and unicodestring, therefore I recommend using ansistring in this context.
What you try to do is impossible, indeed:
highscoresdata.position[1]._MemoryName := StrPLCopy(a, s, Length(a));
That tries to assign a pointer (the result of StrPLCopy, a PWideChar in the last few versions of Delphi) to an array, which is indeed impossible. You can't copy an array like that. I would do:
StrLCopy(highscoresdata.position[1]._MemoryName, PChar(s),
Length(highscoresdata.position[1]._MemoryName));
That should work, and is IMO the simplest solution to copy a string to an array of characters. There is no need to use a as some kind of intermediate, and using Move is, IMO, rather low level and therefore a little tricky (it is easy to forget to multiply by the size of a character, it is unchecked, it does not add a #0, etc.), especially if you don't know what exactly you are doing.
This solution should even work for versions of Delphi before Delphi 2009, as it does not rely on the size of the character.
FWIW, I would not use packed arrays. Packed doesn't have a meaning in current Delphi, but could confuse the compiler and make the types incompatible.