I am using a MemoryStream to construct an array of bytes i need to send to a server.I have thre questions:
1) Is there a better way to construct an array of bytes than this ?
2) Why this pice of code writes bogus in my memory stream ?
var
serial : word;
MS : TMemoryStream;
const
somebytes : array [0..1] of byte = ($72,$72);
...
begin
MS := TMemoryStream.Create();
try
MS.Write(somebytes[0],2);
serial := $3E6C;
MS.Write(serial,2);
finally
MS.Free;
end;
Using the debugger i see that in the stream is added the value $6F32 instead of $3E6C.
3) If i call
MS.Position := 2;
and then i access PByte(MS.Memory)^ why do i get the first byte in the stream instead of the third?
Is there a better way to construct an array of bytes than this?
That's a perfectly reasonable way to do it, in my view.
I see that in the stream is added the value $6F32 instead of $3E6C.
Check again. The correct values are in fact added. But beware of the traps of little endian data types. The 4 bytes added to your stream, in order, are: $72, $72, $6C, $3E.
Why do I get the first byte in the stream instead of the third?
Because the Memory property always refers to the beginning of the stream. It does not take account of the stream's current position.
Related
I am trying to pass an array of bytes from VBscript to my windows Delphi Application and can't seem to find the correct syntax to interpret the passed data.
The requirement is fairly simple as the VBscript snippet below demonstrates
Dim inst,arr(5)
Sub Main
set inst=instruments.Find("EP1")
arr(0) = 0
arr(1) = 1
arr(2) = 2
arr(3) = 3
arr(4) = 4
inst.writebytes arr,5
end Sub
I can get the server to accept the olevariant passed by the script but the data seems garbled, my example server code is shown below and is based on the Stackoverflow question here How to use variant arrays in Delphi
procedure TInstrument.WriteBytes(Data: OleVariant; Length: Integer);
var i,n:integer; Pdat:Pbyte; Adata:PvarArray;
begin
if VarIsArray(data) then
begin
n:=TVarData(Data).VArray^.Bounds[0].ElementCount;
Adata:= VarArrayLock(Data);
Getmem(Pdat,length);
try
for i:=0 to length-1 do
Pdat[i]:=integerArray(Adata.data^)[i];
Finstrument.WriteBytes(Pdat,Length);
finally
freemem(Pdat)
end;
end;
end;
So the idea is to accept the integers passed by the script, convert it to the local data representation (array of byte) then pass it on to my function to use the data.
I have tried several different data types and methods to try and get some ungarbled data out of the variant all to no avail.
What is the correct method of extracting the array data from the passed variant?
Also, TVarData(Data).VArray^.Bounds[0].ElementCount has a value of zero, why would that be?
Arrays created in VBScript are
zero based
untyped
declared with upper bound (not size as you assumed; size of array declared as Dim arr(5) is 6)
include dimension info in them (so you don't need to pass it along with the array)
When used in COM, they are passed as variant arrays of type varVariant (as the Ondrej Kelle points out in his comment). To process such an array in your method you have to assert that:
the value is a single dimensional array
each element can be converted to byte
You can write helper routine for that:
function ToBytes(const Data: Variant): TBytes;
var
Index, LowBound, HighBound: Integer;
ArrayData: Pointer;
begin
if not VarIsArray(Data) then
raise EArgumentException.Create('Variant array expected.');
if VarArrayDimCount(Data) <> 1 then
raise EArgumentException.Create('Single dimensional variant array expected.');
LowBound := VarArrayLowBound(Data, 1);
HighBound := VarArrayHighBound(Data, 1);
SetLength(Result, HighBound - LowBound + 1);
if TVarData(Data).VType = varArray or varByte then
begin
ArrayData := VarArrayLock(Data);
try
Move(ArrayData^, Result[0], Length(Result));
finally
VarArrayUnlock(Data);
end;
end
else
begin
for Index := LowBound to HighBound do
Result[Index - LowBound] := Data[Index];
end;
end;
for loop in the routine will be horribly slow when processing large arrays, so there's optimization for special case (variant array of bytes) that uses Move to copy bytes to result. But this will never happen with VBScript array. You might consider using VB.Net or PowerShell.
Using such a routine has downside of keeping 2 instances of the array in memory - as variant array and as byte array. Use it as a guide when applying it to your use case.
I am trying to organize saving and loading of data changing in size. So the save file needs to store several (unknown and every time different number) of dynamic arrays.
The mistake appears in this MCVE:
procedure TAnnMainF.Button6Click(Sender: TObject);
var
f: file;
ari, aro: array of double;
i, Count: word;
begin
SetLength(aro, random(5) + 1);
for i := 0 to High(aro) do
aro[i] := random(2001) / 2000 - 1;
AssignFile(f, 'c:\delme\1.txt');
ReWrite(f);
Count := Length(aro);
BlockWrite(f, Count, SizeOf(word));
BlockWrite(f, aro[0], SizeOf(double) * Count);
CloseFile(f);
Reset(f);
BlockRead(f, Count, SizeOf(word));
BlockRead(f, ari[0], SizeOf(double) * Count);
CloseFile(f);
end;
This code results in I/O error 998.
I was trying to declare the type TDoubleArray = array of Double; and pass ari as a parameter in BlockRead. I also tried to SetLength(ari, Count) before I call BlockRead without any success.
The Answer to this question did not help me.
The code reads the Count properly but rises an exception at array loading.
What am I doing wrong?
You must set the size of the block in the ReWrite/Reset commands:
ReWrite(f,1);
...
Reset(f,1);
From documentation:
RecSize is an optional expression that can be specified only if F is an untyped file. If F is an untyped file, RecSize specifies the record size to be used in data transfers. If RecSize is omitted, a default record size of 128 bytes is assumed.
This means that reading the data will overflow the allocated buffer, hence the I/O error from the system.
Also read this warning about using ancient file I/O BlockRead/BlockWrite:
Warning: This is an older method that is particularly dangerous to use because of the untyped Buf parameter, leading to potential memory corruption. The record size used by BlockRead and BlockWrite is governed by the optional 2nd parameter to the Reset or Rewrite call that was used to open the file being written. It is preferable to use streams in your applications. For example, a user procedure involving a stream can use both TMemoryStreams and TFileStreams, instead of being limited to using files as with these older routines.
In general the speed difference between BlockRead/Write and streams is insignificant. For larger files, a buffered handler is preferred.
There is an excellent example of a buffered file stream handler from David: Buffered files (for faster disk access)
As #kami/#TomBrunberg noted and what you tried, you must also allocate the length of the ari dynamic array before reading the data.
I'm having a weird issue on using Delphi's TMemoryStream (or TFileStream for that matter). While reading a part of the stream into a byte array. Here's some code as an example.
procedure readfromstream();
var
ms : TMemoryStream;
buffer : array of byte;
recordSize : Integer;
begin
try
begin
ms := TMemeoryStream.Create();
ms.LoadFromFile(<some_path_to_a_binary_file>);
while ms.Position < ms.Size do
begin
buffer := nil;
SetLength(buffer, 4);
ms.ReadBuffer(buffer, 4);
move(buffer[0], recordSize, 4);
SetLength(buffer, recordSize);
ms.Position := ms.Position - 4; // Because I was having issues trying to read the rest of the record into a specific point in the buffer
FillChar(buffer, recordSize, ' ');
ms.ReadBuffer(buffer, recordSize); // Issue line ???
// Create the record from the buffer
end;
finally
begin
ms.Free();
end;
end;
procedure is called as,
// Some stuff happens before it
readfromstream();
// Some stuff happens after it
on debugging, I can see that it reads the stream into the buffer and the record is stored in memory appropriately. The procedure then exits normally and the debugger steps out of the procedure, but I end up straight back into the procedure and it repeats.
By forcing the procedure to exit prematurely I believe the issue involves the ms.ReadBuffer(buffer, recordSize); but I don't see why it would cause the issue.
This procedure is called only once. My test data has only one entry/data.
Any help would be greatly appreciated.
FillChar(buffer, recordSize, ' ');
Here you are overwriting the dynamic array variable, a pointer, rather than writing to the content of the array. That causes a memory corruption. Pretty much anything goes at that point.
The call to FillChar is needless anyway. You are going to read into the entire array anyway. Remove the call to FillChar.
For future reference, to do that call correctly, you write it like this:
FillChar(Pointer(buffer)^, ...);
or
FillChar(buffer[0], ...);
I prefer the former since the latter is subject to range errors when the array length is zero.
And then
ms.ReadBuffer(buffer, recordSize);
makes the exact same mistake, writing to the array variable rather than the array, and thus corrupting memory.
That should be
ms.ReadBuffer(Pointer(buffer)^, recordSize);
or
ms.ReadBuffer(buffer[0], recordSize);
The first 4 lines inside the loop are clumsy. Read directly into the variable:
ms.ReadBuffer(recordSize, SizeOf(recordSize));
I recommend that you perform some sanity checks on the value of recordSize that you read. For instance, any value less than 4 is clearly an error.
There's not a lot of point in moving the stream pointer back and reading again. You can copy recordSize into the first 4 bytes and the array and then read the rest.
Move(recordSize, buffer[0], SizeOf(recordSize));
ms.ReadBuffer(buffer[SizeOf(recordSize)], recordSize - SizeOf(recordSize));
A memory stream also seems wasteful. Why read the entire file into memory? That's going to place stress on your address space for large files. Use a buffered file stream.
Letting the caller allocate the stream would give more flexibility to the caller. They could then read from any type of stream and not be constrained to use a disk file.
Your try/finally block is wrong. You must acquire the resource immediately before the try. As you have it, an exception in the constructor leads to you calling Free on an uninitialized variable.
A better version might be:
procedure ReadFromStream(Stream: TStream);
var
buffer: TArray<byte>;
recordSize: Integer;
begin
while Stream.Position < Stream.Size do
begin
Stream.ReadBuffer(recordSize, SizeOf(recordSize));
if recordSize < SizeOf(recordSize) then
raise ...;
SetLength(buffer, recordSize);
Move(recordSize, buffer[0], SizeOf(recordSize));
if recordSize > SizeOf(recordSize) then
Stream.ReadBuffer(buffer[SizeOf(recordSize)],
recordSize - SizeOf(recordSize));
// process record
end;
end;
Sorry I can't add a comment, being a newb and all :) This reply is based on my understanding of Clayton's code in light of his comment with the recordSize values.
The reason David's code is looping is probably that you are interpreting each four byte "block" is a number. I'll assume your first Stream.Readbuffer is correct and that the first four bytes in the file is a length.
Now, unless I'm mistaken, I expect the recordSize will usually be greater than SizeOf(recordSize), which I think should be 4 (the size of an int). Nevertheless, this line is meaningless here.
The SetLength is correct, given my previous assumption.
Now your Move is where the story hits a snag: you haven't read anything since you read the length! So before the move, you should have:
bytesRead := Stream.Readbuffer(Pointer(buffer)^, recordSize);
Now you can check for EOF:
if bytesRead <> recordSize then
raise...;
...and move the buffer somewhere (if you wish):
Move(buffer[0], dest[0], recordSize);
And you are positioned to read the next recordSize value.
Repeat until EOF.
When I use a large file in memorystream or filestream I see an error which is "out of memory"
How can I solve this problem?
Example:
procedure button1.clıck(click);
var
mem:TMemoryStream;
str:string;
begin
mem:=Tmemorystream.create;
mem.loadfromfile('test.txt');----------> there test.txt size 1 gb..
compressstream(mem);
end;
Your implementation is very messy. I don't know exactly what CompressStream does, but if you want to deal with a large file as a stream, you can save memory by simply using a TFileStream instead of trying to read the whole thing into a TMemoryStream all at once.
Also, you're never freeing the TMemoryStream when you're done with it, which means that you're going to leak a whole lot of memory. (Unless CompressStream takes care of that, but that's not clear from the code and it's really not a good idea to write it that way.)
You can't fit the entire file into a single contiguous block of 32 bit address space. Hence the out of memory error.
Read the file in smaller pieces and process it piece by piece.
Answering the question in the title, you need to process the file piece by piece, byte by byte if that's needed: you definitively do not load the file all at once into memory! How you do that obviously depends on what you need to do with the file; But since we know you're trying to implement an Huffman encoder, I'll give you some specific tips.
An Huffman encoder is a stream encoder: Bytes go in and bits go out. Each unit of incoming data is replaced with it's corresponding bit pattern. The encoder doesn't need to see the whole file at once, because it is in fact only working on one byte each time.
Here's how you'd huffman-compress a file without loading it all into memory; Of course, the actual Huffman encoder is not shown, because the question is about working with big files, not about building the actual encoder. This piece of code includes buffered input and output and shows how you'd link an actual encoder procedure to it.
(beware, code written in browser; if it doesn't compile you're expected to fix it!)
type THuffmanBuffer = array[0..1023] of Byte; // Because I need to pass the array as parameter
procedure DoActualHuffmanEncoding(const EncodeByte:Byte; var BitBuffer: THuffmanBuffer; var AtBit: Integer);
begin
// This is where the actual Huffman encoding would happen. This procedure will
// copy the correct encoding for EncodeByte in BitBuffer starting at AtBit bit index
// The procedure is expected to advance the AtBit counter with the number of bits
// that were actually written (that's why AtBit is a var parameter).
end;
procedure HuffmanEncoder(const FileNameIn, FileNameOut: string);
var InFile, OutFile: TFileStream;
InBuffer, OutBuffer: THuffmanBuffer;
InBytesCount: Integer;
OutBitPos: Integer;
i: Integer;
begin
// First open the InFile
InFile := TFileStream.Create(FileNameIn, fmOpenRead or fmShareDenyWrite);
try
// Now prepare the OutFile
OutFile := TFileStream.Create(FileNameOut, fmCreate);
try
// Start the out bit counter
OutBitPos := 0;
// Read from the input file, one buffer at a time (for efficiency)
InBytesCount := InFile.Read(InBuffer, SizeOf(InBuffer));
while InBytesCount <> 0 do
begin
// Process the input buffer byte-by-byte
for i:=0 to InBytesCount-1 do
begin
DoActualHuffmanEncoding(InBuffer[i], OutBuffer, OutBitPos);
// The function writes bits to the outer buffer, not full bytes, and the
// encoding for a rare byte might be significantly longer then 1 byte.
// Whenever the output buffer approaches it's capacity we'll flush it
// out to the OutFile
if (OutBitPos > ((SizeOf(OutBuffer)-10)*8) then
begin
// Ok, we've got less then 10 bytes available in the OutBuffer, time to
// flush!
OutFile.Write(OutBuffer, OutBitPos div 8);
// We're now possibly left with one incomplete byte in the buffer.
// We'll copy that byte to the start of the buffer and continue.
OutBuffer[0] := OutBuffer[OutBitPos div 8];
OutBitPos := OutBitPos mod 8;
end;
end;
// Read next chunk
InBytesCount := InFile.Read(InBuffer, SizeOf(InBuffer));
end;
// Flush the remaining of the output buffer. This time we want to flush
// the final (potentially incomplete) byte as well, because we've got no
// more input, there'll be no more output.
OutFile.Write(OutBuffer, (OutBitPos + 7) div 8);
finally OutFile.Free;
end;
finally InFile.Free;
end;
end;
The Huffman encoder is not a difficult encoder to implement, but doing it both correctly and fast might be a challenge. I suggest you start with a correct encoder, once you've got both encoding and decoding working figure out how to do a fast encoder.
try something like http://www.explainth.at/en/delphi/mapstream.shtml
Ultimately my goal is to be able to save an array of records data structure in a sqlite BLOB field, but in the interim I am trying to serialize an array of records and store it in a TMemoryStream. In case it may be relevant, I am using Tim Anderson's sqlite wrapper (unicode) with Delphi 2010.
This is my array of record declaration:
type
TPerson = array of packed record
sCountry: string[50];
sFullName: string[100];
sAddress: string[100];
sCity: string[30];
sEmployer: string[100];
end;
var
MyPeople : TPerson;
And here is the code I am currently using to serialize this array of record to a TMemoryStream. The problem is that it doesn't work:
var
i : integer;
ms : TMemoryStream:
ms2 : TMemoryStream;
TestPeople : TPerson;
sldb : TSQLiteDatabase;
begin
ms := TMemoryStream.Create;
ms2 := TMemoryStream.Create;
sldb := TSQLiteDatabase.Create(slDBPath); //sldBPath is a global variable with path to sqlite db
try
i := Length(MyPeople);
ms.Write(i, 4);
ms.Write(pointer(MyPeople)^, i * sizeOf(MyPeople));
///
ms2.Read(i, 4);
SetLength(ms2, i);
ms2.Read(pointer(TestPeople)^, i * sizeOf(TestPeople));
WriteLn('#############' + ms2[0].sFullName); //check if we can read data back
sQuery := 'UPDATE PersonDirectory SET fldPersonBlob = ? WHERE id = "' + 42 + '";';
sldb.UpdateBlob(sQuery, ms);
finally
ms.Free;
ms2.Free;
sqldb.Free;
end;
end;
Any ideas on either how to generally serialize an array of record to a TMemoryStream, or a better way to store an array of records in a sqlite BLOB field?
Make TPerson the packed record, and declare TPersons as array of TPerson.
type
TPerson = packed record
sCountry: string[50];
sFullName: string[100];
sAddress: string[100];
sCity: string[30];
sEmployer: string[100];
end;
TPersons = array of TPerson;
var
MyPeople: TPersons;
This allows you to get SizeOf(TPerson) to get the right size of the record, while Length(MyPeople) will give you the length of the array. Multiply them to get the total byte size. That should get you started.
Also note that some read and write action start from the beginning of the stream, while others continue at the current position. Setting position to 0 before copying the contents of a stream to another is sometimes needed. The documentation will mention this.
Maybe I wrong, but you are writing in ms and reading from ms2.
ms2 is a memory stream, but don't point to same memory content that ms is pointing. I think that ms2 will not read content from ms if you don't change your code.
If you need to put the content of ms into ms2, try
ms2.LoadFromStream(ms);
before you start to reading content.
Of course, you need to know the size of a individual record to do the right math.
You can use an array of records with plain string, then save them into a stream with our TDynArray wrapper.
It will use less space than shortstrings, because it will write only the characters used by the string. For instance, sEmployer defined as string[100] will always use 101 bytes when stored directly. Whereas with our TDynArray, it will only use the number of chars plus one.
Works from Delphi 6 up to XE.
type
TPerson = packed record
sCountry: string;
sFullName: string;
sAddress: string;
sCity: string;
sEmployer: string;
end;
TPersons = array of TPerson;
var
MyPeople: TPersons;
(...)
procedure SavePeopleToStream(Stream: TMemoryStream);
begin
DynArray(TypeInfo(TPersons),MyPeople).SaveToStream(Stream);
end;
You've got the reverse LoadFromStream method available, and much more, in TDynArray.
See this blog entry to get this wrapper.
Additional note: It's funny, this is the same exact feature I'm about to add to our ORM: store dynamic array content in a BLOB field, encoded as binary. But in our case, all will be automated thanks to the Delphi RTTI. And there is also a Client/Server layer, using JSON for transmission, to make this more n-Tier compliant.