I'm attempting to get an array of bytes from a TMemoryStream. I'm struggling to understand how a memory stream works. To my understand I should just be able to loop through the MemoryStream using the Position and Size properties.
My expected result is to populate an array of bytes looping through the memory stream however when adjusting the Position property of the Memory stream it jumps from example 0 to 2 and then from 2 to 6.
Data.Position := 0;
repeat
SetLength(arrBytes, Length(arrBytes) + 1);
Data.Read(arrBytes[High(arrBytes)], Data.Size);
Data.Position := Data.Position + 1;
until (Data.Position >= Data.Size -1);
The above code results in partial or in some cases just no data at all. How can I correctly convert the data from a memory stream to an Array of Byte
When reading data from TMemoryStream or any other stream for that matter the position is automatically advanced by the number of bytes read.
From TMemoryStream.Read documentation
Reads up to Count bytes from the memory stream into Buffer and
advances the current position of the stream by the number of bytes
read.
So if you are reading data from TMemoryStream sequentially you don't have to change the memory position yourself as it is done automatically.
Related
I am trying to organize saving and loading of data changing in size. So the save file needs to store several (unknown and every time different number) of dynamic arrays.
The mistake appears in this MCVE:
procedure TAnnMainF.Button6Click(Sender: TObject);
var
f: file;
ari, aro: array of double;
i, Count: word;
begin
SetLength(aro, random(5) + 1);
for i := 0 to High(aro) do
aro[i] := random(2001) / 2000 - 1;
AssignFile(f, 'c:\delme\1.txt');
ReWrite(f);
Count := Length(aro);
BlockWrite(f, Count, SizeOf(word));
BlockWrite(f, aro[0], SizeOf(double) * Count);
CloseFile(f);
Reset(f);
BlockRead(f, Count, SizeOf(word));
BlockRead(f, ari[0], SizeOf(double) * Count);
CloseFile(f);
end;
This code results in I/O error 998.
I was trying to declare the type TDoubleArray = array of Double; and pass ari as a parameter in BlockRead. I also tried to SetLength(ari, Count) before I call BlockRead without any success.
The Answer to this question did not help me.
The code reads the Count properly but rises an exception at array loading.
What am I doing wrong?
You must set the size of the block in the ReWrite/Reset commands:
ReWrite(f,1);
...
Reset(f,1);
From documentation:
RecSize is an optional expression that can be specified only if F is an untyped file. If F is an untyped file, RecSize specifies the record size to be used in data transfers. If RecSize is omitted, a default record size of 128 bytes is assumed.
This means that reading the data will overflow the allocated buffer, hence the I/O error from the system.
Also read this warning about using ancient file I/O BlockRead/BlockWrite:
Warning: This is an older method that is particularly dangerous to use because of the untyped Buf parameter, leading to potential memory corruption. The record size used by BlockRead and BlockWrite is governed by the optional 2nd parameter to the Reset or Rewrite call that was used to open the file being written. It is preferable to use streams in your applications. For example, a user procedure involving a stream can use both TMemoryStreams and TFileStreams, instead of being limited to using files as with these older routines.
In general the speed difference between BlockRead/Write and streams is insignificant. For larger files, a buffered handler is preferred.
There is an excellent example of a buffered file stream handler from David: Buffered files (for faster disk access)
As #kami/#TomBrunberg noted and what you tried, you must also allocate the length of the ari dynamic array before reading the data.
I'm working with stream files, but "out of memory" error occurred. I think I must read stream, byte by byte. Then I load a file with this method:
fs := TFileStream.Create("c:\a\a.avi", fmOpenRead or fmShareDenyWrite) ;
Next i reset stream position:
fs.positon:=0;
Then i'm trying to read first byte of stream:
var
oneByte:byte;
begin
fs.Read(oneByte,2);
but it doesn't work properly. Where is my mistake?
Byte size is 1 not 2
fs.Read(oneByte, 1);
Such errors can easily be prevented by using SizeOf() function
fs.Read(oneByte, SizeOf(oneByte));
On another note, Read returns the number of bytes read to indicate whether or not the entire read succeeded. You would be expected to check that return value to deal with errors.
The preferred idiom is to use ReadBuffer instead. That will call Read and in case of error raise an exception.
As #David Heffernan pointed out reading file stream byte by byte is not the most efficient way. Take a look at Buffered files (for faster disk access)
I have a function that converts TBitmap (which I draw) to TPngImage and then saves it to stream, so other methods can use it. Png is used because it creates smaller images for report output (excel, html). The problem is that SaveToStream seems to take too much time, 15x more than converting TBitmap to TPngImage or using TStream with png. Here is the code:
var
BitmapImage: TBitmap;
PNGImage: TPngImage;
PngStream: TStream;
begin
// draw on BitmapImage
...
PNGImage := TPngImage.Create;
PNGStream := TMemoryStream.Create;
Try
PNGImage.Assign(BitmapPicture.Bitmap); // Step 1: assign TBitmap to PNG
PNGImage.SaveToStream(PNGStream); // Step 2: save PNG to stream
WS.Shapes.AddPicture(PNGStream,PNGImage.Width,PNGImage.Height); // Step 3: Add PNG from Stream to Excel
finally
PNGImage.Free;
PNGStream.Free;
end;
...
This is tested with 70000 images and here are the timings:
Step 1: 7 s
Step 2: 93 s
Step 3: 6 s
Why is saving to Stream so slow? Any suggestion to optimize this?
Using Delphi XE7
EDIT
Here is example (MCVE) with simple bmp that gets converted to PNG and then saved into stream. Just for the sake of another verification I added SaveToFile, which of course takes longer, but it is saving to disk, so I assume acceptable.
The img1.bmp is 49.5KB, saved PNG is 661 bytes.
link to img1.bmp = http://www.filedropper.com/img1_1
TMemoryStreamAccess = class(TMemoryStream)
end;
procedure TForm1.Button1Click(Sender: TObject);
var BitmapImage:TBitmap;
PNGImage:TPngImage;
PNGStream:TMemoryStream;//TStream;
i,t1,t2,t3,t4,t5,t6: Integer;
vFileName:string;
begin
BitmapImage:=TBitmap.Create;
BitmapImage.LoadFromFile('c:\tmp\img1.bmp');
t1:=0; t2:=0; t3:=0; t4:=0; t5:=0; t6:=0;
for i := 1 to 70000 do
begin
PNGImage:=TPngImage.Create;
PNGStream:=TMemoryStream.Create;
try
t1:=GetTickCount;
PNGImage.Assign(BitmapImage);
t2:=t2+GetTickCount-t1;
t3:=GetTickCount;
TMemoryStreamAccess(PNGStream).Capacity := 1000;
PNGImage.SaveToStream(PNGStream);
// BitmapImage.SaveToStream(PNGStream); <-- very fast!
t4:=t4+GetTickCount-t3;
finally
PNGImage.Free;
PNGstream.Free
end;
end;
showmessage('Assign = '+inttostr(t2)+' - SaveToStream = '+inttostr(t4));
end;
This is tested with 70000 images and here are the timings:
Step 1: 7 s
Step 2: 93 s
Step 3: 6 s
Why is saving to Stream so slow?
Let's crunch some numbers:
Step 1: 7s = 7000ms. 7000 / 70000 = 0.1ms per image
Step 2: 93s = 93000ms. 93000 / 70000 = ~1.33ms per image
Step 3: 6s = 6000ms. 6000 / 70000 = ~0.086ms per image
Do you think 1.33 ms per SaveToStream() is slow? You are just doing a LOT of them, so they add up over time, that's all.
That being said, PNG data in memory is not compressed. It gets compressed when the data is saved. So that is one reason for slowdown. Also, saving the PNG does a lot of writes to the stream, which can cause the stream to perform multiple memory (re)allocations (TPNGImage also performs internal memory allocations during saving), so that is another slowdown.
Any suggestion to optimize this?
There is nothing you can do about the compression overhead, but you can at least pre-set the TMemoryStream.Capacity to a reasonable value before calling SaveToStream() to reduce the memory reallocations that TMemoryStream needs to perform during writing. You don't need to be exact with it. If writing to the stream causes its Size to exceed its current Capacity, it will simply increase its Capacity accordingly. Since you have already processed 70000 images, take the average size of them and add a few more KB to it, and use that as your initial Capacity.
type
TMemoryStreamAccess = class(TMemoryStream)
end;
var
BitmapImage: TBitmap;
PNGImage: TPngImage;
PngStream: TMemoryStream;
begin
// draw on BitmapImage
...
PNGImage := TPngImage.Create;
Try
PNGImage.Assign(BitmapPicture.Bitmap); // Step 1: assign TBitmap to PNG
PNGStream := TMemoryStream.Create;
try
TMemoryStreamAccess(PNGStream).Capacity := ...; // some reasonable value
PNGImage.SaveToStream(PNGStream); // Step 2: save PNG to stream
WS.Shapes.AddPicture(PNGStream, PNGImage.Width, PNGImage.Height); // Step 3: Add PNG from Stream to Excel
finally
PNGStream.Free;
end;
finally
PNGImage.Free;
end;
...
If that still is not fast enough for you, consider using threads to process multiple images in parallel. Don't process them sequentially.
Did you assign the compression level? I didn't notice something like
PNGImage.CompressionLevel := 1;
in your code. It can be in a range of 0 to 9. By default, it is 7. If you set it to 1, it would be significantly faster, while the output stream size increase will be negligible.
Maybe not directly related, but i was having an issue with Read/Write (in RAM) because i was using TMemoryStream (byte by byte).
In my case i could adapt my code to work in pure RAM with SetLength(MyEncripted,TheSize); & SetLength(MyClear,TheSize); instead of TMemoryStream.Read and TMemoryStream.Write.
What i was doing was the easy 'concept' (that can be done with pure RAM string variables) of MyEncripted[i]:=Chr(Ord(MyClear[i]) xor Ord(MyKey[i])); with a TMemoryStream.Write in Byte by Byte logic.
My measured timings:
Using TMemoryStream for 633 KiloBytes => 3'21"
Using RAM String type variables for 633 KiloBytes => Near instantaneous, less than 0.1"
Using TMemoryStream for 6.3 MegaBytes => 33'30"
Using RAM String type variables for 6.3 MegaBytes => Near instantaneous, less than 0.1"
Using TMemoryStream for 63 MegaBytes => 5h35'00"
Using RAM String type variables for 63 MegaBytes => Near instantaneous, less than 0.1"
Using TMemoryStream for 633 MegaBytes => 55h50'00"
Using RAM String type variables for 633 MegaBytes => Less than 1"
Note: I did not go further in timing since 55 hours is really huge and was enough to saw what was happening; but it looks it scales 'linear'.
TMemoryStream is too slow, so slow that something what can be done in RAM taking less than ONE second can take more than TWO days while using TMemoryStream.Write (in byte by byte logic).
So i had decided long time ago to never ever use any TMemoryStream again.
Hope this can help to understand what level of SLOW is TMemoryStream versus direct RAM String variables.
I am using a MemoryStream to construct an array of bytes i need to send to a server.I have thre questions:
1) Is there a better way to construct an array of bytes than this ?
2) Why this pice of code writes bogus in my memory stream ?
var
serial : word;
MS : TMemoryStream;
const
somebytes : array [0..1] of byte = ($72,$72);
...
begin
MS := TMemoryStream.Create();
try
MS.Write(somebytes[0],2);
serial := $3E6C;
MS.Write(serial,2);
finally
MS.Free;
end;
Using the debugger i see that in the stream is added the value $6F32 instead of $3E6C.
3) If i call
MS.Position := 2;
and then i access PByte(MS.Memory)^ why do i get the first byte in the stream instead of the third?
Is there a better way to construct an array of bytes than this?
That's a perfectly reasonable way to do it, in my view.
I see that in the stream is added the value $6F32 instead of $3E6C.
Check again. The correct values are in fact added. But beware of the traps of little endian data types. The 4 bytes added to your stream, in order, are: $72, $72, $6C, $3E.
Why do I get the first byte in the stream instead of the third?
Because the Memory property always refers to the beginning of the stream. It does not take account of the stream's current position.
When I use a large file in memorystream or filestream I see an error which is "out of memory"
How can I solve this problem?
Example:
procedure button1.clıck(click);
var
mem:TMemoryStream;
str:string;
begin
mem:=Tmemorystream.create;
mem.loadfromfile('test.txt');----------> there test.txt size 1 gb..
compressstream(mem);
end;
Your implementation is very messy. I don't know exactly what CompressStream does, but if you want to deal with a large file as a stream, you can save memory by simply using a TFileStream instead of trying to read the whole thing into a TMemoryStream all at once.
Also, you're never freeing the TMemoryStream when you're done with it, which means that you're going to leak a whole lot of memory. (Unless CompressStream takes care of that, but that's not clear from the code and it's really not a good idea to write it that way.)
You can't fit the entire file into a single contiguous block of 32 bit address space. Hence the out of memory error.
Read the file in smaller pieces and process it piece by piece.
Answering the question in the title, you need to process the file piece by piece, byte by byte if that's needed: you definitively do not load the file all at once into memory! How you do that obviously depends on what you need to do with the file; But since we know you're trying to implement an Huffman encoder, I'll give you some specific tips.
An Huffman encoder is a stream encoder: Bytes go in and bits go out. Each unit of incoming data is replaced with it's corresponding bit pattern. The encoder doesn't need to see the whole file at once, because it is in fact only working on one byte each time.
Here's how you'd huffman-compress a file without loading it all into memory; Of course, the actual Huffman encoder is not shown, because the question is about working with big files, not about building the actual encoder. This piece of code includes buffered input and output and shows how you'd link an actual encoder procedure to it.
(beware, code written in browser; if it doesn't compile you're expected to fix it!)
type THuffmanBuffer = array[0..1023] of Byte; // Because I need to pass the array as parameter
procedure DoActualHuffmanEncoding(const EncodeByte:Byte; var BitBuffer: THuffmanBuffer; var AtBit: Integer);
begin
// This is where the actual Huffman encoding would happen. This procedure will
// copy the correct encoding for EncodeByte in BitBuffer starting at AtBit bit index
// The procedure is expected to advance the AtBit counter with the number of bits
// that were actually written (that's why AtBit is a var parameter).
end;
procedure HuffmanEncoder(const FileNameIn, FileNameOut: string);
var InFile, OutFile: TFileStream;
InBuffer, OutBuffer: THuffmanBuffer;
InBytesCount: Integer;
OutBitPos: Integer;
i: Integer;
begin
// First open the InFile
InFile := TFileStream.Create(FileNameIn, fmOpenRead or fmShareDenyWrite);
try
// Now prepare the OutFile
OutFile := TFileStream.Create(FileNameOut, fmCreate);
try
// Start the out bit counter
OutBitPos := 0;
// Read from the input file, one buffer at a time (for efficiency)
InBytesCount := InFile.Read(InBuffer, SizeOf(InBuffer));
while InBytesCount <> 0 do
begin
// Process the input buffer byte-by-byte
for i:=0 to InBytesCount-1 do
begin
DoActualHuffmanEncoding(InBuffer[i], OutBuffer, OutBitPos);
// The function writes bits to the outer buffer, not full bytes, and the
// encoding for a rare byte might be significantly longer then 1 byte.
// Whenever the output buffer approaches it's capacity we'll flush it
// out to the OutFile
if (OutBitPos > ((SizeOf(OutBuffer)-10)*8) then
begin
// Ok, we've got less then 10 bytes available in the OutBuffer, time to
// flush!
OutFile.Write(OutBuffer, OutBitPos div 8);
// We're now possibly left with one incomplete byte in the buffer.
// We'll copy that byte to the start of the buffer and continue.
OutBuffer[0] := OutBuffer[OutBitPos div 8];
OutBitPos := OutBitPos mod 8;
end;
end;
// Read next chunk
InBytesCount := InFile.Read(InBuffer, SizeOf(InBuffer));
end;
// Flush the remaining of the output buffer. This time we want to flush
// the final (potentially incomplete) byte as well, because we've got no
// more input, there'll be no more output.
OutFile.Write(OutBuffer, (OutBitPos + 7) div 8);
finally OutFile.Free;
end;
finally InFile.Free;
end;
end;
The Huffman encoder is not a difficult encoder to implement, but doing it both correctly and fast might be a challenge. I suggest you start with a correct encoder, once you've got both encoding and decoding working figure out how to do a fast encoder.
try something like http://www.explainth.at/en/delphi/mapstream.shtml