Optimal Buffer Stream write process - delphi

Question

We have our own data streaming algorithm that include some metadata+records+fields values.
Currently we use a TStream and write to add values to the stream.
Now I'm wondering if this time cosuming operation could be made faster by using some technic.
Edit: We are only appending data to the end, not moving or seeking.
Some things I was thinking of are:
Not use Streams buf some large memory allocated buffer to copy data to, the problem is if we go beyond the buffer size, then we have to relocate to some new memory space.
Use a Stream prefilled with #0s to some size and then start adding values. The rationale is that Tstream have to allocate it's own buffer every time I do a write (I don't know how it really works, just wondering...)
We are adding strings to the TStream and binary data in the form #0#0#0#1 for example.
The data is then transfered via TCP, so it's not about write to a File.
So what is the best method for this?

Answer 1

First, you are assuming the TStream is the bottleneck. You need to profile your code, such as with AQTime, to determine where the bottleneck really is. Don't make assumptions.
Second, what type of TStream are you actually using? TMemoryStream? TFileStream? Something else? Different stream types handle memory differently. TMemoryStream allocates a memory buffer and grows it by a pre-set amount of bytes whenever the buffer fills up. TFileStream, on the other hand, doesn't use any memory at all, it just writes directly to the file and lets the OS handle any buffering.
Regardless of which type of stream you use, one thing you could try is implement your own custom TStream class that has an internal fixed-size buffer and a pointer to your real destination TStream object. You can then pass an instance of your custom class to your algorithm. Have your class override the TStream::Write() method to copy the input data into its buffer until it fills up, then you can Write() the buffer to the destination TStream and clear the buffer. Your algorithm will never know the difference. Both TMemoryStream and TFileStream would benefit from the extra buffering - fewer larger writes means more efficient memory allocations and file I/O. For example:
type
TMyBufferedStreamWriter = class(TStream)
private
fDest: TStream;
fBuffer: array[0..4095] of Byte;
fOffset: Cardinal;
public
constructor Create(ADest: TStream);
function Read(var Buffer; Count: Longint): Longint; override;
function Write(const Buffer; Count: Longint): Longint; override;
procedure FlushBuffer;
end;
.
uses
RTLConsts;
constructor TMyBufferedStreamWriter.Create(ADest: TStream);
begin
fDest := ADest;
end;
function TMyBufferedStreamWriter.Read(var Buffer; Count: Longint): Longint;
begin
Result := 0;
end;
function TMyBufferedStreamWriter.Write(const Buffer; Count: Longint): Longint;
var
pBuffer: PByte;
Num: Cardinal;
begin
Result := 0;
pBuffer := PByte(#Buffer);
while Count > 0 do
begin
Num := Min(SizeOf(fBuffer) - fOffset, Cardinal(Count));
if Num = 0 then FlushBuffer;
Move(pBuffer^, fBuffer[fOffset], Num);
Inc(fOffset, Num);
Inc(pBuffer, Num);
Dec(Count, Num);
Inc(Result, Num);
end;
end;
procedure TMyBufferedStreamWriter.FlushBuffer;
var
Idx: Cardinal;
Written: Longint;
begin
if fOffset = 0 then Exit;
Idx := 0;
repeat
Written := fDest.Write(fBuffer[Idx], fOffset - Idx);
if Written < 1 then raise EWriteError.CreateRes(#SWriteError);
Inc(Idx, Written);
until Idx = fOffset;
fOffset := 0;
end;
.
Writer := TMyBufferedStreamWriter.Create(RealStreamHere);
try
... write data to Writer normally as needed...
Writer.FlushBuffer;
finally
Writer.Free;
end;

Answer 2

Use a profiler to see where it is actually slow.
If it is indeed because of multiple reallocation to increase the size of the stream, you can avoid that by setting the Size property to a big enough amount upfront.
The only case where using a memory buffer could make an obvious difference is if you're using FileStreams, all other useable streams are already doing this for you.

Answer 3

Overwrite TMemoryStream and remove the restriction of Size and Capacity. And do not call TMemoryStream.Clear but call TMemoryStream.SetSize(0)
type
TMemoryStreamEx = class(TMemoryStream)
public
procedure SetSize(NewSize: Longint); override;
property Capacity;
end;
implementation
{ TMemoryStreamEx }
procedure TMemoryStreamEx.SetSize(NewSize: Integer);
var
OldPosition: Longint;
begin
if NewSize > Capacity then
inherited SetSize(NewSize)
else
begin
OldPosition := Position;
SetPointer(Memory, NewSize);
if OldPosition > NewSize then
Seek(0, soFromEnd);
end;
end;