delphi TFileStream "out of memory" - delphi

Question

I am having trouble with some Delphi code that uses TFileStream to read chunks of data from a file to a dynamic array. The original objective in writing the code is to compare the contents of two files that have the same size but potentially different date and time stamps to see if the contents are the same. This is done by reading the data from each file of the pair into separate dynamic arrays and comparing each byte of one array with the corresponding byte of the other.
The code makes multiple calls to TFileStream.Read. After about 75 calls, the program crashes with an 'Out of Memory' Error message.
It does not seem to matter how large the blocks of data that are read, it seems to be the number of calls that results in the error message.
The code is a function that I have written that is called elsewhere whenever the program encounters two files that it needs to compare (which, for reasons that I won't go into, could be forty or fifty different file pairs). The 'Out of Memory' error occurs whether it is a single file that is being read in small blocks, or multiple files that are being read in their entirety. It seems to be the number of calls that is the determinant of the error.
While I realize that there might be more elegant ways of achieving the comparison of the files than what I have shown below, what I would really like to know is what is wrong with the use of the TFileStream and/or SetLength calls that are causing the memory problems. I have tried freeing the memory after every call (as shown in the code) and it seems to make no difference.
I would be grateful if someone could explain what is going wrong.
function Compare_file_contents(SPN,TPN : String; SourceFileSize : int64) : boolean;
var
SF : TFileStream; //First file of pair for comparison
TF : TFileStream; //Second file of pair
SourceArray : TBytes; // Buffer array to receive first file data
TargetArray : TBytes; //Buffer array to receive second file data
ArrayLength : int64; //Length of dynamic array
Position : int64; //Position within files to start each block of data read
TestPosition : int64; //Position within dynamic arrays to compare each byte
MaxArrayLength : integer; //Maximum size for the buffer arrays
LastRun : Boolean; //End first repeat loop
begin
{ The comparison has an arbitrary upper boundary of 100 MB to avoid slowing the
the overall program. The main files bigger than this will be *.pst files that
will most likely have new dates every time the program is run, so it will take
about the same time to copy the files as it does to read and compare them, and
it will have to be done every time.
The function terminates when it is confirmed that the files are not the same.
If the source file is bigger than 100 MB, it is simply assumed that they are
not identical, thus Result = False. Also, LongInt integers (=integers) have
a range of -2147483648..2147483647, so files bigger than 2 GB will have
overflowed to a negative number. Hence the check to see if the file size is
less than zero.
The outer repeat ... until loop terminates on LastRun, but LastRun should only
be set if SecondLastRun is True, because it will skip the final comparisons in
the inner repeat ... until loop otherwise. }
Result := True;
LastRun := False;
MaxArrayLength := 1024*1024;
if (SourceFileSize > 100*1024*1024) or (SourceFileSize < 0) then Result := False
else
begin
{ The comparison is done by using TFileStream to open and read the data from
the source and target files as bytes to dynamic arrays (TBytes). Then a repeat
loop is used to compare individual bytes until a difference is found or all
of the information has been compared. If a difference is found, Result is
set to False. }
if SourceFileSize > MaxArrayLength then ArrayLength := MaxArrayLength
else ArrayLength := SourceFileSize;
SF := TFileStream.Create(SPN,fmOpenRead);
TF := TFileStream.Create(TPN,fmOpenRead);
Position := 0;
SetLength(SourceArray,ArrayLength);
SetLength(TargetArray,ArrayLength);
try
SF.Read(SourceArray,ArrayLength);
TF.Read(TargetArray,ArrayLength);
Position := SF.Position;
finally
SF.Free;
TF.Free;
end;
repeat
TestPosition := 0;
repeat
if SourceArray[TestPosition] <> TargetArray[TestPosition] then
Result := False;
Inc(TestPosition);
until (Result = False) or (TestPosition = ArrayLength);
if SourceFileSize > Position then
begin
if SourceFileSize - Position - MaxArrayLength > 0 then
ArrayLength := MaxArrayLength
else ArrayLength := SourceFileSize - Position;
SF := TFileStream.Create(SPN,fmOpenRead);
TF := TFileStream.Create(TPN,fmOpenRead);
SF.Position := Position;
TF.Position := Position;
try
SF.Read(SourceArray,ArrayLength);
TF.Read(TargetArray,ArrayLength);
Position := SF.Position;
finally
SF.Free;
TF.Free;
end;
end else LastRun := True;
until (Result = False) or LastRun;
Finalize(SourceArray);
Finalize(TargetArray);
end;
end; { Compare_file_contents }

Answer 1

This routine seems to be far more complicated than it needs to be. Rather than trying to debug it, I offer you my routine that compares streams.
function StreamsEqual(Stream1, Stream2: TStream): Boolean;
const
OneKB = 1024;
var
Buffer1, Buffer2: array [0..4*OneKB-1] of Byte;
SavePos1, SavePos2: Int64;
Count: Int64;
N: Integer;
begin
if Stream1.Size<>Stream2.Size then begin
Result := False;
exit;
end;
SavePos1 := Stream1.Position;
SavePos2 := Stream2.Position;
Try
Stream1.Position := 0;
Stream2.Position := 0;
Count := Stream1.Size;
while Count <> 0 do begin
N := Min(SizeOf(Buffer1), Count);
Stream1.ReadBuffer(Buffer1, N);
Stream2.ReadBuffer(Buffer2, N);
if not CompareMem(#Buffer1, #Buffer2, N) then begin
Result := False;
exit;
end;
dec(Count, N);
end;
Result := True;
Finally
Stream1.Position := SavePos1;
Stream2.Position := SavePos2;
End;
end;
If you wish to add your 100MB size check to this function, it's obvious where and how to do it.
The routine above uses a stack allocated buffer. In contrast your version allocates on the heap. Perhaps your version leads to heap fragmentation.
I realise that this does not answer the direct question that you asked. However, it does solve your problem. I hope this proves useful.