stack overflow error in delphi - delphi

Question

I have a procedure which calls several functions:
procedure TForm1.Button1Click(Sender: TObject);
var
rawData: TRawData;
rawInts: TRawInts;
processedData: TProcessedData;
begin
rawData := getRawData();
rawInts := getRawInts(rawData);
processedData := getProcessedData(rawInts);
end;
The data types are defined like this:
TRawData = array[0..131069] of Byte;
TRawInts = array[0..65534] of LongInt;
TProcessedData = array[0..65534] of Double;
running the program with just:
rawData := getRawData();
rawInts := getRawInts(rawData);
Works totally fine. However, when I try to run:
getProcessedData(rawInts)
I get a stackoverflow error. I don't see why this is. The function code for getProcessedData is very simple:
function getProcessedData(rawInts : TRawInts) : TProcessedData;
var
i: Integer;
tempData: TProcessedData;
scaleFactor: Double;
begin
scaleFactor := 0.01;
for i := 0 to 65534 do
tempData[i] := rawInts[i] * scaleFactor;
Result := tempData;
end;
Why is this causing an error ?

Answer 1

The default maximum stack size for a thread is 1 MB. The three local variables of Button1Click total 131,070 + 65,535 * 4 + 65,535 * 8 = 917,490 bytes. When you call getProcessedData, you pass the parameter by value, which means that the function makes a local copy of the parameter on the stack. That adds SizeOf(TRawInts) = 262,140 bytes to bring the stack to at least 1,179,630 bytes, or about 1.1 MB. There's your stack overflow.
You can reduce the stack use by passing the TRawInts array by reference instead. Then the function won't make its own copy. Zdravko's answer suggests using var, but since the function has no need to modify the passed-in array, you should use const instead.
function getProcessedData(const rawInts: TRawInts): TProcessedData;
Naively, we might expect the tempData and Result variables in getProcessedData to occupy additional stack space, but in reality, they probably won't. First, large return types typically result in the compiler changing the function signature, so it would act more like your function were declared with a var parameter instead of a return value:
procedure getProcessedData(rawInts: TRawInts; var Result: TProcessedData);
Then the call is transformed accordingly:
getProcessedData(rawInts, processedData);
Thus, Result doesn't take up any more stack space because it's really just an alias for the variable in the caller's frame.
Furthermore, sometimes the compiler recognizes that an assignment at the end of your function, like Result := tempData, means that tempData doesn't really need any space of its own. Instead, the compiler may treat your function as though you had been writing directly into Result all along:
begin
scaleFactor := 0.01;
for i := 0 to 65534 do
Result[i] := rawInts[i] * scaleFactor;
end;
However, it's best not to count on the compiler to make those sorts of memory-saving changes. Instead, it's better not to lean so heavily on the stack in the first place. To do that, you can use dynamic arrays. Those will move the large amounts of memory out of the stack and into the heap, which is the part of memory used for dynamic allocation. Start by changing the definitions of your array types:
type
TRawData = array of Byte;
TRawInts = array of Integer;
TProcessedData = array of Double;
Then, in your functions that return those types, use SetLength to assign the length of each array. For example, the function we've seen already might go like this:
function getProcessedData(const rawInts: TRawInts): TProcessedData;
var
i: Integer;
scaleFactor: Double;
begin
Assert(Length(rawInts) = 65535);
SetLength(Result, Length(rawInts));
scaleFactor := 0.01;
for i := 0 to High(rawInts) do
Result[i] := rawInts[i] * scaleFactor;
end;

Answer 2

These objects are all very large. And you appear to be allocating them as local variables. They will reside on the stack which has an fixed size, by default 1MB on Windows. You have allocated enough of these large objects in various parts of the call stack to exceed the 1MB limit. Hence a stack overflow.
Another problem in your code is the way you pass these objects as parameters. Passing large objects as value parameters results in copies being made. Copying an integer or two is nothing to worry about. Copying 65 thousand doubles is wasteful. It hurts performance. Don't do that. Pass references to large objects. Passing as const parameters achieves that.
The stack is well suited for small objects. It is not suited to these large objects. Allocate these objects on the heap. Use dynamic arrays: array of Integer, TArray<Integer> etc.
Do not increase the default stack size for your process. Especially in modern times of multi-core machines this is a recipe for out of memory errors.
Do not use magic constants. Use low() and high() to obtain array bounds.
Do pass input parameters by const. This allows the compiler to make optimisations that are significantly beneficial.

Answer 3

The key issue here is the size of your arrays.
If you use SizeOf you will see that they are probably larger than you think:
program Project3;
{$APPTYPE CONSOLE}
uses
SysUtils;
type
TRawData = array[0..131069] of Byte;
TRawInts = array[0..65534] of Longint;
TProcessedData = array[0..65534] of Double;
begin
try
writeln('TProcessedData:':20, SizeOf(TProcessedData):8);
writeln('TRawData:':20, SizeOf(TRawData):8);
writeln('TRawInts:':20, SizeOf(TRawInts):8);
writeln('Total:':20, SizeOf(TRawInts) + SizeOf(TProcessedData) + SizeOf(TRawData):8);
readln;
except
on E:Exception do
Writeln(E.Classname, ': ', E.Message);
end;
end.
Output:
TProcessedData: 524280
TRawData: 131070
TRawInts: 262140
Total: 917490
So most of the 1MB stack is consumed by the arrays. As some of the stack will already be allocated, you get a stack overflow.
This can be avoid by using dynamic arrays, which have their memory allocated from the heap.
TRawData = array of Byte;
TRawInts = array of Longint;
TProcessedData = array of Double;
...
SetLength(TProcessedData, 65535);
...