Why does my Delphi Indy idHTTpServer stop responding when CLOSE_WAIT? - delphi

Question

The Environment
I've created a web server in Delphi using Indy component TidHTTPServer. I'm using Delphi XE2 which came with Indy version 10.5.8. The
server is running as a desktop app with a form that displays a log of the connections and their requests. It is running on Windows 7
Professional. Requests are for SQL data from a Firebird database. The response is JSON. All traffic is HTTP.
The Challenge
When I was testing it with a small number of users everything worked great. Now that I have rolled it out to about 400 users there are
communication problems. The server stops responding to requests and the only way I can get it to respond again is to reboot the machine it is running on and then restart it. The need to reboot occurs more frequently during
high volume times.
The Symptoms
Using Windows netstat I have noticed that whenever a TCP connection of type CLOSE_WAIT occurs the server stops responding to requests and I have to reboot again
The Test Procedure
I have been able to simulate this hanging even when there is no traffic on the server. I created a web page that sends multiple requests with
a delay between each request.
The web page let's me specify the number of requests to make, how long to wait between each request, and how long to wait before timing out. Even at one millisecond between requests the server seems to respond without issue.
The Test Results
If I set the time out period of each request to a very small number, like 1 msec, I can make my Delphi HTTP Server hang. At a 1 msec timeout requests to my server fail every time, as I would expect. The time out is so short my server can't possibly respond quickly enough.
What I don't understand is that after I force this timeout at the client side, even a relatively small number of requests (fewer than 50), my Delphi web server no longer responds to any requests. When I run netstat on the server machine there are a number of CLOSE_WAIT socket connections. Even after an hour and after closing my server the CLOSE_WAIT socket connections persist.
The Questions
What is going on? Why does my Delphi Indy idHTTPServer stop responding when there are (even just one) CLOSE_WAIT socket connection? The CLOSE_WAITs don't go away and the server does not start responding again. I have to reboot.
What am I not doing?
Here is the results of netstat command showing CLOSE_WAITs:
C:\Windows\system32>netstat -abn | findstr 62000
TCP 0.0.0.0:62000 0.0.0.0:0 LISTENING
TCP 10.1.1.13:62000 9.49.1.3:57036 TIME_WAIT
TCP 10.1.1.13:62000 9.49.1.3:57162 CLOSE_WAIT
TCP 10.1.1.13:62000 9.49.1.3:57215 CLOSE_WAIT
TCP 10.1.1.13:62000 9.49.1.3:57244 CLOSE_WAIT
TCP 10.1.1.13:62000 9.49.1.3:57263 CLOSE_WAIT
TCP 10.1.1.13:62000 9.49.1.3:57279 ESTABLISHED
TCP 10.1.1.13:62000 104.236.216.73:59051 ESTABLISHED
Here is the essence of my web server:
unit MyWebServer;
interface
Uses
...
Type
TfrmWebServer = class(TForm)
...
IdHTTPServer: TIdHTTPServer;
...
procedure IdHTTPServerCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
procedure IdHTTPServerDisconnect(AContext: TIdContext);
procedure btnStartClick(Sender: TObject);
...
dbFirebird : TIBDatabase;
txFireird : TIBTransaction;
...
private
function CreateSomeResponseStringData: string;
end;
implementation
procedure TfrmWebServer.btnStartClick(Sender: TObject);
begin
{set the IP's and proit to listen on}
IdHTTPServer.Bindings.Clear;
IdHTTPServer.Bindings.Add.IP := GetSetting(OPTION_TCPIP_ADDRESS);
IdHTTPServer.Bindings.Add.Port := Str2Int(GetSetting(OPTION_TCPIP_PORT));
{start the web server}
IdHTTPServer.Active := TRUE;
...
dbFirebird.Transactrion := txFirebird;
...
end;
procedure TfrmWebServer.IdHTTPServerCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
var
qryFirebird : TIBSql;
function CreateSomeResponseStringData: string;
begin
qryFirebird := NIL;
qryFirebird := TIBSql.Create(IdHTTPServer);
qryFirebird.Database := dbFirebird;
dbFirebird.Connected := FALSE;
dbFirebird.Connected := TRUE;
qryFirebird.Active := TRUE;
Result := {...whatever string will be returned}
end;
function CreateAnErrorResponse: string;
begin
Result := {...whatever string will be returned}
end;
begin
try
AResponseInfo.ContentText := CreateSomeResponseStringData;
{Clean up: What do I do here to make sure that the connection that was served is:
- properly closed so that I don't run out of resourses?
- anything that needs to be cleaned up is freed so no memory leaks
- TIME_WAIT, CLOSE_WAIT, any other kind of _WAITs are not accumulating?}
except;
AResponseInfo.ContentText := CreateAnErrorResponse;
end;
qryFirebird.Free;
end;
procedure TfrmWebServer.IdHTTPServerDisconnect(AContext: TIdContext);
begin
{Maybe I do the "Clean Up" here? I tried Disconnect as shown but still lots of
TIME_WAIT tcp/ip connections accumulate. even after the app is closed}
AContext.Connection.Disconnect;
end;
end.

Answer 1

There are at least two major issues with this code that could cause the crashing:
The database and transaction objects are global to all threads created by IdHTTPServer. When you disconnect the database it would disconnect for all threads.
If there is a run time error assigning content text this line AResponseInfo.ContentText := CreateAnErrorResponse; is not in an exception block.
Here is how I would fix this:
...
procedure TfrmWebServer.btnStartClick(Sender: TObject);
begin
{set the IP's and port to listen on}
IdHTTPServer.Bindings.Clear;
IdHTTPServer.Default.Port := Str2Int(GetSetting(OPTION_TCPIP_PORT));
IdHTTPServer.Bindings.Add.IP := GetSetting(OPTION_TCPIP_ADDRESS);
{start the web server}
IdHTTPServer.Active := TRUE;
...
end;
procedure TfrmWebServer.IdHTTPServerCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
var
{make these local to each thread}
qryFirebird : TIBSql;
dbFirebird : TIBDatabase;
txFirebird : TIBTransaction;
function CreateSomeResponseStringData: string;
begin
dbFirebird := TIBDatbase.Create(IdHTTPServer);
txFirebird := TIBTransaction.Create(IdHTTPServer);
qryFirebird := TIBSql.Create(IdHTTPServer);
dbFirebird.Transaction := txFirebird;
qryFirebird.Database := dbFirebird;
...Add params that do the log in to database
dbFirebird.Connected := TRUE;
qryFirebird.Active := TRUE;
Result := {...whatever string will be returned}
end;
function CreateAnErrorResponse: string;
begin
Result := {...whatever string will be returned}
end;
begin
try
try
...
AResponseInfo.ContentText := CreateSomeResponseStringData;
...
except;
try
AResponseInfo.ContentText := CreateAnErrorResponse;
except
{give up}
end;
end;
finaly
qryFirebird.Free;
dbFirebird.Free;
txFirebird.Free;
end;
end;
end.