I discovered that when setting the ConnectTimeoout property for a TIdHTTP component, it makes the requests (GET and POST) become about 120ms slower?
Why is this, and can I avoid/bypass this somehow?
Env: D2010 with shipped Indy components, all updates installed for D2010. OS is WinXP (32bit) SP3 with most patches...
My timing routine is:
Procedure DoGet;
Var
Freq,T1,T2 : Int64;
Cli : TIdHTTP;
S : String;
begin
QueryPerformanceFrequency(Freq);
Try
QueryPerformanceCounter(T1);
Cli := TIdHTTP.Create( NIL );
Cli.ConnectTimeout := 1000; // without this we get < 15ms!!
S := Cli.Get('http://127.0.0.1/empty_page.php');
Finally
FreeAndNil(Cli);
QueryPerformanceCounter(T2);
End;
Memo1.Lines.Add('Time = '+FormatFloat('0.000',(T2-T1)/Freq) );
End;
With the ConnectTimeout set in code I get avg. times of 130-140ms, without it's about 5-15ms ...
When ConnectTimeout is zero (and TIdAntifreeze is not in effect), Indy simply connects. Otherwise, TIdIOHandlerStack.ConnectClient calls DoConnectTimeout, which creates a new thread to do the connecting while the calling thread sleeps and processes TIdAntifreeze operations, waiting for the connection to be established. If there's not connection by the time the timeout elapses, it throws an exception.
Threads aren't free, and the calling thread will always sleep before checking whether the connection thread has accomplished its task. The default sleep duration is 125 ms. (To use something else, activate TIdAntifreeze and set its IdleTimeout property lower than 125.)
Related
I have a problem regarding the ConnectTimeout from Indy 10's TIdTCPClient.
When setting the ConnectTimeout higher than 125ms the Connect() procedure will block the current thread for 125ms. If it is less than 125ms, it will block for the given time (e.g. it blocks for 30ms if the timeout is set to 30ms). In both cases, the connection is stable and I can transmit and receive data.
Why is the TIdTCPClient behaving like that?
IMHO the Connect() procedure should exit directly after the connection is successfully made and only block the full duration of the timeout if no connection can be opened.
Here's my code for monitoring the duration of the Connect() procedure.
The timer for calling TimerConnectTimer is set to 250ms.
I am using Lazarus v1.6.4 and Indy 10 under Windows 7 Professional.
procedure TForm1.FormCreate(Sender: TObject);
begin
timer := TEpikTimer.create(self);
timer.Clear;
end;
procedure TForm1.TimerConnectTimer(Sender: TObject);
begin
timer.Start;
client := TIdTCPClient.create();
logTime(0);
// Tested with values between 10ms and 1000ms
client.ConnectTimeout := SpinEdit1.Value;
try
logTime(1);
// choose ip and port for a running server
client.connect('192.168.2.51', 9912);
logTime(2);
except
end;
logTime(3);
try
client.Disconnect();
FreeAndNil(client);
except
end;
logTime(4);
timer.Clear;
end;
procedure TForm1.logTime(ch: integer);
begin
StringGrid1.Cells[0, ch] := FormatFloat('0.00', timer.Elapsed*1000);
end;
When setting the ConnectTimeout higher than 125ms the Connect() procedure will block the current thread for 125ms. If it is less than 125ms, it will block for the given time (e.g. it blocks for 30ms if the timeout is set to 30ms). In both cases, the connection is stable and I can transmit and receive data.
Why is the TIdTCPClient behaving like that?
That is hard to answer without knowing which version of Indy 10 you are using exactly, as the implementation of ConnectTimeout has changed over the years. But, in general:
If ConnectTimeout is set to IdTimeoutDefault or 0, IdTimeoutInfinite is used instead.
If Connect() is called in the main UI thread, TIdAntiFreeze is being used, and an infinite timeout is used, a hard-coded 2min timeout is used instead.
If any timeout is used, Connect() spawns a worker thread to connect the socket to the server, and then waits up to the timeout for the thread to terminate. If the timeout elapses before the thread terminates, Indy closes the socket and terminates the thread.
Assuming you are using a fairly up-to-date version of Indy 10 (at least SVN revision 5382 on Dec 14 2016, or later), and are not using TIdAntiFreeze, then on Windows only, the wait should exit immediately when the worker thread terminates, as Indy makes a single call to WaitForSingleObject() on the thread for the full timeout and exits as soon as WFSO exits.
If you are using TIdAntiFreeze, or are using Indy on a non-Windows platform, or are using a version of Indy 10 prior to SVN rev 5382, the wait calls IndySleep() (and if needed, TIdAntiFreeze.DoProcess()) in a loop at fixed intervals (125ms, or TIdAntiFreeze.IdleTimeOut, whichever is smaller) until the timeout is exceeded or the thread has terminated. In which case, there may be a slight delay before Connect() exits, as each sleep cycle has to complete before Connect() can check if the thread has terminated, and do so for every sleep interval within the overall connect timeout.
IMHO the Connect() procedure should exit directly after the connection is successfully made and only block the full duration of the timeout if no connection can be opened.
That is exactly what it currently does, on Windows at least. If the connection is successful, the thread terminates, and Connect() exits immediately (even if TIdAntiFreeze is used). The current sleep cycle is ended by the thread terminating itself. That is not the case on non-Windows platforms (for now, may be addressed in a future version).
Note that all of the above only applies when a timeout is used. If no timeout is used, no worker thread is used, either. Connect() simply connects the socket directly and lets the attempt block the calling thread until finished, whether successful or not.
Here's my code for monitoring the duration of the Connect() procedure.
I would suggest something more like this instead:
procedure TForm1.TimerConnectTimer(Sender: TObject);
begin
try
timer.Start;
try
logTime(0);
client := TIdTCPClient.create();
try
// choose ip and port for a running server
client.Host := '192.168.2.51';
client.Post := 9912;
// Tested with values between 10ms and 1000ms
client.ConnectTimeout := SpinEdit1.Value;
logTime(1);
client.Connect();
logTime(2);
client.Disconnect();
logTime(3);
finally
FreeAndNil(client);
end;
logTime(4);
finally
timer.Clear;
end;
except
end;
end;
I want to download some large files (GB) from an FTP server.
The download of the first file always works. Then when trying to get the second file I get:
"Socket Error # 10038. Socket operation on non-socket."
The error is on 'Get'. After 'Get' I see these messages (via FTP status event):
Starting FTP transfer
Disconnecting.
Disconnected.
The code is like this:
{pseudo-code}
for 1 to AllFiles do
begin
if Connect2FTP then
begin
FTP.Get(Name, GzFile, TRUE, FALSE); <--- Socket operation on non-socket" error (I also get EIdConnClosedGracefully 'Connection Closed Gracefully' in IDE, F9 will resume execution without problems, but this is OK)
Unpack(GzFile); <--- this takes more than 60 seconds
end;
end;
if FTP.Connected
then FTP.Disconnect;
--
function Connect2FTP(FTP: TIdFTP; RemoteFolder: string; Log: TRichLog): Boolean;
begin
Result:= FTP.Connected;
if NOT Result then
begin { We are already connected }
FTP.Host := MyFTP;
FTP.Username:= usr;
FTP.Password:= psw;
TRY
FTP.Connect;
EXCEPT
on E: Exception DO
Result:= FTP.Connected;
if Result then FTP.ChangeDir(RemoteFolder);
end;
end;
Full code here: http://pastebin.com/RScj86R8 (PAS) or here https://ufile.io/26b54 (ZIP)
I think the problem appears after calling 'Unpack' which takes few minutes.
UPDATE: CONFIRMED: The problem appears after calling 'Unpack'. I removed the call and everything is fine. Pausing (sleep or break point) the program between downloads for a while (I think for more than 60 sec) will create the same problem.
FTP uses multiple socket connections, one for commands/responses, and separate connections for each transfer.
The most likely cause of the socket error is an FTP-unaware proxy/router/firewall sitting in between TIdFTP and the FTP server is closing the command connection after a short period of inactivity. During the Unpack() (or manual pause), no commands/responses are being transmitted on the command connection, it is sitting idle, and thus is subject to being closed by a timeout on such a proxy/router/firewall.
During a transfer, the command connection is sitting idle, no FTP commands/responses are being transmitted on it (unless you abort the transfer), until the transfer is complete. An FTP-unaware proxy/router/firewall may close the command connection prematurely during this time.
To avoid that, TIdFTP has a NATKeepAlive property that can enable TCP keep-alives on the command connection while it is sitting idle. This usually prevents premature closes.
However, when there is no transfer in prgress, TIdFTP disables TCP keep-alives on the command connection if NATKeepAlive.UseKeepAlive is True. TIdFTP uses TCP keep-alives only during transfers, with the assumption that you are not going to perform long delays in between FTP commands. If you need to delay for awhile, either close the FTP connection, or send an FTP command at regular intervals (such as calling TIdFTP.Noop() in a timer/thread).
Alternatively, you can try manually enabling TCP keep-alives after connecting to the server, and set NATKeepAlive.UseKeepAlive to False so TIdFTP will not automatically disable the keep-alives after each transfer, eg:
function Connect2FTP(FTP: TIdFTP; RemoteFolder: string; Log: TRichLog): Boolean;
begin
Result := FTP.Connected;
if not Result then
begin { We are not already connected }
FTP.Host := MyFTP;
FTP.Username:= usr;
FTP.Password:= psw;
try
FTP.Connect;
try
FTP.ChangeDir(RemoteFolder);
// send a TCP keep-alive every 5 seconds after being idle for 10 seconds
FTP.NATKeepAlive.UseKeepAlive := False; // False by default, but just in case...
FTP.Socket.Binding.SetKeepAliveValues(True, 10000, 5000);
except
FTP.Disconnect(False);
raise;
end;
except
Exit;
end;
Result := True;
end;
end;
Note that while most platforms support enabling TCP keep-alives on a per-connection basis (keep-alives are part of the TCP spec), setting keep-alive intervals is platform-specific. At this time, Indy supports setting the intervals on:
Windows 2000+ and WinCE 4.x+, for Win32 and .NET
Linux, when Indy is used in Kylix
Unix/Linux and NetBSD, when Indy is used in FreePascal.
Otherwise, OS default intervals get used, which may or may not be too large in value for this situation, depending on OS configuration.
At this time, the intervals are not customizable in Delphi FireMonkey, except for Windows.
If a particular platform supports setting custom TCP keep-alive intervals, but Indy does not implement them in SetKeepAliveValues(), you can use TIdFTP.Socket.Binding.SetSockOpt() instead to set the values manually as needed. Many platforms do support TCP_KEEPIDLE/TCP_KEEPINTVL or equivalent socket options.
I have an Indy Server TIdTCPServer which has 3 bindings for different ports. If I connect a client to those 3 ports, and then deactivate the server, it gets stuck in what appears to be a deadlock. No matter what I do, it won't respond to my click, it won't even report "not responding" to Windows. If I disconnect the client(s) before deactivating the server, everything goes just perfect. I mean "deactivating" as in Server.Active:= False;.
Has anyone else experienced this? What might be causing it? I have nothing happening in here which crosses over threads which could in turn cause a deadlock (for example GUI updates). I tried an Antifreeze component TIdAntiFreeze but no luck.
TIdTCPServer is a multi-threaded component. A deadlock during server deactivation means that one or more of its client threads is not terminating correctly. That usually means that your server event handlers are doing something they should not be doing, typically either catching and discarding Indy's internal exceptions to itself, synchronizing with the thread context that is busy terminating the server, or deadlocking on something else outside of Indy. Without seeing your actual code, there is no way to know for sure which is actually the case, but it is always user error that causes this kind of deadlock.
TIdAntiFreeze only affects Indy components that run in the context of the main thread. TIdTCPServer does not.
I added this code on Form.OnClose works good!
procedure TformSFTP.FormClose(Sender: TObject; var Action: TCloseAction);
var
iA : Integer;
Context: TidContext;
begin
if sftpServidorFTP.Active then
with sftpServidorFTP.Contexts.LockList do
try
for iA := Count - 1 downto 0 do
begin
Context := Items[iA];
if Context = nil then
Continue;
Context.Connection.IOHandler.WriteBufferClear;
Context.Connection.IOHandler.InputBuffer.Clear;
Context.Connection.IOHandler.Close;
if Context.Connection.Connected then
Context.Connection.Disconnect;
end;
finally
sftpServidorFTP.Contexts.UnlockList;
end;
if sftpServidorFTP.Active then
sftpServidorFTP.Active := False;
end;
Related to a post of mine ( How to retrieve a file from Internet via HTTP? ) about how to easily and robustly download a file from Internet, I have found a possible solution - however is not working as it was supposed to work.
According to MS documentation, the code below is supposed to time-out at 500ms after I disconnect myself from internet. However, it looks like it totally ignores the 'INTERNET_OPTION_RECEIVE_TIMEOUT' setting. The application freezes during download. It takes about 20-30 to this function to realize that there the Internet connection is down and to give the control back to the GUI.
Anybody knows why?
function GetBinFileHTTP (const aUrl: string; const pStream: TStream; wTimeOut: Word= 500; wSleep: Word= 500; wAttempts: Word= 10): Integer;
CONST
BufferSize = 1024;
VAR
hSession, hService: HINTERNET;
Buffer : array[0..BufferSize-1] of Char;
dwBytesRead, dwBytesAvail: DWORD;
lSucc : LongBool;
lRetries, dwTimeOut: Integer;
begin
Result:= 0;
if NOT IsConnectedToInternet then
begin
Result:= -1;
EXIT;
end;
hSession := InternetOpen(PChar(ExtractFileName(Application.ExeName)), INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0); { The INTERNET_OPEN_TYPE_PRECONFIG flag specifies that if the user has configured Internet Explorer to use a proxy server, WinInet will use it as well. }
if NOT Assigned(hSession) then
begin
Result:= -4;
EXIT;
end;
TRY
hService := InternetOpenUrl(hSession, PChar(aUrl), nil, 0, INTERNET_FLAG_RELOAD, 0);
if NOT Assigned(hService) then Exit;
TRY
FillChar(Buffer, SizeOf(Buffer), 0);
{ Set time out }
dwTimeOut:= wTimeOut;
InternetSetOption(hService, INTERNET_OPTION_RECEIVE_TIMEOUT, #dwTimeOut, SizeOf(dwTimeOut)); { use INTERNET_FLAG_RELOAD instead of NIL to redownload the file instead of using the cache }
InternetSetOption(hService, INTERNET_OPTION_CONNECT_TIMEOUT, #dwTimeOut, SizeOf(dwTimeOut));
REPEAT
lRetries := 0;
REPEAT
lSucc:= InternetQueryDataAvailable( hService, dwBytesAvail, 0, 0);
if NOT lSucc
then Sleep( wSleep );
if lRetries > wAttempts
then Result:= -2;
UNTIL lSucc OR (Result= -2);
if NOT InternetReadFile(hService, #Buffer, BufferSize, dwBytesRead) then
begin
Result:= -3; { Error: File not found/File cannot be downloaded }
EXIT;
end;
if dwBytesRead = 0
then Break;
pStream.WriteBuffer(Buffer[0], dwBytesRead);
UNTIL False;
FINALLY
InternetCloseHandle(hService);
end;
FINALLY
InternetCloseHandle(hSession);
end;
Result:= 1;
end;
Here is the documentation:
{
INTERNET_OPTION_CONNECT_TIMEOUT Sets or retrieves an unsigned long integer value that contains the time-out value to use for Internet connection requests. If a connection request takes longer than this time-out value, the request is canceled. When attempting to connect to multiple IP addresses for a single host (a multihome host), the timeout limit is cumulative for all of the IP addresses. This option can be used on any HINTERNET handle, including a NULL handle. It is used by InternetQueryOption and InternetSetOption.
INTERNET_OPTION_RECEIVE_TIMEOUT Sets or retrieves an unsigned long integer value that contains the time-out value to receive a response to a request. If the response takes longer than this time-out value, the request is canceled. This option can be used on any HINTERNET handle, including a NULL handle. It is used by InternetQueryOption and InternetSetOption. For using WinInet synchronously, only the default value for this flag can be changed by calling InternetSetOption and passing NULL in the hInternet parameter.
INTERNET_OPTION_CONTROL_RECEIVE_TIMEOUT - Identical to INTERNET_OPTION_RECEIVE_TIMEOUT. This is used by InternetQueryOption and InternetSetOption.
}
Edit:
I disconnect the Internet by unplugging the cable or (for wireless) from software AFTER the application starts the download (I chose to download large file). It simulates the web site going offline.
The connect timeout obviously isn't applicable in your test because by the time you start your test (i.e., pull the plug), the connection has already been established. Indeed, the connection is already established before you even get around to setting the timeout option.
The validity of the receive timeout is also suspect, because you've already begun receiving the response, too.
The most promising-looking timeout is the disconnect timeout, but MSDN says that's not implemented yet.
It seems to me that the way to go is to use asynchronous operations. Use InternetReadFileEx and use the irf_Async and irf_No_Wait flags. If too much time passes without receiving any data, close the connection. Another option is to stick with your synchronous calls, but then call InternetCloseHandle from another thread if the download takes too long.
There is a documented bug in MS IE code. Can only be solved by using the code in a thread and re-implementing the time out mechanism.
Details:
"This acticle shows a workaround to the InternetSetOption API bug on setting timeout values by creating a second thread.
InternetSetOption Does Not Set Timeout Values"
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q224318
(Link was reported broken. Blame MS not me)
Maybe somebody can help with implementing this bug fix also in Delphi. I personally don't have experience with C. Even the backbone in pseudo-Pascal will be nice.
IMO, you should run this in a thread. Threading does not have to mean looping - it can be a "one and done" thread. Run it that way, and your GUI remains responsive until the thread finishes. I realize that this does not actually answer your question, but it will make your code better.
Also, if you disconnect the internet during the first loop where you're checking for data, I think it will retry 10 times. You should detect, and then quit right away.
Lastly, I don't think you should use EXIT when you've got handles and stuff open. Break instead, so that you still run through the disconnects. I would expect your code to tie up the socket. I saw this recently during a code review when there was an EXIT intead of a BREAK, and it's causing a memory leak because objects are created and never freed. I'd use the same rule here.
Are you sure that you aren't hitting the INTERNET_OPTION_CONNECT_TIMEOUT? It will try to connect first, then receive.
In order to test the connect timeout, it must resolve, but never connect. In order to test the read timeout, it must connect, but never receive any data.
I generally set my connect timeout to 10 seconds, and the read timeout to 30 seconds. Anything longer than that, I consider down anyway.
I haven't programmed in Delphi for a while and frankly didn't think I'll ever have to but...
Here I am, desperately trying to find some information on the matter and it's so scarce nowadays, I can't find anything. So maybe you guys could help me out.
Currently my application uses Synapse library to make HTTP calls, but it doesn't allow for setting a timeout. Usually, that's not a big problem, but now I absolutely must to have a timeout to handle any connectivity issues nicely.
What I'm looking for, is a library (synchronous or not) that will allow making HTTP requests absolutely transparent for the user with no visible or hidden delays. I can't immediately kill a thread right now, and with possibility of many frequent requests to the server that is not responding, it's no good.
EDIT: Thanks everybody for your answers!
You will always have to take delays and timeouts into account when doing network communication. The closest you can get IMHO is to put network communication in a thread. Then you can check if the thread finishes in the desired time and if not just let it finish, but ignore the result (there's no safe way to abort a thread). This has an additional advantage: you can now just use synchronous network calls which are a lot easier to read.
In synapse, the timeout is available from the TSynaClient object, which THttpSend decends from. So all you have to do to adjust for timeout (assuming your using the standard functions) is to copy the function your using, add a new parameter and set the Timeout to what you need. For example:
function HttpGetTextTimeout(const URL: string;
const Response: TStrings;
const Timeout:integer): Boolean;
var
HTTP: THTTPSend;
begin
HTTP := THTTPSend.Create;
try
HTTP.Timeout := Timeout;
Result := HTTP.HTTPMethod('GET', URL);
if Result then
Response.LoadFromStream(HTTP.Document);
finally
HTTP.Free;
end;
end;
Synapse defaults to a timeout of 5000 and does timeout if you wait long enough. Since its tightly contained, synapse runs perfectly fine in threads.
[Known to work on D2010 only]
You can use MSXML to send client requests (add msxml and ole2 to your uses clause). The trick is to use IServerXMLHTTPRequest rather than IXMLHTTPRequest, as the former allows timeouts to be specified. The code below shows the Execute() method of a thread:
procedure TClientSendThread.Execute;
const
LResolveTimeoutMilliseconds = 2000;
LConnectTimeoutMilliseconds = 5000;
LSendTimeoutMilliseconds = 5000;
LReceiveTimeoutMilliseconds = 10000;
var
LHTTPServer: IServerXMLHTTPRequest;
LDataStream: TMemoryStream;
LData: OleVariant;
begin
{Needed because this is inside a thread.}
CoInitialize(nil);
LDataStream := TMemoryStream.Create;
try
{Populate ....LDataStream...}
LData := MemoryStreamToOleVariant(LDataStream);
LHTTPServer := CreateOleObject('MSXML2.ServerXMLHTTP.3.0') as IServerXMLHTTPRequest;
LHTTPServer.setTimeouts(
LResolveTimeoutMilliseconds,
LConnectTimeoutMilliseconds,
LSendTimeoutMilliseconds,
LReceiveTimeoutMilliseconds
);
LHTTPServer.open('POST', URL, False, 0, 0);
LHTTPServer.send(LData);
FAnswer := LHTTPServer.responseText;
finally
FreeAndNil(LDataStream);
CoUninitialize;
end;
end;
I recently discovered an extremely annoying behavior of this MSXML technique in which GET requests will not be re-sent if the URL remains unchanged for subsequent sendings; in other words, the client is caching GET requests. This does not happen with POST.
Obviously, once the timeouts occur, the Execute method completes and the thread is cleaned up.
Synapse can be configured to raise an Exception when network errors occur.
RaiseExcept
Check http://synapse.ararat.cz/doc/help/blcksock.TBlockSocket.html#RaiseExcept:
If True, winsock errors raises
exception. Otherwise is setted
LastError value only and you must
check it from your program! Default
value is False.