I haven't programmed in Delphi for a while and frankly didn't think I'll ever have to but...
Here I am, desperately trying to find some information on the matter and it's so scarce nowadays, I can't find anything. So maybe you guys could help me out.
Currently my application uses Synapse library to make HTTP calls, but it doesn't allow for setting a timeout. Usually, that's not a big problem, but now I absolutely must to have a timeout to handle any connectivity issues nicely.
What I'm looking for, is a library (synchronous or not) that will allow making HTTP requests absolutely transparent for the user with no visible or hidden delays. I can't immediately kill a thread right now, and with possibility of many frequent requests to the server that is not responding, it's no good.
EDIT: Thanks everybody for your answers!
You will always have to take delays and timeouts into account when doing network communication. The closest you can get IMHO is to put network communication in a thread. Then you can check if the thread finishes in the desired time and if not just let it finish, but ignore the result (there's no safe way to abort a thread). This has an additional advantage: you can now just use synchronous network calls which are a lot easier to read.
In synapse, the timeout is available from the TSynaClient object, which THttpSend decends from. So all you have to do to adjust for timeout (assuming your using the standard functions) is to copy the function your using, add a new parameter and set the Timeout to what you need. For example:
function HttpGetTextTimeout(const URL: string;
const Response: TStrings;
const Timeout:integer): Boolean;
var
HTTP: THTTPSend;
begin
HTTP := THTTPSend.Create;
try
HTTP.Timeout := Timeout;
Result := HTTP.HTTPMethod('GET', URL);
if Result then
Response.LoadFromStream(HTTP.Document);
finally
HTTP.Free;
end;
end;
Synapse defaults to a timeout of 5000 and does timeout if you wait long enough. Since its tightly contained, synapse runs perfectly fine in threads.
[Known to work on D2010 only]
You can use MSXML to send client requests (add msxml and ole2 to your uses clause). The trick is to use IServerXMLHTTPRequest rather than IXMLHTTPRequest, as the former allows timeouts to be specified. The code below shows the Execute() method of a thread:
procedure TClientSendThread.Execute;
const
LResolveTimeoutMilliseconds = 2000;
LConnectTimeoutMilliseconds = 5000;
LSendTimeoutMilliseconds = 5000;
LReceiveTimeoutMilliseconds = 10000;
var
LHTTPServer: IServerXMLHTTPRequest;
LDataStream: TMemoryStream;
LData: OleVariant;
begin
{Needed because this is inside a thread.}
CoInitialize(nil);
LDataStream := TMemoryStream.Create;
try
{Populate ....LDataStream...}
LData := MemoryStreamToOleVariant(LDataStream);
LHTTPServer := CreateOleObject('MSXML2.ServerXMLHTTP.3.0') as IServerXMLHTTPRequest;
LHTTPServer.setTimeouts(
LResolveTimeoutMilliseconds,
LConnectTimeoutMilliseconds,
LSendTimeoutMilliseconds,
LReceiveTimeoutMilliseconds
);
LHTTPServer.open('POST', URL, False, 0, 0);
LHTTPServer.send(LData);
FAnswer := LHTTPServer.responseText;
finally
FreeAndNil(LDataStream);
CoUninitialize;
end;
end;
I recently discovered an extremely annoying behavior of this MSXML technique in which GET requests will not be re-sent if the URL remains unchanged for subsequent sendings; in other words, the client is caching GET requests. This does not happen with POST.
Obviously, once the timeouts occur, the Execute method completes and the thread is cleaned up.
Synapse can be configured to raise an Exception when network errors occur.
RaiseExcept
Check http://synapse.ararat.cz/doc/help/blcksock.TBlockSocket.html#RaiseExcept:
If True, winsock errors raises
exception. Otherwise is setted
LastError value only and you must
check it from your program! Default
value is False.
Related
Is it possible to read the HTTP headers (specifically the GET header) in the Indy TIdHTTPServer.OnConnect event without interfering with the subsequent OnCommandGet event?
If I try to pull them with a loop of ReadLn's then OnCommandGet never fires. I need an advanced peek at them without pulling them from the input buffer.
Is it possible to read the HTTP headers (specifically the GET header) in the Indy TIdHTTPServer.OnConnect event without interfering with the subsequent OnCommandGet event?
It it possible, in that you could use the TIdIOHandler.WaitFor() method to wait for the header terminator to arrive in the TIdIOHandler.InputBuffer, returning everything received before it without removing anything from the buffer, eg:
procedure TMyForm.IdHTTPServer1Connect(AContext: TIdContext);
var
headers: String;
begin
header := AContext.Connection.IOHandler.WaitFor(EOL+EOL, False);
...
end;
However, this has some limitations:
it assumes each line is terminated by the byte sequence $0D $0A and thus the header is terminated by the byte sequence $0D $0A $0D $0A. This is technically true per the HTTP standards, and will usually be the case. However, some clients do terminate lines with just $0A and thus the header would be terminated by $0A $0A. TIdHTTPServer would normally handle that just fine, but using WaitFor() would not.
A more robust solution would involve using TIdIOHandler.CheckForDataOnSource() in a loop, manually scanning the TIdIOHandler.InputBuffer until either $0D $0A $0D $0A or $0A $0A is found in the buffer.
this won't work if there are multiple HTTP requests on the same connection, which can happen if HTTP keep-alives or HTTP pipelining are used. You would be "peeking" the header of only the 1st HTTP request on the connection.
If I try to pull them with a loop of ReadLn's then OnCommandGet never fires.
Correct, because TIdHTTPServer expects to be the one to read them off the InputBuffer. If you read them yourself beforehand, there won't be anything left for TIdHTTPServer to read, so it won't know what each HTTP request even looks like.
I need an advanced peek at them without pulling them from the input buffer.
Why? What do you want to do with them, if you could get them?
You should check if the TIdHTTPServer.OnHeadersAvailable event suits your needs. It is fired at the beginning of every HTTP request, after the headers have been read from the InputBuffer but before the request body has been read.
I got it working by peeking at the Inputbuffer per Remy's suggestion:
procedure TForm1.IdHTTPServer1Connect(AContext: TIdContext);
var
s: string;
Done: boolean;
begin
Done := False;
repeat
Sleep(10);
if AContext.Connection.IOHandler.CheckForDataOnSource then
begin
s := AContext.Connection.IOHandler.InputBuffer.AsString;
if (Pos(#13#10#13#10, s) > 0) or (Pos(#10#10, s) > 0) then Done := True;
end;
until Done;
...
end;
One issue I could see happening is a bot making a TCP connection on my port and that loop never ending since no headers are coming. I would need to add some kind of timeout check.
The other suggestion of using OnHeadersAvailable wouldn't work for me because it's called right before OnCommandGet each time (i.e., multiple times per connection when KeepAlive is True) so I might as well just put tests in OnCommandGet if I went that route.
EDIT:
I also just tried doing this in the OnConnect handler:
s := AContext.Connection.IOHandler.WaitFor(#10, False, True, nil, 1000);
Since I only need the GET line and it will always be first (right?) if it's included at all, I just need to find the first line feed character. That solves the line terminator problem and there's a timeout parameter that solves the bot issue. While this does read the first line header, it also causes an immediate disconnect and CommandGet never gets called. What am I doing wrong?
I'm using TIdHTTP in my class to handle web APIs (TWebAPI). Since it might happen that this class is used in a TIdHTTPServer.OnCommandGet event handler, I need to make sure that TWebAPIis thread safe.
Do I need to wrap a PUT/GET/POST inside a TMultiReadExclusiveWriteSynchronizer?
EDIT: Code-Sample
TWebAPI
TWebAPI=class(TObject)
FSocket:TidHTTP;
end;
procedure TWebAPI.Send;
Var
Response:TSTringStream;
begin
FSocket.Get(fURL,Response);
end;
Main Program
TMain=class(TForm)
Server:TidHTTPServer;
WebAPI:TWebAPI;
end;
procedure TMain.ServerCommandGet(....)
begin
WebAPI.Send;
end;
So my WebAPI would be used in different threads on each command the server gets. SHould I use the CriticalSection in TMain, or implement it in TWebAPI like this?
TWebAPI=class(TObject)
FSocket:TidHTTP;
FLock:TCriticalSection;
end;
procedure TWebAPI.Send;
Var
Response:TSTringStream;
begin
FLock.Aquire;
try
FSocket.Get(fURL,Response);
finally
FLock.Release;
end;
end;
A single TIdHTTP could be protected by a TCriticalSection or TMonitor. There is no R/W involved, so don't use TMultiReadExclusiveWriteSynchronizer - just a mutex/lock. But if you use a single TIdHTTP you will need a mutex/lock, so all HTTP calls will be serialized, which may be not a good idea on a multi-threaded server.
I would maintain one connection per thread, or write a connection pool. Perhaps just a given TIdHTTP each time may not be too bad. At least it will be safe and there will be room for improvement later on. Reopening a TCP/HTTP connection is quick in practice.
I tried to employ Indy 10.5.5 (shipped with Delphi 2010) for:
connecting to telnet server
performing username/password authentication (gaining access to the command shell)
executing a command with returning resulting data back to application
and had no success, additionally i'm completely lost in spaghetti logic of Indy's internals and now have no idea why it didnt work or how i supposed to send strings to the server and grab the results. Need some sample code to study.
Formal form of the question: Where can i get 3-rd party contributed demo covering TIdTelnet component? (indyproject.org demos webpage do not have one)
The main problem with Telnet is that it DOES NOT utilize a command/response model like most other Internet protocols do. Either party can send data at any time, and each direction of data is independant from the other direction. This is reflected in TIdTelnet by the fact that it runs an internal reading thread to receive data. Because of this, you cannot simply connect, send a command, and wait for a response in a single block of code like you can with other Indy components. You have to write the command, then wait for the OnDataAvailable event to fire, and then parse the data to determine what it actually is (and be prepared to handle situations where partial data may be received, since that is just how TCP/IP works).
If you are connecting to a server that actually implements a command/response model, then you are better off using TIdTCPClient directly instead of TIdTelnet (and then implement any Telnet sequence decoding manually if the server really is using Telnet, which is rare nowadays but not impossible). For Indy 11, we might refactor TIdTelnet's logic to support a non-threaded version, but that is undecided yet.
done with indy.
no comments.. just som old code :-)
telnet don't like the send string kommand.. use sendch.
telnetdude.Host := 1.1.1.1;
try
telnetdude.connect;
except
on E: Exception do begin
E.CleanupInstance;
end; {except}
if telnetdude.Connected then begin
for i := 1 to length(StringToSend) do telnetdude.sendch(StringToSend[i]);
telnetdude.sendch(#13);
end;
end; {while}
end; {if}
if telnetdude.Connected then telnetdude.Disconnect;
end;
I hope this helps anyone looking for answers to a similar question.
Firstly, It would seem the typical command/response model (as mentioned above, does indeed NOT apply).
So I just got it working for some very simple application (rebooting my router).
Specific additions to above code from Johnny Lanewood (and perhaps some clarification)
a) You have to send #13 to confirm the command
b) I got "hangs" on every command I sent / response I requested UNTIL I enabled ThreadedEvent. (this was my big issue)
c) the OnDataAvailable event tells you when new data is available from the Telnet Server - however there are no guarantees as to what this data is - i.e. it's pretty what you get in the command line / what ever is appended to the previous responses. But is is NOT a specific response line to your command - it's whatever the telnet server returns (could be welcome info, ASCII drawings etc etc.)
Given (c) above, one would rather check the OnDataAvailable event and parse the data (knowing what you'd expect). When the output stops (i.e. you need build a mechanism for this), you can parse the data and determine whether the server is ready for something new from the client. For the purpose of my code below, I set a read timemout and I just used Sleep(2000) - ignorantly expecting no errors and that the server would be ready after the sleep for the next command.
My biggest stumbling block was ThreadedEvent := True (see above in b)
Thus, my working solution (for specific application, and possibly horrible to some).
lIDTelnet := TIdTelnet.Create(nil);
try
lIdTelnet.ReadTimeout := 30000;
lIDTelnet.OnDataAvailable := TDummy.Response;
lIDTelnet.OnStatus := TDummy.Status;
lIdTelnet.ThreadedEvent := True;
try
lIDTelnet.Connect('192.168.0.1', 23);
if not lIDTelnet.Connected then
Raise Exception.Create('192.168.0.1 TELNET Connection Failed');
Sleep(2000);
lIdtelnet.SendString(cst_user + #13);
Sleep(2000);
lIdtelnet.SendString(cst_pass + #13);
Sleep(2000);
lIdtelnet.SendString(cst_reboot + #13);
Sleep(2000);
if lIDTelnet.Connected then
lIDTelnet.Disconnect;
except
//Do some handling
end;
finally
FreeAndNil(lIdTelnet);
end;
and then
class procedure TDummy.Response(Sender: TIdTelnet; const Buffer: TIdBytes);
begin
Write(TDummy.ByteToString(Buffer));
end;
class function TDummy.ByteToString(
const aBytes: TIdBytes): String;
var
i : integer;
begin
result := '';
for i := 0 to Length(aBytes) -1 do
begin
result := result + Char(aBytes[i]);
end;
end;
I'm trying to use the TUdpSocket in Delphi. What I want to do is connect to a UDP server, send some data and wait for answer. Data is sent correctly, but the control does not receive anything. I don't know why. I've been struggling with this problem for many hours now and I'm going to give up :-(.
I tried to use TIdUDPClient, but the situation is the same. Data is sent properly, but none is received.
Only does TIdUDPServer work more or less properly for it both sends and receives data. Unfortunately data reception is handled by a separate thread (main or other, depending on the ThreadedEvent property) which forces me to use synchronization and complicate the whole code. I would like to handle UDP connection in my own thread. Just send some data and call WaitForData() to wait for an answer and then handle it in the same thread.
And if opossible, I don't want to use any third party controls, but if it is the only solution, I accept it.
Thank you very, very much for your help in advance.
---- Examples ---
i) TUDPSocket:
var
lR, lW, lE: boolean;
begin
UdpSocket1.LocalPort := '1600';
UdpSocket1.RemotePort := '1600';
UdpSocket1.RemoteHost := '127.0.0.1';
UdpSocket1.Connect;
UdpSocket1.Sendln('test');
UdpSocket1.Select(#lR, #lW, #lE, 2000);
if lR then
ShowMessage(UdpSocket1.Receiveln());
end;
As you can see, the control should receive the data it transmits. And apparently it does, for the lR evaluates to true after the Select() method is called. But Receiveln() returns an empty string, as ReceiveBuf() does. When i start a UDP server and send some data to it, it is received properly, so I'm certain that the data is really sent.
You should need nothing more than this:
function SayHi(Host: String; Port: Integer): String;
var
Client: TIdUDPClient;
begin
Client := TIdUDPClient.Create(nil);
try
Client.Host := Host;
Client.Port := Port;
Client.Send('Hello');
Result := Client.ReceiveString(1000);
finally
Client.Free;
end;
end;
which will, in the same thread, send a UDP packet and either receive a UDP packet or raise an exception after a timeout.
If the above doesn't work, check (using something like Wireshark that the client's actually sending the data. Then check on the server that it's actually receiving the packet.
Send() ultimately (on Windows) calls WinSock's sendto(), and ReceiveString() uses select() and recvfrom(). You could sort've simulate a TIdUDPServer by
while not Terminated do begin
S := Client.ReceiveString(1000);
DoSomething(S);
end;
Another possibility would be to use Synapse for your communication. It is a simple communications library which performs blocking calls, so works well in a single worker thread where you want to stop and wait for data rather than rely on an event to fire. The latest versions in SVN have support for the latest versions of Delphi.
Related to a post of mine ( How to retrieve a file from Internet via HTTP? ) about how to easily and robustly download a file from Internet, I have found a possible solution - however is not working as it was supposed to work.
According to MS documentation, the code below is supposed to time-out at 500ms after I disconnect myself from internet. However, it looks like it totally ignores the 'INTERNET_OPTION_RECEIVE_TIMEOUT' setting. The application freezes during download. It takes about 20-30 to this function to realize that there the Internet connection is down and to give the control back to the GUI.
Anybody knows why?
function GetBinFileHTTP (const aUrl: string; const pStream: TStream; wTimeOut: Word= 500; wSleep: Word= 500; wAttempts: Word= 10): Integer;
CONST
BufferSize = 1024;
VAR
hSession, hService: HINTERNET;
Buffer : array[0..BufferSize-1] of Char;
dwBytesRead, dwBytesAvail: DWORD;
lSucc : LongBool;
lRetries, dwTimeOut: Integer;
begin
Result:= 0;
if NOT IsConnectedToInternet then
begin
Result:= -1;
EXIT;
end;
hSession := InternetOpen(PChar(ExtractFileName(Application.ExeName)), INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0); { The INTERNET_OPEN_TYPE_PRECONFIG flag specifies that if the user has configured Internet Explorer to use a proxy server, WinInet will use it as well. }
if NOT Assigned(hSession) then
begin
Result:= -4;
EXIT;
end;
TRY
hService := InternetOpenUrl(hSession, PChar(aUrl), nil, 0, INTERNET_FLAG_RELOAD, 0);
if NOT Assigned(hService) then Exit;
TRY
FillChar(Buffer, SizeOf(Buffer), 0);
{ Set time out }
dwTimeOut:= wTimeOut;
InternetSetOption(hService, INTERNET_OPTION_RECEIVE_TIMEOUT, #dwTimeOut, SizeOf(dwTimeOut)); { use INTERNET_FLAG_RELOAD instead of NIL to redownload the file instead of using the cache }
InternetSetOption(hService, INTERNET_OPTION_CONNECT_TIMEOUT, #dwTimeOut, SizeOf(dwTimeOut));
REPEAT
lRetries := 0;
REPEAT
lSucc:= InternetQueryDataAvailable( hService, dwBytesAvail, 0, 0);
if NOT lSucc
then Sleep( wSleep );
if lRetries > wAttempts
then Result:= -2;
UNTIL lSucc OR (Result= -2);
if NOT InternetReadFile(hService, #Buffer, BufferSize, dwBytesRead) then
begin
Result:= -3; { Error: File not found/File cannot be downloaded }
EXIT;
end;
if dwBytesRead = 0
then Break;
pStream.WriteBuffer(Buffer[0], dwBytesRead);
UNTIL False;
FINALLY
InternetCloseHandle(hService);
end;
FINALLY
InternetCloseHandle(hSession);
end;
Result:= 1;
end;
Here is the documentation:
{
INTERNET_OPTION_CONNECT_TIMEOUT Sets or retrieves an unsigned long integer value that contains the time-out value to use for Internet connection requests. If a connection request takes longer than this time-out value, the request is canceled. When attempting to connect to multiple IP addresses for a single host (a multihome host), the timeout limit is cumulative for all of the IP addresses. This option can be used on any HINTERNET handle, including a NULL handle. It is used by InternetQueryOption and InternetSetOption.
INTERNET_OPTION_RECEIVE_TIMEOUT Sets or retrieves an unsigned long integer value that contains the time-out value to receive a response to a request. If the response takes longer than this time-out value, the request is canceled. This option can be used on any HINTERNET handle, including a NULL handle. It is used by InternetQueryOption and InternetSetOption. For using WinInet synchronously, only the default value for this flag can be changed by calling InternetSetOption and passing NULL in the hInternet parameter.
INTERNET_OPTION_CONTROL_RECEIVE_TIMEOUT - Identical to INTERNET_OPTION_RECEIVE_TIMEOUT. This is used by InternetQueryOption and InternetSetOption.
}
Edit:
I disconnect the Internet by unplugging the cable or (for wireless) from software AFTER the application starts the download (I chose to download large file). It simulates the web site going offline.
The connect timeout obviously isn't applicable in your test because by the time you start your test (i.e., pull the plug), the connection has already been established. Indeed, the connection is already established before you even get around to setting the timeout option.
The validity of the receive timeout is also suspect, because you've already begun receiving the response, too.
The most promising-looking timeout is the disconnect timeout, but MSDN says that's not implemented yet.
It seems to me that the way to go is to use asynchronous operations. Use InternetReadFileEx and use the irf_Async and irf_No_Wait flags. If too much time passes without receiving any data, close the connection. Another option is to stick with your synchronous calls, but then call InternetCloseHandle from another thread if the download takes too long.
There is a documented bug in MS IE code. Can only be solved by using the code in a thread and re-implementing the time out mechanism.
Details:
"This acticle shows a workaround to the InternetSetOption API bug on setting timeout values by creating a second thread.
InternetSetOption Does Not Set Timeout Values"
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q224318
(Link was reported broken. Blame MS not me)
Maybe somebody can help with implementing this bug fix also in Delphi. I personally don't have experience with C. Even the backbone in pseudo-Pascal will be nice.
IMO, you should run this in a thread. Threading does not have to mean looping - it can be a "one and done" thread. Run it that way, and your GUI remains responsive until the thread finishes. I realize that this does not actually answer your question, but it will make your code better.
Also, if you disconnect the internet during the first loop where you're checking for data, I think it will retry 10 times. You should detect, and then quit right away.
Lastly, I don't think you should use EXIT when you've got handles and stuff open. Break instead, so that you still run through the disconnects. I would expect your code to tie up the socket. I saw this recently during a code review when there was an EXIT intead of a BREAK, and it's causing a memory leak because objects are created and never freed. I'd use the same rule here.
Are you sure that you aren't hitting the INTERNET_OPTION_CONNECT_TIMEOUT? It will try to connect first, then receive.
In order to test the connect timeout, it must resolve, but never connect. In order to test the read timeout, it must connect, but never receive any data.
I generally set my connect timeout to 10 seconds, and the read timeout to 30 seconds. Anything longer than that, I consider down anyway.