Delphi send HTTP GET request without wait for the server response - delphi

I want send HTTP request (GET) without wait for the server response.
I have tried with IdHTTP by ignoring the response, eg.:
aIdHTTP := TIdHTTP.create;
try
aIdHTTP.Get('https://stackoverflow.com/');
finally
aIdHTTP.free;
free
but it always wait for the server response...
A trick is close the connection by raising an exception in the TIdHTTP.OnWork event when AWorkMode is wmRead, this close the connection sooner... but wait for the "first byte" response.
There is a way for send data without waiting for the response?

The short answer is NO, well not with TIdHTTP, anyway. HTTP is a protocol, and TIdHTTP follows the protocol, which includes reading the server's response.
As you discovered, you could raise an exception in the TIdHTTP.OnWork event, but that will only work once at least 1 byte is received from the server. You could specify a short TIdHTTP.ReadTimeout value to avoid that, though.
Another option is to close the socket connection in the TIdHTTP.OnHeadersAvailable event, but that requires TIdHTTP reading the server's complete response headers.
IF the response is using HTTP 1.1 chunking, you could enable the hoNoReadChunked flag in the TIdHTTP.HTTPOptions property. But that also requires reading the server's complete response header.
Really, the only way to accomplish what you are asking for is to not use TIdHTTP at all. Use TIdTCPClient instead, and manually send your own HTTP request and then close the connection after sending it, eg:
aIdClient := TIdTCPClient.Create;
try
aIdClient.Host := 'stackoverflow.com';
aIdClient.Port := 443;
aIdClient.IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(aIdClient);
TIdSSLIOHandlerSocketOpenSSL(aIdClient.IOHandler).PassThrough := False;
aIdClient.Connect;
try
aIdClient.IOHandler.WriteLn('GET / HTTP/1.1');
aIdClient.IOHandler.WriteLn('Host: stackoverflow.com');
aIdClient.IOHandler.WriteLn('Connection: close');
aIdClient.IOHandler.WriteLn;
finally
aIdClient.Disconnect;
end;
finally
aIdClient.free;
end;
But what is the point of requesting a file and not actually downloading it?
If you just want to check that the server can be connected to, you don't have to send anything at all
aIdClient := TIdTCPClient.Create;
try
aIdClient.Host := 'stackoverflow.com';
aIdClient.Port := 443;
aIdClient.Connect;
aIdClient.Disconnect;
finally
aIdClient.free;
end;
If you are just trying to avoid downloading a file's actual content over HTTP, you could use TIdHTTP.Head() instead of TIdHTTP.Get():
aIdHTTP.Head('https://stackoverflow.com/');

Related

TIdHTTP.Get timeouts while the same call done with Postman succeeds: possible reasons?

I call a webapi with a Delphi app, in some pcs, the call timeouts, while in other it works fine.
The request done with Postman works fine.
It is a simple custom ping webservice (URL is in Edit1.Text in the code below), in fact the answer is a textual "Pong".
This is the Delphi code of the call:
errormsg := '';
{
old way of setting custom headers
IdHTTP1.Request.CustomHeaders.AddValue('X-HTTP-Method-Override', 'ForwardCommand');
IdHTTP1.Request.CustomHeaders.AddValue('Connection', 'keep-alive');
IdHTTP1.Request.CustomHeaders.AddValue('Accept', '*/*');
IdHTTP1.Request.CustomHeaders.AddValue('User-Agent', 'QualibusSilent');
IdHTTP1.Request.CustomHeaders.AddValue('Content-Type', 'text/plain');
}
//better way of setting custom headers
IdHTTP1.Request.MethodOverride := 'ForwardCommand';
IdHTTP1.Request.Connection := 'keep-alive';
IdHTTP1.Request.UserAgent := 'myCustomUserAgent';
IdHTTP1.Request.ContentType := 'text/plain';
IdHTTP1.Request.Accept := '*/*';
IdSSLIOHandlerSocketOpenSSL1 := TIdSSLIOHandlerSocketOpenSSL.Create(IdHTTP1);
IdSSLIOHandlerSocketOpenSSL1.SSLOptions.Mode := sslmClient;
IdSSLIOHandlerSocketOpenSSL1.SSLOptions.SSLVersions:=
[sslvTLSv1,sslvTLSv1_1,sslvTLSv1_2];
startTime := GetTickCount;
Try
sHTML := IdHTTP1.Get(Edit1.Text);
Except
On E:Exception do
errormsg := e.Message;
End;
EndTime := GetTickCount;
ShowMessage('Time taken: ' +
IntToStr(endTime-startTime)+#13#10+'Error:'+errormsg);
Basically it is a call where instead of GET I do a custom method (ForwardCommand) that I call with X-HTTP-Method-Override.
In the code above I tried to add many headers so that the call is really as the Postman one.
If the call is done directly to the IP address it works, but if I call the https URL it timeouts, there is no evidence of error in the proxy server.
Checking the logs at the webserver side it seems the call is not performed at all.
And this occurs only from some Windows 10 machines, while in the majority of them the call is performed correctly.
Could you please suggest which could be the cause of the error? What should I try to change in the Delphi code to avoid the timeout so that Delphi behaves like Postman?
Thanks.
As stated in comments:
Why when Tidhttp uses proxyParams timeout does not occur and the call succeeds?
...
I finally got the reason for the Postman vs Indy behavior: Proxy. By passing proxy IP and port to TIdHTTP it works, Postman manages to retrieve the system proxy automatically and therefore it works.
There is no "system proxy" on Windows, however there is a proxy in the WinInet API, which is what Internet Explorer (and Edge?) relies on.
In any case, it sounds like the failing PCs don't have direct access to the Internet to begin with, only through a proxy. Indy has no concept of any "system proxy" on any platform, so you will have to assign the proxy settings to TIdHTTP manually, as you have discovered.

TIdHTTPProxyServer raising "Unknown Protocol" RSHTTPUnknownProtocol exception

I'm recoding an old Delphi XE program using Delphi 10.3 Rio. It uses the TIdHTTPProxyServer Indy component listening on 127.0.0.1:80.
with IdHTTPProxyServer.Bindings.Add do begin
IP := '127.0.0.1';
Port := 80;
end;
IdHTTPProxyServer.Active := True;
For testing, I added 127.0.0.1 localtest123.com and 127.0.0.1 www.localtest123.com to the hosts file and disabled the DNS cache service. Then in multiple browers I requested http://localtest123.com/ and http://www.localtest123.com/. With OutputDebugString() I can see the connections accepted but then raises an "Unknown Protocol" error.
I debugged the exception in the TIdHTTPProxyServer.CommandPassThrough procedure in IdHTTPProxyServer.pas. It seems LURI.Protocol is an empty string which is why the RSHTTPUnknownProtocol is raised.
LContext := TIdHTTPProxyServerContext(ASender.Context);
LContext.FCommand := ASender.CommandHandler.Command; //<-'GET'
LContext.FTarget := ASender.Params.Strings[0]; //<-'/'
LContext.FOutboundClient := TIdTCPClient.Create(nil);
try
LURI := TIdURI.Create(LContext.Target); //<-'/'
try
TIdTCPClient(LContext.FOutboundClient).Host := LURI.Host; //<-''
if LURI.Port <> '' then begin //<-''
TIdTCPClient(LContext.FOutboundClient).Port := IndyStrToInt(LURI.Port, 80);
end
else if TextIsSame(LURI.Protocol, 'http') then begin //<-'' {do not localize}
TIdTCPClient(LContext.FOutboundClient).Port := IdPORT_HTTP;
end
else if TextIsSame(LURI.Protocol, 'https') then begin //<-'' {do not localize}
TIdTCPClient(LContext.FOutboundClient).Port := IdPORT_https;
end else begin
raise EIdException.Create(RSHTTPUnknownProtocol);
end;
I'm probably missing something but TIdHTTPProxyServer just works without much code so I have to ask for help on this exception. Thanks in advance!
You can't just redirect the domains in your HOSTS file and expect things to magically work. That is not how proxying works.
You must explicitly configure web browsers to make HTTP requests through an HTTP proxy so that they format the proper requests that a proxy would understand. Sending an HTTP request directly to a target web server is handled differently than sending the same HTTP request through a proxy.
You are getting the exception because the browser requests are not targeting your proxy properly.
For example, when a browser sends an HTTP GET request directly to a target web server, it connects directly to that server and then sends a request that looks something like this:
GET /path HTTP/1.1
Host: server.com
But, when it sends the same request through an HTTP proxy, it connects to the proxy and sends a request that looks more like this instead:
GET http://server.com/path HTTP/1.1
That extra path information in the GET line is missing in your browser requests, because you do not have your browsers configured for proxying, thus the exception when TIdHTTPProxyServer is trying to determine the info it needs to make a connection to the target web server and forward the current request to it.
This is fundamentally to how HTTP works, and how TIdHTTPProxyServer is designed to work.
Things are a bit more complicated when HTTPS is involved, but I'm leaving that detail out for now, as it is not relevant to your question about the exception.
UPDATE: in comments, you say:
In the XE version it never raised an exception when checking for the protocol which would still work today because I manually set the host and port in DoHTTPBeforeCommand.
In that old version, there was no exception raised, because TIdHTTPProxyServer did not check the protocol yet to differentiate between HTTP and HTTPS. You were able to manually fill in missing info when a request was received that was not specifically targeting your proxy. That is why things worked for you before.
In a later version, TIdHTTPProxyServer was updated to differentiate between HTTP and HTTPS when no port is explicitly specified in the request, so a default port is set based on protocol requested. That check happens before DoHTTPBeforeCommand() is called.
To get the old behavior back, you would have to alter TIdHTTPProxyServer's source code to delay the raising of the exception until after DoHTTPBeforeCommand() returns, so you have a chance to fill in missing values again.
If you file a feature request for that, I might consider adding it to Indy's official code.

How do I get Indy requests to show up in Fiddler?

Trying to inspect the header information being posted from my app but fiddler doesn't seem to pick anything up.
I am also using a web service in my app and when I invoke some of the APIs I can see these requests in Fiddler, however, when doing custom requests using Indy 10 nothing seems to be picked up.
Does Indy use WinInet? If not, that's the problem. Fiddler2 inserts itself as a proxy in your internet settings, but programs like SoapUI that use their own communication stack don't use WinInet, and therefore don't (auto-magically) pass through Fiddler2. So you may need to mess with proxy settings.
I use a construction to handle the requests in Fiddler:
try
// lHTTP.IOHandler := lIOHandler; - even without this line works
lHTTP.ProxyParams.ProxyServer := '127.0.0.1';
lHTTP.ProxyParams.ProxyPort := 8888;
sResponse := lHTTP.Post('<URL>', slRequest);
Memo1.Lines.Text := sResponse;
finally
// lIOHandler.Free;
end;

Indy SSL to plain socket pump

I have to provide an SSL front for a plain TCP server, so I'm making a "pump" application that will provide an SSL connection to the outside while the original server can stay plain.
I'm trying to use the Indy components to support my SSL needs, but I can't seem to get any data from the SSL port. I assigned the TIdTCPServer.OnExecute to the following event handler:
procedure TForm1.IdTCPServer1Execute(AContext: TIdContext);
var
c:TIdTCPClient;
cs,ss:TIdIOHandler;
b:TBytes;
begin
c:=TIdTCPClient.Create(Self);
try
c.Host:='127.0.0.1';
c.Port:=60675;
c.ConnectTimeout:=500;
c.Connect;
ss:=c.IOHandler;
cs:=AContext.Connection.IOHandler;
while (cs.Connected) and (ss.Connected) do
begin
if cs.CheckForDataOnSource(1) then
begin
try
cs.ReadBytes(b,1,False);
except on e:Exception do
Memo1.Lines.Add(e.Message); //BAD out of Thread context
end;
if Length(b)>0 then
ss.Write(b);
end;
if ss.CheckForDataOnSource(1) then
begin
ss.ReadBytes(b,1,False);
if Length(b)>0 then
cs.Write(b);
end;
end;
finally
c.Free;
end;
end;
The TCP server has an SSL handler attached. I did the same on a plain HTTP server and it worked fine, so I'm assuming my SSL setup is not the issue.
cs=Client Side (the server socket) and ss=Server side (the client for the TCP server I'm trying to add SSL to).
Now, I know it needs cleanup and doing 1ms waits isn't pretty, but before I can attack that issue, I'd like to receive some data.
Neither of my ReadBytes get called. When I used cs.Readable(), I get true just once, but I still couldn't read.
What can I do to make a pump? Why am I not getting data?
Try using the TIdMappedPortTCP component instead of TIdTCPServer directly. TIdMappedPortTCP handles all the work of passing data back and forth between a client and another server. By default, the outbound connection to the second server is not encrypted, even if the inbound connection to TIdMappedPortTCP is encrypted.

Delphi Indy IdTcpClient read operation returning truncated data for one specific request

This is an interesting problem that I’ve not been able to solve yet.
I am writing a client that communicates across the Internet to a server. I am using the TIdTcpClient Internet Direct (Indy) component in Indy 10 using RAD Studio 2007 native personality.
To get data from the server, I issue an HTTP request using SSL over port 443 where my request details are contained in the HTTP message body. So far, so good. The code works like charm, with one exception.
There is one request that I am submitting that should produce a response of about 336 KB from the server (the HTTP response header contains Content-Length: 344795). The problem is that I am getting only 320KB back. The response, which is in XML, is clearly truncated in the middle of an XML element.
For what it’s worth, the XML is simple text. There are no special characters that can account for the truncation. My TIdTcpClient component is simply reporting that, after receiving the partial response, that the server closed the connection gracefully (which is how every response is expected to be completed, even those that are not truncated, so this is not a problem).
I can make nearly identical calls to the same server where the response is also more than a few K bytes, and all of these work just fine. One request I make returns about 850 KB, another returns about 300 KB, and so on.
In short, I encounter this problem only with the one specific request. All other requests, of which there are many, receive a complete response.
I have talked to the creator of the service, and have supplied examples of my request. He reported that the request is correct. He also told me that when he issues my same request to his server that he gets a complete response.
I’m at a loss. Either the creator of the service is mistaken, and there is actually a problem with the response on that end, or there is something peculiar about my request.
Is there a solution here that I'm missing? Note that I’ve also used a number of other read mechanisms (ReadString, ReadStrings, ReadBytes, etc) and all produce the same result, a truncation of this one specific response at the 320KB mark.
The code is probably not relevant, but I’ll include it anyway. Sorry, but I cannot include the XML request, as it includes proprietary information. (ReadTimeout is set to 20 seconds, but the request returns in about 1 second, so it's not a timeout issue.)
function TClient.GetResponse(PayloadCDS: TClientDataSet): String;
var
s: String;
begin
try
try
s := GetBody(PayloadCDS);
IdTcpClient1.Host := Host;
IdTcpClient1.Port := StrToInt(Port);
IdTcpClient1.ReadTimeout := ReadTimeout;
IdTcpClient1.Connect;
IdTcpClient1.IOHandler.LargeStream := True;
//IdTcpClient1.IOHandler.RecvBufferSize := 2000000;
IdTcpClient1.IOHandler.Write(s);
Result := IdTcpClient1.IOHandler.AllData;
except
on E: EIdConnClosedGracefully do
begin
//eat the exception
end;
on e: Exception do
begin
raise;
end;
end;
finally
if IdTcpClient1.Connected then
IdTcpClient1.Disconnect;
end;
end;
Since you are sending an HTTP request, you should be using the TIdHTTP component instead of the TIdTCPClient component directly. There are a lot of details about the HTTP protocol that TIdHTTP manages for you that you would have to handle manually if you continue using TIdTCPClient directly.
If you are going to continue using TIdTCPClient directly, then you should at least stop using the TIdIOHandler.AllData() method. Extract the 'Content-Length' reply header (you can Capture() the headers into a TStringList or TIdHeaderList and then use its Values[] property) and then pass the actual reported byte count to either TIdIOHandler.ReadString() or TIdIOHandler.ReadStream(). That will help ensure that the reading I/O does not stop reading prematurely because of the server's disconnect at the end of the reply.
As I mentioned in my original question, this connection uses SSL. This requires that you use an IdSSLIOHandlerSocketOpenSSL component, which you assign to the IOHandler property of the IdTcpClient component. All of this was in my original design, and as I mentioned, many of my requests were being responded to correctly.
Over the weekend I discovered another request which was returning an incomplete response. I captured that original HTTP request, and using a tool called SOAP UI, I submitted that request to the server. Using SOAP UI, the server returned a complete answer. Clearly, something was wrong with my client, and not the server.
I finally found the solution by just fiddling around with various properties. The property that finally corrrected the problem was in the SSLOptions property of the IdSSLIOHandlerSocketOpenSSL class. The default value of SSLOptions.Method is sslvSSLv2. When I changed Method to sslvSSLv23, all responses returned complete.
Why I was able to retrieve some responses in full and not all before I made this change is still a mystery to me. Nonetheless, setting IdSSLIOHandlerSocketOpenSSL.SSLOptions.Method to sslvSSLv23 solved my problem.
Thank you Remy Lebeau, for your suggestions.

Resources