I have a server that handle both http and https request.
The common routine between client and server is based on two request:
1) The user request a file with an url, eg.:
https://example.com/get?id=10
2) The server makes some check and redirect in the response headers to a new url:
location: https://example.com/files/10.zip
3) The browser, after redirect to the new url, request the file:
https://example.com/files/10.zip
4) The server, sends the file "10.zip" in the response stream:
ResponseInfo.ContentStream := TFileStream.Create('C:\10.zip', fmOpenRead or fmShareDenyNone);
all routine is of two request (1, 4), and for this reason i have disabled session and keepAlive:
IdHTTPServer.KeepAlive := False;
IdHTTPServer.AutoStartSession := False;
IdHTTPServer.SessionState := False;
I think keep alive has huge benefit only on multiple round trip, this seem a good idea into my scenario when user use http, but, maybe this isn't a good idea when the user use https, because each request without KeepAlive is a new request and the initial setup costs of a HTTPS connection are far higher.
Enabling KeepAlive on IdHTTPServer has some performance benefit into my scenario?
Apache 2.4 has default KeepAliveTimeout at 5 seconds. For IndyServer is good a value in order of some seconds?
Related
I'm trying to do a POST request through proxy using https.
Code looks like:
FHttp := TIdHttp.Create(nil);
FHttp.ProxyParams.ProxyServer := Host;
FHttp.ProxyParams.ProxyPort := Port;
FHttp.ProxyParams.ProxyUsername := User;
FHttp.ProxyParams.ProxyPassword := Password;
FHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
FHandler.SSLOptions.Method := sslvTLSv1_2;
FHandler.PassThrough := true;
FHttp.IOHandler := FHandler;
FHttp.HandleRedirects := true;
FHttp.Request.ContentType := 'application/x-www-form-urlencoded';
FHttp.Request.Connection := 'keep-alive';
FHttp.Request.ProxyConnection := 'keep-alive';
...
FParams.Add('username=user');
FParams.Add('password=pwd');
FHttp.Post('https://my.service/login', FParams);
Proxy server is Squid.
Code generates error "Socket Error # 10054 Connection reset by peer."
Now, the interesting part comes:
If not using proxy at all (i.e. not setting FHttp.ProxyParams settings) - everything is OK.
If not setting any POST parameters (i.e. empty FParams), but still using proxy - everything is OK.
The most strange one: If I'm debugging the Indy code step by step (TIdCustomHTTP.DoRequest method) - everything is OK with above example (proxy settings + parameters).
POST parameters are not sent properly for some reason?
And why step 3 is happening?
Indy is up to date, just pulled from repository
UPDATE
After intercepting TIdHTTP calls (thanks Remy) there is a little bit more clarity. (failing log, working log).
Short version: when doing debug, Indy does 3 CONNECT + POST + DISCONNECT requests (because there are redirection on the service I believe) and it works.
When running test without debug - CONNECT + DISCONNECT + POST - and it fails obviously (i.e. POST is executed without CONNECT in front).
See attached log files for details.
You have found some logic bugs in TIdHTTP that need to be fixed. I have opened a new ticket for that:
#315: Bugs in TIdHTTP proxy handling
Here is what I see happening in your "failing" scenario:
TIdHTTP connects to the proxy, sends a CONNECT request that successfully connects to my.service.com:443, then sends a POST request (using HTTP 1.0 rather than HTTP 1.1 a).
a) to send a POST request with HTTP 1.1, you have to set the TIdHTTP.ProtocolVersion property to pv1_1, AND enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property. Otherwise, TIdHTTP.Post() forces the ProtocolVersion to pv1_0.
The HTTP server replies with a 302 Found response redirecting to a different URL, including a Keep-Alive header indicating the server will close the connection if a new request is not sent in the next 5 seconds.
When TIdHTTP is done processing the POST response, it knows it is going to re-send the same request to a new URL. On the next loop iteration, it sees that the target server is the same, and the proxy is still connected, and so the connection is not closed, and the code that would have sent a new CONNECT request is skipped.
Just before the POST request is sent, the Response.KeepAlive property is checked to know whether or not to close the socket connection anyway. The KeepAlive property getter sees the ProtocolVersion property is pv1_0 and that there is no Proxy-Connection: keep-alive header present in the response (even though there is a Connection: keep-alive header), so it returns False, and then the socket connection is closed.
TIdHTTP then re-connects to the proxy again, but does not send a new CONNECT request before sending the POST request. The proxy does not know what to do with the POST, so it fails the request with a 400 Bad Request response.
Here is what I see happening in your "working" scenario:
Everything is the same as above, up to the point where the 1st POST request is processed. Then there is a delay of roughly 16 seconds (likely since you are stepping through code) - more than the 5-second Keep-Alive delay allows - so the HTTP server closes its connection with the proxy, which then closes its connection to TIdHTTP.
By the time TIdHTTP is ready to send the 2nd POST request, it knows it has been disconnected from the proxy, so it re-connects to the proxy, sends a new CONNECT request, and then sends the POST request.
Until I can fix the bugs properly, try the following:
enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property to allow TIdHTTP.Post() to use HTTP 1.1. That in itself may fix the issue with the connection being closed unnecessarily before sending the 2nd POST request to the redirected URL.
if that doesn't solve the issue, try editing IdHTTP.pas yourself and recompile Indy, to update the TIdCustomHTTP.ConnectToHost() method to force a Disconnect() if the Response.KeepAlive property is False BEFORE the local LUseConnectVerb variable is set to not Connected in the case where ARequest.UseProxy is ctSSLProxy (and ctProxy, too). That way, the 2nd POST request will disconnect from the proxy and re-connect with a new CONNECT request.
I am using the TIdHTTP component and it's GET function.
The GET function sends a complete request, which is fine.
However I would like to spare/save some traffic from a GET response and only want to receive the Responsecode which is in the first "line" of a HTTP response.
Is there a possibility of disconnecting the connection in order to save traffic from any further content?
As mentioned, I only need the responsecode from a website.
I alternatively thought about using Indy's TCP component (with SSL IOHandler) and craft an own HTTP Request Header and then receive the responsecode and disconnect on success - but I don't know how to do that.
TIdHTTP has an OnHeadersAvailable event that is intended for this very task. It is triggered after the response headers have been read and before the body content is read, if any. It has a VContinue output parameter that you can set to False to cancel any further reading.
Update: Something I just discovered: When setting VContinue=False in the OnHeadersAvailable event, TIdHTTP will set Response.KeepAlive=False and skip reading the response body (OK so far), but after the response is done being processed, TIdHTTP checks the KeepAlive value, and the property getter returns True if the socket hasn't been closed on the server's end (HTTP 1.1 uses keep-alives by default). This causes TIdHTTP to not close its end of the socket, and will leave any response body unread. If you then re-use the same TIdHTTP object for a new HTTP request, it will end up processing any unread body data from the previous response before it sees thee response headers of the new request.
You can work around this issue by setting the Request.Connection property to 'close' before calling TIdHTTP.Get(). That tells the server to close its end of the socket connection after sending the response (although, I just found that when requesting an HTTPS url, especially after an HTTP request directs to HTTPS, TIdHTTP clears the Request.Connection value!). Or, simply call TIdHTTP.Disconnect() after TIdHTTP.Get() exits.
I have now updated TIdHTTP to:
no longer clear the Request.Connection when preparing an HTTPS request.
close its end of the socket connection if either:
OnHeadersAvailable returns VContinue=False
the Request.Connection property (or, if connected to a proxy, the Request.ProxyConnection property) has been set to 'close', regardless of the server's response.
Usually you would use TIdHttp.Head, because HEAD requests are intended for doing just that.
If the server does not accept HEAD requests like in OP's case, you can assign the OnWorkBegin event of your TIdHttp instance, and call TIdHttp(Sender).Disconnect; there. This immediately closes the connection, the download does not continue, but you still have the meta data like response code, content length etc.
We are doing POST requests to a web service.
Works fine.
However, we notice that the requests are always HTTP 1.0, which causes our web server to decline to gzip the responses.
If the requests are HTTP 1.1, then the responses are gzipped.
How do we properly ask Indy 10 to issue HTTP 1.1 POST requests?
Thank you!
Include the hoKeepOrigProtocol option into the HTTPOptions property set (set it to True). Except that keep the ProtocolVersion property set to pv1_1 (which is the default value).
In the TIdCustomHTTP.Post method code there's a comment explaining the current behavior:
Currently when issuing a POST, IdHTTP will automatically set the
protocol to version 1.0 independently of the value it had initially.
This is because there are some servers that don't respect the RFC to
the full extent. In particular, they don't respect sending/not sending
the Expect: 100-Continue header. Until we find an optimum solution
that does NOT break the RFC, we will restrict POSTS to version 1.0.
A few lines below is the change to the version 1.0 with the following comment:
// If hoKeepOrigProtocol is SET, it is possible to assume that the developer
// is sure in operations of the server
if not (hoKeepOrigProtocol in FOptions) then begin
if Connected then begin
Disconnect;
end;
FProtocolVersion := pv1_0;
end;
And the above code is skipped (the version is not changed) if you have the hoKeepOrigProtocol option included in the HTTPOptions.
just need to write:
idHTTP.HTTPOptions:= [hoKeepOrigProtocol];
I'm new to this delphi datasnap stuff and there are barely any useful documentations about it.
I've setup a simple rest server by following this guide. Now I want to define my own authentication protocol.
As far as I know. All requests came from remote clients will be processed through WebModule methods before invoking server class methods. So far, I was able to capture the client IP adress and the requesting URL path like this:
procedure TWebModule1.WebModuleBeforeDispatch(Sender: TObject;
Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
begin
ShowMessage(request.RemoteAddr); // adress where request came from
ShowMessage(request.PathInfo); // url path
Response.SetCustomHeader('access-control-allow-origin','*');
if FServerFunctionInvokerAction <> nil then
FServerFunctionInvokerAction.Enabled := AllowServerFunctionInvoker;
end;
How am I able to:
Reject requests depend on specific IP addresses and return 401 status code to clients before they reach the class methods?
Redirect client to another methods? For example: Redirect this resquest:
request = new XMLHttpRequest();
request.open('GET', 'http://localhost:8080/datasnap/rest/TServerMethods1/ReverseString/blah', true);
To another method instead of ReverseString.
Complex redirection scenarios can be handled with an Apache HTTP server as reverse proxy with the mod_rewrite module.
For example, this answer shows that client IP addresses can be used to trigger a redirection.
https://stackoverflow.com/a/1073877/80901
RewriteCond %{REMOTE_ADDR} !^10\.0\.1\.1$
RewriteRule ^ /maintenance.html
this would send all requests with some exception to a maintenance page.
Using Apache (or a different HTTP proxy server) also allows to keep the DataSnap server running continuously, while Apache configuration changes are applied. This will remove your server down time virtually to zero.
the question is extended from the question which I asked before.
Currently I can let my proxy server with TIdHTTPProxyServer forward the request to another proxy server which also using TIdHTTPProxyServer. But when I tried to let it forward the request to other proxy servers from the free proxy servers list on web. Most servers which can be used through IE proxy setting return 403 error message to my proxy server. In fact I have tried about 20 valid proxy servers, but only two can accept the requests from my proxy server. I can not figure out why it happens.
Below is the code I use in HTTPBeforeCommand of TIdHTTPProxyServer.
TIdIOHandlerStack* tempIO = new TIdIOHandlerStack(AContext->OutboundClient);
TIdConnectThroughHttpProxy* tempProxy = new TIdConnectThroughHttpProxy(AContext->OutboundClient);
tempProxy->Enabled = true;
tempProxy->Host = ProxyServerAddress;
tempProxy->Port = ProxyServerPort ;
tempIO->TransparentProxy = tempProxy;
AContext->OutboundClient->IOHandler = tempIO;
After monitoring the behaviors of TIdHTTPProxyServer using wireshark, I found TIdHTTPProxyServer always send a CONNECT request to other proxy servers at first berfore foward the real requests(the browser does not do that).
And then receive 403 response for most proxy servers. But still do not know how to make it works.
Updated on 2012/08/07
Hi, I am not really familiar with those HTTP stuffs, so I just record what I saw in wireshark here. It seems IE uses GET/POST commands for HTTP requests and CONNECT command for HTTPS requests. And most proxy servers block Connect commands when they are not HTTPS request (For example, CONNECT www.google.com.tw:80 HTTP/1.0). That is why TIdConnectThroughHttpProxy always does not work.
Below is my workaround, I made a little bit changes in IdHTTPProxyServer.pas. Hope it is useful for someone else who meets the same problems.
For CONNECT commands, still use TIdConnectThroughHttpProxy
Within TIdHTTPProxyServer.CommandCONNECT
if UseProxy = True then
begin
tempIO := TIdIOHandlerStack.Create(LContext.FOutboundClient);
tempProxy := TIdConnectThroughHttpProxy.Create(LContext.FOutboundClient);
tempProxy.Enabled := True;
tempProxy.Host := UseProxyAddr;
tempProxy.Port := UseProxyPort ;
tempIO.TransparentProxy := tempProxy;
LContext.FOutboundClient.IOHandler := tempIO;
end;
For GET/POST commands, I should directly send GET www.google.com.tw:80 HTTP/1.0 to other proxy servers instead of sending CONNECT www.google.com.tw:80 HTTP/1.0 at first.
Within TIdHTTPProxyServer.CommandPassThrough
if UseProxy = True then
begin
TIdTCPClient(LContext.FOutboundClient).Host := UseProxyAddr;
TIdTCPClient(LContext.FOutboundClient).Port := UseProxyPort;
end else
begin
TIdTCPClient(LContext.FOutboundClient).Host := LURI.Host;
TIdTCPClient(LContext.FOutboundClient).Port := IndyStrToInt(LURI.Port, 80);
end;
Also within TIdHTTPProxyServer.CommandPassThrough, I should let header Proxy-Connection = close
LContext.Connection.IOHandler.Capture(LContext.Headers, '', False);
LContext.Headers.Values['Proxy-Connection'] := 'close';
Finally within TIdHTTPProxyServer.TransferData
if AContext.TransferSource = tsClient then
begin
if UseProxy = True then
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Target + ' HTTP/1.0');
end else
begin
ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Document + ' HTTP/1.0');
end;
end;
It is my first time to implement delphi, and hope there are better solutions.
I believe you should not use pin-holing - http://www.indyproject.org/docsite/html/TIdConnectThroughHttpProxy.html
CONNECT command is not how WWW works. No browser use it. It is how non-WWW programs try to break-through the firewalls and open direct access to all areas of internet beyond WWW.
Don't use "transparent proxy" classes.
Use regular HTTP proxy, like in How to download a file over HTTPS using Indy 10 and OpenSSL?
BTW, there is no such event handler as u name "HTTPBeforeCommand"
http://www.indyproject.org/docsite/html/!!MEMBEROVERVIEW_TIdHTTPProxyServer.html