I'm trying to do a POST request through proxy using https.
Code looks like:
FHttp := TIdHttp.Create(nil);
FHttp.ProxyParams.ProxyServer := Host;
FHttp.ProxyParams.ProxyPort := Port;
FHttp.ProxyParams.ProxyUsername := User;
FHttp.ProxyParams.ProxyPassword := Password;
FHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
FHandler.SSLOptions.Method := sslvTLSv1_2;
FHandler.PassThrough := true;
FHttp.IOHandler := FHandler;
FHttp.HandleRedirects := true;
FHttp.Request.ContentType := 'application/x-www-form-urlencoded';
FHttp.Request.Connection := 'keep-alive';
FHttp.Request.ProxyConnection := 'keep-alive';
...
FParams.Add('username=user');
FParams.Add('password=pwd');
FHttp.Post('https://my.service/login', FParams);
Proxy server is Squid.
Code generates error "Socket Error # 10054 Connection reset by peer."
Now, the interesting part comes:
If not using proxy at all (i.e. not setting FHttp.ProxyParams settings) - everything is OK.
If not setting any POST parameters (i.e. empty FParams), but still using proxy - everything is OK.
The most strange one: If I'm debugging the Indy code step by step (TIdCustomHTTP.DoRequest method) - everything is OK with above example (proxy settings + parameters).
POST parameters are not sent properly for some reason?
And why step 3 is happening?
Indy is up to date, just pulled from repository
UPDATE
After intercepting TIdHTTP calls (thanks Remy) there is a little bit more clarity. (failing log, working log).
Short version: when doing debug, Indy does 3 CONNECT + POST + DISCONNECT requests (because there are redirection on the service I believe) and it works.
When running test without debug - CONNECT + DISCONNECT + POST - and it fails obviously (i.e. POST is executed without CONNECT in front).
See attached log files for details.
You have found some logic bugs in TIdHTTP that need to be fixed. I have opened a new ticket for that:
#315: Bugs in TIdHTTP proxy handling
Here is what I see happening in your "failing" scenario:
TIdHTTP connects to the proxy, sends a CONNECT request that successfully connects to my.service.com:443, then sends a POST request (using HTTP 1.0 rather than HTTP 1.1 a).
a) to send a POST request with HTTP 1.1, you have to set the TIdHTTP.ProtocolVersion property to pv1_1, AND enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property. Otherwise, TIdHTTP.Post() forces the ProtocolVersion to pv1_0.
The HTTP server replies with a 302 Found response redirecting to a different URL, including a Keep-Alive header indicating the server will close the connection if a new request is not sent in the next 5 seconds.
When TIdHTTP is done processing the POST response, it knows it is going to re-send the same request to a new URL. On the next loop iteration, it sees that the target server is the same, and the proxy is still connected, and so the connection is not closed, and the code that would have sent a new CONNECT request is skipped.
Just before the POST request is sent, the Response.KeepAlive property is checked to know whether or not to close the socket connection anyway. The KeepAlive property getter sees the ProtocolVersion property is pv1_0 and that there is no Proxy-Connection: keep-alive header present in the response (even though there is a Connection: keep-alive header), so it returns False, and then the socket connection is closed.
TIdHTTP then re-connects to the proxy again, but does not send a new CONNECT request before sending the POST request. The proxy does not know what to do with the POST, so it fails the request with a 400 Bad Request response.
Here is what I see happening in your "working" scenario:
Everything is the same as above, up to the point where the 1st POST request is processed. Then there is a delay of roughly 16 seconds (likely since you are stepping through code) - more than the 5-second Keep-Alive delay allows - so the HTTP server closes its connection with the proxy, which then closes its connection to TIdHTTP.
By the time TIdHTTP is ready to send the 2nd POST request, it knows it has been disconnected from the proxy, so it re-connects to the proxy, sends a new CONNECT request, and then sends the POST request.
Until I can fix the bugs properly, try the following:
enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property to allow TIdHTTP.Post() to use HTTP 1.1. That in itself may fix the issue with the connection being closed unnecessarily before sending the 2nd POST request to the redirected URL.
if that doesn't solve the issue, try editing IdHTTP.pas yourself and recompile Indy, to update the TIdCustomHTTP.ConnectToHost() method to force a Disconnect() if the Response.KeepAlive property is False BEFORE the local LUseConnectVerb variable is set to not Connected in the case where ARequest.UseProxy is ctSSLProxy (and ctProxy, too). That way, the 2nd POST request will disconnect from the proxy and re-connect with a new CONNECT request.
Related
I have a server that handle both http and https request.
The common routine between client and server is based on two request:
1) The user request a file with an url, eg.:
https://example.com/get?id=10
2) The server makes some check and redirect in the response headers to a new url:
location: https://example.com/files/10.zip
3) The browser, after redirect to the new url, request the file:
https://example.com/files/10.zip
4) The server, sends the file "10.zip" in the response stream:
ResponseInfo.ContentStream := TFileStream.Create('C:\10.zip', fmOpenRead or fmShareDenyNone);
all routine is of two request (1, 4), and for this reason i have disabled session and keepAlive:
IdHTTPServer.KeepAlive := False;
IdHTTPServer.AutoStartSession := False;
IdHTTPServer.SessionState := False;
I think keep alive has huge benefit only on multiple round trip, this seem a good idea into my scenario when user use http, but, maybe this isn't a good idea when the user use https, because each request without KeepAlive is a new request and the initial setup costs of a HTTPS connection are far higher.
Enabling KeepAlive on IdHTTPServer has some performance benefit into my scenario?
Apache 2.4 has default KeepAliveTimeout at 5 seconds. For IndyServer is good a value in order of some seconds?
I am using the TIdHTTP component and it's GET function.
The GET function sends a complete request, which is fine.
However I would like to spare/save some traffic from a GET response and only want to receive the Responsecode which is in the first "line" of a HTTP response.
Is there a possibility of disconnecting the connection in order to save traffic from any further content?
As mentioned, I only need the responsecode from a website.
I alternatively thought about using Indy's TCP component (with SSL IOHandler) and craft an own HTTP Request Header and then receive the responsecode and disconnect on success - but I don't know how to do that.
TIdHTTP has an OnHeadersAvailable event that is intended for this very task. It is triggered after the response headers have been read and before the body content is read, if any. It has a VContinue output parameter that you can set to False to cancel any further reading.
Update: Something I just discovered: When setting VContinue=False in the OnHeadersAvailable event, TIdHTTP will set Response.KeepAlive=False and skip reading the response body (OK so far), but after the response is done being processed, TIdHTTP checks the KeepAlive value, and the property getter returns True if the socket hasn't been closed on the server's end (HTTP 1.1 uses keep-alives by default). This causes TIdHTTP to not close its end of the socket, and will leave any response body unread. If you then re-use the same TIdHTTP object for a new HTTP request, it will end up processing any unread body data from the previous response before it sees thee response headers of the new request.
You can work around this issue by setting the Request.Connection property to 'close' before calling TIdHTTP.Get(). That tells the server to close its end of the socket connection after sending the response (although, I just found that when requesting an HTTPS url, especially after an HTTP request directs to HTTPS, TIdHTTP clears the Request.Connection value!). Or, simply call TIdHTTP.Disconnect() after TIdHTTP.Get() exits.
I have now updated TIdHTTP to:
no longer clear the Request.Connection when preparing an HTTPS request.
close its end of the socket connection if either:
OnHeadersAvailable returns VContinue=False
the Request.Connection property (or, if connected to a proxy, the Request.ProxyConnection property) has been set to 'close', regardless of the server's response.
Usually you would use TIdHttp.Head, because HEAD requests are intended for doing just that.
If the server does not accept HEAD requests like in OP's case, you can assign the OnWorkBegin event of your TIdHttp instance, and call TIdHttp(Sender).Disconnect; there. This immediately closes the connection, the download does not continue, but you still have the meta data like response code, content length etc.
We are doing POST requests to a web service.
Works fine.
However, we notice that the requests are always HTTP 1.0, which causes our web server to decline to gzip the responses.
If the requests are HTTP 1.1, then the responses are gzipped.
How do we properly ask Indy 10 to issue HTTP 1.1 POST requests?
Thank you!
Include the hoKeepOrigProtocol option into the HTTPOptions property set (set it to True). Except that keep the ProtocolVersion property set to pv1_1 (which is the default value).
In the TIdCustomHTTP.Post method code there's a comment explaining the current behavior:
Currently when issuing a POST, IdHTTP will automatically set the
protocol to version 1.0 independently of the value it had initially.
This is because there are some servers that don't respect the RFC to
the full extent. In particular, they don't respect sending/not sending
the Expect: 100-Continue header. Until we find an optimum solution
that does NOT break the RFC, we will restrict POSTS to version 1.0.
A few lines below is the change to the version 1.0 with the following comment:
// If hoKeepOrigProtocol is SET, it is possible to assume that the developer
// is sure in operations of the server
if not (hoKeepOrigProtocol in FOptions) then begin
if Connected then begin
Disconnect;
end;
FProtocolVersion := pv1_0;
end;
And the above code is skipped (the version is not changed) if you have the hoKeepOrigProtocol option included in the HTTPOptions.
just need to write:
idHTTP.HTTPOptions:= [hoKeepOrigProtocol];
In Delphi, I'm using Indy's TIdHTTPWebBrokerBridge coupled with TIdHTTP to send/receive data via HTTP. On the Server, I don't have any fancy handling, I always just respond with a simple content stream. If there's any issues, I only return information about that issue in the response content (such as authentication failed, invalid request, etc.). So, on the client side, can I assume that every successful request I make to this server will always have a response code of 200 (OK)?
I'm wondering because on the client, the requests are wrapped inside functions which return just a boolean for the success of the request.
Inside this function:
IdHTTP.Get(SomeURL, AStream);
Result:= IdHTTP.ResponseCode = 200;
This function handles any and every request which could possibly fetch data. If there were any issues in the request, This function should return False. In my scenario, since I always return some sort of content on the server, would the client always receive a response code of 200 in this function?
I guess the real question is, if I always return some sort of content and handle all exceptions on the server, then will the server always return status code of 200 to each request?
"Does every successful HTTP request always return status code 200?"
See w3.org: HTTP/1.1 Status Code Definitions (RFC 2616)
The answer is No. All 2xx are considered successful.
That may depend on the HTTP method used.
Should your web-server application always return 200 upon success? That may as well depend on the request method and the signal it intends for the client . e.g.
for PUT method (emphasis is mine):
If an existing resource is modified, either the 200 (OK) or 204 (No
Content) response codes SHOULD be sent to indicate successful
completion of the request.
for POST method:
The action performed by the POST method might not result in a resource
that can be identified by a URI. In this case, either 200 (OK) or 204
(No Content) is the appropriate response status, depending on whether
or not the response includes an entity that describes the result.
If a resource has been created on the origin server, the response
SHOULD be 201 (Created) and contain an entity which describes the
status of the request and refers to the new resource, and a Location
header (see section 14.30). Responses to this method are not
cacheable, unless the response includes appropriate Cache-Control or
Expires header fields. However, the 303 (See Other) response can be
used to direct the user agent to retrieve a cacheable resource.
As you can learn from the RCF, every method SHOULD have it's own success status codes, depending on the implementation.
Your other question:
"can I assume that every successful request I make to this server will always have a response code of 200 (OK)?"
You can always expect Status code 200, if your web server always responds with Status 200. Your web server application controls what response it returns to the client.
That said, Status code 200 is the Standard response for successful HTTP requests (The actual response will depend on the request method used), and in the real world of web servers, SHOULD be set as default upon successful request, unless told otherwise (As explained in Remy's answer).
To answer your specific question:
can I assume that every successful request I make to this server will always have a response code of 200 (OK)?
The answer is Yes, because TIdHTTPWebBrokerBridge wraps TIdHTTPServer, which always sets the default response code to 200 for every request, unless you overwrite it with a different value yourself, or have your server do something that implicitly replies with a different response code (like Redirect() which uses 302, or SmartServeFile() which uses 304), or encounter an error that causes TIdHTTPServer to assign a 4xx or 5xx error response code.
However, in general, what others have told you is true. On the client side, you should handle any possible HTTP success response code, not just 200 by itself. Don't make any assumptions about the server implementation.
In fact, TIdHTTP already handles that for you. If TIdHTTP encounters a response code that it considers to be an error code, it will raise an EIdHTTPProtocolException exception into your code. So if you don't get an exception, assume the response is successful. You don't need to check the response code manually.
If there is a particular response code that normally raises an exception but you do not want it to, you can specify that value in the optional AIgnoreReplies parameter of TIdHTTP.Get() or TIdHTTP.DoRequest(). Or, if you are are using an up-to-date Indy 10 SVN revision, a new hoNoProtocolErrorException flag was recently added to the TIdHTTP.HTTPOptions property so the EIdHTTPProtocolException exception is not raised for any response code.
Successful resposes are 2xx List_of_HTTP_status_codes
i did the following. Process straight all 200`s and LOG exceptions. worked, not a single non 200 - except unauthorized and timeouts (password or sometimes unavaliable server). but many/all responses will be considered for a wide range of mainstream apps.
while (iRedo < 3) do begin
s := Self.HTTPComponent.Get( sUrl );
if self.HTTPComponent.ResponseCode = 200 then begin
break;
end;
// IDEIA - log what happend if not 200
logWhatHappend( s, HTTPComponent ); // then log content, headers, etc
inc( iRedo ); sleep( 5 );
end;
I'm implementing a REST API using ASP.NET MVC, and a little stumbling block has come up in the form of the Expect: 100-continue request header for requests with a post body.
RFC 2616 states that:
Upon receiving a request which
includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
This sounds to me like I need to make two responses to the request, i.e. it needs to immediately send a HTTP 100 Continue response, and then continue reading from the original request stream (i.e. HttpContext.Request.InputStream) without ending the request, and then finally sending the resultant status code (for the sake of argument, lets say it's a 204 No Content result).
So, questions are:
Am I reading the specification right, that I need to make two responses to a request?
How can this be done in ASP.NET MVC?
w.r.t. (2) I have tried using the following code before proceeding to read the input stream...
HttpContext.Response.StatusCode = 100;
HttpContext.Response.Flush();
HttpContext.Response.Clear();
...but when I try to set the final 204 status code I get the error:
System.Web.HttpException: Server cannot set status after HTTP headers have been sent.
The .NET framework by default always sends the expect: 100-continue header for every HTTP 1.1 post. This behavior can be programmatically controlled per request via the System.Net.ServicePoint.Expect100Continue property like so:
HttpWebRequest httpReq = GetHttpWebRequestForPost();
httpReq.ServicePoint.Expect100Continue = false;
It can also be globally controlled programmatically:
System.Net.ServicePointManager.Expect100Continue = false;
...or globally through configuration:
<system.net>
<settings>
<servicePointManager expect100Continue="false"/>
</settings>
</system.net>
Thank you Lance Olson and Phil Haack for this info.
100-continue should be handled by IIS. Is there a reason why you want to do this explicitly?
IIS handles the 100.
That said, no it's not two responses. In HTTP, when the Expect: 100-continue comes in as part of the message headers, the client should be waiting until it receives the response before sending the content.
Because of the way asp.net is architected, you have little control over the output stream. Any data that gets written to the stream is automatically put in a 200 response with chunked encoding whenever you flush, be it that you're in buffered mode or not.
Sadly all this stuff is hidden away in internal methods all over the place, and the result is that if you rely on asp.net, as does MVC, you're pretty much unable to bypass it.
Wait till you try and access the input stream in a non-buffered way. A whole load of pain.
Seb