When Indy get a 404, I cannot get the response content, its always empty.
For example, using Indy to access https://www.etsy.com/people/car, I get a 404 and contentstream (or assigned stream) is just empty.
Doing the same with THttpClient, I also get a 404, but also get the content of the 404 page in the response stream.
So my question is, how can I get the response content with Indy when its a 404?
using Indy to access https://www.etsy.com/people/car, I get a 404 and contentstream (or assigned stream) is just empty.
This is the default behavior, by design, to prevent corrupting the output with unexpected error data that was not requested.
how can I get the response content with Indy when its a 404?
There are two possible ways:
By default, TIdHTTP raises an EIdHTTPProtocolException exception on an HTTP error. The response data will be in its ErrorMessage property.
You can avoid the raised exception by either:
enabling the hoNoProtocolErrorException flag in the TIdHTTP.HTTPOptions property.
using the overloaded version of TIdHTTP.Get() that has an AIgnoreReplies parameter, passing 404 (or any other error response code you want) in that parameter.
Either way, by default the response data will still be discarded. You can avoid that by enabling the recently added hoWantProtocolErrorContent flag in the TIdHTTP.HTTPOptions property (see New TIdHTTP flags and OnChunkReceived event).
Related
I am using the TIdHTTP component and it's GET function.
The GET function sends a complete request, which is fine.
However I would like to spare/save some traffic from a GET response and only want to receive the Responsecode which is in the first "line" of a HTTP response.
Is there a possibility of disconnecting the connection in order to save traffic from any further content?
As mentioned, I only need the responsecode from a website.
I alternatively thought about using Indy's TCP component (with SSL IOHandler) and craft an own HTTP Request Header and then receive the responsecode and disconnect on success - but I don't know how to do that.
TIdHTTP has an OnHeadersAvailable event that is intended for this very task. It is triggered after the response headers have been read and before the body content is read, if any. It has a VContinue output parameter that you can set to False to cancel any further reading.
Update: Something I just discovered: When setting VContinue=False in the OnHeadersAvailable event, TIdHTTP will set Response.KeepAlive=False and skip reading the response body (OK so far), but after the response is done being processed, TIdHTTP checks the KeepAlive value, and the property getter returns True if the socket hasn't been closed on the server's end (HTTP 1.1 uses keep-alives by default). This causes TIdHTTP to not close its end of the socket, and will leave any response body unread. If you then re-use the same TIdHTTP object for a new HTTP request, it will end up processing any unread body data from the previous response before it sees thee response headers of the new request.
You can work around this issue by setting the Request.Connection property to 'close' before calling TIdHTTP.Get(). That tells the server to close its end of the socket connection after sending the response (although, I just found that when requesting an HTTPS url, especially after an HTTP request directs to HTTPS, TIdHTTP clears the Request.Connection value!). Or, simply call TIdHTTP.Disconnect() after TIdHTTP.Get() exits.
I have now updated TIdHTTP to:
no longer clear the Request.Connection when preparing an HTTPS request.
close its end of the socket connection if either:
OnHeadersAvailable returns VContinue=False
the Request.Connection property (or, if connected to a proxy, the Request.ProxyConnection property) has been set to 'close', regardless of the server's response.
Usually you would use TIdHttp.Head, because HEAD requests are intended for doing just that.
If the server does not accept HEAD requests like in OP's case, you can assign the OnWorkBegin event of your TIdHttp instance, and call TIdHttp(Sender).Disconnect; there. This immediately closes the connection, the download does not continue, but you still have the meta data like response code, content length etc.
I am using Delphi 7 and Indy 9 to implement a trivial HTTP server. When I upload a file to the server using the POST method, I would like to know the content-length before the OnCommandGet event, in order to show a progress bar.
The event sequence is the following:
On the OnPostStream I create a file stream and hook the OnWork event. (here I need the content-length to set the progress bar)
The POST body is uploaded and the OnWork fires with the transferred bytes.
The OnCommandGet even fires and I get the TIdHTTPRequestInfo object wuth the correct content length, but at this point it is too late.
I modified the Indy sources to get a copy of the TIdHTTPRequestInfo in the ASender.Data field passed to the OnPostStream handler.
Everything works, but I would like not to touch Indy.
Is there a cleaner solution?
Thank you
Is there a cleaner solution?
Not for Indy 9, no. Modifying Indy's source code to expose access to the request object (or at least the request headers) is the only option.
In Indy 10, TIdHTTPServer actually provides 2 ways that you can get the Content-Length value before the POST data is read - there is a new OnHeadersAvailable event that has an AHeaders parameter, and the OnCreatePostStream event has a new AHeaders parameter as well (so different stream classes can be used depending on the type of request being made). You can grab the Content-Length value from the provided TIdHeaderList object in either event and store the value in ASender.Data as needed for later use.
HOWEVER - do keep in mind that TIdHTTPServer in Indy 10 supports HTTP 1.1, and HTTP 1.1 servers are required to allow HTTP 1.1 clients to post data using the chunked transfer encoding. In which case, there will be no Content-Length header present (or it will be 0), and thus the data length will not be known until the final chunk has been received. Fortunately, you can look at the Transfer-Encoding header in the OnHeadersAvailable/OnCreatePostStream event to detect that condition and tailor your progress events accordingly.
In Delphi, I'm using Indy's TIdHTTPWebBrokerBridge coupled with TIdHTTP to send/receive data via HTTP. On the Server, I don't have any fancy handling, I always just respond with a simple content stream. If there's any issues, I only return information about that issue in the response content (such as authentication failed, invalid request, etc.). So, on the client side, can I assume that every successful request I make to this server will always have a response code of 200 (OK)?
I'm wondering because on the client, the requests are wrapped inside functions which return just a boolean for the success of the request.
Inside this function:
IdHTTP.Get(SomeURL, AStream);
Result:= IdHTTP.ResponseCode = 200;
This function handles any and every request which could possibly fetch data. If there were any issues in the request, This function should return False. In my scenario, since I always return some sort of content on the server, would the client always receive a response code of 200 in this function?
I guess the real question is, if I always return some sort of content and handle all exceptions on the server, then will the server always return status code of 200 to each request?
"Does every successful HTTP request always return status code 200?"
See w3.org: HTTP/1.1 Status Code Definitions (RFC 2616)
The answer is No. All 2xx are considered successful.
That may depend on the HTTP method used.
Should your web-server application always return 200 upon success? That may as well depend on the request method and the signal it intends for the client . e.g.
for PUT method (emphasis is mine):
If an existing resource is modified, either the 200 (OK) or 204 (No
Content) response codes SHOULD be sent to indicate successful
completion of the request.
for POST method:
The action performed by the POST method might not result in a resource
that can be identified by a URI. In this case, either 200 (OK) or 204
(No Content) is the appropriate response status, depending on whether
or not the response includes an entity that describes the result.
If a resource has been created on the origin server, the response
SHOULD be 201 (Created) and contain an entity which describes the
status of the request and refers to the new resource, and a Location
header (see section 14.30). Responses to this method are not
cacheable, unless the response includes appropriate Cache-Control or
Expires header fields. However, the 303 (See Other) response can be
used to direct the user agent to retrieve a cacheable resource.
As you can learn from the RCF, every method SHOULD have it's own success status codes, depending on the implementation.
Your other question:
"can I assume that every successful request I make to this server will always have a response code of 200 (OK)?"
You can always expect Status code 200, if your web server always responds with Status 200. Your web server application controls what response it returns to the client.
That said, Status code 200 is the Standard response for successful HTTP requests (The actual response will depend on the request method used), and in the real world of web servers, SHOULD be set as default upon successful request, unless told otherwise (As explained in Remy's answer).
To answer your specific question:
can I assume that every successful request I make to this server will always have a response code of 200 (OK)?
The answer is Yes, because TIdHTTPWebBrokerBridge wraps TIdHTTPServer, which always sets the default response code to 200 for every request, unless you overwrite it with a different value yourself, or have your server do something that implicitly replies with a different response code (like Redirect() which uses 302, or SmartServeFile() which uses 304), or encounter an error that causes TIdHTTPServer to assign a 4xx or 5xx error response code.
However, in general, what others have told you is true. On the client side, you should handle any possible HTTP success response code, not just 200 by itself. Don't make any assumptions about the server implementation.
In fact, TIdHTTP already handles that for you. If TIdHTTP encounters a response code that it considers to be an error code, it will raise an EIdHTTPProtocolException exception into your code. So if you don't get an exception, assume the response is successful. You don't need to check the response code manually.
If there is a particular response code that normally raises an exception but you do not want it to, you can specify that value in the optional AIgnoreReplies parameter of TIdHTTP.Get() or TIdHTTP.DoRequest(). Or, if you are are using an up-to-date Indy 10 SVN revision, a new hoNoProtocolErrorException flag was recently added to the TIdHTTP.HTTPOptions property so the EIdHTTPProtocolException exception is not raised for any response code.
Successful resposes are 2xx List_of_HTTP_status_codes
i did the following. Process straight all 200`s and LOG exceptions. worked, not a single non 200 - except unauthorized and timeouts (password or sometimes unavaliable server). but many/all responses will be considered for a wide range of mainstream apps.
while (iRedo < 3) do begin
s := Self.HTTPComponent.Get( sUrl );
if self.HTTPComponent.ResponseCode = 200 then begin
break;
end;
// IDEIA - log what happend if not 200
logWhatHappend( s, HTTPComponent ); // then log content, headers, etc
inc( iRedo ); sleep( 5 );
end;
How i can determine the size in bytes of a remote file hosted in the web, before download it, using Delphi?
Thanks In advance.
You could use Indy.
First include IdHTTP.
You can retrieve the size this way:
procedure TFormMain.Button1Click(Sender: TObject);
var
Http: TIdHTTP;
begin
Http := TIdHTTP.Create(nil);
try
Http.Head('http://live.sysinternals.com/ADExplorer.exe');
ShowMessage(IntToStr(Http.Response.ContentLength));
finally
Http.Free;
end;
end;
Short answer: use the HTTP HEAD command, available in the TIdHttp component of Indy Delphi.
Details:
HTTP protocol defines a HEAD method.
9.4 HEAD
The HEAD method is identical to GET
except that the server MUST NOT return
a message-body in the response. The
metainformation contained in the HTTP
headers in response to a HEAD request
SHOULD be identical to the information
sent in response to a GET request.
This method can be used for obtaining
metainformation about the entity
implied by the request without
transferring the entity-body itself.
This method is often used for testing
hypertext links for validity,
accessibility, and recent
modification.
The response to a HEAD request MAY be
cacheable in the sense that the
information contained in the response
MAY be used to update a previously
cached entity from that resource. If
the new field values indicate that the
cached entity differs from the current
entity (as would be indicated by a
change in Content-Length, Content-MD5,
ETag or Last-Modified), then the cache
MUST treat the cache entry as stale.
HEAD Asks for the response identical to the one that would correspond to a GET request, but without the response body, retrieving the complete response headers, without the entire content.
The HTTP response headers retrieved are documented in List of HTTP headers on Wikipedia.
http://en.wikipedia.org/wiki/List_of_HTTP_headers
HTTP Headers form the core of an HTTP request,
and are very important in an HTTP response.
They define various characteristics of the data
that is requested or the data that has been provided.
The headers are separated from the request or
response body by a blank line. HTTP headers
can be near-arbitrary strings, but only some
are commonly understood.
One of the headers that is always present for a valid URL to retrieve a content is
the Content-Length header.
14.13 Content-Length
The Content-Length entity-header field indicates the
size of the entity-body, in decimal number of OCTETs,
sent to the recipient or, in the case of the HEAD method,
the size of the entity-body that would have been sent
had the request been a GET.
Content-Length = "Content-Length" ":" 1*DIGIT
An example is
Content-Length: 3495
Applications SHOULD use this field to indicate the
transfer-length of the message-body, unless this is
prohibited by the rules in section 4.4.
Any Content-Length greater than or equal to zero is a
valid value. Section 4.4 describes how to determine
the length of a message-body if a Content-Length is not given.
Note that the meaning of this field is significantly
different from the corresponding definition in MIME,
where it is an optional field used within the
"message/external-body" content-type. In HTTP,
it SHOULD be sent whenever the message's length
can be determined prior to being transferred,
unless this is prohibited by the rules in section 4.4.
From Delphi, drop a TIdHttp component to your form. And paste the following code in one of your delphi event process methods.
var
url: string; // must contain a fully qualified url
contentLength: integer;
begin
....
contentLength:=0;
try
Idhttp1.Head(url);
contentLength:=idhttp1.response.ContentLength;
except end;
....
Be aware that not ALL servers will return a valid content size for a head request. If the content length = 0, then you will ONLY know if you issue a GET request. For example the HEAD request against the Google logo returns a 0 content-length, however a GET returns the proper length, but also retrieves the image. Some servers will return content-length as the length of the packet following the header.
You can use Synapse to get at this information also. Note that the data is transfered, but the buffer is thrown away. This is a much more reliable method, but at the cost of additional bandwidth.
var
HTTP : tHTTPSend;
begin
HTTP := THTTPSend.Create;
try
HTTP.HTTPMethod('GET',url);
DownloadSize := HTTP.DownloadSize;
finally
HTTP.Free;
end;
end;
I'm implementing a REST API using ASP.NET MVC, and a little stumbling block has come up in the form of the Expect: 100-continue request header for requests with a post body.
RFC 2616 states that:
Upon receiving a request which
includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
This sounds to me like I need to make two responses to the request, i.e. it needs to immediately send a HTTP 100 Continue response, and then continue reading from the original request stream (i.e. HttpContext.Request.InputStream) without ending the request, and then finally sending the resultant status code (for the sake of argument, lets say it's a 204 No Content result).
So, questions are:
Am I reading the specification right, that I need to make two responses to a request?
How can this be done in ASP.NET MVC?
w.r.t. (2) I have tried using the following code before proceeding to read the input stream...
HttpContext.Response.StatusCode = 100;
HttpContext.Response.Flush();
HttpContext.Response.Clear();
...but when I try to set the final 204 status code I get the error:
System.Web.HttpException: Server cannot set status after HTTP headers have been sent.
The .NET framework by default always sends the expect: 100-continue header for every HTTP 1.1 post. This behavior can be programmatically controlled per request via the System.Net.ServicePoint.Expect100Continue property like so:
HttpWebRequest httpReq = GetHttpWebRequestForPost();
httpReq.ServicePoint.Expect100Continue = false;
It can also be globally controlled programmatically:
System.Net.ServicePointManager.Expect100Continue = false;
...or globally through configuration:
<system.net>
<settings>
<servicePointManager expect100Continue="false"/>
</settings>
</system.net>
Thank you Lance Olson and Phil Haack for this info.
100-continue should be handled by IIS. Is there a reason why you want to do this explicitly?
IIS handles the 100.
That said, no it's not two responses. In HTTP, when the Expect: 100-continue comes in as part of the message headers, the client should be waiting until it receives the response before sending the content.
Because of the way asp.net is architected, you have little control over the output stream. Any data that gets written to the stream is automatically put in a 200 response with chunked encoding whenever you flush, be it that you're in buffered mode or not.
Sadly all this stuff is hidden away in internal methods all over the place, and the result is that if you rely on asp.net, as does MVC, you're pretty much unable to bypass it.
Wait till you try and access the input stream in a non-buffered way. A whole load of pain.
Seb