I have a function that uploads a file with a TIdHTTP Post request. It uses a TIdMultiPartFormDataStream for the file so is using the function
function TIdCustomHTTP.Post(AURL: string; ASource: TIdMultiPartFormDataStream): string;
This part works correctly, but I'd like to slow the rate it is sending so it uses less bandwidth.
Is there a way to loop a writebytes call or something so I could stick some sleeps in the middle and slow it down? I don't care about the response from the request.
Indy has a TIdInterceptThrottler component for this very purpose. You can assign it to the TIdHTTP.Intercept property, and then set the TIdInterceptThrottler.SendBitsPerSec property as needed.
Related
Is it possible to read the HTTP headers (specifically the GET header) in the Indy TIdHTTPServer.OnConnect event without interfering with the subsequent OnCommandGet event?
If I try to pull them with a loop of ReadLn's then OnCommandGet never fires. I need an advanced peek at them without pulling them from the input buffer.
Is it possible to read the HTTP headers (specifically the GET header) in the Indy TIdHTTPServer.OnConnect event without interfering with the subsequent OnCommandGet event?
It it possible, in that you could use the TIdIOHandler.WaitFor() method to wait for the header terminator to arrive in the TIdIOHandler.InputBuffer, returning everything received before it without removing anything from the buffer, eg:
procedure TMyForm.IdHTTPServer1Connect(AContext: TIdContext);
var
headers: String;
begin
header := AContext.Connection.IOHandler.WaitFor(EOL+EOL, False);
...
end;
However, this has some limitations:
it assumes each line is terminated by the byte sequence $0D $0A and thus the header is terminated by the byte sequence $0D $0A $0D $0A. This is technically true per the HTTP standards, and will usually be the case. However, some clients do terminate lines with just $0A and thus the header would be terminated by $0A $0A. TIdHTTPServer would normally handle that just fine, but using WaitFor() would not.
A more robust solution would involve using TIdIOHandler.CheckForDataOnSource() in a loop, manually scanning the TIdIOHandler.InputBuffer until either $0D $0A $0D $0A or $0A $0A is found in the buffer.
this won't work if there are multiple HTTP requests on the same connection, which can happen if HTTP keep-alives or HTTP pipelining are used. You would be "peeking" the header of only the 1st HTTP request on the connection.
If I try to pull them with a loop of ReadLn's then OnCommandGet never fires.
Correct, because TIdHTTPServer expects to be the one to read them off the InputBuffer. If you read them yourself beforehand, there won't be anything left for TIdHTTPServer to read, so it won't know what each HTTP request even looks like.
I need an advanced peek at them without pulling them from the input buffer.
Why? What do you want to do with them, if you could get them?
You should check if the TIdHTTPServer.OnHeadersAvailable event suits your needs. It is fired at the beginning of every HTTP request, after the headers have been read from the InputBuffer but before the request body has been read.
I got it working by peeking at the Inputbuffer per Remy's suggestion:
procedure TForm1.IdHTTPServer1Connect(AContext: TIdContext);
var
s: string;
Done: boolean;
begin
Done := False;
repeat
Sleep(10);
if AContext.Connection.IOHandler.CheckForDataOnSource then
begin
s := AContext.Connection.IOHandler.InputBuffer.AsString;
if (Pos(#13#10#13#10, s) > 0) or (Pos(#10#10, s) > 0) then Done := True;
end;
until Done;
...
end;
One issue I could see happening is a bot making a TCP connection on my port and that loop never ending since no headers are coming. I would need to add some kind of timeout check.
The other suggestion of using OnHeadersAvailable wouldn't work for me because it's called right before OnCommandGet each time (i.e., multiple times per connection when KeepAlive is True) so I might as well just put tests in OnCommandGet if I went that route.
EDIT:
I also just tried doing this in the OnConnect handler:
s := AContext.Connection.IOHandler.WaitFor(#10, False, True, nil, 1000);
Since I only need the GET line and it will always be first (right?) if it's included at all, I just need to find the first line feed character. That solves the line terminator problem and there's a timeout parameter that solves the bot issue. While this does read the first line header, it also causes an immediate disconnect and CommandGet never gets called. What am I doing wrong?
I have a JSON-RPC service which for one of the requests returns a continuous stream of JSON objects.
I.e. :
{id:'1'}
{id:'2'}
//30 minutes of no data
{id:'3'}
//...
Of course, there's no Content-Length because the stream is endless.
I'm using custom TStream descendant to receive and parse the data. But internally TIdHttp buffers the data and does not pass it to me until RecvBufferSize bytes are received.
This results in:
{id:'1'} //received
{id:'2'} //buffered by Indy but not received
//30 minutes of no data
{id:'3'} //this is where Indy commits {id:'2'} to me
Obviously this won't do because the message which mattered 30 minutes ago should have been delivered 30 minutes ago.
I'd like Indy to do just what sockets do: read up to RecvBufferSize or less if there's data available and return immediately.
I've found this discussion from 2005 where some poor soul tried to explain the problem to Indy developers but they didn't understand him. (Read it; it's a sad sight)
Anyway, he worked around this by writing custom IOHandler descendant, but that was back in 2005, maybe there are some ready solutions today?
Sounds to me like a WebSocket task, since your connection is not plain HTTP question/answer oriented any more, but a stream of content.
See WebSocket server implementations for Delphi for some code.
There is at least one based on Indy, from the author of AsmProfiler.
AFAIK there are two kind of stream in websockets: binary and text. I suspect your JSON stream is some text content, from the websocket point of view.
Another option is to use long-pooling or some older protocols, which are more rooter-friendly - when the connection switch to websockets mode, it is no standard HTTP any more, so some "sensible" packet-inspection tools (on a corporate network) may identify it as a security attack (e.g. DoS), so may stop the connection.
You do not need to write a IOHandler descendant, it is already possible with the TIdTCPClient class. It exposes a TIdIOHandler object, which has methods to read from the socket. These ReadXXX methods block until the requested data has been read or a timeout occurs. As long as the connection exists, ReadXXX can be executed in a loop and whenever it receives a new JSON object, pass it to the application logic.
Your example looks like all JSON objects only have one line. JSON objects however could be multi-lined, in this case the client code needs to know how they are separated.
Update: in a similar Stackoverflow question (for .Net) for a 'streaming' HTTP JSON web service, the most upvoted solution used a lower-level TCP client instead of a HTTP client: Reading data from an open HTTP stream
While using TCP stream was an option, in the end I went with original solution of writing custom TIdIOHandlerStack descendant.
The motivation was that with TIdHTTP I know what doesn't work and only need to fix that, while switching to lower level TCP means new problems can arise.
Here's the code that I'm using, and I'm going to discuss the key points here.
New TIdStreamIoHandler has to inherit from TIdIOHandlerStack.
Two functions need to be rewritten: ReadBytes and ReadStream:
function TryReadBytes(var VBuffer: TIdBytes; AByteCount: Integer;
AAppend: Boolean = True): integer; virtual;
procedure ReadStream(AStream: TStream; AByteCount: TIdStreamSize = -1;
AReadUntilDisconnect: Boolean = False); override;
Both are modified Indy functions which can be found in IdIOHandler.TIdIOHandler. In ReadBytes the while clause has to be replaced with a singe ReadFromSource() request, so that TryReadBytes returns after reading up to AByteCount bytes in one go.
Based on this, ReadStream has to handle all combinations of AByteCount (>0, <0) and ReadUntilDisconnect (true, false) to cyclically read and then write to stream chunks of data arriving from the socket.
Note that ReadStream need not terminate prematurely even in this stream version if only part of the requested data is available in the socket. It just has to write that part to the stream instantly instead of caching it in FInputBuffer, then block and wait for the next part of data.
There is actually a length data right before the content of packet which transferred in chunked encoding transfer mode. Using this length data, IOhandler of idhttp read one packet by one packet to stream. The minimum meaningful unit is a packet, So there should be no need to read characters one by one from a packet and then no need to change the functions of IOHandler. The only problem is idhttp wouldn't stop an turn the stream data to next step because of the endless of stream data: there is no ending packet. So the solution is using idhttp onwork event to trigger a reading from stream and setting the stream position to zero in order to avoid overflow .like this:
//add a event handler to idhttp
IdHTTP.OnWork := IdHTTPWork;
procedure TRatesStreamWorker.IdHTTPWork(ASender: TObject; AWorkMode: TWorkMode; AWorkCount: Int64);
begin
.....
ResponseStringStream.Position :=0;
s:=ResponseStringStream.ReadString(ResponseStringStream.Size) ;//this is the packet conten
ResponseStringStream.Clear;
...
end;
procedure TForm1.ButtonGetStreamPricesClick(Sender: TObject);
var
begin
.....
source := RatesWorker.RatesURL+'EUR_USD';
RatesWorker.IdHTTP.Get(source,RatesWorker.ResponseStringStream);
end;
Yet use a custom write() function of Tstream may be a better solution for this kind of requirement.
I want to load a url directly into a string without any data stream,what is the best way, internet open url works but it seems not clear.
I don't want to use any component for reading some short messages
Delphi 6 and later ship with Indy, which has a TIdHTTP client component, eg:
uses
..., IdHTTP;
var
Reply: String;
begin
Reply := IdHTTP1.Get('http://test.com/postaccepter?=msg1=3444&msg2=test');
...
end;
Or:
uses
..., IdHTTP;
var
Reply: TStream;
begin
Reply := TMemoryStream.Create;
try
IdHTTP1.Get('http://test.com/postaccepter?=msg1=3444&msg2=test', Reply);
Reply.Position := 0;
...
finally
Reply.Free;
end;
end;
Depending on your needs.
You can use Synapse, a very light weight library that has a simple function call to get just what your asking for:
uses
classes, httpsend;
var
Response : TStringlist;
begin
if HttpGetText(URL,Response) then
DoSomethingWithResponse(Response.Text);
end;
I would suggest getting the latest copy from SVN, which is more current and contains support for the latest versions of Delphi. There are also simple functions for posting form data, or retrieving binary resources. These are implemented as simple functions and are a great template if you want to do something extra, or that is not directly supported.
You can use our SynCrtSock unit, which is even lighter than Synapse.
See http://synopse.info/fossil/finfo?name=SynCrtSock.pas
It is a self-contained unit (only one dependency with the WinSock unit), and it works from Delphi 6 up to Delphi XE.
You have these two functions available to get your data in one line of code:
/// retrieve the content of a web page, using the HTTP/1.1 protocol and GET method
function HttpGet(const server, port: AnsiString; const url: TSockData): TSockData;
/// send some data to a remote web server, using the HTTP/1.1 protocol and POST method
function HttpPost(const server, port: AnsiString; const url, Data, DataType: TSockData): boolean;
Or you can use textfile-based commands (like readln or writeln) to receive or save data.
TSockData is just a wrapper of RawByteString (under Delphi 2009/2010/XE) or AnsiString (up to Delphi 2007).
If you need also to write a server, you have dedicates classes at hand, resulting in fast processing and low resource consummation (it uses a Thread pool, and is implemented over I/O Completion Ports).
If I'm already using XML in an application (and the MSXML2_TLB), I generally use IXmlHttpRequest to perform http operations. If you open and send the request, you can either use the response data as XML DOM using the ResponseXML, as text using ResponseText or as a data-stream using ResponseStream, see here for an example how to use this in Delphi: http://yoy.be/item.asp?i142
I haven't programmed in Delphi for a while and frankly didn't think I'll ever have to but...
Here I am, desperately trying to find some information on the matter and it's so scarce nowadays, I can't find anything. So maybe you guys could help me out.
Currently my application uses Synapse library to make HTTP calls, but it doesn't allow for setting a timeout. Usually, that's not a big problem, but now I absolutely must to have a timeout to handle any connectivity issues nicely.
What I'm looking for, is a library (synchronous or not) that will allow making HTTP requests absolutely transparent for the user with no visible or hidden delays. I can't immediately kill a thread right now, and with possibility of many frequent requests to the server that is not responding, it's no good.
EDIT: Thanks everybody for your answers!
You will always have to take delays and timeouts into account when doing network communication. The closest you can get IMHO is to put network communication in a thread. Then you can check if the thread finishes in the desired time and if not just let it finish, but ignore the result (there's no safe way to abort a thread). This has an additional advantage: you can now just use synchronous network calls which are a lot easier to read.
In synapse, the timeout is available from the TSynaClient object, which THttpSend decends from. So all you have to do to adjust for timeout (assuming your using the standard functions) is to copy the function your using, add a new parameter and set the Timeout to what you need. For example:
function HttpGetTextTimeout(const URL: string;
const Response: TStrings;
const Timeout:integer): Boolean;
var
HTTP: THTTPSend;
begin
HTTP := THTTPSend.Create;
try
HTTP.Timeout := Timeout;
Result := HTTP.HTTPMethod('GET', URL);
if Result then
Response.LoadFromStream(HTTP.Document);
finally
HTTP.Free;
end;
end;
Synapse defaults to a timeout of 5000 and does timeout if you wait long enough. Since its tightly contained, synapse runs perfectly fine in threads.
[Known to work on D2010 only]
You can use MSXML to send client requests (add msxml and ole2 to your uses clause). The trick is to use IServerXMLHTTPRequest rather than IXMLHTTPRequest, as the former allows timeouts to be specified. The code below shows the Execute() method of a thread:
procedure TClientSendThread.Execute;
const
LResolveTimeoutMilliseconds = 2000;
LConnectTimeoutMilliseconds = 5000;
LSendTimeoutMilliseconds = 5000;
LReceiveTimeoutMilliseconds = 10000;
var
LHTTPServer: IServerXMLHTTPRequest;
LDataStream: TMemoryStream;
LData: OleVariant;
begin
{Needed because this is inside a thread.}
CoInitialize(nil);
LDataStream := TMemoryStream.Create;
try
{Populate ....LDataStream...}
LData := MemoryStreamToOleVariant(LDataStream);
LHTTPServer := CreateOleObject('MSXML2.ServerXMLHTTP.3.0') as IServerXMLHTTPRequest;
LHTTPServer.setTimeouts(
LResolveTimeoutMilliseconds,
LConnectTimeoutMilliseconds,
LSendTimeoutMilliseconds,
LReceiveTimeoutMilliseconds
);
LHTTPServer.open('POST', URL, False, 0, 0);
LHTTPServer.send(LData);
FAnswer := LHTTPServer.responseText;
finally
FreeAndNil(LDataStream);
CoUninitialize;
end;
end;
I recently discovered an extremely annoying behavior of this MSXML technique in which GET requests will not be re-sent if the URL remains unchanged for subsequent sendings; in other words, the client is caching GET requests. This does not happen with POST.
Obviously, once the timeouts occur, the Execute method completes and the thread is cleaned up.
Synapse can be configured to raise an Exception when network errors occur.
RaiseExcept
Check http://synapse.ararat.cz/doc/help/blcksock.TBlockSocket.html#RaiseExcept:
If True, winsock errors raises
exception. Otherwise is setted
LastError value only and you must
check it from your program! Default
value is False.
This is an interesting problem that I’ve not been able to solve yet.
I am writing a client that communicates across the Internet to a server. I am using the TIdTcpClient Internet Direct (Indy) component in Indy 10 using RAD Studio 2007 native personality.
To get data from the server, I issue an HTTP request using SSL over port 443 where my request details are contained in the HTTP message body. So far, so good. The code works like charm, with one exception.
There is one request that I am submitting that should produce a response of about 336 KB from the server (the HTTP response header contains Content-Length: 344795). The problem is that I am getting only 320KB back. The response, which is in XML, is clearly truncated in the middle of an XML element.
For what it’s worth, the XML is simple text. There are no special characters that can account for the truncation. My TIdTcpClient component is simply reporting that, after receiving the partial response, that the server closed the connection gracefully (which is how every response is expected to be completed, even those that are not truncated, so this is not a problem).
I can make nearly identical calls to the same server where the response is also more than a few K bytes, and all of these work just fine. One request I make returns about 850 KB, another returns about 300 KB, and so on.
In short, I encounter this problem only with the one specific request. All other requests, of which there are many, receive a complete response.
I have talked to the creator of the service, and have supplied examples of my request. He reported that the request is correct. He also told me that when he issues my same request to his server that he gets a complete response.
I’m at a loss. Either the creator of the service is mistaken, and there is actually a problem with the response on that end, or there is something peculiar about my request.
Is there a solution here that I'm missing? Note that I’ve also used a number of other read mechanisms (ReadString, ReadStrings, ReadBytes, etc) and all produce the same result, a truncation of this one specific response at the 320KB mark.
The code is probably not relevant, but I’ll include it anyway. Sorry, but I cannot include the XML request, as it includes proprietary information. (ReadTimeout is set to 20 seconds, but the request returns in about 1 second, so it's not a timeout issue.)
function TClient.GetResponse(PayloadCDS: TClientDataSet): String;
var
s: String;
begin
try
try
s := GetBody(PayloadCDS);
IdTcpClient1.Host := Host;
IdTcpClient1.Port := StrToInt(Port);
IdTcpClient1.ReadTimeout := ReadTimeout;
IdTcpClient1.Connect;
IdTcpClient1.IOHandler.LargeStream := True;
//IdTcpClient1.IOHandler.RecvBufferSize := 2000000;
IdTcpClient1.IOHandler.Write(s);
Result := IdTcpClient1.IOHandler.AllData;
except
on E: EIdConnClosedGracefully do
begin
//eat the exception
end;
on e: Exception do
begin
raise;
end;
end;
finally
if IdTcpClient1.Connected then
IdTcpClient1.Disconnect;
end;
end;
Since you are sending an HTTP request, you should be using the TIdHTTP component instead of the TIdTCPClient component directly. There are a lot of details about the HTTP protocol that TIdHTTP manages for you that you would have to handle manually if you continue using TIdTCPClient directly.
If you are going to continue using TIdTCPClient directly, then you should at least stop using the TIdIOHandler.AllData() method. Extract the 'Content-Length' reply header (you can Capture() the headers into a TStringList or TIdHeaderList and then use its Values[] property) and then pass the actual reported byte count to either TIdIOHandler.ReadString() or TIdIOHandler.ReadStream(). That will help ensure that the reading I/O does not stop reading prematurely because of the server's disconnect at the end of the reply.
As I mentioned in my original question, this connection uses SSL. This requires that you use an IdSSLIOHandlerSocketOpenSSL component, which you assign to the IOHandler property of the IdTcpClient component. All of this was in my original design, and as I mentioned, many of my requests were being responded to correctly.
Over the weekend I discovered another request which was returning an incomplete response. I captured that original HTTP request, and using a tool called SOAP UI, I submitted that request to the server. Using SOAP UI, the server returned a complete answer. Clearly, something was wrong with my client, and not the server.
I finally found the solution by just fiddling around with various properties. The property that finally corrrected the problem was in the SSLOptions property of the IdSSLIOHandlerSocketOpenSSL class. The default value of SSLOptions.Method is sslvSSLv2. When I changed Method to sslvSSLv23, all responses returned complete.
Why I was able to retrieve some responses in full and not all before I made this change is still a mystery to me. Nonetheless, setting IdSSLIOHandlerSocketOpenSSL.SSLOptions.Method to sslvSSLv23 solved my problem.
Thank you Remy Lebeau, for your suggestions.