I am using the Internet Direct TIdTCPClient component to communicate with a remote service to retrieve a message that is normally about 5k is size. During a typical operation I will send about 400 requests to the service, each one taking about 1 second to complete. Most of the time everything works perfectly. However, about one percent of the time the request takes 189 seconds and I receive no data at all. For ease of discussion, I'll call this a failure.
I am particularly interested in understanding exactly what is happening when the failure occurs so that I can take my evidence to the publisher of the service. First of all, the failure is not reproducible. If I re-send a failed request there is a very high probability (maybe 99 percent) that it will work.
I also capture the request that I send when a failure occurs, so I am able to confirm that the request is well formed.
I am assuming that during a failure that I am getting some data, just not all of it. Here is why. My IdTCPClient has a 30 second timeout (I've even set it to 5 seconds, but that didn't make a difference). When the failure occurs, it always fails after 189 seconds (plus about 500 milliseconds).
As a result, I am thinking that during a failure my component is receiving a trickle of data, which is why my client is not timing out. And, I am assuming that the disconnection is happening at the service since none of my timeout values are ever set to 189 seconds. On the other hand, reading IOHandler.AllData does not raise an exception (not even an EIdConnClosedGracefully exception). Am I interpreting this evidence correctly?
What I want to do is to confirm that I am getting some data, just not all of it, before the service terminates the connection. Furthermore, I want to know what that partial data looks like as I believe it can help identify the source of the failure.
Currently, my request is similar to the following:
//ExceptionName is a temporary global variable
//that I am using while trying to solve this issue
ExceptionName = 'no exception';
try
s := GetRequest(id);
IdTcpClient1.Host := Host;
IdTcpClient1.Port := StrToInt(Port);
IdTcpClient1.ReadTimeout := ReadTimeout;
try
IdTcpClient1.Connect;
except
on e: exception do
begin
ExceptionName := e.ClassName;
raise EConnectionFailure.Create('Connection refused: ' + e.Message)
end;
end;
IdTcpClient1.IOHandler.Writeln(s);
try
Result := IdTcpClient1.IOHandler.AllData;
except
on E: EIdConnClosedGracefully do
begin
ExceptionName := e.ClassName;
//eat this exception
end;
on e: Exception do
begin
ExceptionName := e.ClassName;
raise;
end;
end;
finally
if IdTcpClient1.Connected then
IdTcpClient1.Disconnect;
end;
Using IOHandler.AllData to read the data is very convenient, but I cannot retrieve any data following a failure (AllData returns an empty string). I've tested IOHandler.InputBufferIsEmpty after a failure, and it returns True.
I've also tried other methods to read the data, such as IOHandler.ReadStream (this produced the same result as reading AllData). I also used IOHandler.ReadBytes and IOHandler.ReadByte (in conjunction with IOHandler.CheckForDataOnSource). Nothing has worked.
Am I wrong about partial data transmission? If so, why would I see a consistent 189.nnnn seconds delay before the failure.
If partial data transmission is a possibility, what approach should I take to capture every byte of data received before the failure.
I am using Delphi 2009 for this project and Indy 10, but I don't think the version has anything to do with it. I don't think this is an Indy issue.
Edit: I have inspected the communication between my Indy client and the server using WireShark. When one of these failures occur, after sending my request the server sent two [ACK] packets followed by silence for just over 189 seconds. After that delay, the response included [FIN, PSH, ACK] but no application data.
When the communication worked normally, the two ACK packets returned by the server in response to my request was followed by an application data packet.
Edit: Have reported the issue to the publisher of the Web service, and am waiting for a reply.
Edit: Ok, the publisher of the Web service has responded. They acknowledged problems on their end and have addressed some of those. We are no longer getting timeouts. Most responses are received in about 2 seconds, with a few taking slightly longer. The publisher is working to fix the remaining issues.
Thank you everyone for your input.
You need to run Fiddler2 and watch the traffic. It inserts itself as a proxy and sniffs evertying that uses the WinInet stack.
Then you know whether you've gotten any data or not, and exactly what you're sending and receiving.
http://www.fiddler2.com/fiddler2/
Related
I am working on the project where I communicate with another device over COM port.
For incoming data I am using VaComm1RXchar event, there I store the message into the array and increment msgIndex which represents number of messages.
Then I call the function where I work with this message.
Inside this function is this timeout cycle where I wait for this message:
while MsgIndex < 1 do
begin
stop := GetTickCount;
if (stop - start)> timeout then
begin
MessageBox(0, 'Timeout komunikace !', 'Komunikace', MB_OK);
exit(false);
end;
sleep(10);
end;
The strange thing for me is that, when have it like this above then it always end with timeout. But when I put on there before this while cycle a ShowMessage('Waiting') then It works correcly.
Does anyone know what can caused this and how can I solve it? Thanks in advance!
We can deduce that the VaComm1RXchar event is a synchronous event and that by blocking your program in a loop you are preventing the normal message processing that would allow that event to execute.
Showing a modal dialog box, on the other hand, passes message handling to that dialog so the message queue is properly serviced and your Rx events and handled normally.
You can be certain this is the case if this also works (please never write code like this - it's just to prove the point):
while MsgIndex < 1 do begin
stop := GetTickCount;
if (stop - start)> timeout then begin
MessageBox(0, 'Timeout komunikace !', 'Komunikace', MB_OK);
exit(false);
end;
Application.ProcessMessages; // service the message queue so that
sleep(10); // your Rx event can be handled
end;
If there is a lesson here it is that RS-232 communication really needs to be done on a background thread. Most all implementations of "some chars have been received" events lead to dreadful code for the very reason you are discovering. Your main thread needs to be free to process the message that characters have been received but, at the same time, you must have some parallel process that is waiting for those received characters to complete a cogent instruction. There does not exist a sensible solution in an event-driven program to both manage the user interface and the communication port at the same time on one thread.
A set of components like AsyncPro**, for example, wrap this functionality into data packets where synchronous events are used but where the components manage start and end string (or bytes) detection for you on a worker thread. This removes one level of polling from the main thread (ie: you always get an event when a complete data packet has arrived, not a partial one). Alternatively, you can move the communication work to a custom thread and manage this yourself.
In either case, this is only partly a solution, of course, since you still cannot stick to writing long procedural methods in synchronous event handlers that require waiting for com traffic. The second level of polling, which is managing a sequence of complete instructions, will still have you needing to pump the message queue if your single procedure needs to react to a sequence of more than one comport instruction. What you also need to think about is breaking up your long methods into shorter pieces, each in response to specific device messages.
Alternatively, for heavily procedural process automation, it is also often a good idea to move that work to a background thread as well. This way your worker threads can block on synchronization objects (or poll in busy loops for status updates) while they wait for events from hardware. One thread can manage low level comport traffic, parsing and relaying those commands or data packets, while a second thread can manage the higher level process which is handling the sequence of complete comport instructions that make up your larger process. The main thread should primarily only be responsible for marshalling these messages between the workers, not doing any of the waiting itself.
See also : I do not understand what Application.ProcessMessages in Delphi is doing
** VAComm may also support something like this, I don't know. The API and documentation are not publicly available from TMS for ASync32 so you'll need to consult your local documentation.
What happens is that a modal message loop executes when the dialog is showing. That this message loop changes behaviour for you indicates that your communications with the device require the presence of a message loop.
So the solution for you is to service the message queue. A call to Application.ProcessMessages in your loop will do that, but also creates other problems. Like making your UI become re-entrant.
Without knowing more about your program, I cannot offer more detailed advice as to how you should solve this problem.
I have situation when I know how much bytes I expect to get from client (in this code it is 100). If client does't gives me 100 I don't need this packet at all, because it is corrupted.
I have situation when client gives less than 100 bytes and disconnects. In this case I'm catching exception and everything would be fine except that after this exception my ServerTCPExecute is called for many times uncontinuously. There are no any other clients connected. Why it is so? If I do TIdYarnOfThread(AContext.Yarn).Thread.Terminate; everything goes fine. But I'm not sure it is good solution.
procedure Form1.ServerTCPExecute(AContext: TIdContext);
begin
try
AContext.Connection.IOHandler.ReadBytes(b, 100, False);
except
//TIdYarnOfThread(AContext.Yarn).Thread.Terminate;
end;
end;
Solution: do not use try ... except to catch and ignore all exceptions within OnExecute.
This will swallow the Indy exception which happens when the client disconnects, the server will execute the next iteration of OnExecute if the IOHandler.InputBuffer still has data in it, and ReadBytes raises another exception because the peer disconnected.
If any exception is raised and leaves the method, the Indy TCP server will close and clean up the connection.
Indy uses exceptions for error handling and notifications in OnExecute, so do not suppress them with an "empty" exception handler. If you need to catch exceptions, re-raise any EIdException-derived exceptions you catch and let the server handle them.
p.s.: the posted code seems to use a non-local variable b. Because TIdTCPServer is a multi-threaded component, non-local variables must always be accessed in a thread-safe way, for example using a critical section, or putting the variable inside of the TIdContext object.
Is there a way to prevent Indy from raising exceptions on any/all HTTP statuses?
My understanding of the IgnoreReplies array is that it'll prevent those statuses from ever coming back, but that's not what I want. I want ALL statuses coming back, and none of them to raise an exception. There are plenty of REST services that return 404 for example, and that's considered a perfectly "valid" response.
I really don't want 1/2 of my code in exception handlers, so is there way to get the current Indy to just return everything?
Sorry, but you have to use the IgnoreReplies parameter to tell TIdHTTP which specific HTTP reply codes to not raise an exception for. There is currently no way to tell TIdHTTP to not raise an exception for all possible replies generically. If you want to receive a non-success reply, like 404, without raising an exception then you have to tell TIdHTTP that, eg:
IdHTTP.Get('http://host/path', [404]);
Note that this syntax is only available for GET requests, not other requests, like POST (unless you call the protected TIdHTTP.DoRequest() method directly).
Indy is specifically designed around exception handling. You have to embrace exceptions, not avoid them, if you want to use Indy effectively. What is so troublesome about wrapping TIdHTTP.Get() or TIdHTTP.Post() in a small try/except block? If you don't want to handle all possible exceptions, then you can have the except block handle EIdHTTPProtocolException by itself.
try
IdHTTP.Get('http://host/path');
except
on E: EIdHTTPProtocolException do
begin
if E.ErrorCode <> 404 then Raise;
end;
end;
Update: thinking about it more, it might be worth adding a new flag to the TIdHTTP.HTTPOptions property to disable the exception for all status codes. I'm considering it.
Update: a new hoNoProtocolErrorException flag has now been added to the TIdHTTP.HTTPOptions property:
IdHTTP.HTTPOptions := IdHTTP.HTTPOptions + [hoNoProtocolErrorException];
IdHTTP.Get('http://host/path');
// use IdHTTP.ResponseCode as needed...
Update: in addition, there is also a hoWantProtocolErrorContent flag as well. When hoNoProtocolErrorException is enabled and an HTTP error occurs, if hoWantProtocolErrorContent is enabled then the error content will be returned to the caller (either in a String return value or in an AResponseContent output stream, depending on which version of Get()/Post() is being called), otherwise the content will be discarded.
I'm writing a program to perform checks on websites to test they're current status, e.g. 200 OK. I'm using Indy HTTP shipped with Delphi 2010 for this task and have the current code.
procedure TCheckSiteIndividual.Execute;
var
http : TIdhttp;
begin
try
try
http := Tidhttp.create(nil); //Create indy
http.Get(CSiteURL,nil); //Send Request To Check Site
except
on E: EIdHTTPProtocolException do
status := http.ResponseText; // or: code := E.ErrorCode;
end;
status := http.ResponseText; //Return Status of Site
synchronize(updateform); //Update data to form
finally
http.Free;
end;
end;
The CSiteURL variable is set in the Thread Constructor, and the status variable is a variable to the entire thread.
The try, except parts of this code came from Remy Lebeau - TeamB here which was a question I asked when initially fiddling around with this idea. The problem I'm having is that this code only works if the site returns a 200 OK code. Anything else causes an exception, including if I disable the internet. This also causes exceptions on sites that redirect e.g. www.google.com causes an exception during debugging I presume causes it's redirecting to www.google.co.uk for me, however if I continue past the exception then it returns a 301 FOUND code with no problems.
The final result I'm looking for it for it to give me a return based on whether the site is online or having error (e.g. HTML RETURN CODE) additionally giving feedback when the site doesn't exist (e.g. a non-registered URL) or just can't be contacted.
The other peices of advice I'm looking for are into how to get some more information back from the request. Specifically at the moment I'm looking to also get the IP of the site that was checked.
It would also be nice to have it work without needing to have http:// before any URL, so for example www.google.com will work which at the moment it doesn't because of unknown protocol.
Basically it would be great if someone had a link to some nice documentation for IndyHTTP because it seems to be sparse,
Yes, of course it causes an exception. That's how Indy works, and that's why Remy advised you to catch exceptions.
What you're confused by now is that even though you have code to catch exceptions, the Delphi debugger still intercepts them and pauses execution of your program to notify you that an exception occurred. I've written about that phenomenon before.
Since you're apparently not interested in knowing about non-status-200 exceptions while you're debugging — because your program is designed to handle all those errors itself — your best option is probably to add EIdHTTPProtocolException to the debugger's list of ignored exceptions. The easiest way to do that is to choose "Ignore this exception type" the next time the debugger notifies you.
I'm new to delphi and this is my first project.
Here's a little bit of code:
procedure TForm1.Button2Click(Sender: TObject);
responseStringFromServer:TStringStream;
begin
try
if IdTCPClient1.Connected then
begin
dataSentToDevice:= 'http/1.0 content-length: 344 content-type: text/xml <?xml version="1.0" encoding="UTF-8" ?> ...'
IdTCPClient1.IOHandler.WriteLn(dataSentToDevice);
responseStringFromServer := TStringStream.Create;
IdTCPClient1.IOHandler.ReadStream(responseStringFromServer);
...
I have a device connected to local network. I manage to connect to it succesfully. Sending commands is working too, but when i do
IdTCPClient1.IOHandler.ReadStream(responseStringFromServer);
then it waits til device is done processing and an exception occurs: "Connection Closed Gracefully". So i'm not able to read data the device is supposed to send me. Device is not shutting down. I've read other posts and i understand that device itself drops connection.
I have a demo program that communicates with it and it works fine. I need to get response xml before it drops the connection.
Also the request is http and i am using IdTCPClient (i need to use xml request, i don't know how to do it with TidHTTP).
May it be that after device is sending response it drops the connection, so that my tcpclient gets connection dropped flag before receiving data.
Any help would be appreciated!
Also the request is http and i am
using IdTCPClient (i need to use xml
request, i don't know how to do it
with TidHTTP).
Working with IdHTTP is simple...
Drop an instance of it on your form,
select it.
In Object Inspector, go to
ProtocolVersion property, and set it
to pv1_0, then open its
Request property set, and set
Request.ContentType to text/xml, and
Request.ContentEncoding to UTF-8,
and set other properties if
required.
Add a button to your form and
double-click on it.
In your code, create an instance of
TStringStream, and load your XML
content into it.
Assign your stream to
IdHttp.Request.Source.
Call IdHttp.Get() method by giving
it a host address.
IdHttp.Get() returns a string which
is the response the server sent you.
The way you are calling ReadStream(), it will interpret the first 4 bytes (or 8 bytes if the TIdIOHandler.LargeStream property is True) as an Integer (or Int64) in network-byte order that specifies the length of the data, and then it will try to read that many bytes. Is the device actually sending such a length value? If not, then ReadStream() will attempt to read the wrong number of bytes. An EIdConnClosedGracefully exception means the device is closing the socket on its end. So either the device is closing the connection immediately after sending its data, or it is timing out waiting for you to send the next command, which you cannot do since you are blocked waiting for the wrong data from the previous command.
As said by #Remy Lebeau and the documentation (press F1 when your edit caret is over the ReadStream sentence):
AByteCount indicates the number of bytes from the IOHandler to be read into AStream. When AByteCount contains -1 and AReadUntilDisconnect contains False, the byte count is read as an Integer value from the IOHandler. The size of AStream is adjusted to match the size expected from the IOHandler.
This is very useful when you're connecting to servers written also in INDY WriteStream in one end, ReadStream in the other, or any other language if you send that byte cound as expected by INDY client.
If you're reading from a device, be sure the device is sending that info up-front the stream or change the way you read the data. If you know the size of the stream, just pass the second parameter: AByteCount. If the device will close the channel when the stream ends, pass -1 as the second parameter and True as the third parameter: AReadUntilDisconnect.
If the device send text of unknown length but using a known terminator (like CR/LF), you better use ReadLn method of IOHandler to get that string.
Take a look at if neither fits your needs, take a look on Read* methods of TIOHandlerClass and RTFM for each to find the correct one to get the data sent by the device.