I'm writing a program to perform checks on websites to test they're current status, e.g. 200 OK. I'm using Indy HTTP shipped with Delphi 2010 for this task and have the current code.
procedure TCheckSiteIndividual.Execute;
var
http : TIdhttp;
begin
try
try
http := Tidhttp.create(nil); //Create indy
http.Get(CSiteURL,nil); //Send Request To Check Site
except
on E: EIdHTTPProtocolException do
status := http.ResponseText; // or: code := E.ErrorCode;
end;
status := http.ResponseText; //Return Status of Site
synchronize(updateform); //Update data to form
finally
http.Free;
end;
end;
The CSiteURL variable is set in the Thread Constructor, and the status variable is a variable to the entire thread.
The try, except parts of this code came from Remy Lebeau - TeamB here which was a question I asked when initially fiddling around with this idea. The problem I'm having is that this code only works if the site returns a 200 OK code. Anything else causes an exception, including if I disable the internet. This also causes exceptions on sites that redirect e.g. www.google.com causes an exception during debugging I presume causes it's redirecting to www.google.co.uk for me, however if I continue past the exception then it returns a 301 FOUND code with no problems.
The final result I'm looking for it for it to give me a return based on whether the site is online or having error (e.g. HTML RETURN CODE) additionally giving feedback when the site doesn't exist (e.g. a non-registered URL) or just can't be contacted.
The other peices of advice I'm looking for are into how to get some more information back from the request. Specifically at the moment I'm looking to also get the IP of the site that was checked.
It would also be nice to have it work without needing to have http:// before any URL, so for example www.google.com will work which at the moment it doesn't because of unknown protocol.
Basically it would be great if someone had a link to some nice documentation for IndyHTTP because it seems to be sparse,
Yes, of course it causes an exception. That's how Indy works, and that's why Remy advised you to catch exceptions.
What you're confused by now is that even though you have code to catch exceptions, the Delphi debugger still intercepts them and pauses execution of your program to notify you that an exception occurred. I've written about that phenomenon before.
Since you're apparently not interested in knowing about non-status-200 exceptions while you're debugging — because your program is designed to handle all those errors itself — your best option is probably to add EIdHTTPProtocolException to the debugger's list of ignored exceptions. The easiest way to do that is to choose "Ignore this exception type" the next time the debugger notifies you.
Related
I've got a Delphi 6 client-server application where the server is essentially:
initialize;
repeat
try
getInput;
putOutput;
except
on e: exception do writeln(e.message);
end
until false;
During development, bugs (such as a pointer exception) were rare but the server would be able to keep running. Now that it's in production the server seems debugged and it's tempting to remove the try...except because it adds substantial execution overhead to the server, but then if an exception were to occur it would stop the server.
One solution would be to have a parent process simply restart the entire server, and bypassing initialization on restart would effectively reproduce what try...except accomplished, but there wouldn't be any record of what caused the exception, and restarting with or without initialization actually creates a lot of unwanted "state" issues, especially with client connections.
Is there some way to override or amend exception handling so that when the exception happens it's recorded somewhere (such as in the system event log or %errorlevel%) before the programs quits and is restarted or (better:) recovers, but without the overhead of an all-encompassing try...except?
The ideal solution would record the exception but resume execution at the level of the repeat...until. Perhaps by doing something like saving the address of the repeat in some accessible global, then make any exception unwind the call stack and resume at that saved address? (Which I guess is what try..except essentially does, but is there a way to avoid the overhead it imposes when there aren't exceptions?)
Expanding on Serg's suggestion:
repeat
try
repeat
getInput;
putOutput;
until false;
except
on e: exception do writeln(e.message);
end
until false;
Has the same behaviour but moves the try .. except outside the innermost loop.
However, I'd be amazed if this had any discernible impact on performance.
Expanding on David and Sergs suggestions...
If the complexity of your loop is as low as your posted code suggests (or could be reduced to such a level of simplicity) then it may be practical to provide both behaviours, subject to a command line switch to either enable or disable runtime errors via exception handling.
Parse the command line at startup to determine the mode of operation and set an indicator variable accordingly. e.g using an enum:
type
TServerMode = (smHighThroughput, smDiagnostic);
var
ServerMode: TServerMode;
if FindCmdLineSwitch('d') then
ServerMode := smDiagnostic
else
ServerMode := smHighThroughput;
You could of course use a setting in a configuration file or database table if a command line parameter is not desirable or practical for whatever reason.
However you go about it, you provide the ability to run the server in "high throughput" mode or "recoverable diagnostic" mode with a minimum of duplicated syntax:
try
case ServerMode of
smDiagnostic : repeat
try
getInput;
putOutput;
except
on e: Exception do .... // etc
end;
until FALSE;
smHighThroughput : repeat
getInput;
putOutput;
until FALSE;
end;
except
on e: Exception do .... // etc
end;
Note that even in HighThroughput mode the above example captures information about any exception that causes server termination. The Diagnostic mode logs all exceptions but attempts to keep the server running.
You can then default to HighThroughput mode with the option of enabling Diagnostic mode in the event that you need to diagnose a server that seems unusually unstable.
Having said all that, I would try to definitively quantify the impact of the try..except on the loop before going to great lengths to engineer around that impact.
Drop an ApplicationEvents on a central place in your application.
Implement the OnException Event.
Delete the local try...except to remove overhead.
In the exception event you can do anything including log the exception.
If you want to add something special to the exception it is possible to still have a local try..except and modify the E.Message and then call raise. Then the global exception handler will be called with the modified message.
Is there a way to prevent Indy from raising exceptions on any/all HTTP statuses?
My understanding of the IgnoreReplies array is that it'll prevent those statuses from ever coming back, but that's not what I want. I want ALL statuses coming back, and none of them to raise an exception. There are plenty of REST services that return 404 for example, and that's considered a perfectly "valid" response.
I really don't want 1/2 of my code in exception handlers, so is there way to get the current Indy to just return everything?
Sorry, but you have to use the IgnoreReplies parameter to tell TIdHTTP which specific HTTP reply codes to not raise an exception for. There is currently no way to tell TIdHTTP to not raise an exception for all possible replies generically. If you want to receive a non-success reply, like 404, without raising an exception then you have to tell TIdHTTP that, eg:
IdHTTP.Get('http://host/path', [404]);
Note that this syntax is only available for GET requests, not other requests, like POST (unless you call the protected TIdHTTP.DoRequest() method directly).
Indy is specifically designed around exception handling. You have to embrace exceptions, not avoid them, if you want to use Indy effectively. What is so troublesome about wrapping TIdHTTP.Get() or TIdHTTP.Post() in a small try/except block? If you don't want to handle all possible exceptions, then you can have the except block handle EIdHTTPProtocolException by itself.
try
IdHTTP.Get('http://host/path');
except
on E: EIdHTTPProtocolException do
begin
if E.ErrorCode <> 404 then Raise;
end;
end;
Update: thinking about it more, it might be worth adding a new flag to the TIdHTTP.HTTPOptions property to disable the exception for all status codes. I'm considering it.
Update: a new hoNoProtocolErrorException flag has now been added to the TIdHTTP.HTTPOptions property:
IdHTTP.HTTPOptions := IdHTTP.HTTPOptions + [hoNoProtocolErrorException];
IdHTTP.Get('http://host/path');
// use IdHTTP.ResponseCode as needed...
Update: in addition, there is also a hoWantProtocolErrorContent flag as well. When hoNoProtocolErrorException is enabled and an HTTP error occurs, if hoWantProtocolErrorContent is enabled then the error content will be returned to the caller (either in a String return value or in an AResponseContent output stream, depending on which version of Get()/Post() is being called), otherwise the content will be discarded.
I am using the Internet Direct TIdTCPClient component to communicate with a remote service to retrieve a message that is normally about 5k is size. During a typical operation I will send about 400 requests to the service, each one taking about 1 second to complete. Most of the time everything works perfectly. However, about one percent of the time the request takes 189 seconds and I receive no data at all. For ease of discussion, I'll call this a failure.
I am particularly interested in understanding exactly what is happening when the failure occurs so that I can take my evidence to the publisher of the service. First of all, the failure is not reproducible. If I re-send a failed request there is a very high probability (maybe 99 percent) that it will work.
I also capture the request that I send when a failure occurs, so I am able to confirm that the request is well formed.
I am assuming that during a failure that I am getting some data, just not all of it. Here is why. My IdTCPClient has a 30 second timeout (I've even set it to 5 seconds, but that didn't make a difference). When the failure occurs, it always fails after 189 seconds (plus about 500 milliseconds).
As a result, I am thinking that during a failure my component is receiving a trickle of data, which is why my client is not timing out. And, I am assuming that the disconnection is happening at the service since none of my timeout values are ever set to 189 seconds. On the other hand, reading IOHandler.AllData does not raise an exception (not even an EIdConnClosedGracefully exception). Am I interpreting this evidence correctly?
What I want to do is to confirm that I am getting some data, just not all of it, before the service terminates the connection. Furthermore, I want to know what that partial data looks like as I believe it can help identify the source of the failure.
Currently, my request is similar to the following:
//ExceptionName is a temporary global variable
//that I am using while trying to solve this issue
ExceptionName = 'no exception';
try
s := GetRequest(id);
IdTcpClient1.Host := Host;
IdTcpClient1.Port := StrToInt(Port);
IdTcpClient1.ReadTimeout := ReadTimeout;
try
IdTcpClient1.Connect;
except
on e: exception do
begin
ExceptionName := e.ClassName;
raise EConnectionFailure.Create('Connection refused: ' + e.Message)
end;
end;
IdTcpClient1.IOHandler.Writeln(s);
try
Result := IdTcpClient1.IOHandler.AllData;
except
on E: EIdConnClosedGracefully do
begin
ExceptionName := e.ClassName;
//eat this exception
end;
on e: Exception do
begin
ExceptionName := e.ClassName;
raise;
end;
end;
finally
if IdTcpClient1.Connected then
IdTcpClient1.Disconnect;
end;
Using IOHandler.AllData to read the data is very convenient, but I cannot retrieve any data following a failure (AllData returns an empty string). I've tested IOHandler.InputBufferIsEmpty after a failure, and it returns True.
I've also tried other methods to read the data, such as IOHandler.ReadStream (this produced the same result as reading AllData). I also used IOHandler.ReadBytes and IOHandler.ReadByte (in conjunction with IOHandler.CheckForDataOnSource). Nothing has worked.
Am I wrong about partial data transmission? If so, why would I see a consistent 189.nnnn seconds delay before the failure.
If partial data transmission is a possibility, what approach should I take to capture every byte of data received before the failure.
I am using Delphi 2009 for this project and Indy 10, but I don't think the version has anything to do with it. I don't think this is an Indy issue.
Edit: I have inspected the communication between my Indy client and the server using WireShark. When one of these failures occur, after sending my request the server sent two [ACK] packets followed by silence for just over 189 seconds. After that delay, the response included [FIN, PSH, ACK] but no application data.
When the communication worked normally, the two ACK packets returned by the server in response to my request was followed by an application data packet.
Edit: Have reported the issue to the publisher of the Web service, and am waiting for a reply.
Edit: Ok, the publisher of the Web service has responded. They acknowledged problems on their end and have addressed some of those. We are no longer getting timeouts. Most responses are received in about 2 seconds, with a few taking slightly longer. The publisher is working to fix the remaining issues.
Thank you everyone for your input.
You need to run Fiddler2 and watch the traffic. It inserts itself as a proxy and sniffs evertying that uses the WinInet stack.
Then you know whether you've gotten any data or not, and exactly what you're sending and receiving.
http://www.fiddler2.com/fiddler2/
Solution found, see my comment below
D5, odbc to mysql database
This code:
with QryCmdPerf do begin
Close;
ParamByName('ACCTID').AsInteger:= AcctId;
ParamByName('FROMDT').AsString:= MySQLDate(FromDt);
ParamByName('TODT').AsString:= MySQLDate(ToDt);
Open;
first;
try
edit;
FieldByName('PnL').AsFloat:= 97979;
ApplyUpdates;
except
close;
end;
end; // with
(specifically the "ApplyUpdates") causes a popup to appear with the text "Update Failed" if the PnL field already has the value 97979, evidently because of this code:
procedure TUpdateSQL.ExecSQL(UpdateKind: TUpdateKind);
begin
with Query[UpdateKind] do
begin
Prepare;
ExecSQL;
if RowsAffected <> 1 then DatabaseError(SUpdateFailed);
end;
end;
in DBTables.pas. Anyway, I want to be able to issue ApplyUpdates, and not have to worry about a popup if it doesn't do any updating. But if "try...except" doesn't work, what will?
TIA
You're confusing the dialog displayed by the debugger with a dialog displayed by your program. Please see this article I wrote a few years ago:
Why do I continue getting error messages even after I have written an exception handler?
It describes several ways to avoid the debugger interfering:
Use "advanced breakpoints" to temporarily disable the debugger around the code that throws exceptions.
Configure the debugger to ignore certain exception types. (Read the debugger's message more carefully to see exactly what exception class you're dealing with.)
Configure the debugger not to interrupt on any exceptions.
Turn off integrated debugger entirely.
The short answer is, you have to set up an eventhandler for OnUpdateError or no amount of "try...except" blocks will block the popup. The long answer is it appears to be a bug with odbc. The repro is here: http://www.codeupload.com/3919 for anyone who wants to take a look at it. You can skip the MySQL stuff, any odbc database will do.
There are two things that can be going wrong here.
Option 1
First, some "very bad code" may be short-circuiting the unwinding of the call stack on exceptions. E.g. ApplyUpdates or one of its child routines may also have a try...except block that calls Application.HandleException directly.
To test this, if you put a breakpoint on QryCmdPerf.Close, do you reach it?
If not, then Application.HandleException (or worse Application.ShowException) has been called directly.
Solving this requires a custom exception handler hooked to the Application.OnException event. You may have to set temporary state to know when this particular exception can be ignored.
Yes it's messy, that why calling Application.HandleException directly, is "very bad code".
Option 2
If you do reach the breakpoint, but the exception is being raised again, then it should be a lot simpler to solve.
The Close method is probably attempting to save any pending changes, so is effectively applying the updates again. Rather than simply closing the data set, call CancelChanges or equivalent.
I am using the following code to copy text to the clipboard:
Clipboard.Open;
try
Clipboard.AsText := GenerateClipboardText;
finally
Clipboard.Close;
end;
Seemingly at random I get "Cannot open clipboard: Access Denied" errors. I'm guessing that these errors are caused by other application locking the clipboard, but I never seem to be doing anything with other applications that should cause the locks.
Strangely my users seem to be reporting more of the errors with Vista and Windows 7 than with XP.
Is there a way to check if the clipboard is locked before trying to access it?
This is not a Delphi problem. Because the clipboard can be locked any moment, even if you check, if the clipboard is currently not locked, it might become locked directly after the check.
You have two possibilities here:
Don't use the Delphi clipboard class. Instead use raw API functions, where you have a little more fine-grained control over possible error situations.
Expect your code to fail by adding an exception handler. Then add some retry code, i.e. retry to set the text three times, perhaps with exponential backoff, before throwing your own error.
I'd recommend the second solution, because it'd be the more Delphi-like approach and in the end will result in cleaner code.
var
Success : boolean;
RetryCount : integer;
begin
RetryCount := 0;
Success := false;
while not Success do
try
//
// Set the clipboard here
//
Success := True;
except
on E: EClipboardException do
begin
Inc(RetryCount);
if RetryCount < 3 then
Sleep(RetryCount * 100)
else
raise Exception.Create('Cannot set clipboard after three attempts');
end else
raise; // if not a clipboard problem then re-raise
end;
end;
Strangely my users seem to be
reporting more of the errors with
Vista and Windows 7 than with XP
This may have to do with how Vista/Win7 deal with clipboard viewer notification. While they still support the XP "clipboard viewer chain", which sends one notification message that must be re-sent to each listener in turn (and if one app fails to do this, the other apps aren't notified). Starting with Vista, apps are notified directly. And there's nothing to keep them from trying to access the clipboard all at once.
Analogy: I have 3 children. I have a cake. With XP rules, I tell the oldest child to have some cake, then tell the next oldest child to have a slice. She gets her slice, tells her brother, he gets his, and tells his brother, who gets his, and everything proceeds in an orderly fashion.
Problem: The middle child takes the cake to his room, doesn't tell the youngest, and the youngest misses out.
With Vista/Windows7, that system still exists. But newer apps can request to be notified immediately, by me, as soon as the cake arrives in the kitchen. I shout "cake is ready!" and they all show up at the same time and try to grab some. But there's only one serving knife, so they have to keep reaching for the knife, failing to get it, and waiting for the next opportunity.
Try to check GetClipboardOwner, if it's not null and not your Application.Handle, you cannot Open to modify it's content.
And even it seems good to go, it might not be anymore when you actually do it.
So add a try except in a loop until you get it or give up nicely (notifying the user for instance).
There is no way to check for something and then depending on the result do something else with the expectation that it could not fail, because unless the check and the action are one atomic operation there is always the possibility that another process or thread does the same in parallel.
This holds whether you try to open the clipboard, open a file, create or delete a directory - you should simply try to do it, maybe several times in a loop, and gracefully handle errors.
First of all please notice that this probably isn't a problem in your application. Other applications locked the clipboard or messed up the notification chain and now your application fails to access it. When I do have problems like this I restart the computer and they magically go away... well... at least until I run again the application that creates the problem.
This code (not checked in Delphi) may help you. It won't fix the problem is the notification chain is broken (nothing except a PC restart will ever fix it) but it will fix the problem if an application is locking the clipboard for a while. Increase the MaxRetries if that pesky application keeps the clipboard locked for A REALLY LONG time (seconds):
procedure Str2Clipboard(CONST Str: string; iDelayMs: integer);
CONST
MaxRetries= 5;
VAR RetryCount: Integer;
begin
RetryCount:= 0;
for RetryCount:= 1 to MaxRetries DO
TRY
inc(RetryCount);
Clipboard.AsText:= Str;
Break;
EXCEPT
on Exception DO
if RetryCount = MaxRetries
then RAISE Exception.Create('Cannot set clipboard')
else Sleep(iDelayMs)
END;
end;
Also, it may be a good idea to drop the 'raise' and convert it to a function and use it like this:
if not Str2Clipboard
then Log.AddMsg('Dear user, other applications are blocking the clipboard. We have tried. We really did. But it didn''t work. Try again in a few seconds.');
I guess you are running your app on Win 8 or higher.
Just right-click on your App .exe file, go to Compatibility tab and change compatibility mode on Windows XP or lower versions. It'll work, guaranteed!