We have an exception that pops up on our website that is getting reported to BugzScout many times a day. The functionality that produces the exception still does what it's intended to do, so we just want to stop FogBugz from piling up all of these occurrences until we have a chance to dig into the issue and prevent the exception.
That said, is there a way to set up a filter on the FogBugz side of things to ignore a list of exceptions that get reported? I know I can set up some logic in our app's BugzScout class so it stops sending those messages, but it would be nice to know if FogBugz does this already before I put the time into building that filter locally. We are using the hosted FogBugz On Demand version of the product if that makes a difference.
In the BugzScout case itself, you should be able to set "Stop Reporting" for the Scout Will setting. This way, only the occurrences will increment when the exception is reported. The case will not reopen or notify anyone.
It sounds a bit from your description that there are many different exceptions reporting to the same ScoutDescription. As much as possible, you should use version numbers and exception line numbers to make sure that exceptions are reported separately. I can elaborate on this if you want.
Related
In my application I have about 100 continuous Esper filter queries with events being sent in. At some point for an unknown reason some of statemens stop matching events and never match any further event without ever throwing an exception (nothing logged in log4j default logging setup). This is not reproducible in a small example, and I realize that it's difficult to pinpoint a problem like this, but I'm writing this in the hopes of this being a known and/or fixed issue.
I would suggest to review your application code to make sure it is still sending events and the listener/subscriber code to make sure it is still processing output events, i.e. exception handling and logging. Or perhaps an OOM occurs and logging doesn't happen and thus you may want to check heap memory use. Also look at console out see if the JVM has encountered an issue.
I've started using SmartInspect in my Delphi applications because my users were running into bugs/problems I couldn't reproduce on my machine. When I have a general idea of the problem I'll monitor the application in a few specific places to confirm what is or is not working.
When the bug doesn't have an obvious cause, I feel lost. I don't know where to start logging in order to narrow down the problem. Are there common techniques or best practices for using a logger?
SmartInspect seems to be quite powerful, but I don't know quite what to log or how to organise my logs so the data is meaningful and useful for catching bugs.
NOTE: I'm using SmartInspect but I assume the answers should be suitable for any logging package.
Here are some guidelines I tried to implement in my own OpenSource logging unit, but it's fairly generic, and as you state, it should be suitable for any logging package:
Make several levels (we use sets) of logging, to tune the logging information needed;
Log all exceptions, even the handled one with a try...except block - and add an exception classes list not worth logging (e.g. EConvertError) - e.g. our unit is able to log all exceptions via a global exception "hook" (no try..except to add in your code), and handle a list of exception classes to be ignored;
Log all "fatal" errors, like database connection issues, or wrong SQL syntax - should be done though "log all exceptions" previous item;
For such exceptions, log the stack trace to know about the calling context;
Be able to log all SQL statements, or database access;
Add a generic User Interface logging, to know which main functions of the software the User did trigger (e.g. for every toolbar button or menu items): it's very common that the user said 'I have this on my screen/report, but I didn't do anything'... and when you see the log, you will discover that the "anything" was done. ;)
Monitor the main methods of your application, and associated parameters;
Logging is an evolving feature: use those general rules above, then tune your logging from experiment, according to your debugging needs.
For UI-driven applications here are the main things I instrument first:
ActionManager or ActionList's events when an action executes (gives me a user clicked here then here then here list).
Unhandled Exceptions with tracebacks using JCL debug go right in my main log, whereas if I was using MadExcept or EurekaLog, exceptions have their own log.
Background thread starts, stops and significant history events
Warnings, errors, API function failures, file access failures, handled (caught) exceptions.
Current memory usage can be useful for long running processes to see if there are memory leaks (which could lead to an out of memory error some day).
I have seen Ryan Bates talk about exception_logger and exception notification. Are there any other good ones to consider? What do you like and dislike about these?
Also, do these exception notifiers log exceptions if you catch them?
Thanks!
I would highly recommend using Hoptoad. We use the free account for our production system which is plenty fine, and is just a life saver. I find it highly superior to the ExceptionNotifier and ExceptionLogger plugins. It's dead simple to install, and then you get automatic email notification, as well as really nice presentation of the exceptions in their web UI.
But, also, one nice thing is that it doesn't email you every time you get the exception, just the first time, but then it queues up the others, so when you go look at it on the web, you'll see that say it happened 7 times or whatever. This alone is really great in that if for some reason you introduce something that is all of a sudden firing off lots of exceptions, and you have a high traffic site, you won't have now completely inundated your email inbox with the notices.
Another other cool thing is that it integrates with Capistrano deployments, and records when your deployments were done, so this gives you a bit more info, and may be the key to tracking down what is wrong (e.g. you deploy, and then shortly thereafter you start getting notices, you know that the problem is in whatever you just introduced).
Finally, there's even an API to interact with it. Lots of goodness, and it's the only thing I use now - even with a New Relic RPM account, I still use Hoptoad for all my exception notification.
I have been using ExceptionLogger since one year. I didn't like exception_notification because I don't like to receive tons of email. Also, ExceptionLogger is somehow an improved version of ExceptionNotification.
Unfortunately ExceptionLogger hasn't been updated since a while. I recently started to work on a custom tool that takes advantage of Rails 2.3 Engine feature.
Also there are a couple of webapps dedicated to solve this problem:
Exceptional
Hoptoad
Hoptoad is my preferred exception notifier by a landslide.
It's intelligent about the emails it sends me, and it's web interface does a nice job of automatically organizing all of the exceptions.
If I make a change that causes an error, I'll likely get several of the same error messages, so Hoptoad sends me an email, and in the web interface I might see something like 4 of the same error, I can click through to the details, patch the bug, and with one click check them off as fixed.
It's a really nice clean way of handling exceptions and strikes me as far less intrusive/ghetto.
I also use Hoptoad and it seems to functions ok after a slightly fiddly setup (but then I need to log rake exceptions too).
However, if I hadn't won a Hoptoad account, I'd prefer to use Exceptional as it has much better notification options (such as SMS)
I use ExceptionLogger as well.
If you catch aka handle an exception it is really no longer an exception, you could as well log it yourself.
I've used Exceptional in the past, their offering is quite similar to Hoptoad.
I was pretty happy with Exceptional, apart from the UI upgrade they did about a year or two ago. I thought the previous layout was better for tracking errors.
Now considering Hoptoad for a project that includes an iPhone component, as they list iOS among supported platforms.
I finishing up a web application and I'm trying to implement some logging. I've never seen any good examples of what to log. Is it just exceptions? Are there other things I should be logging? What type of information do you find useful for finding and fixing bugs.
Looking for some specific guidance and best practices.
Thanks
Follow up
If I'm logging exceptions what information specifically should I be logging? Should I be doing something more than _log.Error(ex.Message, ex); ?
Here is my logical breakdown of what can be logged within and application, why you might want to and how you might go about doing it. No matter what I would recommend using a logging framework such as log4net when implementing.
Exception Logging
When everything else has failed, this should not. It is a good idea to have a central means of capturing all unhanded exceptions. This shouldn't
be much harder then wrapping your entire application in a giant try/catch unless you are using more than on thread. The work doesn't end here
though because if you wait until the exception reaches you a lot of useful information would have gone out of scope. At the very least you should
try to collect specific pieces of the application state that are likely to help with debugging as the stack unwinds. Your application should always be prepared to produce this type of log output, especially in production. Make sure to take a look at ELMAH if you haven't already. I haven't tried it but I have heard great things
Application Logging
What I call application logs includes any log that captures information about what your application is doing on a conceptual level such as "Deleted Order" or "A User Signed On". This kind of information can be useful in analyzing trends, auditing the system, locking it down, testing, security and detecting bugs of coarse. It is probably a good idea to plan on leaving these logs on in production as well, perhaps at variable levels of granularity.
Trace Logging
Trace logging, to me, represents the most granular form of logging. At this level you focus less on what the application is doing and more on how it is doing it. This is one step above actually walking through the code line by line. It is probably most helpful in dealing with concurrency issues or anything for that matter which is hard to reproduce. You wouldn't want to always have this running, probably only turning it on when needed.
Lastly, as with so many other things that usually only get addressed at the very end, the best time to think about logging is at the beginning of a project so that the application can be designed with it in mind. Great question though!
Some things to log:
business actions, such as adding/deleting items. Talk to your app's business owner to come up with a list of things that are useful. These should make sense to the business, not to you (for example: when user submitted report, when user creates a new process, etc)
exceptions
exceptions
exceptions
Some things to NOT to log:
do not log information simply for tracking user usage. Use an analytics tool for that (which tracks the client in javascirpt, not in the client)
do not track passwords or hashes of passwords (huge security issue)
Maybe you should log page/resource accesses which are not yet defined in your application, but are requested by clients. That way, you may be able to find vulnerabilities.
It depends on the application and its audience. If you are managing sales or trading stocks, you probably should log more info than say a personal blog. When you need the log most is when an error is happening in your production environment, but can't reproduce it locally. Having log level and log hierarchy would help in such situations, because you can dynamically increase the log level. See log4j's documentation and log4net.
My few cents.
Besides using log severity and exceptions properly, consider structuring your log statements so that you could easily look though the log data in the future. For example - extracting meaningful info quickly, doing queries etc. There is no problem to generate an ocean of log data, the problem is to convert this data into information. So, structuring and defining it beforehand helps in later usage. If you use log4j, I would also suggest using mapped diagnostic context (MDC) - this helps a lot for tracking session contexts. Aside from trace and info, I would also use debug level where I usually keep temp. items. Those could be filtered out or disabled when not needed.
You probably shouldn't be thinking of this at this stage, rather, logging is helpful to consider at every stage of development to help diffuse potential bugs before they arise. Depending on your program, I would try to capture as much information as possible. Log everything. You can always stop logging certain components or processes if you don't reference that data enough. There is no such thing as too much information.
From my (limited) experience, if you don't want to make a specific error table for each possible error type, construct a generic database table that accepts general information as well as a string that you can populate with exception data, confirmation messages during successful yet important processes, etc. I've used a generic function with parameters for this.
You should also consider the ability to turn logging off if necessary.
Hope this helps.
I beleive when you log an exception you should also save current date and time, requested url, url refferer and user IP address.
I have just started using an exception logger (EurekaLog) in my (Delphi) application. Now my application sends me lots of error messages via e-mail every day. Here's what I found out so far
lots of duplicate errors
multiple mails from the same PC
While this is highly valuable input to improve my application, I am slightly overwhelmed by the sheer amount of information I'm getting.
What are your best practices for handling mails from your application?
If you get to much information as it is currently the case you are not getting any information at all.
So I would say categorize your errors into groups, like WARNINGS, FATAL ERRORS, etc.
Then limit your emails to the most important messages (FATAL).
Apart from that review your logs on a regular basis (day, week ...).
What I've done with my exception logging, which uses madExcept as the core, but my own transport mechanism, is have them all go into a database. The core information is all extracted from each report and put into fields, and the whole report is stored as well. The stack trace is automatically analysed to remove the un-interesting functions, leaving a list of only my functions that have failed.
With this happening automatically, I can now "ignore" each individual message coming in, but see the bigger picture in a grid that shows me simply which functions are having the most problems. I can then focus on them, look for the causes, and fix them.
My display app is also able to filter out reports in builds before a certain number if I choose, so that I can tell it not to include "MyWidget.BadProc" before build 75 once I've fixed it.
This has helped me improve my app, and hit the problems that people found most problematic, without having to guess.
It would very much depend on what the errors are that are being sent back. The obvious one being if there are errors in your application, they need fixing and patches/updates sent to your clients.
If they are exceptions that you know can happen and do not required you to be notified you can add "Exception Filters" in the Eureaka Log options to specify how they should be handled (or ignored!).
Another option is to use EurekaLog Variables (where you can add the exception description etc) in the mail Subject line and then use your email client to filter based on this.
I did this using madExcept. It's really useful for tracking down problems we couldn't reproduce ourselves.
Which makes me ask why you are getting so many? Untrapped exceptions should be few and far between. Especially if the user sees an error dialog. I was responsible for several applications, each with hundreds of installs and I would rarely get e-mail notices.
If they are mostly from a very small number of PCs, I'd work with some of those users to find out what they're doing differently, or how their setup might be generating exceptions.
If they are from all over the place, it's probably a bug that got through your testing.
Either way, use the details to fix your code or, at the very least, anticipate known exceptions and trap them properly (no empty try..except).
Fixing the hot spot problems will cut way down on the number of e-mails you get, making the occasional notice much more manageable.
I think that you should throw out all duplicates. Leave only count of the reports. I.e. if you get, say 100 reports, but there are only 4 unique problems - leave only 4 reports, throw out other 96 reports, but use their count to sort reports by severity. For example, 6 reports for fourth problem, 10 reports for third, 20 for second and 60 for first. So, you should fix first problem with 60 reports and only then switch to second.
I believe that EurekaLog has BugID in its reports. Same problem has the same BugID. This will allow you to sort reports with duplicates. EurekaLog Viewer also can sort out duplicates.