I have seen Ryan Bates talk about exception_logger and exception notification. Are there any other good ones to consider? What do you like and dislike about these?
Also, do these exception notifiers log exceptions if you catch them?
Thanks!
I would highly recommend using Hoptoad. We use the free account for our production system which is plenty fine, and is just a life saver. I find it highly superior to the ExceptionNotifier and ExceptionLogger plugins. It's dead simple to install, and then you get automatic email notification, as well as really nice presentation of the exceptions in their web UI.
But, also, one nice thing is that it doesn't email you every time you get the exception, just the first time, but then it queues up the others, so when you go look at it on the web, you'll see that say it happened 7 times or whatever. This alone is really great in that if for some reason you introduce something that is all of a sudden firing off lots of exceptions, and you have a high traffic site, you won't have now completely inundated your email inbox with the notices.
Another other cool thing is that it integrates with Capistrano deployments, and records when your deployments were done, so this gives you a bit more info, and may be the key to tracking down what is wrong (e.g. you deploy, and then shortly thereafter you start getting notices, you know that the problem is in whatever you just introduced).
Finally, there's even an API to interact with it. Lots of goodness, and it's the only thing I use now - even with a New Relic RPM account, I still use Hoptoad for all my exception notification.
I have been using ExceptionLogger since one year. I didn't like exception_notification because I don't like to receive tons of email. Also, ExceptionLogger is somehow an improved version of ExceptionNotification.
Unfortunately ExceptionLogger hasn't been updated since a while. I recently started to work on a custom tool that takes advantage of Rails 2.3 Engine feature.
Also there are a couple of webapps dedicated to solve this problem:
Exceptional
Hoptoad
Hoptoad is my preferred exception notifier by a landslide.
It's intelligent about the emails it sends me, and it's web interface does a nice job of automatically organizing all of the exceptions.
If I make a change that causes an error, I'll likely get several of the same error messages, so Hoptoad sends me an email, and in the web interface I might see something like 4 of the same error, I can click through to the details, patch the bug, and with one click check them off as fixed.
It's a really nice clean way of handling exceptions and strikes me as far less intrusive/ghetto.
I also use Hoptoad and it seems to functions ok after a slightly fiddly setup (but then I need to log rake exceptions too).
However, if I hadn't won a Hoptoad account, I'd prefer to use Exceptional as it has much better notification options (such as SMS)
I use ExceptionLogger as well.
If you catch aka handle an exception it is really no longer an exception, you could as well log it yourself.
I've used Exceptional in the past, their offering is quite similar to Hoptoad.
I was pretty happy with Exceptional, apart from the UI upgrade they did about a year or two ago. I thought the previous layout was better for tracking errors.
Now considering Hoptoad for a project that includes an iPhone component, as they list iOS among supported platforms.
Related
We're writing an application which uses our http api, and occasionally it will encounter errors, failed connections, timeouts, etc.
We'd like to, at least in beta, be able to capture these incidents, and forward them to ourselves somehow. Since obviously many of these issues could be due to the actual connection being down, this would need to queue these incidents and send them when a connection is available.
I tried googling for an answer for this but to no avail, came across a bunch of solutions which catch exceptions, but not just random "incidents" (could really just be a string we log somewhere, we'd just include all the details in it).
Short of writing my own coredata (or something) backed queue, I'm at a loss at what a solution for this could be.
Does anyone know of any libs/services which could help with this?
You might want to look into Testflight, or less general purpose, Parse. Not quite sure, but maybe HockeyKit offers a solution for this, too.
You can take a look at Bugfender, it's a product we have built to solve this problem. We have found that while developing an app there are a lot of issues that are not crashes, so we decided to make our own product to help us on this.
It's easy to integrate and you can get the devices logs. Our service works offline and online, we have spent a lot of time to make it reliable and easy to use.
Compared to other products, you don't need any crash to get the logs, you choose when you want them.
I'm running a rails 3.0 application on Heroku and using the New Relic addon/service.
I have been looking at the transaction traces feature (available in the pro version) to understand a little more about the performance characteristics of the application. However, a significant portion of time (30-50%) is "uninstrumented time". After making a few stabs by putting method_tracers in some places and going through the reasonably slow cycle to test whether I get more info, I'm feeling this is going nowhere fast.
It seems in the PHP new relic agent they have a great feature to get very detailed traces without needing to guess where to put method tracers: http://newrelic.com/docs/php/php-agent-faq#top100
Is there anything similar to this for ruby?
Note: I'm already using rpm_contrib to get some more info and have garbage collection stats enabled. Also, this is not about fixing a performance problem, just understanding how to better use the performance tools available and scratch a niggling itch about that uninstrumented time.
There isn't currently anything similar for Ruby. I'll mention it to the Ruby engineer when I get a chance. My guess is unless a lot of requests come in for it, it won't be at the top of the list for a while, though. In the meantime, you can use the method tracers to figure out the uninstrumented time.
Hope that helps.
Method tracers can work well, but if you have a lot of code in your controller, try a binary search using trace_execution_scoped, which records the time spent in a block of code:
http://newrelic.github.com/rpm/NewRelic/Agent/MethodTracer/InstanceMethods/TraceExecutionScoped.html#method-i-trace_execution_scoped
Add a couple calls to this, give each metric a sensible name like "Custom/MySlowControllerAction/block0" (first argument to trace_execution_scoped), and repeat.
The metrics you name will show up not just in Transaction Traces, but also in the Performance Breakdown for the controller action under the Web Transactions tab, so you'll see average time in that block of code across all requests, not just the slow ones.
We have an exception that pops up on our website that is getting reported to BugzScout many times a day. The functionality that produces the exception still does what it's intended to do, so we just want to stop FogBugz from piling up all of these occurrences until we have a chance to dig into the issue and prevent the exception.
That said, is there a way to set up a filter on the FogBugz side of things to ignore a list of exceptions that get reported? I know I can set up some logic in our app's BugzScout class so it stops sending those messages, but it would be nice to know if FogBugz does this already before I put the time into building that filter locally. We are using the hosted FogBugz On Demand version of the product if that makes a difference.
In the BugzScout case itself, you should be able to set "Stop Reporting" for the Scout Will setting. This way, only the occurrences will increment when the exception is reported. The case will not reopen or notify anyone.
It sounds a bit from your description that there are many different exceptions reporting to the same ScoutDescription. As much as possible, you should use version numbers and exception line numbers to make sure that exceptions are reported separately. I can elaborate on this if you want.
I finishing up a web application and I'm trying to implement some logging. I've never seen any good examples of what to log. Is it just exceptions? Are there other things I should be logging? What type of information do you find useful for finding and fixing bugs.
Looking for some specific guidance and best practices.
Thanks
Follow up
If I'm logging exceptions what information specifically should I be logging? Should I be doing something more than _log.Error(ex.Message, ex); ?
Here is my logical breakdown of what can be logged within and application, why you might want to and how you might go about doing it. No matter what I would recommend using a logging framework such as log4net when implementing.
Exception Logging
When everything else has failed, this should not. It is a good idea to have a central means of capturing all unhanded exceptions. This shouldn't
be much harder then wrapping your entire application in a giant try/catch unless you are using more than on thread. The work doesn't end here
though because if you wait until the exception reaches you a lot of useful information would have gone out of scope. At the very least you should
try to collect specific pieces of the application state that are likely to help with debugging as the stack unwinds. Your application should always be prepared to produce this type of log output, especially in production. Make sure to take a look at ELMAH if you haven't already. I haven't tried it but I have heard great things
Application Logging
What I call application logs includes any log that captures information about what your application is doing on a conceptual level such as "Deleted Order" or "A User Signed On". This kind of information can be useful in analyzing trends, auditing the system, locking it down, testing, security and detecting bugs of coarse. It is probably a good idea to plan on leaving these logs on in production as well, perhaps at variable levels of granularity.
Trace Logging
Trace logging, to me, represents the most granular form of logging. At this level you focus less on what the application is doing and more on how it is doing it. This is one step above actually walking through the code line by line. It is probably most helpful in dealing with concurrency issues or anything for that matter which is hard to reproduce. You wouldn't want to always have this running, probably only turning it on when needed.
Lastly, as with so many other things that usually only get addressed at the very end, the best time to think about logging is at the beginning of a project so that the application can be designed with it in mind. Great question though!
Some things to log:
business actions, such as adding/deleting items. Talk to your app's business owner to come up with a list of things that are useful. These should make sense to the business, not to you (for example: when user submitted report, when user creates a new process, etc)
exceptions
exceptions
exceptions
Some things to NOT to log:
do not log information simply for tracking user usage. Use an analytics tool for that (which tracks the client in javascirpt, not in the client)
do not track passwords or hashes of passwords (huge security issue)
Maybe you should log page/resource accesses which are not yet defined in your application, but are requested by clients. That way, you may be able to find vulnerabilities.
It depends on the application and its audience. If you are managing sales or trading stocks, you probably should log more info than say a personal blog. When you need the log most is when an error is happening in your production environment, but can't reproduce it locally. Having log level and log hierarchy would help in such situations, because you can dynamically increase the log level. See log4j's documentation and log4net.
My few cents.
Besides using log severity and exceptions properly, consider structuring your log statements so that you could easily look though the log data in the future. For example - extracting meaningful info quickly, doing queries etc. There is no problem to generate an ocean of log data, the problem is to convert this data into information. So, structuring and defining it beforehand helps in later usage. If you use log4j, I would also suggest using mapped diagnostic context (MDC) - this helps a lot for tracking session contexts. Aside from trace and info, I would also use debug level where I usually keep temp. items. Those could be filtered out or disabled when not needed.
You probably shouldn't be thinking of this at this stage, rather, logging is helpful to consider at every stage of development to help diffuse potential bugs before they arise. Depending on your program, I would try to capture as much information as possible. Log everything. You can always stop logging certain components or processes if you don't reference that data enough. There is no such thing as too much information.
From my (limited) experience, if you don't want to make a specific error table for each possible error type, construct a generic database table that accepts general information as well as a string that you can populate with exception data, confirmation messages during successful yet important processes, etc. I've used a generic function with parameters for this.
You should also consider the ability to turn logging off if necessary.
Hope this helps.
I beleive when you log an exception you should also save current date and time, requested url, url refferer and user IP address.
I have just started using an exception logger (EurekaLog) in my (Delphi) application. Now my application sends me lots of error messages via e-mail every day. Here's what I found out so far
lots of duplicate errors
multiple mails from the same PC
While this is highly valuable input to improve my application, I am slightly overwhelmed by the sheer amount of information I'm getting.
What are your best practices for handling mails from your application?
If you get to much information as it is currently the case you are not getting any information at all.
So I would say categorize your errors into groups, like WARNINGS, FATAL ERRORS, etc.
Then limit your emails to the most important messages (FATAL).
Apart from that review your logs on a regular basis (day, week ...).
What I've done with my exception logging, which uses madExcept as the core, but my own transport mechanism, is have them all go into a database. The core information is all extracted from each report and put into fields, and the whole report is stored as well. The stack trace is automatically analysed to remove the un-interesting functions, leaving a list of only my functions that have failed.
With this happening automatically, I can now "ignore" each individual message coming in, but see the bigger picture in a grid that shows me simply which functions are having the most problems. I can then focus on them, look for the causes, and fix them.
My display app is also able to filter out reports in builds before a certain number if I choose, so that I can tell it not to include "MyWidget.BadProc" before build 75 once I've fixed it.
This has helped me improve my app, and hit the problems that people found most problematic, without having to guess.
It would very much depend on what the errors are that are being sent back. The obvious one being if there are errors in your application, they need fixing and patches/updates sent to your clients.
If they are exceptions that you know can happen and do not required you to be notified you can add "Exception Filters" in the Eureaka Log options to specify how they should be handled (or ignored!).
Another option is to use EurekaLog Variables (where you can add the exception description etc) in the mail Subject line and then use your email client to filter based on this.
I did this using madExcept. It's really useful for tracking down problems we couldn't reproduce ourselves.
Which makes me ask why you are getting so many? Untrapped exceptions should be few and far between. Especially if the user sees an error dialog. I was responsible for several applications, each with hundreds of installs and I would rarely get e-mail notices.
If they are mostly from a very small number of PCs, I'd work with some of those users to find out what they're doing differently, or how their setup might be generating exceptions.
If they are from all over the place, it's probably a bug that got through your testing.
Either way, use the details to fix your code or, at the very least, anticipate known exceptions and trap them properly (no empty try..except).
Fixing the hot spot problems will cut way down on the number of e-mails you get, making the occasional notice much more manageable.
I think that you should throw out all duplicates. Leave only count of the reports. I.e. if you get, say 100 reports, but there are only 4 unique problems - leave only 4 reports, throw out other 96 reports, but use their count to sort reports by severity. For example, 6 reports for fourth problem, 10 reports for third, 20 for second and 60 for first. So, you should fix first problem with 60 reports and only then switch to second.
I believe that EurekaLog has BugID in its reports. Same problem has the same BugID. This will allow you to sort reports with duplicates. EurekaLog Viewer also can sort out duplicates.