ASP.NET MVC Application crashing randomly - asp.net-mvc

Question

I created a MVC 2 Application to work as a RSS Feeder and deliver News Contents for a number of applications to consume.
Everything was doing fine until yesterday when suddenly my application started throwing a random error which doesn't tell me much of what's going on (or at least I don't understand it).
This error only occurs in Production and cannot be reproduced in staging or my local machine.
Here's the stack trace:
Error executing child request for handler
'System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper'.errorPath:/Android/Edition/2011-11-22/P1
HostIP:##.##.##.## at
System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler,
TextWriter writer, Boolean preserveForm, Boolean setPreviousPage,
VirtualPath path, VirtualPath filePath, String physPath, Exception
error, String queryStringOverride) at
System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter
writer, Boolean preserveForm, Boolean setPreviousPage) at
System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter
writer, Boolean preserveForm) at
System.Web.HttpServerUtilityWrapper.Execute(IHttpHandler handler,
TextWriter writer, Boolean preserveForm) at
System.Web.Mvc.ViewPage.RenderView(ViewContext viewContext) at
System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
at System.Web.Mvc.ControllerActionInvoker.<>c_DisplayClass14.b_11()
at
System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter
filter, ResultExecutingContext preContext, Func1 continuation) at
System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter
filter, ResultExecutingContext preContext, Func1 continuation) at
System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext
controllerContext, IList1 filters, ActionResult actionResult) at
System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext
controllerContext, String actionName) at
System.Web.Mvc.Controller.ExecuteCore() at
System.Web.Mvc.MvcHandler.<>c__DisplayClass8.b__4() at
System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass1.b__0() at
System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass81.b__7(IAsyncResult
_) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.End() at
System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
at
System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step,
Boolean& completedSynchronously)
This error only occurs sometimes and can throw at any action of any controller.
I would like to have found the reason in a specific action but it only throws sometimes and can throw for any action.

Answer 1

Such byzantine errors are hard to debug. The stack trace you're seeing does not help in diagnosing the error.
First, you should improve error logging. In your global.asax, create an implementation of the Application_Error hook that logs the exception and all inner exceptions to a file (I wouldn't log this to the database, because the db connection could be the culprit). Make sure that this code is solid: it should focus on these critical errors and not log every 404 page. Also, make sure that the logging code itself does not create any problems (it should be highly error tolerant).
The cause for this kind of problems is usually access to some kind of static variables. Out of all keywords, I believe static is by far the most dangerous one because it is so subtle.
Some common errors I've seen.
Caching Someone wanted to be smart and cache some data in a static dictionary or so. Unfortunately, the locking code is flawed. An exception occurs only if the code for some user tries to add sth. to the cache, but it's already there:
if(_dict.ContainsKey(cacheKey) == false)
{
// second thread adds data to the dictionary here
_dict.Add(cacheKey, cacheData); // exception
}
This can also happen in a 3rd party library that uses caching or pooling. Access static variables with care.
Uncommon Code Paths Something unusual happens that calls code that is not called very often, and that code is flawed. If you have testing with high code coverage, check the spots that are not covered by the tests.
DB Connection Lost A socket reset on the db connection could lead to exceptions. Many database connection libraries / drivers fix this quickly (i.e. the next request will be fine). This is hard to handle; basically it shouldn't happen but there are many reasons why it could occur.

Answer 2

This error tells you, that the true error is somewhere in some child action, that is action rendered by Html.RenderAction().
There is no such thing as "random" error - there has to be scenario where it happens, you just dont know it because the conditions vary maybe.
You should identify all your child actions, and test them with various inputs.. after you identify the problematic input set, try disabling child actions one by one and find the error this way.