We have just launched a new MVC5 web site. The site uses Entity Framework for its data and also implements a couple of WebApi services for some simple AngularJS pages used on the web site.
The site has gone through development and testing without a problem, but now it is installed on an IIS 8.5 production server we are seeing the following entries in the IIS (WAS) event logs:
Here is first error:
A worker process serving application pool 'xxx' has requested a recycle
because it reached its private bytes memory limit.
Around 90 seconds later we see this error:
A worker process '4880' serving application pool 'xxx' failed to stop
a listener channel for protocol 'http' in the allotted time. The data
field contains the error number.
Which is immediately (the same time to the second) followed by a third error:
A process serving application pool 'xxx' exceeded time limits during
shut down. The process id was '4880'.
Finally, we see another Application Pool reccycle event:
A worker process serving application pool 'xxx' has requested a recycle
because it reached its private bytes memory limit.
We are currently seeing this problem approximately once per day and it does not seem to be related to site traffic/loading.
The reason we set the Application Pool to recycle on a Private Bytes consumption exceeded 4,194,304 KB (4 GB) - it normally (for perhaps 36 hours) sits at less than 1 GB, was because we had noticed that occasionally the Application Pools Private Memory consumption would increase linearly. Again we did not see this during development or local testing.
We have tried running load tests of several hundred concurrent users across the application, but have been unable to replicate this error sequence.
We have also run the application locally for extended periods of time with ReSharper's dotMemory profiler and memory snapshots do not reveal any problems.
Are there any tools/techniques available that we can run on the production server that would give us more information on what is happening?
Related
I have an ASP.NET MVC5 web application that is consuming excessive amounts of memory on our production server (excessive in that it increases over time and doesn't seem to stop until we recycle the application pool at ~8 GB, the max we've let it reach is 30 GB). The application does not perform like this on my dev machine or on our test server.
The production server is Windows Server 2012 R2 running IIS 8.5.9600.
I have written a small test tool which creates 50 concurrent threads and sends 1000 sequential requests each thread. In development the web application's memory stays around 400 MB, as it does in our test environment. On the production server, the memory just increases on and on. It doesn't matter what the endpoint does, I've just got it returning a vanilla .cshtml Razor view.
I've been trying figure out what could be causing the memory leak for some time and tried a few things:
Taking memory dumps and using a profiler (I've tried several different profilers). They all indicate that managed memory is within a reasonable amount (~100 - 200 MB), even on memory dumps of 8 GB!
Deploying a copy of the default "empty" ASP.NET MVC app generated by Visual Studio and running my test tool pointing to that. Same symptoms; memory is stable in dev and test environments but increases on the production server. I'm going to let it run for a while and see how high it goes, but so far it's 3 GB and climbing with each request.
The production server does have 96 GB of RAM, and from my understanding of how IIS uses this, it can get very greedy. But my dev machine has 32 GB and the max application pool size I've seen is around 600 MB and then it gets GC'd and reduces back to around ~400 MB.
What is taking up all the memory? Is this normal behaviour for IIS?
Update:
I've created a new VM server on Azure with similar specifications (112 GB RAM) and the memory stops at around 400 MB also. There must be something specific to our production server causing the problem.
In our case, it was our hosting provider who had installed some monitoring tool which used the .NET profiling API poorly.
I suggest anyone who is observing similar symptoms in their app to try configuring a new server instance.
When i work in dashboard (edit content items or settings) sometimes i am redirected to login page.
It happens only on a virtual hosting. There is memory limit of 1280 Mb. So, Sometimes, iis logger gives log event:
A worker process with process id of '49292' serving application pool '...' has requested a recycle because it reached its virtual memory limit.
A worker process serving application pool '...' has requested a recycle because it reached its private bytes memory limit.
I don't now if drop of authorization happens because of memory limit. But on my local machine with same limit and same log event all works fine.
How can i fix drop of authorization? And why does it may happen?
1280MB should be about 4 times what an Orchard instance needs under ordinary circumstances, so the first thing to do is probably to take a memory profile and find out what's eating so much memory. You very likely have a memory leak somewhere. This is the problem you should be focusing on: appdomain restarts are expensive and leave your application unresponsive for seconds.
Now yes, that still shouldn't drop authorization, which makes me think that you also have a misconfigured server. Most likely, you didn't configure a machine key, causing a new one to be generated at each restart, which makes existing authentication tokens invalid.
But really, your real problem is that memory footprint.
Yesterday I got a trial account on webhosting.net's Jelastic v2.2.2 and configured an environment with a minimum of 0 cloudlets (max 8, i.e., all dynamic, no reserved). Then I deployed a Grails war which was using 3 cloudlets after it started up (around 350 MB). It worked great, and I was very impressed.
However, I did not access my app overnight, and the billing history shows it kept using 3 dynamic cloudlets every hour, even with 0 requests (i.e., 0 MB paid traffic) for 14 hours. Is there some way I can get my Jelastic environment to sleep (i.e., hibernation) after some period with no requests (e.g., after an hour or two)? Then, when it gets a request, I'd like it to automatically wake up (i.e., allocate some cloudlets and restore memory from disk). I see how to stop and restart it manually, but I would like it to work automatically, for any requester.
edit: I found the following documentation, but does it not work for Tomcat/Grails?
Hibernation
Jelastic’s hibernation feature delivers even better utilization of cluster resources. Optimal use of resources is achieved by suspending non-active containers and returning released resources back to the cluster.
Because they are in sleep mode, hibernated containers do not consume resources (only disk space). As a result you save money while your containers are in hibernate mode. If applications are needed again the platform returns them to a running state again in just a few seconds.
It takes a little time to awaken your environment from sleep, so it's not suitable to work how you describe for production use - you would effectively lose visitors because it would seem like your service is offline due to the delays for that first access.
For that reason the 'sleep' function is only active for trial accounts, and the inactivity time before sleep is set by the hosting provider (so you should contact them directly for help on that point).
Of course you should also remember that accesses from search engine spiders etc. may keep your environment awake.
I have an ASP.NET MVC application running on IIS6 with enabled wildcard mapping. After performing some load tests I digged into log files with focus on the slow requests. I have a log file from the load testing application, IIS log file and the log file from the IHttpModule I develop for this purpose which records time of Application.BeginRequest and Application.EndRequest.
When I compared data for slow requests in the IIS log file and the log file of my module I've discovered some odd behavior.
var request_processing_start = iis_log_file.time - iis_log_file.taken
var aspnet_processing_start = my_module_log_file.begin_request_called
var aspnet_processing_end = my_module_log_file.end_request_called
var request_processing_end = iis_log_file.time
For all requests values of aspnet_processing_end and request_processing_end match pretty close (difference no more then few milliseconds). However some requests have timespan between request_processing_start and aspnet_processing_start up to 30 seconds. What is the cause of this lag and a how can I prevent it?
Some more details of load test I've done:
Number of requests: on long time average 2 per seconds, in the peaks no more then 20 per second. (This is really low, we cannot blame wildcard mapping)
Processor usage: flat line bellow 5% (We cannot blame lack of cpu power)
Free memory: more than 1GB (Memory isn't also issue).
The lag I described is occurring only when the server is under load. When the load testing application isn't running it cannot be reproduced.
I have an Umbraco website that I have to restart every morning in order for the users to be able to publish content. Is there any solutions available that will help me get around doing this each morning?
1 - Document why do you "have to" restart IIS every morning
like the web app can't re-establish connection with SQL
or one process gets so huge that it's obvious it's leaking
or one process heats up with huge CPU usage and IIS keeps dropping requests
etc. etc. have to check log files, EventLog, SQL Server has it's own log
2 - Document usage patters of the site
like does it sit idle for 8-10 h or is busy all night
if it's busy then log files (including IIS log) will provide some info on when a problem started
if it's idle for a long time, check that AppPool for the site has automatic recycling of worker process set say after 1h of inactivity - you can also set diferent recycling tactics
if it's SQL connection after along idle period - Kerberos ticket for the account expired.
you do have a domain account under which that AppPool runs I hope
to fix that, look at DB connection string (normally in web.config) and check MSDN for params
or bring up a new web site or app that's going to keep pinging a web method which will just do a little query ( like a count on some table) and return the result as a kind of admin heartbeat -- this helps only if you acsually see SQL connection issue
3 - Check if you have multiple sites / web apps running on the server
that each has it's own AppPool and that they run under a domain account
that each app has it's own, separate folder for logs and any other writable files
that each AppPool has recycling tactics that's good for actual usage pattern
needs different recycling tactics if it's busy all the time
ask sor some mininal kind of heartbeat web service to be developed and pinged for ops needs
running as part of each web app and using the same SQL connection
if you don't have the budget for this raise some hell
makes you feel good :-)