I have an ASP.NET MVC5 web application that is consuming excessive amounts of memory on our production server (excessive in that it increases over time and doesn't seem to stop until we recycle the application pool at ~8 GB, the max we've let it reach is 30 GB). The application does not perform like this on my dev machine or on our test server.
The production server is Windows Server 2012 R2 running IIS 8.5.9600.
I have written a small test tool which creates 50 concurrent threads and sends 1000 sequential requests each thread. In development the web application's memory stays around 400 MB, as it does in our test environment. On the production server, the memory just increases on and on. It doesn't matter what the endpoint does, I've just got it returning a vanilla .cshtml Razor view.
I've been trying figure out what could be causing the memory leak for some time and tried a few things:
Taking memory dumps and using a profiler (I've tried several different profilers). They all indicate that managed memory is within a reasonable amount (~100 - 200 MB), even on memory dumps of 8 GB!
Deploying a copy of the default "empty" ASP.NET MVC app generated by Visual Studio and running my test tool pointing to that. Same symptoms; memory is stable in dev and test environments but increases on the production server. I'm going to let it run for a while and see how high it goes, but so far it's 3 GB and climbing with each request.
The production server does have 96 GB of RAM, and from my understanding of how IIS uses this, it can get very greedy. But my dev machine has 32 GB and the max application pool size I've seen is around 600 MB and then it gets GC'd and reduces back to around ~400 MB.
What is taking up all the memory? Is this normal behaviour for IIS?
Update:
I've created a new VM server on Azure with similar specifications (112 GB RAM) and the memory stops at around 400 MB also. There must be something specific to our production server causing the problem.
In our case, it was our hosting provider who had installed some monitoring tool which used the .NET profiling API poorly.
I suggest anyone who is observing similar symptoms in their app to try configuring a new server instance.
Related
We have the following issue in our application hosted in IIS.
Our website performance is going drastically down when the concurrent users hit count increased to 100s. (Our application is hosted in the IIS version 8.5)
Server - Windows server 2008
IIS - 8.5
Application - ASP.NET MVC/Angular
DB - SQL server 2016
But the speed is resumed back when we restart the Application pool on which the application is running.
When the application performance is going down, the following attributes were observed and they look normal.
1) CPU usage - the max was at 70% when the peak load hits.
2) Memory usage - the server memory usage was not more that 40%
3) Checked if there any deadlocks in the DB and looked normal
4) Verified the code and connections were properly handled using 'using statement'
Please let me know what exactly could causing the slowness when the load increase and why its addressed while restarting the app pool.
I know for this scenario, I cant give code sample and need immediate guidance from the experts. Please let me know if you need more details to help.
Thank you!!!
We have just launched a new MVC5 web site. The site uses Entity Framework for its data and also implements a couple of WebApi services for some simple AngularJS pages used on the web site.
The site has gone through development and testing without a problem, but now it is installed on an IIS 8.5 production server we are seeing the following entries in the IIS (WAS) event logs:
Here is first error:
A worker process serving application pool 'xxx' has requested a recycle
because it reached its private bytes memory limit.
Around 90 seconds later we see this error:
A worker process '4880' serving application pool 'xxx' failed to stop
a listener channel for protocol 'http' in the allotted time. The data
field contains the error number.
Which is immediately (the same time to the second) followed by a third error:
A process serving application pool 'xxx' exceeded time limits during
shut down. The process id was '4880'.
Finally, we see another Application Pool reccycle event:
A worker process serving application pool 'xxx' has requested a recycle
because it reached its private bytes memory limit.
We are currently seeing this problem approximately once per day and it does not seem to be related to site traffic/loading.
The reason we set the Application Pool to recycle on a Private Bytes consumption exceeded 4,194,304 KB (4 GB) - it normally (for perhaps 36 hours) sits at less than 1 GB, was because we had noticed that occasionally the Application Pools Private Memory consumption would increase linearly. Again we did not see this during development or local testing.
We have tried running load tests of several hundred concurrent users across the application, but have been unable to replicate this error sequence.
We have also run the application locally for extended periods of time with ReSharper's dotMemory profiler and memory snapshots do not reveal any problems.
Are there any tools/techniques available that we can run on the production server that would give us more information on what is happening?
I'm developing an API on Appfog and want to know what to focus on (more memory with one instances or more instances with lower memory).
Appfog gives you free 2GB of RAM and up to 16 instances if each instances get 128 MB RAM.
My application uses PHP, MySql and Memcachier.
I want to launch it soon and want to know which configuration is best for my server.
What is the benefit with more RAM or instances?
Thanks for helping :)
Best Regards,
Johnny
You want as many instances as your app will run without running out of memory :). More instances means better performance and uptime. However, if an instance runs out of memory it will be shut down leaving your app running with fewer instances until they all collapse. You can diagnose this problem with the af apps and af logs <appname> --all commands. If the app is running at < 100% regularly then the instance memory budget may be too low. When there are down instances, the logs command may reveal memory limit reached errors.
Memory Recommendations
Here are some memory recommendations to start out with: Wordpress with several installed plugins will need > 512mb to be stable. For lean custom PHP apps 128mb is usually sufficient but should be watched. If an app is using a framework try 256mb. These memory limits may seem high but it's really the peak memory usage not the average usage.
Load Test
Load testing using Seige can help find a memory / instance balance. It does this by determining if your app is peaking out over the memory limit. Scale the app down to 1 instance and siege with 5, 10, and 15 concurrent connections progressively increasing by 5 until the app falls over. If the app does stop, bump the memory up and try again.
I have a webserver with 10 mvc sites that I have created. All sites "link" to the same code in a folder. So all 10 sites run on same code. This is done have different languages on the site.
My problem is that 1 site goes up to use 40-60% of the CPU for a few minutes. Then it drops down and another sites rises in CPU. Normally the sites use 300mb in w3wp process, but sometimes rises to 8-900mb!
There's 20GB of memory, 18,9 is constantly used. There is also a SQL server running that takes 13GB.
The site is a MVC 3 site. Uses caching on the SQL data.
Every site has its own pool.
I know its a very vague question. But can anyone see any obvious problems with this setup?
Why does the CPU rise periodically?
There are about constantly 100-150 users on the sites.
Any ideas or thought would be very appreciated.
CPU utilization is result of web application so application developers would know best where is the problem. If application developers aren't "in-house" you as administrator will have to find is there problem in setup (less likely) or in application (more likely in my opinion).
Since IIS NEED tool from Microsoft like IIS Tracer, debugging high CPU utilization with "DebugDiag & friends" require lots of knowledge and effort.
Here is one nice article how to troubleshoot high CPU usage
Do your web apps use caching ?
OutputCache duration="180" varybyparam="*"
Sorry if this isn't helpful, but I'm administrator and when IIS server have high CPU usage in 95% problem is in poor SQL queries or bad programming which we fix after analyzing source code.
I run a social community site for card players. I currently have 7,000+ members and getting 2,000 visitors/15k+ pageviews a day. Recently the site has started to really slow down during peak hours of the day and I am starting to think my site needs some serious performance optimizations in the code and settings. I really don't want to purchase a second server to run the site as I am pretty sure my current server should be able to handle this kind of load easily.
During peak hours, when the pages load, they still load very quickly. The problem is that a lot of times it will timeout and give a "website not available" error in the browser. Then you refresh it and it loads up quickly. Then a couple of pageviews later it will do it again. My CPU and RAM usage do not even get very high during these times, so I must believe it is in my IIS settings or something. I have done some searching and cannot find any good answers or ideas of what a fix could be.
Here are some stats of my setup:
ASP.NET MVC 2 w/ Output Caching and Partial View caching
IIS 7
Windows Web Server 2008 RC2 64-Bit
AMD Athlon II X2
4GB of RAM
My heavier pages on the site have quite a bit of database reads and a lot of image requests. I am not sure if this is the problem, because when a page does load it is VERY fast.
I did purchase a new server I am building and was thinking about switching everything to this instead. The new server I just got is gonna run an Intel Xeon X3430 2.4GHz Quad-Core w/ HT and 8GB RAM.
I am looking for a few possible things I could look into for this problem and if there are any possible solutions or settings I could implement to stop the "website not available" messages and also help my server handle future traffic increases as the site grows. Would upgrading the server to this new one make the difference?
It looks like this is more of an IIS issue than your code or hardware. There is a default setting for max concurrent connections per cpu and queue length that you may be reaching.
See Optimising IIS Performance and someone with a similar problem (and resolution).