We have the following issue in our application hosted in IIS.
Our website performance is going drastically down when the concurrent users hit count increased to 100s. (Our application is hosted in the IIS version 8.5)
Server - Windows server 2008
IIS - 8.5
Application - ASP.NET MVC/Angular
DB - SQL server 2016
But the speed is resumed back when we restart the Application pool on which the application is running.
When the application performance is going down, the following attributes were observed and they look normal.
1) CPU usage - the max was at 70% when the peak load hits.
2) Memory usage - the server memory usage was not more that 40%
3) Checked if there any deadlocks in the DB and looked normal
4) Verified the code and connections were properly handled using 'using statement'
Please let me know what exactly could causing the slowness when the load increase and why its addressed while restarting the app pool.
I know for this scenario, I cant give code sample and need immediate guidance from the experts. Please let me know if you need more details to help.
Thank you!!!
Related
I have an ASP.NET MVC5 web application that is consuming excessive amounts of memory on our production server (excessive in that it increases over time and doesn't seem to stop until we recycle the application pool at ~8 GB, the max we've let it reach is 30 GB). The application does not perform like this on my dev machine or on our test server.
The production server is Windows Server 2012 R2 running IIS 8.5.9600.
I have written a small test tool which creates 50 concurrent threads and sends 1000 sequential requests each thread. In development the web application's memory stays around 400 MB, as it does in our test environment. On the production server, the memory just increases on and on. It doesn't matter what the endpoint does, I've just got it returning a vanilla .cshtml Razor view.
I've been trying figure out what could be causing the memory leak for some time and tried a few things:
Taking memory dumps and using a profiler (I've tried several different profilers). They all indicate that managed memory is within a reasonable amount (~100 - 200 MB), even on memory dumps of 8 GB!
Deploying a copy of the default "empty" ASP.NET MVC app generated by Visual Studio and running my test tool pointing to that. Same symptoms; memory is stable in dev and test environments but increases on the production server. I'm going to let it run for a while and see how high it goes, but so far it's 3 GB and climbing with each request.
The production server does have 96 GB of RAM, and from my understanding of how IIS uses this, it can get very greedy. But my dev machine has 32 GB and the max application pool size I've seen is around 600 MB and then it gets GC'd and reduces back to around ~400 MB.
What is taking up all the memory? Is this normal behaviour for IIS?
Update:
I've created a new VM server on Azure with similar specifications (112 GB RAM) and the memory stops at around 400 MB also. There must be something specific to our production server causing the problem.
In our case, it was our hosting provider who had installed some monitoring tool which used the .NET profiling API poorly.
I suggest anyone who is observing similar symptoms in their app to try configuring a new server instance.
I have a MVC application that is running on a hosted server. I have an issue that the host is forcing app pool recycles. They say that my RAM is limited to 128mb. While this isn't very much I'm not so sure that my app is causing this - problem is I need to understand how it works!
My app runs at about 25mb with no significant memory leak.
What I don't really understand is how this can get up to 128mb very quickly - seems to be a problem when an admin user is logged in - the user identity is wiped almost immediately.
How does memory usage fluctuate with number of users on the site?
Thanks for any help
I have a webserver with 10 mvc sites that I have created. All sites "link" to the same code in a folder. So all 10 sites run on same code. This is done have different languages on the site.
My problem is that 1 site goes up to use 40-60% of the CPU for a few minutes. Then it drops down and another sites rises in CPU. Normally the sites use 300mb in w3wp process, but sometimes rises to 8-900mb!
There's 20GB of memory, 18,9 is constantly used. There is also a SQL server running that takes 13GB.
The site is a MVC 3 site. Uses caching on the SQL data.
Every site has its own pool.
I know its a very vague question. But can anyone see any obvious problems with this setup?
Why does the CPU rise periodically?
There are about constantly 100-150 users on the sites.
Any ideas or thought would be very appreciated.
CPU utilization is result of web application so application developers would know best where is the problem. If application developers aren't "in-house" you as administrator will have to find is there problem in setup (less likely) or in application (more likely in my opinion).
Since IIS NEED tool from Microsoft like IIS Tracer, debugging high CPU utilization with "DebugDiag & friends" require lots of knowledge and effort.
Here is one nice article how to troubleshoot high CPU usage
Do your web apps use caching ?
OutputCache duration="180" varybyparam="*"
Sorry if this isn't helpful, but I'm administrator and when IIS server have high CPU usage in 95% problem is in poor SQL queries or bad programming which we fix after analyzing source code.
I run a social community site for card players. I currently have 7,000+ members and getting 2,000 visitors/15k+ pageviews a day. Recently the site has started to really slow down during peak hours of the day and I am starting to think my site needs some serious performance optimizations in the code and settings. I really don't want to purchase a second server to run the site as I am pretty sure my current server should be able to handle this kind of load easily.
During peak hours, when the pages load, they still load very quickly. The problem is that a lot of times it will timeout and give a "website not available" error in the browser. Then you refresh it and it loads up quickly. Then a couple of pageviews later it will do it again. My CPU and RAM usage do not even get very high during these times, so I must believe it is in my IIS settings or something. I have done some searching and cannot find any good answers or ideas of what a fix could be.
Here are some stats of my setup:
ASP.NET MVC 2 w/ Output Caching and Partial View caching
IIS 7
Windows Web Server 2008 RC2 64-Bit
AMD Athlon II X2
4GB of RAM
My heavier pages on the site have quite a bit of database reads and a lot of image requests. I am not sure if this is the problem, because when a page does load it is VERY fast.
I did purchase a new server I am building and was thinking about switching everything to this instead. The new server I just got is gonna run an Intel Xeon X3430 2.4GHz Quad-Core w/ HT and 8GB RAM.
I am looking for a few possible things I could look into for this problem and if there are any possible solutions or settings I could implement to stop the "website not available" messages and also help my server handle future traffic increases as the site grows. Would upgrading the server to this new one make the difference?
It looks like this is more of an IIS issue than your code or hardware. There is a default setting for max concurrent connections per cpu and queue length that you may be reaching.
See Optimising IIS Performance and someone with a similar problem (and resolution).
I have a website that is hanging every 5 or 10 requests. When it works, it works fast, but if you leave the browser sit for a couple minutes and then click a link, it just hangs without responding. The user has to push refresh a few times in the browser and then it runs fast again.
I'm running .NET 3.5, ASP.NET MVC 1.0 on IIS 7.0 (Windows Server 2008). The web app connects to a SQLServer 2005 DB that is running locally on the same instance. The DB has about 300 Megs of RAM and the rest is free for web requests I presume.
It's hosted on GoGrid's cloud servers, and this instance has 1GB of RAM and 1 Core. I realize that's not much, but currently I'm the only one using the site, and I still receive these hangs.
I know it's a difficult thing to troubleshoot, but I was hoping that someone could point me in the right direction as to possible IIS configuration problems, or what the "rough" average hardware requirements would be using these technologies per 1000 users, etc. Maybe for a webserver the minimum I should have is 2 cores so that if it's busy you still get a response. Or maybe the slashdot people are right and I'm an idiot for using Windows period, lol. In my experience though, it's usually MY algorithm/configuration error and not the underlying technology's fault.
Any insights are appreciated.
What diagnistics are available to you? Can you tell what happens when the user first hits the button? Does your application see that request, and then take ages to process it, or is there a delay and then your app gets going and works as quickly as ever? Or does that first request just get lost completely?
My guess is that there's some kind of paging going on, I beleive that Windows tends to have a habit of putting non-recently used apps out of the way and then paging them back in. Is that happening to your app, or the DB, or both?
As an experiment - what happens if you have a sneekly little "howAreYou" page in your app. Does the tiniest possible amount of work, such as getting a use count from the db and displaying it. Have a little monitor client hit that page every minute or so. Measure Performance over time. Spikes? Consistency? Does the very presence of activity maintain your applicaition's presence and prevent paging?
Another idea: do you rely on any caching? Do you have any kind of aging on that cache?
Your application pool may be shutting down because of inactivity. There is an Idle Time-out setting per pool, in minutes (it's under the pool's Advanced Settings - Process Model). It will take some time for the application to start again once it shuts down.
Of course, it might just be the virtualization like others suggested, but this is worth a shot.
Is the site getting significant traffic? If so I'd look for poorly-optimized queries or queries that are being looped.
Your configuration sounds fine assuming your overall traffic is relatively low.
To many data base connections without being release?
Connecting some service/component that is causing timeout?
Bad resource release?
Network traffic?
Looping queries or in code logic?