I have a MVC3, .NET4.5 asp.net web application hosted on Azure Websites.
I am experimenting with "Free", "Shared" and "Standard" scaling configurations.
I have noticed that after a period of inactivity the compiled code get dropped from memory, or the app pool gets recycled forcing a JIT recompile.
My main question is what is time period before the compiled code gets dropped forcing a recompile? I assume this is as a result of the application pool recycling? I have come across this on standard shared hosts such as DiscountASP.
My second question is: What is the best approach to minimise this issue as I would not like my users bumping into this recompilation lag? My initial thoughts are precompilation.
Many thanks in advance.
EDIT:
I have a found a related SO post on this here: App pool timeout for azure web sites
However it seems, as like standard Shared hosting, one cannot change App Pool recycling. One has more flexibility with the "Standard" scale option, since it is dedicated. So the likely options at present are:
1) Precompilation
2) Use of "Keep alive" ping sites.
EDIT2:
1) "Keep Alive" approach seems to be working. I have a 10 minute monitor running.
I believe the inactivity period is 20 minutes by default. I haven't used web sites yet so I'm not famailiar with rescrtictions on changing settings but one quick way to keep your site activie is to use a uptime monitoring service like Pingdom (you can check one site for free at time of writing), this will ping your site regularly and prevent it from becoming idle.
Related
There's a similar question about app harbor on StackOverflow, but the user didn't try to use new relic to overcome the problem.
I deployed my ASP.NET MVC project on App Harbor. It's very easy to configure and you can even set automatic deployments from Git. However, as my website is still mainly used only by me, I was getting very long cold starts (over 15 secs). To avoid it, I installed New Relic. The idea was to simultaneously to monitor the application but also to create periodic pings that, according to "a lot of people", would drastically reduce the loading time.
It's not working. I have New Relic correctly pinging my application every minute, but I still get very long cold starts. For instance, 5 min ago, I've got a cold start of 16 seconds. 1 minute after, I got the page loaded in less than a second.
I know I could have used Pingdom or StillAlive to achieve the same result:
How do I improve app performance on AppHarbor?
I wouldn't like to do it because I like New Relic and I don't want to have a lot of add-on's on app harbor as they will slow down my website. Do you have any idea what might be causing it?
I'm not familiar with AppHarbor's setup. But if it's using IIS, the pinging is just keeping the application pool from reaching the idle timeout. But there the default IIS setting for the application pool to be recycled every 29 hours no matter the number of requests. And it's normally in the best interest to let it recycle once in a while, so working around it may not be in your best interest.
Your best bet is to reduce the number of things happening on application start. Precompiling your views is a good place to start. And heck, Stack Exchange/Stack Overflow precompiles views to avoid the application start up cost.
Currently after a build/deployment of our app (58 projects, large asp.net MVC 3 front end) takes ~15-20secs to load as it goes through the whole 'recycling the app pool' (release configuration).
We do have a web farm if that alters people's answers, but the question really is:
What are people doing in large scale applications where a maintenance window isn't viable (we're a 24/7 very active website) to minimize that initial 'first hit' on the app pool recycle after a deploy?
We've used a number of tools to analyze that startup time and there doesn't really seem to be any way to bring it down so what I'm looking for are what techniques do people employ in order to minimize the impact of a large application deploy affecting users.
By default - if you change 15 files in an ASP.NET application at once (even via FTP) then the app pool is automatically recycled. You can change the number of files but as soon as web.config and bin files are changed then it needs to recycle. So in my opinion the ideal solution for an environment like yours would be as follows:
4 web servers (this is an arbitrary number)
each server has a status.aspx that the load balancer looks at - use TeamCity to take 2 of these servers "off line" (off the load balancer) and wait 20 seconds for the traffic to filter across. A distributed cache will help keep user experience problems
Use TeamCity to deploy to those 2 servers - run your automated tests etc. and once you are happy put those back into the farm and take the other 2 offline and deploy to those
This can all be scripted / automated. The only issue with this is any schema changes that are not backwards compatible may not allow running the new version site in parallel with old version of the site for the 20 seconds for the load balancer to kick back in
This is good old fashioned Canary Releasing - there are some patterns here http://continuousdelivery.com/patterns/ to help take into consideration. Id also suggest a copy of that continuous delivery book - its like a continuous delivery bible and has got me out of a few situations :)
At the very base you could run a tinyget script against the application after completion of deployment which will "warm up" the application however if a customer hits your site before the script can run, they will still face a delay. What do you currently have in place, what post deployment steps do you have in place?
In a farm environment you could stage deployments too, so take one server out of load balance, update it and then bring that online after deployment and take the other out, complete the deployment and then reintroduce into the farm. How is your SQL Server setup - clustered?
copy and paste from my post here
We operate a Blue/Green deployment strategy on a 4 tier architecture which has a web site over 4 servers at the top tier. Due to the complexity the architecture introduced for deployments, we needed a way to deploy without disturbing any traffic to the "live" site. Following Fowler's advice, but not quite in the same way, we came up with a solution that means we have 2 sites on each server (a blue and a green, or in our case site A and site B). The live site has the appropriate host header, and once we have deployed and tested to the non-live site, we then flip the headers of the 2 sites so that what was once live is now the non-live site, and vice-versa. The effect is, a robust deployment that can be done in business hours and with the highest level of confidence.
This of course complicates your configuration and deployment slightly, but it's worth the effort. I guess it kind of goes without saying that you want to script both the deployment, and the host header swapping.
Firstly, unless you're running Google or something bigger, does a 15-20s load time at 3am for a handful of users really impact that much? I'd say the effort invested in eliminating the occasional lag would far outweigh the 15-20s inconvenience of a couple of users.
I consider it a necessary evil of using ASP.NET unfortunately. Using a pre-compiled site (.DLLs instead of the code-behind files) will lessen the time but not necessarily eliminate it.
The best thing you can do is use something like a status notification bar to warn users they may experience some "issues" during "essential maintenance".
But even then, I'd say in terms of user experience it'd be better to keep quiet and have a handful of people blame their "slow internet" when your site takes 20s to load on one occasion, than announce to all and sundry that it will be slow.
You can also try this approach : http://weblogs.asp.net/scottgu/archive/2009/09/15/auto-start-asp-net-applications-vs-2010-and-net-4-0-series.aspx
without knowing anything about your site, my first thought is that you might be able to break it down into smaller sites so that they start faster individually.
second, with your web farm, i assume you have some sort of load balancing device in front of that from which you can pull machines out of the pool when they are being deployed. don't put them back in the pool until after you have sent a request against the site to get it started up. you should be able to script this such that you are pretty much clicking a button that takes a machine out, deploys to it, and sends a request after it's back up and happy.
You can consider using aspnet_compiler.exe to precompile your application, because I think the delay after deployment is caused by the compilation phase rather than "whole recycling the app pool".
I have an Umbraco website that I have to restart every morning in order for the users to be able to publish content. Is there any solutions available that will help me get around doing this each morning?
1 - Document why do you "have to" restart IIS every morning
like the web app can't re-establish connection with SQL
or one process gets so huge that it's obvious it's leaking
or one process heats up with huge CPU usage and IIS keeps dropping requests
etc. etc. have to check log files, EventLog, SQL Server has it's own log
2 - Document usage patters of the site
like does it sit idle for 8-10 h or is busy all night
if it's busy then log files (including IIS log) will provide some info on when a problem started
if it's idle for a long time, check that AppPool for the site has automatic recycling of worker process set say after 1h of inactivity - you can also set diferent recycling tactics
if it's SQL connection after along idle period - Kerberos ticket for the account expired.
you do have a domain account under which that AppPool runs I hope
to fix that, look at DB connection string (normally in web.config) and check MSDN for params
or bring up a new web site or app that's going to keep pinging a web method which will just do a little query ( like a count on some table) and return the result as a kind of admin heartbeat -- this helps only if you acsually see SQL connection issue
3 - Check if you have multiple sites / web apps running on the server
that each has it's own AppPool and that they run under a domain account
that each app has it's own, separate folder for logs and any other writable files
that each AppPool has recycling tactics that's good for actual usage pattern
needs different recycling tactics if it's busy all the time
ask sor some mininal kind of heartbeat web service to be developed and pinged for ops needs
running as part of each web app and using the same SQL connection
if you don't have the budget for this raise some hell
makes you feel good :-)
I have a project in asp.net mvc, my hosting is using IIS6, and the first request after the website sit's idle is very slow.
I looked at, http://forums.asp.net/t/1418959.aspx and asked the hosting for this settings.
They say that the actual settings are:
"The pool is set with Idle Timeout disabled, Rapid-fail enabled and with a single worker process."
But still slow at the first request. Do you have any other clues?
Thanks in advance,
Alfredo
You are probably a victim of worker process recycling. Ask your host how often the worker processes are recyled.
When a worker process is recycled, it has to recompile and restart the entire web application, and that's what causes the slowdown.
This is natural.
IIS is often configured to shut down the website if it's a certain age or if there hasn't been a request in awhile. Your website has to be loaded (and possibly compiled) when the first request comes after asp.net has been shut down by IIS.
The common solution is to precompile your website before publishing it to the server.
Just a guess, but perhaps you are caching some data, that needs to be refreshed after the site has been idle for some time ?
If this is not the case, then my guess would be that the worker process has been shut down for some reason (it could be for some other reason than the idle timeout in IIS). If you need to check whether this might be the case, you could add some code to the Application_Start event that logs the startup event to a file or whatever logging you have in place. After some time in operation, you can examine the logs and see, how many Application_Start events has occured.
I have a website that is hanging every 5 or 10 requests. When it works, it works fast, but if you leave the browser sit for a couple minutes and then click a link, it just hangs without responding. The user has to push refresh a few times in the browser and then it runs fast again.
I'm running .NET 3.5, ASP.NET MVC 1.0 on IIS 7.0 (Windows Server 2008). The web app connects to a SQLServer 2005 DB that is running locally on the same instance. The DB has about 300 Megs of RAM and the rest is free for web requests I presume.
It's hosted on GoGrid's cloud servers, and this instance has 1GB of RAM and 1 Core. I realize that's not much, but currently I'm the only one using the site, and I still receive these hangs.
I know it's a difficult thing to troubleshoot, but I was hoping that someone could point me in the right direction as to possible IIS configuration problems, or what the "rough" average hardware requirements would be using these technologies per 1000 users, etc. Maybe for a webserver the minimum I should have is 2 cores so that if it's busy you still get a response. Or maybe the slashdot people are right and I'm an idiot for using Windows period, lol. In my experience though, it's usually MY algorithm/configuration error and not the underlying technology's fault.
Any insights are appreciated.
What diagnistics are available to you? Can you tell what happens when the user first hits the button? Does your application see that request, and then take ages to process it, or is there a delay and then your app gets going and works as quickly as ever? Or does that first request just get lost completely?
My guess is that there's some kind of paging going on, I beleive that Windows tends to have a habit of putting non-recently used apps out of the way and then paging them back in. Is that happening to your app, or the DB, or both?
As an experiment - what happens if you have a sneekly little "howAreYou" page in your app. Does the tiniest possible amount of work, such as getting a use count from the db and displaying it. Have a little monitor client hit that page every minute or so. Measure Performance over time. Spikes? Consistency? Does the very presence of activity maintain your applicaition's presence and prevent paging?
Another idea: do you rely on any caching? Do you have any kind of aging on that cache?
Your application pool may be shutting down because of inactivity. There is an Idle Time-out setting per pool, in minutes (it's under the pool's Advanced Settings - Process Model). It will take some time for the application to start again once it shuts down.
Of course, it might just be the virtualization like others suggested, but this is worth a shot.
Is the site getting significant traffic? If so I'd look for poorly-optimized queries or queries that are being looped.
Your configuration sounds fine assuming your overall traffic is relatively low.
To many data base connections without being release?
Connecting some service/component that is causing timeout?
Bad resource release?
Network traffic?
Looping queries or in code logic?