I am a litle bit lost here.
I am using a grails application deployed in tomcat with memcached-session-store. That it uses spymemcached.
I am also using melody plugin to monitor the app.
In the righter-upper part, there is a http-sessions graph that only grows.
We need to know if this is a potential problem. For now, and without know, we daily restart the webservers. And as a last test we are going to let the http-sessions grows to see if in the future it tends to clean it self.
This is the graph that I am talking about:
So: is a problem? Do I have to configure memcached, tomcat, grails, memcached-session-store or spymemcached to expirate the sessions with a less expiration time? I couldn't find in Interet how to do that.
Any pointer would help.
thanks in advance
AFAICS there were 117 concurrent sessions at max, which are not too many generally. You can also limit the maximum number of active sessions in your context.xml/server.xml via maxActiveSessions for the manager btw.
Some questions:
Do you experience any issues (e.g. running out of memory or anything else)?
How much memory is available to your jvm?
How much memory is used by the jvm?
Do you know how many sessions your app can handle?
What's your session expiration?
What's the size of your (serialized) sessions? You can e.g. check jmx stats from memcached-session-manager (see JMXStatistics).
Finally I'd say that you should keep your tomcats up and running until you experience any real problem.
Related
I have a VPS which is hosting (currently) 5 different rails applications, all with different domains. To make them work I've added one server {} listener per app in my nginx config file. I've left everything else as default, for instance there's only one nginx worker process.
Concurrently, I also have 2 rails workers for one of the apps.
Now, this works as is, but performances are low, in particular speed. How could I make my apps quicker by adhering to my constraints?
Thanks!
Your problem is that you are deep into swap. The slowness you experiencing switching apps is the system loading the requested app from swap into physical memory.
To address this, you can observe who is hogging the memory (also using 'top'), and address that. It's possible you'll find some things to tune, but also quite possible you'll find that you are near the physical limits of what's possible without significant architectural changes.
If your time is worth much, your best course of action will be to upgrade to an instance with at least a 1GB of memory, because you are already using nearly that much.
The nginx "worker_processes" should be set to the number of cores that you have available to work with. You mentioned you had it set to 1. Do you have more cores than that?
I need to host a lot of simple rails/sinatra/padrino applications of different ruby versions each with 0..low hits per day. They belong to different owners and should be well isolated from each other.
When an app is hit it should respond in reasonably short time, but I expect several simultaneous visitors are hitting the same site to be a rare case.
I'm going to create separate os user for each application. Surely I'd like to put them as many per server as it's possible. Thus I need to choose the web server with the lowest memory footprint, which can run applications on behalf of different users with different ruby versions and gemsets.
I consider webrick,nginx+passenger,thin,apache+passenger. I suppose the reliability of all choices is sufficient for such a job, and while performance isn't an issue, the memory consumption is.
I found many posts regarding performance issues, but most of them discuss the performance tuning and issues. I couldn't find a comparison of web servers memory usage when idle.
Is "in process" webrick the best choice? Which one would you choose for that job?
And I couldn't figure out how to resolve subdomains to application ports with webrick. Shall I use nginx or apache for that?
I don't have much experience with hosting myself, but using Webrick for production is not a good idea I think. You can also check out mongrel which I saw used in production. In most cases though you will probably want to choose between thin and unicorn. Check out this http://cmelbye.github.com/2009/10/04/thin-vs-unicorn.html or google around. Good luck :-)
Why not use Heroku? Its free and gets you out of the hassle of server configuration and maintenance.
I have a Rails app that uses MySQL, MongoDB, NodeJS (and SocketIO). Right now, the app (everything) is hosted inside 1 box. I would like to know what I should do when the number of users grow. What factors should I take into account to determine whether I need to host a separate element in another box (like MySQL, Node, Mongo in each of its separate box). Should I just make that one single box bigger? Is there a best-practice method that I can go with?
If you guys can provide me with reference, guides, research regarding this topic. Please do. I am super noob at deployment and server configuration.
We faced this dilemma at work a short while ago and found that simply upgrading to a more powerful single box sufficed and would give us room to grow further by up to 3-4 times.
The most important thing would be to identify your potential bottlenecks.
In our case there were 2 bottlenecks. Disk I/O and the database's ability to utilise memory.
On our new server we had the hard drive array configured in such a way as to maximise the disk I/O and we upgraded the database software to allow it to use more memory. In fact the DBMS now keeps the entire database in memory and only performs write operations to the disk as needed. This significantly improved performance.
The short answer is move everything to its own box. The longer answer is: it depends on your app's usage.
I recommend you use Nagios or similar to monitor your app's resource utilization -- that is, how much CPU and RAM each of your services use. When one starts to each up too much resources (and your page load speed is negatively affected), move that to its own box.
Then continue to monitor that box, beef up when necessary or shard out.
The high scalability blog is good for reading on what other people have done.
I have noticed that after my Grails app has been deployed for about 2 weeks, performance degrades significantly, and I have to redeploy. I am using the Spring Security plugin and caching users. My first inclination is that it has something to do with this and the session cache size, but I'm not sure how to go about verifying this.
Does it sound like I'm on the right track? Has anyone else experienced this and narrowed down the problem? Any help would be great.
Thanks!
Never guess where to optimize, it's going to be wrong.
Get a heap dump and a profile it a little (VisualVM worked fine for me).
It might be a memory leak, like it happened to me. What is your environment - OS, webserver, Grails?
I would recommend getting YourKit (VisualVM has limited information) and use this to profile your application in production (if possible).
Alternatively you could create a performance test (with JMeter for example) and performance test the pieces of your application that you suspect is causing the performance degredation.
Monitoring memory,cpu,threads,gc and such while running some simple JMeter performance tests will definitely find the culprit. This way you can easily re-test your system over time and see if you have incorporated new "performance killing" bugs.
Performance testing tools/services:
JMeter
Grinder
Selenium (Can performance test with selenium grid, need hw though)
Browsermob (Commercial, which uses Selenium + Selenium-Grid)
NeoLoad by NeoTys (Commercial, trial version available)
HP Loadrunner (Commercial, The big fish on the market, trial version available)
I'd also look into installing the app-info plugin and turning on a bunch of the options (especially around hibernate) to see if things are getting out of control there. Could be something that's filling the hibernate session but never closing a transaction.
Another area to look at is if you're doing anything with the groovy template engine. That has a known memory leak, that's sort of unfixable if you're not caching the class/results. Recently fixed a problem around this on our app. If you're seeing any perm gen errors, this could be the case.
Try to install Javamelody plugin. In our case it helped to find problem with GC.
I have a website that is hanging every 5 or 10 requests. When it works, it works fast, but if you leave the browser sit for a couple minutes and then click a link, it just hangs without responding. The user has to push refresh a few times in the browser and then it runs fast again.
I'm running .NET 3.5, ASP.NET MVC 1.0 on IIS 7.0 (Windows Server 2008). The web app connects to a SQLServer 2005 DB that is running locally on the same instance. The DB has about 300 Megs of RAM and the rest is free for web requests I presume.
It's hosted on GoGrid's cloud servers, and this instance has 1GB of RAM and 1 Core. I realize that's not much, but currently I'm the only one using the site, and I still receive these hangs.
I know it's a difficult thing to troubleshoot, but I was hoping that someone could point me in the right direction as to possible IIS configuration problems, or what the "rough" average hardware requirements would be using these technologies per 1000 users, etc. Maybe for a webserver the minimum I should have is 2 cores so that if it's busy you still get a response. Or maybe the slashdot people are right and I'm an idiot for using Windows period, lol. In my experience though, it's usually MY algorithm/configuration error and not the underlying technology's fault.
Any insights are appreciated.
What diagnistics are available to you? Can you tell what happens when the user first hits the button? Does your application see that request, and then take ages to process it, or is there a delay and then your app gets going and works as quickly as ever? Or does that first request just get lost completely?
My guess is that there's some kind of paging going on, I beleive that Windows tends to have a habit of putting non-recently used apps out of the way and then paging them back in. Is that happening to your app, or the DB, or both?
As an experiment - what happens if you have a sneekly little "howAreYou" page in your app. Does the tiniest possible amount of work, such as getting a use count from the db and displaying it. Have a little monitor client hit that page every minute or so. Measure Performance over time. Spikes? Consistency? Does the very presence of activity maintain your applicaition's presence and prevent paging?
Another idea: do you rely on any caching? Do you have any kind of aging on that cache?
Your application pool may be shutting down because of inactivity. There is an Idle Time-out setting per pool, in minutes (it's under the pool's Advanced Settings - Process Model). It will take some time for the application to start again once it shuts down.
Of course, it might just be the virtualization like others suggested, but this is worth a shot.
Is the site getting significant traffic? If so I'd look for poorly-optimized queries or queries that are being looped.
Your configuration sounds fine assuming your overall traffic is relatively low.
To many data base connections without being release?
Connecting some service/component that is causing timeout?
Bad resource release?
Network traffic?
Looping queries or in code logic?