I have recently installed Nginx + Thin on my deployment server, but i am not sure how this will perform in last requests & responses situation. lets say 1000/req per sec.
so the speed on thin is good with 10-100 req /per sec
I wanted to know on higher volumes of data being processed on the request/response cluster.
Guide me on this :-)
Multiple thin processes and nginx are capable of providing lots of speed, depending on what your application is doing. So, the problem will be your application code, the speed of your application server, and your database server.
Scaling Rails has been recently covered in depth by the Scaling Rails Screencasts. I recommend you start there. My 5 step program to scaling Rails would be:
First step is to have the tools to look at what is slow in your application. Do not spend time optimizing everything in your application when you don't know what the problem is.
The easiest way to be able to handle lots of requests/second is with page caching.
If you can't do that, cache everything possible (fragment caching, use memcached to cache data, etc), to speed up your application.
After that, optimize your application as best as possible, make SQL queries fast, index everything, etc.
If you still need more speed, throw more hardware at the problem. Get a big, powerful database server, a bunch of app servers, and proxy your requests across them. You can start here, too, but it will only delay the optimization process.
If you have a single server I think that the main key is, apart from everything already mentioned, is don't skimp on the specs of it. Trying to get too much to run on too little is just a recipe for disaster.
It is also a good idea to get monit or God monitoring your thin instances, I started out with God, but it leaked memory pretty bad on Ruby 1.8.6 so I stop using it in favour of monit. Monit is written in C I believe and has a tiny memory footprint so I'd recommend that one.
If all that seems like a bit much to keep nginx and thin playing nicely you may want to look into an all in one solution like Passenger or LiteSpeed. I have very little experience with these so can offer no substancial advice for them.
Related
I've been searching around for performance tests on the new passenger 5 as I read here it became way faster.
I tried to find other ressources confirming this but no luck. Has anyone tried to install it and see the difference?
Passenger 5 scores better on custom-picked benchmarks because it has a built-in caching layer ("turbocaching") that can avoid actually running your application code for identical requests in a short timeframe; it will not make your actual application code run any faster. This caching layer is only active in certain constrained situations, and is not likely to provide much benefit in the vast majority of actual cases. If you aren't careful, the caching layer may actually end up breaking your application - I demonstrated several security vulnerabilities due to the caching layer to Phusion during the 5 beta phase (which they fixed, at the cost of the caching layer not being able to cache nearly as much). IMO, the Raptor/Passenger 5 benchmarks are deceptive marketing fluff, and the caching layer exists primarily to win Hello World benchmarks, and you should probably just ignore them.
That said, the speed of your application server is almost certainly insignificant in the scope of your overall application performance. Passenger is a great platform because it's extremely user-friendly, well-documented, has an absolutely fantastic installer, and handles a lot of the annoying crap for you out of the box. You should use Passenger if you need the functionality it provides and don't want to screw around with a ton of config stuff. If it doesn't fit your use case, use something else that does.
If every last microsecond is of prime concern to you, you should measure your application's performance under various webservers and various workloads, and then pick the one that performs the best. Otherwise, use whatever you like the most and then switch once performance becomes an actual problem.
Footnote: If you do use Passenger 5, be sure to read the Turbocaching security changes article to be sure you don't accidentally make your application vulnerable to user data theft (or otherwise introduce bugs) through the turbocaching layer.
I recently migrated my app from Puma to Passenger.
I have to say that I'm really happy about this move, especially because I'm hosting it on Heroku.
Since I updated to Ruby 2.2, I had some memory issues with Puma due to Heroku memory limitation (512M).
I tried a couple of different configurations but without success.
Since I moved to Passenger, I saw the memory usage go down to almost half what I was consuming with Puma with the same number of server instances (3 in my case).
Regarding the response time, it seems to be pretty much the same as before but with those memory improvements. Although, I reduced my number of Heroku dynos to handle the same number of requests.
In conclusion, on my personal experience, Passenger helped me a lot reducing the memory used by the app but it didn't helped improving the average response time. Another good point is, regarding the Heroku pricing, it helped me a lot to reduce the cost of my hosting.
I know this post is not really exhaustive without any benchmarks etc... But I thought that maybe you could find it interesting to have a personal experience on a migration from Puma to Passenger.
Hope it helps :)
First of all, let's just be clear, when we talk about server performance, the question is about how well the server scales as usage increases. If your server has one or very few human users, for most apps, you will get the same end-user experience regardless of which server you use because the bottle-neck will be the performance of Ruby, not the app server.
For medium and large apps:
So let's talk about scaling. The more CPU power and RAM you have, the more you can scale. Most servers run out of RAM faster than CPU power. So the key is to minimize the amount of RAM used per request. Each request the server receives will be handled by a process or a thread. Processes use a lot of RAM, threads use very little RAM. So the goal is simply to have lots of threads and few processes.
Puma and Passenger Enterprise are both multi-threaded servers which will scale approximately the same. (Passenger's benchmarks claim to use 5MB less RAM per process, but this is negligible.) Passenger itself (the free version) is single threaded and will not scale as well as the paid Passenger Enterprise version or as well as Puma.
So if you want maximum performance, you're choosing between Puma and Passenger Enterprise. The question then becomes, is Passenger Enterprise worth the financial cost vs is Puma worth the technical expertise cost. The answer depends on what's in your brain(s), what's in your bank account(s), and your general opportunity cost.
Passenger Enterprise has some nice tools that hold your hand if you're not an expert system administrator. Puma also has some tools but they are not as powerful as what Passenger Enterprise provides. Puma requires system-level expertise if you want to get the same control and insights as Passenger Enterprise. (You can also use Puma without all the bells-and-whistles of Passenger Enterprise, but I want to keep this comparison "apples-to-apples").
Personally, I'm a low-level guy who loves configuring servers so I prefer to trade my time rather than my money for an awesome Ruby server. Therefore, I use Puma. If you're not interested in configuring low-level stuff (or if you're a company that realizes software licenses are cheaper than administrator-developers), you might be better off with a paid Passenger Enterprise license.
For the guy running a $5 VPS (or similar low-resource, low-traffic environment):
What I said above is more for higher-traffic apps on servers with considerable resources. You're just trying to get by with basics, so it does not really apply to you.
Think of Puma and Passenger Enterprise as jumbo jets that can move a lot of people very fast. That's way more than your little app needs. What you really need is the server equivalent to a Honda Civic. For that, you should consider either the free version of Passenger or Thin. Use free Passenger if you want easy setup and decent tools. Use thin if you want similar performance but have a willingness to engineer the server.
In this case, I see no reason to use anything but free Passenger unless you are looking for a challenge.
As others said, Passenger by itself doesn't make your app faster per se. Passenger itself has become a lot faster in version 5, but the app server is only a part of the response time. If your app is slow, then it doesn't matter how fast Passenger itself is.
Having said that, Passenger differentiates itself from other app servers by actively helping you to make your app faster. Passenger's turbocache is one way through which Passenger helps you. The article Dynamic Site as fast as a Static Generated One with Raptor demonstrates a good use case for the turbocache. Passenger also provides an optimization guide that gives you tips on how to optimize your app using Passenger settings.
Just wanted to get people's opinions on using Unicorn vs Thin as a rails server. Most of the articles/benchmarks I found online seem very incomplete, so it would nice to have a centralized place to discuss it.
Unicron is a multi-processes server, while thin is an event based/non-blocking server. Event-based servers are great... if your code is asynchronous/non-blocking - vanilla rails is blocking. So unless you use non-blocking rails libraries, I really don't see the advantage of using Thin. Even worse, in a non-blocking server, if your i/o loop is blocking you're going to block the entire loop and not be able to handle any more requests until the blocking call returns. Blocking libraries are going to slow thin down!
Why did Heroku choose Thin as their default server (for cedar)? They are smart guys, so I'm sure they had a reason.
Bellow is a link that suggests replacing Thin with 4 Unicorn workers - this makes perfect sense to me.
4 Unicron workers on Heroku
Thin is easy to configure - not optimal, but it just works in the Heroku environment.
Unicorn can be more efficient, but it needs to be configured: How many workers? Preload App? What do you pick?
I have released Unicorn Heroku apps with workers set to 3, 5 and 8 - just based on how big each app is - how much code, how much memory is used and how much traffic you get all go into picking this number, and you need to monitor over time to make sure you got the number right, and your app isn't running out of memory.
Preload false - this will make your app start slower, but when Unicorn restarts a worker, this is 'safer' with network connections (memcache, postgres, mongo etc)
Preload true - this is better, but you need to handle server re-connections correctly in the pre and post fork code.
Thin has none of these issues out of the box, but you only get process of execution.
Summary: It's really hard to configure Unicorn out of the box to work well (or at all) for everyone, whereas Thin can just work to get people running with fewer support requests.
Recently (only a few months ago) the folks behind Phusion Passenger add support to Heroku. Definitely this is an alternative you should try and see if fits your needs.
Is blazing fast even with 1 dyno and the drop in response time is palpable.
A simple Passenger Ruby Heroku Demo is hosted on github.
The main benefits that Passengers on Heroku claims are:
Static asset acceleration through Nginx - Don't let your Ruby app serve static assets, let Nginx do it for you and offload your app for the really important tasks. Nginx will do a much better job.
Multiple worker processes - Instead of running only one worker on a dyno, Phusion Passenger runs multiple worker on a single dyno, thus utilizing its resources to its fullest and giving you more bang for the buck. This approach is similar to Unicorn's. But unlike Unicorn, Phusion Passenger dynamically scales the number of worker processes based on current traffic, thus freeing up resources when they're not necessary.
Memory optimizations - Phusion Passenger uses less memory than Thin and Unicorn. It also supports copy-on-write virtual memory in combination with code preloading, thus making your app use even less memory when run on Ruby 2.0.
Request/response buffering - The included Nginx buffers requests and responses, thus protecting your app against slow clients (e.g. mobile devices on mobile networks) and improving performance.
Out-of-band garbage collection - Ruby's garbage collector is slow, but why bother your visitors with long response times? Fix this by running garbage collection outside of the normal request-response cycle! This concept, first introduced by Unicorn, has been improved upon: Phusion Passenger ensures that only one request at the same time is running out-of-band garbage collection, thus eliminating all the problems Unicorn's out-of-band garbage collection has.
JRuby support - Unicorn's a better choice than Thin, but it doesn't support JRuby. Phusion Passenger does.
Hope this helps.
Heroku does not use intelligent routing - it will randomly assign jobs to dynos regardless of whether the dyno is busy. Thus, if your dyno cannot handle multiple jobs at once, you will get latency (perhaps massive latency) even if you are paying for lots of other dynos that are free. " That's right — if your app needs 80 dynos with an intelligent router, it needs 4,000 with a random router. "
http://news.rapgenius.com/James-somers-herokus-ugly-secret-lyrics
Heroku says they are working on this, and their plan is to make it easier to use Unicorn. They basically said "Oops, we didn't notice that this was a problem for a few years... and now that we look, it's definitely a problem for Thin... so I guess you need to use a different program than the one we've been pushing all this time."
http://news.rapgenius.com/Jesper-joergensen-routing-performance-update-lyrics
From the official Heroku explanation (second link above):
"Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn.
Rails apps deployed to Cedar with Thin can rather quickly end up with request queuing problems. Because the Cedar router no longer does any queuing on behalf of the app, requests queued at the dyno must wait until the single Rails process works its way through the queue. Many customers have run into this issue and we failed to take action and provide them with a better approach to deploying Rails apps on Cedar."
Also of interest is that their performance tools, including New Relic, have not been reporting time spent in the dyno queue.
http://news.rapgenius.com/Lemon-money-trees-rap-genius-response-to-heroku-lyrics
Oops.
Recently our ruby on rails application was upgraded to 2.3.8. We also replaced Mongrel/Mongrel cluster with Phusion Passenger during upgrade.
Whenever we try to deploy our application, it seems to be responding faster initially but response time gradually increases. We also noticed that cpu usage on the database box spiking to 400% and lot of requests waiting in global queue. This seems to be happening only in our production environment.
Can anyone let me know how I should go about debugging this issue?
Is there anyway we can restrict the number of connections between passenger and DB?
Also is there a way we can setup connection pooling in passenger?
Thanks,
Sivakumar.
I don't think the issue is with Passenger. If you are having a large spike on your DB box CPU, your issue probably lies there. Without more information about your database it's hard to give you specifics, but here are a few things you could try:
Run top and check how much of your total memory the mysqld or other database process is consuming. If this amount is not high then you probably need to tune MySQL settings to take advantage of the RAM on your DB box.
Analyze your running database queries with the mytop command. You might have some queries that are hogging all of your system resources or causing lots of swapping.
Look through your MySQL slow log and see if you have queries that are taking over 1 second to run.
Check your database engines. Are you using MYISAM and/or InnoDB? You might need to tune your database differently so that it can allocate the proper amount of memory and resources to each engine.
Consult a DBA. They'll be able to take a look at your usage and application and tell you more definitively if the issue lies with your database or application.
P.S: I would recommend upgrading to Passenger 3, it performs better than Passenter 2.
I have a simple Rails app deployed on a 500 MB Slicehost VPN. I'm the only one who uses the app. When I run it on my laptop, it's fast enough. But the deployed version is insanely slow. It take 6 to 10 seconds to load the login screen.
I would like to find out why it's so slow. Is it my code? (Don't think so because it's much faster locally, but maybe.) Is it Slicehost's server being overloaded? Is it the Internet?
Can someone suggest a technique or set of steps I can take to help narrow down the cause of this problem?
Update:
Sorry forgot to mention. I'm running it under CentOS 5 using Phusion Passenger (AKA mod_rails or mod_rack).
If it is just slow on the first time you load it is probably because of passenger killing the process due to inactivity. I don't remember all the details but I do recall reading people who used cron jobs to keep at least one process alive to avoid this lag that can occur with passenger needed to reload the environment.
Edit: more details here
Specifically - pool idle time defaults to 2 minutes which means after two minutes of idling passenger would have to reload the environment to serve the next request.
First, find out if there's a particularly slow response from the server. Use Firefox and the Firebug plugin to see how long each component (including JavaScript and graphics) takes to download. Assuming the main page itself is what is taking all the time, you can start profiling the application. You'll need to find a good profiler, and as I don't actually work in Ruby on Rails, I can't suggest any: google "profile ruby on rails" for some options.
As YenTheFirst points out, the server software and config you're using may contribute to a slowdown, but A) slicehost doesn't choose that, you do, as Slicehost just provides very raw server "slices" that you can treat as dedicated machines. B) you're unlikely to see a script that runs instantly suddenly take 6 seconds just because it's running as CGI. Something else must be going on. Check how much RAM you're using: have you gone into swap? Is the login slow only the first time it's hit indicating some startup issue, or is it always that slow? Is static content served slow? That'd tend to mean some network issue (either on the Slicehost side, or your local network) is slowing things down, assuming you're not in swap.
When you say "fast enough" you're being vague: does the laptop version take 1 second to the Slicehost 6? That wouldn't be entirely surprising, if the laptop is decent: after all, the reason slices are cheap is because they're a fraction of a full server. You're using probably 1/32 of an 8 core machine at Slicehost, as opposed to both cores of a modern laptop. The Slicehost cores are quick, but your laptop could be a screamer compared to 1/4 of core. :)
Try to pint point where the slowness lies
1/ application is slow, or infrastructure (network + web server)
put a static file on your web server, and access it through your browser
2/ If it is fast, it is probable a problem with application + server configuration.
database access is slow
try a page with a simpel loop: is it slow?
3/ If it slow, it is probably your infrastructure. You can check:
bad network connection: do a packet capture (with Wireshark for example) and look for retransmissions, duplicate packets, etc.
DNS resolution is slow?
server is misconfigured?
etc.
What is Slicehost using to serve it?
Fast options are things like: Mongrel, or apache's mod_rails (also called passenger phusion or
something like that)
These are dedicated servers (or plugins to servers) which run an instance of your rails app.
If your host isn't using that, then it's probably defaulting to CGI. Rails comes with a simple CGI script that will serve the page, but it reloads the app for every page.
(edit: I suspect that this is the most likely case, that your app is running off of the CGI in /webapp_directory/public/dispatch.cgi, which would explain the slowness. This tends to be a default deployment on many hosts, since it doesn't require extra configuration on their part, but it doesn't give good performance)
If your host supports "Fast CGI", rails supports that too. Fast CGI will open a CGI session, and keep it open for multiple pages, so you get much better performance, but it's not nearly as good as Mongrel or mod_rails.
Secondly, is it in 'production' or 'development' mode? The easy way to tell is to go to a page in your app that gives an error. If it shows you a stack trace, it's in development mode, which is slower than production mode. Mongrel and mod_rails have startup options to determine whether to run the app in production or development mode.
Finally, if your database is slow for whatever reason, that will be a big bottleneck as well. If you do have a good deployment (Mongrel/mod_rails/etc.) in production mode, try looking into that.
Do you have a lot of data in your DB? I would double check that you have indexed all the appropriate columns- because this can make a huge difference. On your local dev system, you probably have a lot more memory than on your 500 mb slice, which would result in the DB running a lot slower if you have big, un indexed tables. You can also run the slow queries logger in MySql to pinpoint columns without indexes.
Other than that, yes- passenger will need to spool up a process for you if you have not been using the site recently. If this is the case, you should see a significant speed increase on second, and especially third and later page loads.
You might want to run a local virtual machine with 500 MB. Are you doing a lot of client-server interaction? Delays over the WAN are significant
You might want to check out RPM (there's a free "lite" version too) and/or New Relic's Tune Up.
Your CPU time is guaranteed by Slicehost using the Xen virtualization system, so it's not that. Don't have the other answers for you, sorry! Might try 'top' on a console while you're trying to access the page.
If you are using FireFox and doing localhost testing (or maybe even on LAN) you may want to try editing the network.dns.disableIPv6 setting.
Type about:config in the address bar and filter for network.dns.disableIPv6 and double-click to set to true.
This bug has been reported mainly from Vista OS's, but some others as well.
You could try running 'top' when you SSH in to see which process is heavy. If you also have problems logging you, perhaps you may try getting Statistics in the Slicehost manager.
If you discover it is MySQL's fault, consider decreasing the number of servers it can spawn.
512 seems decent for Rails application, you might have to check if you misconfigured too.
I am actually developing and application that has around 15 modules, all of them using the same database.
I am using Apache + Mongrel, I cannot use Passenger because I am working on Windows (please forgive me for this deadly sin!)
Which of the following is a better approach?
Deploy multiple small rails
applications using a virtual server
and a pair of mongrels for each
application.
Deploy only a big rails application
I am worried about the number of running mongrels and the memory/cpu load.
I'd suggest deploying a monolithic Rails application.
I use the request_routing plugin to drive 3 domains sharing the same database from one, big Rails application.
I'm running 4 mongrels, which seems to be enough for now, but YMMV.
It depends on hwo many simultaneous clients you expect to have. One mongrel, one client at a time (until Rails 2.2) since Rails isn't currently threaded.
Two is enough mongrels if you don't expect more than a few simultaneous users. You can raise that number by using page caching to bypass mongrel for pages that don't have user-specific dynamic content.
The only way to be truly sure is to test the system.
In my experience you'll need at least 4 mongrels for a moderately active site of just a few users at a time.
It would seem like one application would best fit your scenario... as others have said...
A good rule of thumb would be that the average behaving mongrel will consume 60mb of memory (or less)... take your total RAM available, subtract out for any other services (database, memcache, etc) and then figure out how many pieces of the pie you can have left from the remaining memory.
You can always scale them up or down from there...
It sounds like it would be a much better use of your hardware to integrate all modules into one comprehensive rails apps.
IMHO the primary weakness of Rails is the amount of resources needed to run a low or very low traffic app. On the other hand a few mongrels go a long way to serve a whole lot of traffic.