I have RoR app, which uses Passenger + Nginx.
Also I have ImageMagick for some regular tasks with images.
In one day (yesterday-tomorrow) I saw that my app crashes, in case out of memory!
ImageMagick write to log 'Cannot allocate memory'.
When I check free RAM, I saw that there only ~120MB free. (Summary is 1GB).
Most part used Passenger.
I had restarted it, and all became OK.
What reason of this can be?
First of all, did you use all functions of ImageMagick, if not change ImageMagick to MiniMagick (you save part of memory).
Next thing image processing, how do you work with images? Always you can do it asynchronously (resque/sidekiq/rake task in cron - "smaller solution"), you can maybe save some MB..
Passenger, is quite memory demanding. Try something smaller like Thin / Puma.
Are you sure, your code is without memory-leaks? If you're using ruby 2.1+ there is several tools to detect it, for example excelent article, or if not, try to run your application in JRuby with Java memory profiler, for example VisualVM
some other question to think, did you really need full Rails? Rails is big and require quite a lot memory, maybe Sinatra or for simple API Grape can be sufficient...
Related
We are running 2 rails application on server with 4GB of ram. Both servers use rails 3.2.1 and when run in either development or production mode, the servers eat away ram at incredible speed consuming up-to 1.07GB ram each day. Keeping the server running for just 4 days triggered all memory alarms in monitoring and we had just 98MB ram free.
We tried active-record optimization related to bloating but still no effect. Please help us figure out how can we trace the issue that which of the controller is at fault.
Using mysql database and webrick server.
Thanks!
This is incredibly hard to answer, without looking into the project details itself. Though I am quite sure you won't be using Webrick in your target production build(right?), so check if it behaves the same under Passenger or whatever is your choice.
Also without knowing the details of the project I would suggest looking at features like generating pdfs, csv parsing, etc. Seen a case, where generating pdf files have been eating resources in a similar fashion, leaving like 5mb of not garbage collected memory for each run.
Good luck.
I've been using Ruby Enterprise Edition and Passenger (for Apache, since I run Apache anyway for other things) for some time, but I'm wondering if there's a new trend about what to use on servers nowadays.
For example I've heard about Thin, Unicorn... I also know that 1.9.2 is faster than REE, but I wonder about RAM consumption. I'd rather have it consume less RAM even at the expense of some speed.
Thanks for all advice.
If you want minimal memory you should try Thin.
It does not have master worker as Unicorn or Passenger, thus uses less memory.
Suppose you have a very small app that needs to run on a small VM, then you can use 1 thin worker + nginx. I ran several rails 3.2 apps using Thin+nginx+postgres on 256MB VMs without swapping.
Unicorn is faster but it needs a master worker. It's good if you want to run on Heroku, you can set it 2 or 3 workers and be within the 512MB limit.
If your app is very big and you have too many long running requests, I would check out jRuby and Thinidad/Torquebox.
I converted a few apps from MRI+Sidekiq to jruby+Trinidad+Trinidad_Scheduler. I get about 100-200 req/sec using a pool of 50 threads in a trinidad server!
What I like about jRuby is that you can combine everything on one Rails Server. You can put together on the same JavaVM the cache_store with EHcache, Scheduling, Background processing and real multithreading.
You don't need to run redis, memcached, resque or sidekiq separately.
Im not saying they are not good, I love sidekiq and resque, but you can decrease your complexity by combining everything on one process and have high concurrency.
A more advanced and Enterprise solution is Torquebox, it has support for clustering and is super scalable. But I've had problems with my app crashing on torquebox, so i'm sticking to Trinidad for now.
The disadvantages of jRuby? MEmory! A Trinidad server will use minimum 512MB, up to 2-3GB ram.
Also, for Single Thread server, a single request from a rails app running Ruby-1.9.3 is about twice as fast as the same request on jRuby.
Another option is Puma, you can get full multithreading on MRI with puma. I myself could not get it stable enough on my apps.
So, it all depends on your requirements, memory usage, full threading and concurrency.
Apart from Passenger, have a look at Unicorn, Trinidad, Puma and Torquebox. Those seems to be the top rails servers right now.
There is an great book with an introduction of converting your Rails app to jRuby and deploy your app using several methods such as trinidad.
http://pragprog.com/book/jkdepj/deploying-with-jruby
The Torquebox Documentation is amazingly good. It's very detailed and explains really good how to use all Torquebox features.
http://torquebox.org/documentation/
I Hope that sharing my experience has helped.
Passenger is still extremely strong, especially being REE will naturally support 1.9 in the near future. The fact that your application can crash, however it won't affect anything else on your machine is an amazing feature to have. Deploying code is extremely easy because the server will continue to accept connections, which means less frustration/stress for you.
However, in terms of comparisons:
Here is a great resource is check out various comparisons(including memory consumption) with all the new servers.
It compares Thin, Unicorn, Passenger, TorqueBox, Glassfish, and Trinidad:
http://torquebox.org/news/2011/03/14/benchmarking-torquebox-round2/
Mike Lewis' link does a good job of comparing those different ruby servers. My personal experience has been with nginx/REE/Passenger and its been good. I haven't tried the others, so I can't comment on that.
However, I can speak on RAM usage. Your biggest savings of RAM will come from using 32-bit servers. In my experience (3x 3GB app servers), 64-bit REE/passenger processes took up to 2x as much RAM as their 32-bit counterparts. We saw a significant performance increase moving from 64 to 32 bit servers, everything else staying the same. Unless your application requires 64-bit, I would suggest running your application servers (not database) in 32-bit.
Passenger is still a very good choice to use so you are not behind the times or anything. It is also actively supported and has a very good development team that contributes a lot to the community. We have been using Unicorn and it has been very good. Our favorite functionality is to be able to upgrade apps/ruby/nginx without dropping a connection.
hi
I am planning to run jruby (1.5.3 latest) on mongrel but how much memory will it require on x64 server for a simple web site ? and how many instances will be required ?
10000 page views per day
for the same requirement what would be the numbers for ruby.
any reference production data would be welcome.
You probably won't use mongrel with jruby, at least i've never heard of it. We run an app using trinidad, which wraps tomcat7 and for similar performance to what you're looking for I use a 1gb heap.
Mongrel has really gone out of favour for more robust setups using passenger or thin or unicorn for instance.
If you're limited with memory, from my experience CRuby is the way to go. Try REE or ruby-1.9.2 with Passenger3 and nginx. It's a super simple setup and very fast.
JRuby definitely takes more memory, but if you have java requirements you don't have much choice.
10000 page views you should get away with a small ec2 instance (if that's what your instances refers to)
It's really hard to give a definitive answer though as it all depends on what type of app you're running. is it cpu intensive calculations, or memory intensive data?? who knows
From my experience, CRuby tends to be much simpler than JRuby, easier for local use (ie tests run significantly faster in cruby) and also very fast.
I have recently installed Nginx + Thin on my deployment server, but i am not sure how this will perform in last requests & responses situation. lets say 1000/req per sec.
so the speed on thin is good with 10-100 req /per sec
I wanted to know on higher volumes of data being processed on the request/response cluster.
Guide me on this :-)
Multiple thin processes and nginx are capable of providing lots of speed, depending on what your application is doing. So, the problem will be your application code, the speed of your application server, and your database server.
Scaling Rails has been recently covered in depth by the Scaling Rails Screencasts. I recommend you start there. My 5 step program to scaling Rails would be:
First step is to have the tools to look at what is slow in your application. Do not spend time optimizing everything in your application when you don't know what the problem is.
The easiest way to be able to handle lots of requests/second is with page caching.
If you can't do that, cache everything possible (fragment caching, use memcached to cache data, etc), to speed up your application.
After that, optimize your application as best as possible, make SQL queries fast, index everything, etc.
If you still need more speed, throw more hardware at the problem. Get a big, powerful database server, a bunch of app servers, and proxy your requests across them. You can start here, too, but it will only delay the optimization process.
If you have a single server I think that the main key is, apart from everything already mentioned, is don't skimp on the specs of it. Trying to get too much to run on too little is just a recipe for disaster.
It is also a good idea to get monit or God monitoring your thin instances, I started out with God, but it leaked memory pretty bad on Ruby 1.8.6 so I stop using it in favour of monit. Monit is written in C I believe and has a tiny memory footprint so I'd recommend that one.
If all that seems like a bit much to keep nginx and thin playing nicely you may want to look into an all in one solution like Passenger or LiteSpeed. I have very little experience with these so can offer no substancial advice for them.
I have a simple Rails app deployed on a 500 MB Slicehost VPN. I'm the only one who uses the app. When I run it on my laptop, it's fast enough. But the deployed version is insanely slow. It take 6 to 10 seconds to load the login screen.
I would like to find out why it's so slow. Is it my code? (Don't think so because it's much faster locally, but maybe.) Is it Slicehost's server being overloaded? Is it the Internet?
Can someone suggest a technique or set of steps I can take to help narrow down the cause of this problem?
Update:
Sorry forgot to mention. I'm running it under CentOS 5 using Phusion Passenger (AKA mod_rails or mod_rack).
If it is just slow on the first time you load it is probably because of passenger killing the process due to inactivity. I don't remember all the details but I do recall reading people who used cron jobs to keep at least one process alive to avoid this lag that can occur with passenger needed to reload the environment.
Edit: more details here
Specifically - pool idle time defaults to 2 minutes which means after two minutes of idling passenger would have to reload the environment to serve the next request.
First, find out if there's a particularly slow response from the server. Use Firefox and the Firebug plugin to see how long each component (including JavaScript and graphics) takes to download. Assuming the main page itself is what is taking all the time, you can start profiling the application. You'll need to find a good profiler, and as I don't actually work in Ruby on Rails, I can't suggest any: google "profile ruby on rails" for some options.
As YenTheFirst points out, the server software and config you're using may contribute to a slowdown, but A) slicehost doesn't choose that, you do, as Slicehost just provides very raw server "slices" that you can treat as dedicated machines. B) you're unlikely to see a script that runs instantly suddenly take 6 seconds just because it's running as CGI. Something else must be going on. Check how much RAM you're using: have you gone into swap? Is the login slow only the first time it's hit indicating some startup issue, or is it always that slow? Is static content served slow? That'd tend to mean some network issue (either on the Slicehost side, or your local network) is slowing things down, assuming you're not in swap.
When you say "fast enough" you're being vague: does the laptop version take 1 second to the Slicehost 6? That wouldn't be entirely surprising, if the laptop is decent: after all, the reason slices are cheap is because they're a fraction of a full server. You're using probably 1/32 of an 8 core machine at Slicehost, as opposed to both cores of a modern laptop. The Slicehost cores are quick, but your laptop could be a screamer compared to 1/4 of core. :)
Try to pint point where the slowness lies
1/ application is slow, or infrastructure (network + web server)
put a static file on your web server, and access it through your browser
2/ If it is fast, it is probable a problem with application + server configuration.
database access is slow
try a page with a simpel loop: is it slow?
3/ If it slow, it is probably your infrastructure. You can check:
bad network connection: do a packet capture (with Wireshark for example) and look for retransmissions, duplicate packets, etc.
DNS resolution is slow?
server is misconfigured?
etc.
What is Slicehost using to serve it?
Fast options are things like: Mongrel, or apache's mod_rails (also called passenger phusion or
something like that)
These are dedicated servers (or plugins to servers) which run an instance of your rails app.
If your host isn't using that, then it's probably defaulting to CGI. Rails comes with a simple CGI script that will serve the page, but it reloads the app for every page.
(edit: I suspect that this is the most likely case, that your app is running off of the CGI in /webapp_directory/public/dispatch.cgi, which would explain the slowness. This tends to be a default deployment on many hosts, since it doesn't require extra configuration on their part, but it doesn't give good performance)
If your host supports "Fast CGI", rails supports that too. Fast CGI will open a CGI session, and keep it open for multiple pages, so you get much better performance, but it's not nearly as good as Mongrel or mod_rails.
Secondly, is it in 'production' or 'development' mode? The easy way to tell is to go to a page in your app that gives an error. If it shows you a stack trace, it's in development mode, which is slower than production mode. Mongrel and mod_rails have startup options to determine whether to run the app in production or development mode.
Finally, if your database is slow for whatever reason, that will be a big bottleneck as well. If you do have a good deployment (Mongrel/mod_rails/etc.) in production mode, try looking into that.
Do you have a lot of data in your DB? I would double check that you have indexed all the appropriate columns- because this can make a huge difference. On your local dev system, you probably have a lot more memory than on your 500 mb slice, which would result in the DB running a lot slower if you have big, un indexed tables. You can also run the slow queries logger in MySql to pinpoint columns without indexes.
Other than that, yes- passenger will need to spool up a process for you if you have not been using the site recently. If this is the case, you should see a significant speed increase on second, and especially third and later page loads.
You might want to run a local virtual machine with 500 MB. Are you doing a lot of client-server interaction? Delays over the WAN are significant
You might want to check out RPM (there's a free "lite" version too) and/or New Relic's Tune Up.
Your CPU time is guaranteed by Slicehost using the Xen virtualization system, so it's not that. Don't have the other answers for you, sorry! Might try 'top' on a console while you're trying to access the page.
If you are using FireFox and doing localhost testing (or maybe even on LAN) you may want to try editing the network.dns.disableIPv6 setting.
Type about:config in the address bar and filter for network.dns.disableIPv6 and double-click to set to true.
This bug has been reported mainly from Vista OS's, but some others as well.
You could try running 'top' when you SSH in to see which process is heavy. If you also have problems logging you, perhaps you may try getting Statistics in the Slicehost manager.
If you discover it is MySQL's fault, consider decreasing the number of servers it can spawn.
512 seems decent for Rails application, you might have to check if you misconfigured too.