I have had a Rails 3 app deployed on Elastic Beanstalk for close to 2 years now. For the most part, I haven't had any issues; however, I recently upgraded to one of their new Ruby configurations (64bit Amazon Linux 2014.09 v1.0.9 running Ruby 2.1 (Passenger Standalone)) and I've been fighting an issue for several days where one of more Ruby processes will consume the CPU - to the point where my site becomes unresponsive. I was using a single m3.medium instance, but I've since moved to a m3.large, which only buys me some time to manually log into the EC2 instance and kill the run away process(es). I would say this happens once or twice a day.
The only thing I had an issue with when moving to the new Ruby config was that I had to add the following to my .ebextensions folder so Nokogiri could install (w/bundle install)...
commands:
build_nokogiri:
command: "bundle config build.nokogiri --use-system-libraries"
I don't think this would cause these hanging processes, but I could be wrong. I also don't want to rule out something unrelated to the Elastic Beanstalk upgrade, but I can't thing of any other significant change that would cause this problem. I realize this isn't a whole lot of information, but has anyone experienced anything similar to this? Anyone have suggestions for tracing these processes to their root cause?
Thanks in advance!
Since you upgraded your beanstalk configuration, I guess you also upgraded Ruby/Rails version. This bumped up all gem versions. The performance issue probably originate from one of these changes (and not the Hardware change).
So this brings us into the domain of RoR performance troubleshooting:
1. Check the beanstalk logs for errors. If you're lucky you'll find a configuration issue this way. give it an hour.
2. Assuming all well there, try to setup the exact same version on your localhost (passenger + ruby 2.1 + gems version). If you're lucky, you will witness the same slowness and be able to debug.
3. If you'd like to shoot straight for production debugging, I suggest you'd install newrelic (or any other application monitoring tool) and then drill into the details of the slowness in their dashboard. I found it extremely useful.
I was able to resolve my run away Ruby process issue by SSHing into my EC2 instance and installing/running gdb. Here's a link - http://isotope11.com/blog/getting-a-ruby-backtrace-from-gnu-debugger with the steps I followed. I did have to sudo yum install gdb before.
gdb uncovered an infinite loop in a section of my code that was looping through days in a date range.
Related
This would be on an AWS server. I have not been able to find a single guide that will walk me through this.
One of these guides was promising, up until the point where I was asked to do:
wget https://raw.githubusercontent.com/puma/puma/master/tools/jungle/init.d/puma
wget https://raw.githubusercontent.com/puma/puma/master/tools/jungle/init.d/run-puma
But neither of these repos seem to exist. I realize this is not a programming question, but I am trying to switch from Passenger, to what Rails is supposed to be using natively, but I can't really find much information about deploying to a production environment.
Well
6 month ago the good person answered as a comment on Rails Server with Apache + Puma (via reverse proxy)
I'm sure that you have solved the problem already, but for anyone else coming here in the future, using init.d has been deprecated in favour of systemd (starting with version 5 of puma).
The init scripts that you mentioned are still available. Go to the project on GitHub, and select the most recent 4.x branch. Under tools, you will find the last release of the files.
I upgraded to Lion few weeks ago, and it completely screwed by Ruby on Rails environment. I have installing RVM, different ruby versions and can't seem to find a solution for it... I think it was one of the worst decisions I could do upgrading to Lion. It only brought problems to me.
Anyway, I have realised that rendering a page of my application (which works perfectly well on deployed server and locally too in other machines) increases the ruby process memory in 20-30mb which is kind of crazy. So you can imagine that after a while, my ruby process reaches 2gb of memory in use and my computer is not usable anymore.
I have seen many people with problems upgrading to Lion but I have not been able to find a solution for my case.
Any had the same problem? Any ideas how could I try to solve this issue?
Thanks
You could use the memprof gem (No longer maintained and doesn't work for Ruby above version 1.8.7) and memprof.com (Broken Link) to get to the bottom of the issue.
Also you could experiment with using Passenger, Unicorn or Thin instead of the default Webrick to see if that gives you different behaviour.
I do not know how you might fix the memory leak, but can propose one way to contain it and further troubleshoot it.
If you are willing to learn Docker, you can contain your development environment inside a Docker container, all while accessing the code on your local machine, just like a shared folder in Vagrant.
When you run the Docker container that runs, you can specify a limit on the amount of memory that container can use. Your rails server process might crash and stop the container, but at least you won't have to restart your machine.
Maybe that will give you more leeway for troubleshooting the problem in greater depth.
Docker Run Reference, see the section "Runtime constraints on CPU and memory".
I have been trying to install Passanger for apache on my VPS (running Ubuntu 10.10) and it seems to fail due to g++ as it fails at the point it compiles the module. The installation hangs for a while then:
g++: Internal error: Killed (program cc1plus)
After looking around on the net it seems this could be due to a lack of memory (I only have 256mb).
Is there any way around this? Or, is this in fact probably not the problem? I just need this server for a few days and so far it has been such a mess around just setting everything up so I'm not sure I really want to have to start again with a high RAM VPS. Any suggestions?
Use the debian package instead of building passenger yourself (apt-get install libapache2-mod-passenger).
256M seems pretty tight for a rails site though. I can't imagine you being able to run more than one or two passenger children (my passenger apache instances are using over 200M each).
I run into this problem, too. Then I tried to install the Debian package. But this package is used with Ruby 1.8 and is not compatible with 1.9 (I got problems with writing hash_name: value instead of :hash_name => value.
I'm trying to upgrade my VPS, but still looking for a way not to do so
I admit that I don't follow much of anything "right" on deploying test versus production code. I have been using ASP.NET, and I typically run it locally in Visual Studio, it works, I upload it, I test it again on the production server.
I have read several people say that deploying Rails apps is harder and there are special programs/ways on the ruby site about deploying RoR. I've only toyed with RoR. What is special about deployment? You don't just copy and paste the code and run it (from development machine to the production)? Is it because one is in Apache and the other running on the built in server?
This will be on a Mac Server if it matters.
Deploying RoR is not difficult anymore, especially with Phusion Passenger.
What is somewhat difficult, is getting a automated production environment setup with capistrano, vlad, etc. If you don't mind simply copying your code to the server, you can do that just fine. Most people choose not to do it that way because you lose out on a lot of the benefits that the automated deployment tools give you.
I guess people consider a Rails app harder to deploy than say some PHP apps or such where you just plop the code somewhere and point Apache or whatever at it. But, as mentioned above, you could do that now with Phusion Passenger.
We use Nginx+Passenger, but not for simplicity of deployment. Capistrano is our deploy tool of choice, and really, unless you have a very simple app, you're going to want something like Capistrano anyway. For example, with our deploy, we do a slew of things:
run any database migrations
generate release notes automatically, based on all the commits to Git between the last deploy and this one
notify various people via email (with differing lists depending on whether the deploy is to our staging environment or production) - we do this via cap_gun which integrates with Capistrano.
Notify New Relic RPM of the deploy so it can mark it in our RPM analysis
Notify Hoptoad of the deploy, so it too can have that data when reporting any exceptions
produce our sitemap.xml file, and ping Google to tell them there's a new one
update crontab files (I store our crontab files for each server in our git repo, and then on deploy it sees if there is a new version and updates accordingly, etc.).
flush/restart memcached
There are other ways aside from Capistrano, but it's a proven tool, with a lot of flexibility, yet pretty simple to setup a vanilla configuration.
So, my take is that once you get into any app that is beyond just the very simplest of apps, you're going to need/want to be doing things other than just simply updating the code. In the beginning though, if you just need the code updates, and maybe Rails migrations, then you can do simpler things like Passenger and code sync, or look at tools like Heroku or Engine Yard's stuff where they do a deploy by doing a Git clone (and then offer some additional abilities).
Another super easy way to deploy is with http://heroku.com/
Some of the issues you face with deploying rails to production:
Database connection.
You need to be sure that the database connector is set up for the production environment.
Database migrations.
You have to run database migrations against the production database even though you may have already run them in production/testing/staging
Ruby version. The version or sub-version or Ruby can trip you up, e.g. An error occurred while installing debugger-linecache (1.1.1), and Bundler cannot continue
Gem dependency.
Your production environment may have different packages and gems from development. Bundler will figure this out for the most part and install the dependencies but occasionally there are still issues that you have to resolve manually.
Dependencies.
Some gems on some machine have particular dependencies. I have seen frequent problems with using gems on my unix box that work on OSX and vice-versa.
Note the last 3 shouldn't affect you if on the same machine but I included them based on the title and to be comprehensive.
It's not especially hard. If you stick to conventions then with a little bit of configuration it boils down to this:
cap deploy
...however there is sometimes a bit of effort needed up front to get the workflow in place.
The good news is that lots of people have packaged up solutions and stacks for RoR that you can just plug and play. For example, google ec2onrails - this is a packaged Ubuntu image and set of capistrano tasks for running rails apps in Amazon's EC2 cloud, with lots of common stuff set up already out of the box.
Choose a good hosting provider and you should be able to find something similar for that also.
An easy way to deploy Rails apps is to use Phusion Passenger. Deployment doesn't get much easier than that for any programming language or framework. You can do that on a Mac server.
Another really easy way to deploy rails is with jruby and the glassfish gem.
I'm having ruby instances from mod_rails go "rogue" -- these processes are no longer listed in passenger-status and utilize 100% cpu.
Other than installing god/monit to kill the instance, can anyone give me some advice on how to prevent this? I haven't been able to find anything in the logs that helps.
If you're using Linux, you can install the "strace" utility to see what the Ruby process is doing that's consuming all the CPU. That will give you a good low-level view. It should be available in your package manager. Then you can:
$ sudo strace -p 22710
Process 22710 attached - interrupt to quit
...lots of stuff...
(press Ctrl+C)
Then, if you want to stop the process in the middle and dump a stack trace, you can follow the guide on using GDB in Ruby at http://eigenclass.org/hiki.rb?ruby+live+process+introspection, specifically doing:
gdb --pid=(ruby process)
session-ruby
stdout_redirect
(in other terminal) tail -f /tmp/ruby_debug.(pid)
eval "caller"
You can also use the ruby-debug Gem to remotely connect to debug sockets you open up, described in http://duckpunching.com/passenger-mod_rails-for-development-now-with-debugger
There also seems to be a project on Github concerned with debugging Passenger instances that looks interesting, but the documentation is lacking:
http://github.com/ddollar/socket-debugger/tree/master
I had a ruby process related to Phusion Passenger, which consumed lots of CPU, even though it should have been idle.
The problem went away after I ran
date -s "`date`"
as suggested in this thread. (That was on Debian Squeeze)
Apparently, the problem was related to a leap second, and could affect many other applications like MySQL, Java, etc. More info in this thread on lklm.
We saw something similar to this with very long running SQL queries.
MySQL would kill the queries because they exceeded the long running limit and the thread never realized that the query was dead.
You may want to check the database logs.
This is a recurring issue with passenger. I've seen this problem many times helping people that ran ruby on rails with passenger. I don't have a fix but you might want to try this http://www.modrails.com/documentation/Users%20guide%20Apache.html#debugging_frozen