Running a pretty sizable rails app, we've recently gotten around to upgrading it to rails 3.
Our stack is ruby-1.9.3p484, rails 3.2.16 and passenger 4.0.23 running on top of apache.
After throwing some traffic at a couple of our machines, we started noticing a few really strange errors coming down.
Things like random methods not being defined on objects that would obviously have them, instance variables being nil inside AR associations, and objects just being randomly replaced with 'false'. Just all around strange behavior.
Inspecting apache's logs gave us another bit of info, namely that as these errors were coming in, more often than not, their respective processes would keel over as well, on random bits of the app.
Sometimes it would be just a ruby node getting passed in as null, other times it would be just some random string overflowing, just random stuff getting mangled about.
None of this happened during testing, so the only 'reliable' way of reproducing this thus far has been to just throw traffic at the respective machines and see when / if they start exhibiting this behavior.
Having gone through all of this, here's a list of things we've ruled out up to now:
passenger's oob garbage collection
rails 3 itself ( apparently we'd been getting these before as well, but they were far enough apart to not set off any alarms )
serialization / shoving things in and out of memcached
libxml - there were some reports about of version 2.5.0 causing memory corruption, upgrading to 2.7.0 didn't really make a difference
turning off prelinking ( this can cause memory corruption, per https://www.ruby-forum.com/topic/205897 )
Returning the GC settings to stock seems to have alleviated the problem, but we don't really have anything conclusive in that regard. It would seem though that more collections result in a lower occurrence rate for the issue.
Any thoughts on what might be causing this or we could use to help us pinpoint the issue?
I've had 1.9.3-p484 segfault at_exit on test runs as well but I haven't looked into it yet. In my case it seems to be triggered by certain dependencies for the test suite.
We've also had problems with another project on Rails 3 that ended up abandoning their port to Rails 3 and sticking with 2.3 ):
Have you tried running the app outside of Apache / Passenger?
Related
Recently I upgraded our EC2 instances to c3.large in order to boost more unicorn workers to handle increased traffic to our site. (6xworkers per machine - 2 ec2 instances).
When I did this, I started seeing duplicate records being created! It looks like somehow, that maybe 2 unicorn workers are attempting to process the same request?
In my nginx logs I see ONE [POST] request to a particular rails controller - yet two records are being created in the database. Obviously I'm hitting some kind of race condition issue - but have no idea how to debug this. It doesn't happen all the time, but when it does it's frustrating as well. I also noticed that the two database records are within 5 seconds of each other. (And yes I disable the button when the user clicks the submit button).
Unicorn 4.8.2
Ruby 1.9.3
Rails 3.2.14
Any help would be great! Thanks!
Although it may not fix your problem, upgrade to Ruby 2.0.0. There are some blog mentioning that is a good idea to upgrade to Ruby 2.0.0 if you are using Unicorn.
For example: https://www.digitalocean.com/community/articles/how-to-optimize-unicorn-workers-in-a-ruby-on-rails-app
Specifically:
If you are using Ruby 1.9, you should seriously consider
switching to Ruby 2.0. To understand why, we need to understand a
little bit about forking.
Forking and Copy-on-Write (CoW)
When a child process is forked, it is
the exact same copy as the parent process. However, the actual
physical memory copied need not be made. Since they are exact copies,
both child and parent processes can share the same physical memory.
Only when a write is made-- then we copy the child process into
physical memory.
So how does this relate to Ruby 1.9/2.0 and Unicorn?
Recall the Unicorn uses forking. In theory, the operating system would
be able to take advantage of CoW. Unfortunately, Ruby 1.9 does not
make this possible. More accurately, the garbage collection
implementation of Ruby 1.9 does not make this possible. An extremely
simplified version is this — when the garbage collector of Ruby 1.9
kicks in, a write would have been made, thus rendering CoW useless.
Without going into too much detail, it suffices to say that the
garbage collector of Ruby 2.0 fixes this, and we can now exploit CoW.
Or something like that. I've upgraded several apps from Rails 3.0 to Rails 3.1. They are are running on Thin with the multithreaded option enabled, they are all configured with threadsafe!. Worked great in Rails 3.0. But in 3.1, after a few requests, things start slowing down. After a few more requests, the pages hangs for N seconds (where N is my db timeout) and I get this error:
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
Those default values of 5 and 5 have been find in the past and should still be fine. Also increasing them does not fix the problem, though it takes a little longer to manifest. I should stress that during these times I have been the only one accessing the apps. When I drop Thin down to single-threaded mode, everything works as expected.
This occurs with MySQL, SQLite, Ruby 1.8.7 and Ruby 1.9.2. Thin is unchanged. The only variable I can find is the upgrade to Rails 3.1. Is there anything I can do, or is this a regression in Rails?
Looks like this is a bug. There's a patch and pull request for it. Hopefully Rails will merge it in. Until then, multi-threaded Rails apps won't work on Thin.
https://github.com/rails/rails/pull/1670
I have Rails 3, with webrick, running a sqlite3 database. On my standard linux desktop, doing Word.all (Word is my model), I have no problems, even though there are 10,000 entries in my database. I have scopes to display them 2000 at a time, to make things more tractable.
On my Windows 7 laptop, it's a very different story. I can only get about 400 Words at a time, or I get that "Not enough space" error.
I can open up window's task manager, and the memory barely even blips. On the console, the command returns almost instantly (it clearly has not done anything to several hundred entries before giving up).
What is going on here? My laptop isn't exactly much worse than my desktop, so I don't think i"m actually hitting any RAM limits... Is there some weird Ruby thing going on?
EDIT: It's not just a server issue either, i see the same thing in the rails console, as well... So, webrick might not be an issue...
If it were code, you'd think I"d see it across platforms, not just on my laptop... Even then, how can it be my code if all I'm typing is Word.all (no custom code) and the database is clearly set up right (I don't have issues getting any individual entry, just not too many at a time)
-jenny
webrick is a very simple web application server think to work only for development.
I have no experience of it on windows os , but,anyway, i have many probel with it when test limit situation. you could try the mongrel gem , if the problem persist it's something in your code.
I have done a few projects using Ruby on Rails. I am going to use JRuby on Rails and hosting it on GAE. In that case what are the stuff that I need to know while developing JRuby apps. I read that
JRuby has the same syntax
I can access Java libraries
JRuby does not have access to some gems/plugins
JRuby app would take some time to load for the first time, so I have to keep it alive by sending
request every 5 mins or so
I cannot use ActiveRecord and instead I must DataMapper
Please correct if I am wrong about any of the statements I have made and Is there anything else that I must know?. Do I need to start reading about JRuby from the scratch or I can go about as usual developing Ruby apps?
I use JRuby everyday.
True:
JRuby has the same syntax
JRuby does not have access to some gems/plugins
I can access Java libraries
Some gems/plugins have jruby-specific versions, some don't work at all. In general, I have found few problems and as the libraries and platforms have matured a lot of the problems have gone away (JRuby has become a lot better).
You can access Java, but in general why would you want to?
False:
JRuby app would take some time to load for the first time, so I have to keep it alive by sending request every 5 mins or so
I cannot use ActiveRecord and instead I must DataMapper
Although I guess it is possible to imagine a server setup where the initial startup/warmup cost of the JVM means you need to ping the server, there is nothing inherent in JRuby that makes this true. If you need to keep the server alive, you should look at your deployment environment. Something similar happens in shared-hosting with passenger where an app can go out of memory after a period of inactivity.
Also, we use ActiveRecord with no problems at all.
afaik, rails 3 is 100% compatible with jruby, so there should be no problem on that path.
like every new platform, you should make yourself comfortable with it by playing around with jruby. i recommend using RVM to do that.
as far as you questions go:
JRuby is just an other runtime like MRI or Rubinus
since JRuby is within the JVM using Java is very easy, but you can also use RJB from MRI
some gems are not compatible, when they use native c libraries, that do not run on JRuby
the JVM and your application container need startup time and some time to load your app, but that is all, there is no need for keep alive, that is wrong
you can use whatever you want, most gems are updated to be compatible with JRuby
#TobyHede mostly covered issues that you thought of you might have so I'll leave it at that.
As for other things to have in mind, it's simply a different interpreter and funny discrepancies will crop up that will take some adaptation.
some methods are implemented differently, such as sleep 10.seconds will throw exception (you have to sleep 10.seconds.to_i) and I remember getting NoMethodError on Symbol class when switching from MRI to JRuby (don't remember which method wasn't implemented), just have in mind slight variations will be there
you will experience hangs and exceptions in gems that otherwise worked for you (pry for example when listing more then one page)
some gems may work differently, pry (again) will exit if you press ctrl+c for example, pretty annoying
slightly slower load times of everything and no zeus
you'll get occasional java exception stack traces with no indication on which line of ruby code it happened
Timeout.timeout often will not work as expected when its wrapped around net code and stars align badly (this has mostly been fixed in jruby core, but it seems to still be an issue with gems that do their own netcode in pure java)
hidden problems with thread-safety in third party code How do you choose gems for a high throughput multithreaded Rails app? - stay away from EventMachine for example
threads will be awesome (due to nativeness and no gil) and fibers will suck (due to no coroutine support in JVM they're ordinary threads), this is why you often won't get a performance boost with celluloid when compared to MRI
you used to run your rails with MRI Ruby as processes in an OS, you knew how to track their PIDs, bloat, run times, kill them, monitor them etc, this part is not evident when you switch to JRuby because everything has turned to threads in a single process. Java world has very good tools to handle these issues, but its something you'll have to learn
killall -9 ruby doesn't do the trick with jruby when your console hangs (which it does more often then before), you have to ps -ef and then track the proper processes without killing your netbeans etc (minor, but annoying)
due to my last point, knowing Java and the JVM will help you get out of tight spots in certain situations (depending on what you intend to do this may be something you actually really need), choice of deployment server will increase or decrease this need (torquebox for example is a bit notorious for this, other deployment options might be simpler, see http://thenerdings.blogspot.com/2012/09/pulling-plug-on-torquebox-and-jruby-for.html)
...
Also, see what jruby team says about differences, https://github.com/jruby/jruby/wiki/DifferencesBetweenMriAndJruby
But yeah, otherwise its "just the same as MRI Ruby" :)
I'm developing a new site based on Ruby on Rails 3 beta. I knew this might be a bad idea considering it's just beta, but I still thought it might work.
Now though I'm having HUGE problems with Rails consuming huge ammounts of memory.
For my application today it consumes about 10 mb per request and it doesn't seem to release it either. So I thought this might be because of bloat in my application and thus I created a test app just to compare.
For my test app I just generated a model with a scaffold and then created about 20 records on this model.
I then went to the index page and hit refresh and I could immediately see memory taking off! Less than my app but still about 1-3 mb per request.
I'm working in OSX Leopard, with Ruby 1.8.7, Rails 3.0.0.beta and a SQLLite db for development.
Does anyone recognize my problem?
I would really appreciate some help here. :/
Thanks!
Well, you should consider how a production Rails app would be served. For example, the above setting (with regards to caching) is typically enabled for the production environment and you should also compare performance with your app running under Passenger (Apache or Nginx).
I do believe there is an easy means to force Passenger to play nicely in dev mode as well.
There were some memory leakage issues in the Rails 3 betas. Is there a reason you're not on 3.0.6?
Edit: D'oh, just saw the date this was asked.