We have a big rails app and since a few days, one by one, our ruby process seems to block in a loop and eat 100% of the CPU until the passenger server die raising 502 erros.
Do you know the best ways to figure that out why?
I've tried New Relic but it's only performance stuff, and our errors are too numerous to try to guess what is the problem. (We have a lot of request a day and lots of UTF-8 BSON Errors because we are using UTF-8 urls)
Using:
Rails 3.2.6 with Ruby 1.9.2p290
Passenger 3.0.13
MongoDB 2.0.1 with Mongoid 2.4.11
Nginx
FreeBSD 8.2
A handy way to find out where your ruby is stuck is to attach gdb to the running process and call rb_backtrace(). That will print the current stack trace to stderr, which is likely directed to a log file. Here's a short walk through I found on a blog. The stack trace should help you localize where your process is stuck and allow you to figure out what's going on via code inspection or by adding logging instrumentation around the critical code path.
Aman Gupta has a gdb to ruby hooks library gdb.rb I've found handy at times.
Related
I am running Mac OSX 12.1. I have some rails 6 apps and they are working normally in regard to memory usage.
But rails 5.2.6 has a strange memory issue. When I start a rails server or run rspec, the usage of the CPU is normal i.e. low when not much is happening but higher when large tests or heavy server usage occurs.
However, when I use rails console the CPU usage climbs almost immediately to over 100%, and pretty soon the Mac fan starts to run. It stays this way even if I exit the console. I have to do killall ruby to stop all ruby processes. I am monitoring memory using top -o cpu. Is this a known issue with rails 5.2.6? Is there a way to stop this happening?
I was experiencing the same problem for quite a while. I've eventually noticed that it's a problem with the listen gem. Updating it is enough to fix it.
Update it to the latest version possible. If I remember correctly, it has to be at least v3.3 for the problem to go away.
I suggest to check what initializes when you do rails c. You def should profile your startup times. Start checking your config/initializers folder and also consider Profiling Rails Boot Time. Also check how to Speed up Rails boot time.
I don't believe this is specific to 5.2.6 but make sure you use at least ruby 2.7.0 ish which also affects your performance.
I want to use Ruby on Rails via rbenv with Apache2.4 on FreeBSD9.1, running ruby-2.0.0p247 and rails 3.2.13.
My problem is that rails needs a hook into apache, which would normally be accomplished using passenger. However, due to a bug in the OS, passenger won't work. Linkage below detailing the specific OS bug. I'd like to get things moving well before FreeBSD9.2 arrives (9.2 resolves the issue).
Is there an alternative to passenger? How can I accomplish my goal in the least exotic way possible?
urlpassenger on github commit: check for broken freebsd version
urlCatching C++ std::bad_cast doesn't work in FreeBSD 9.1
urlWhy does catching std::bad_cast not work on FreeBSD 9?
Conveniently, and even after much initial research, after posting this question, another useful answer was discovered. While quite comprehensive, I'm not sure it entirely answers my original question. Any help on this matter is appreciated.
Note: I work at Phusion.
You can use any of the other app servers described at Ruby on Rails Server options. Unicorn, Puma, Thin... they will all get the job done, though not necessarily as easy as Phusion Passenger.
But if you prefer Phusion Passenger then there's still hope. A pull request came in a couple of days ago which works around the FreeBSD bug: https://github.com/phusion/passenger/pull/87
I haven't got time yet to review the pull request (I'm too busy doing my yearly taxes) but perhaps you can try it out yourself. Installation instructions for the git repo are documented in the readme.
Can someone think of a way to find out where our rails production server hangs? Its CPU is at 99% so I assume it gets lost in a while/for/each loop. Unfortunately we can't find a candidate loop.
The problem does not occur in development and our test suit now has 100% code coverage.
We were already attaching to Ruby via gdb, but didn't know where to go from there.
Any ideas?
Once you have attached with gdb to the busy looping process then call rb_backtrace from gdb:
> call rb_backtrace()
The output from rb_backtrace will be similar to a crash report, and the output will go to your log files similar to a standard Ruby error backtrace.
From there you should hopefully already be a step closer to the solution since you will see where in your Ruby code the process is stuck.
You can check out some more tips here:
http://isotope11.com/blog/getting-a-ruby-backtrace-from-gnu-debugger
This is not a clean solution but at least the following resolved the problem for us: we migrated to the 'thin' webserver and removed 'devise'.
I am getting up and running with Spree in production on a 256MB Slicehost slice with MySQL, Mongrel, and Nginx. I am getting this error every time I try to access any URL in the app:
SystemStackError (stack level too deep):
compass (0.8.17) lib/compass/app_integration/rails/action_controller.rb:7:in `process'
vendor/plugins/haml/rails/./lib/sass/plugin/rack.rb:44:in `call'
(Some more test url requests are in this pastie)
What are the advanced (but free and open source) debugging strategies/tools I should use while in production mode so I can do better at solving this problem?
It's slightly different each time depending on the controller. I have tried increasing my stack size, but that doesn't help because there's probably just some loop I can't find.
The question is, how do I get a more detailed report of the exact output while in production?
I am using Deprec and Capistrano to execute all server side commands. I am running a mongrel cluster of 3. The log files I see are:
current/log$ ls
development.log
testsite.com-error.log
mongrel.8000.log
mongrel.8001.log
mongrel.8002.log
mongrel.log
production.log
testsite.com-access.log
I'm just looking for some advanced debugging strategies/tools while in production mode so I can do better at solving this problem. Maybe some cap tasks for streaming/debugging (or conditionally break-pointing!) the production app (really, a staging app) locally (do people do that?), or something that spits out every single thing that executes in code?
Thanks
I'd check all of your gems and plugins and any other dependencies your application might have: I'd guess that you're missing something (or have the wrong version of something) and it's failing badly enough that it can't even spit out a can't find require message.
check your action controller. On line 7 is most likely where the loop occurs.
Seems you have initiated some endless recursion on line 7 of your controller. Check if you are calling the same method inside the method.
We are trying to update a Rails Server to release 5.1.
Server starts fine; but on the first request, goes completely dead; and has to be killed with signal 9.
Doesn't matter if its Puma or Webrick.
Doesn't matter if its 5.1.0 or 5.1.7
Doesn't matter if its development or production mode.
Eventually I saw the process size was 90GB and growing!
I've tried rbtrace, but struggled to get anything meaningful out of it.
I'm on osx, so strace isn't available, and I've struggled to get dtrace or dtruss to work, or produce anything meaningful.
So looking for a way to get this rails server to tell me what it's problem is....
Let me know what additional information is salient.
After quite a long process, I found a solution that didn't so much find the source of the issue, but provided a process to work around it.
First off, I used
rails app:update
And accepted all of the overwrites. Then using git, I walked through all of the removed code from my config file and returned just the required sections [like config/routes.rb, and ActionMailer config, for example].
Application then started right up, no issue.
This also led me to
http://railsdiff.org/5.0.7.2/5.1.0
Which is pretty critical for Rails upgrading. This is well worth consuming:
https://github.com/rails/rails/issues/31377#issuecomment-350422347