Has anyone experienced this? We upgraded a project from Rails 5.2 to Rails 6.0.0 and after this, the memory consumption skyrocketed... In the release candidate environment, it works fine, but in production, the container dies because of the memory usage... The instances we have in 5.2 can do just fine with 1GB RAM, but the instances we test for Rails 6.0.0 dies immediately, even if we give them 4GB ram.
We already tested https://github.com/schneems/derailed_benchmarks, but the require gems just increased like 10 MiB in total.
We have some heavy queries but we don't know why this happens in Rails 6 and not in Rails 5.
This hugely depends on what gems you have.
Also, I would start by making use of a monitoring tool like Scout or NewRelic to understand where/how is the leak happening.
Related
I am running Mac OSX 12.1. I have some rails 6 apps and they are working normally in regard to memory usage.
But rails 5.2.6 has a strange memory issue. When I start a rails server or run rspec, the usage of the CPU is normal i.e. low when not much is happening but higher when large tests or heavy server usage occurs.
However, when I use rails console the CPU usage climbs almost immediately to over 100%, and pretty soon the Mac fan starts to run. It stays this way even if I exit the console. I have to do killall ruby to stop all ruby processes. I am monitoring memory using top -o cpu. Is this a known issue with rails 5.2.6? Is there a way to stop this happening?
I was experiencing the same problem for quite a while. I've eventually noticed that it's a problem with the listen gem. Updating it is enough to fix it.
Update it to the latest version possible. If I remember correctly, it has to be at least v3.3 for the problem to go away.
I suggest to check what initializes when you do rails c. You def should profile your startup times. Start checking your config/initializers folder and also consider Profiling Rails Boot Time. Also check how to Speed up Rails boot time.
I don't believe this is specific to 5.2.6 but make sure you use at least ruby 2.7.0 ish which also affects your performance.
We have completed upgrading our app from Rails 4.2 to 5.2. When we run load testing on the 5.2 version it can only handle half the load of the 4.2 version. In looking at NewRelic stats during the load tests it seems to be slower everywhere - pretty much every request, ActiveRecord calls, redis calls, ruby, etc. We have confirmed it is not related to other upgrades that happened in addition - ruby upgrade, upgrading pg gem, or upgrading puma. While researching, the only performance issues I have found related to the upgrade have been fixed.
Has anyone run into something similar or have pointers on where to look?
What we have tried so far:
1. Check non-rails related upgrades that happened at the same time:
- Upgrade 4.2 branch to same version of ruby to see if that has any impact (no impact)
- Downgrade puma and pg gems in Rails 5 branch (no impact)
2. Examine performance traces for slower transactions and DB queries. Remove the slowest interactions from the load test to see if the overall slowest continues (it does).
3. Test if slowness appears in Rails 5.1 (it does).
What we are planning to try:
1. Test if slowness appears in Rails 5.0.
2. See if slowest can be detected in single user use rather than load test.
3. Use https://github.com/tleish/ruby-prof-rails to see if we can get more statistics to examine.
4. Downgrade all gems except the ones we absolutely need for the Rails 5 upgrade and see if problem still exists.
This ended up being a combination of Rack::Timeout, Heroku and Puma. Under heavy load, we would sometimes hit our Rake::Timeout value of 28 seconds. For some reason after the upgrade, the 2 seconds between the Rake::Timeout value and Heroku's 30 second router timeout (H12) was not enough. As a result, processes were getting killed by Heroku before Rake::Timeou which was causing a cascade effect in Puma that made a ton of other requests on the same server also timeout. We fixed it by adjusting our Rake::Timeout value to 25 seconds and everything worked.
I'm having a problem with Ruby on Rails running extremely slowly. I'm using Ruby 2.1.3p242 and Rails 4.2.1 on a Windows 8 machine.
Whenever I run anything that requires rails to boot (including tests) it takes a long time to get up and running. I put some calls to Benchmark in config/environment.rb on a clean install of rails:
require File.expand_path('../application', __FILE__)
User cpu System Cpu Total Cpu elapsed time
0.000000 0.000000 0.000000 (0.000000)
Rails.application.initialize!
15.282000 2.891000 18.173000 ( 18.201173)
Clearly Rails.application.initialize is taking an absurdly long time considering its a clean install.
Thanks in advance for your help
Edit-1: I'm running on a dual core i3 4010u#1.7GHZ with 4gb of RAM. I don't think my machine is too bad as it runs most things very well.
Edit-2: I ran ruby-prof on Rails.application.initialize and found the culprit. A process was taking up 85% of the run time:
<Module::SecureRandom>#random_bytes
<Module::OpenSSL::Random>#random_bytes
This is apparently occuring in Ruby21/lib/ruby/2.1.0/securerandom.rb#62
I looked up line 62 in that file and this is what I found:
return OpenSSL::Random.random_bytes(n)
So anyone have any idea what this means?
Edit-2: I ran ruby-prof on Rails.application.initialize and found the
culprit. A process was taking up 85% of the run time:
<Module::SecureRandom>#random_bytes
<Module::OpenSSL::Random>#random_bytes
Yeah, the OpenSSL code for seeding the random number generator is problematic on Windows. See Random Numbers and Windows Issues on the OpenSSL wiki.
return OpenSSL::Random.random_bytes(n)
So anyone have any idea what this means?
Ruby is returning random numbers. In this case, OpenSSL will autoseed itself before retuning random number with RAND_poll since no other seed was provided.
Ruby should not call RAND_poll or allow it to be implicitly called by the library. If the random number generator has not been seeded, then the library will automatically seed itself by calling RAND_poll internally.
Rather, Ruby should read bytes from the OS using CryptGenRandom, and then call OpenSSL's RAND_seed. That will avoid the call to RAND_poll.
Putting this in my config/application.rb (before require 'rails/all') speeds up rails s by 10-15 seconds on windows.
require 'securerandom'
SecureRandom.hex(16)
I have been a windows rails dev for a while now. I have never solved this rails startup issue. Running a microsecond Rspec test takes rails 22 seconds to load on my PC.
When I (temporarily) comment out the line in securerandom.rb (all ruby versions) and replace it with a hardcoded return the startup time is reduced to 10 seconds.
#return OpenSSL::Random.random_bytes(n)
return "\xD3\x04F\f0\xD6{G\xB9\x81"
We have a big rails app and since a few days, one by one, our ruby process seems to block in a loop and eat 100% of the CPU until the passenger server die raising 502 erros.
Do you know the best ways to figure that out why?
I've tried New Relic but it's only performance stuff, and our errors are too numerous to try to guess what is the problem. (We have a lot of request a day and lots of UTF-8 BSON Errors because we are using UTF-8 urls)
Using:
Rails 3.2.6 with Ruby 1.9.2p290
Passenger 3.0.13
MongoDB 2.0.1 with Mongoid 2.4.11
Nginx
FreeBSD 8.2
A handy way to find out where your ruby is stuck is to attach gdb to the running process and call rb_backtrace(). That will print the current stack trace to stderr, which is likely directed to a log file. Here's a short walk through I found on a blog. The stack trace should help you localize where your process is stuck and allow you to figure out what's going on via code inspection or by adding logging instrumentation around the critical code path.
Aman Gupta has a gdb to ruby hooks library gdb.rb I've found handy at times.
Why is Ruby, and Ruby on Rails (1.8.6 One Click Installer, local database) so ruddy slow on Windows?
ruby script/server - 30 seconds
rake test - 45 seconds
etc.
Yet, when I pop over to a much slower linux box, it's virtually instantaneous. I've checked everything - no significant CPU processes running, no network issues... and so on.
Heck, I'd be happy with just a verbose output that at least told me where it was breaking down. Any suggestions?
In general Ruby's MRI interpreter is just not optimized for speed on windows. You might also be running it in development mode on windows vs production mode on the other machines. Rails runs much slower in development mode since it reloads all your classes on every request.
1.8.6 is a very old ruby version. Released almost 3 years ago. You should strongly consider upgrading to 1.9 (or at least 1.8.7). Or switching to JRuby. All of these options will likely lead to a significant performance improvement.
1.8.7 should be fully compatible with 1.8.6. 1.9 has a completely new interpreter that runs 2.5 times faster (Though it has a tendency to occasionally crash on windows). JRuby may be the ideal solution for you since you can run it in either compatibility for 1.8 or 1.9 and it is very stable, but it does not support gems with C extensions and requires a different database adapter.
One last option would be to try running Rails inside of a VMWare with CentOS or another Linux distribution.
The reason is that file stat's in windows are dreadfully slow, and, since Ruby is written on Linux (and optimized for Linux), there hasn't been much work to make it faster.
Using the rubyinstaller.org (1.8.6 or 1.9.x) can make it faster--I'd recommend 1.8.6 since 1.9 has some slowdowns of its own.
If you're looking to get really aggressive, you can try my faster_gem_script gem, which tries to cache the heck out of require based look ups and thus speed things up. Do it with a scratch version of ruby, though :)
Unfortunately Jruby also isn't known for its exceedingly fast lookups. Hopefully this situation will change someday. Until then my faster_gem_script and faster_require are the only way I know of to try to get some speedup.
For a speedup you could try my loader speeder upper (helps rails run faster in doze): https://github.com/rdp/faster_require
Also checkout spork, which works in doze, and jruby also works well.
-rp
UPDATE: Thanks (in part) to some really great work on Fenix by Luis Lavena, Ruby 1.9.3-p327 is much, much faster on Windows. rake used to take 110+ seconds to execute on 1.9.3-p125, and now takes ~20 seconds on p327. Rails is finally usable on Windows!!
Use RubyInstaller to install..
I like taking this approach:
slow rails stack
In my case its
finisher_hook: 22.463 sec
That is the culprit