I started along the process of trying to speed up my tests, mostly by following this railscast's suggestions. So you don't have to watch it, those are:
Replaced "bundle exec spec" with "bin/rspec spec"
Got rid of excess before filters
Tagged slow tests as such for separate running
Replaced unnecessary Factory Creates with Builds
Used Zeus to speed up start-up time
Applied VCR to examples that reached out to external APIs
Deferred Garbage Collection
(I skipped parallel testing for now, cause I want to get to the bottom of this issue, and parallel testing seems like it would just minimize/mask it)
At any rate, I can't figure out whether these changes have accomplished much, because the runtimes for my tests are highly variable. (They are variable even when I go outside of the branch I've been working on for these tests -- seems like something preexisting is causing the problem). I've run the exact same suite of tests now and gotten a wide range of runtimes, from 52 seconds to almost 1m45! Again, the EXACT same tests.
Moreover, the general trend seems to be that if I run a test a few times in a row, the runtimes go up by ~20 second intervals each time. Then if I change something small in spec_helper or wait for a while (I think the latter -- I know this has happened when I've made small changes to garbage collection), the times go back down.
My guess is that this problem is related to garbage collection -- but ideally, I'm doing that more efficiently now. I had it collect garbage every ~15 seconds,
spec_helper.rb
config.before(:all) { DeferredGarbageCollection.start }
config.after(:all) { DeferredGarbageCollection.reconsider }
support/deferred_garbage_collection.rb
class DeferredGarbageCollection
DEFERRED_GC_THRESHOLD = (ENV['DEFER_GC'] || 15.0).to_f
##last_gc_run = Time.now
def self.start
GC.disable if DEFERRED_GC_THRESHOLD > 0
end
def self.reconsider
if DEFERRED_GC_THRESHOLD > 0 && Time.now - ##last_gc_run >= DEFERRED_GC_THRESHOLD
GC.enable
GC.start
GC.disable
##last_gc_run = Time.now
end
end
end
and then, when that was causing the aforementioned problems, I switched to a more straightforward every-ten-test collection system:
spec_helper.rb
config.after(:each) do
counter += 1
if counter > 9
GC.enable
GC.start
GC.disable
counter = 0
end
end
config.after(:suite) do
counter = 0
end
I've tried to isolate this to a group of tests (models/requests/controllers), but they all seem to display some amount of variability in relation to roughly how much time they eat up.
Any ideas what's going wrong here?
EDIT -- proof/example of what's going on here:
Finished in 22.88 seconds
48 examples, 0 failures
➜ my_app git:(faster-tests) ✗>zeus test spec/models
................................................
Finished in 34.89 seconds
48 examples, 0 failures
➜ my_app git:(faster-tests) ✗>zeus test spec/models
................................................
Finished in 44.68 seconds
48 examples, 0 failures
➜ my_app git:(faster-tests) ✗>zeus test spec/models
................................................
Finished in 14.36 seconds
48 examples, 0 failures
➜ my_app git:(faster-tests) ✗>zeus test spec/models
................................................
Finished in 18.74 seconds
48 examples, 0 failures
➜ my_app git:(faster-tests) ✗>
Note that it eventually seems to reset.
I'm not sure about all the other things you have done but I can comment on zeus. I did not alter anything in my environment except add zeus and I can run 50 examples in a couple of seconds. For tests to take as long as yours when running zeus seems wrong and I'm guessing thats your problem. After you run zeus start are all the zeus commands green? I also use zeus rspec spec/.... to run my specs instead of zeus test.
I used this railscast and this thoughtbot article along with the zeus README. Remeber to use the appropriate Ruby version mentioned in the railscast and make sure that you dont have spork anymore if you were using spork. Also remember to install zeus outside of your Gemfile, zeus should not use Bundler. Anyway, my suggestion would be to focus on zeus first and get that working correctly. It has really helped for me
Related
Trying to run rspec (bin/rspec) from my project root, and suddenly it is not running the full test suite - this is usually over 1,000 examples.
Finished in 1.88 seconds (files took 5.17 seconds to load)
1 example, 0 failures
I can run individual files or sub-directories (e.g. spec/system/*), and it works fine in other projects run as normal.
The only thing I can think of is that I did enter the test console (bin/rails c test) recently where I made a user, then went onto the test server in localhost, afterwards destroying the user when I realised it wasn't what I needed. I've also reset the test database and cleared my tmp cache just in case.
Any ideas on how to troubleshoot the cause would be appreciated!
Have you changed file names? To run rspec by default your filenames should end with _spec.rb
Problem now fixed - solution:
I tried running each folder, and I did notice a difference in the focus: true:
bin/rspec spec/controllers:
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
bin/rspec:
Run options: include {:focus=>true}
Commented out the lines in spec_helper to check, and it ran everything correctly then:
# config.filter_run_including focus: true
# config.run_all_when_everything_filtered = true
Searching for focus showed nothing in specs, so I added something to print the metadata in my spec_helper
config.before do |x|
p x.metadata
end
and found the one example it was focusing on, and lo and behold the example it happened to have accidentally been pre-pended with 'fit' instead of 'it' from a prior pulled commit which was causing the focus.
Thanks for the help :-)
In my application, I need to execute a task every 5 minutes. The task itself is quite lean, so that's not the problem: the point is that loading the Rails environment uses almost all the CPU of the server. Right now, I'm using whenever with a code like this in my schedule.rb:
every 5.minutes do
runner Task.perform
end
But the cost is too big. Is there anyway to execute this kind of tasks in a preloaded environment? Or any other better solution?
Consider using a background process (many gems, notably delayed job) and implementing a simple "cron" in your code with a polling loop something like
def perform
last = Time.now
while true
if last.sec % (5 * 60) != 0
do_task
last = Time.now
end
end
end
Warning: this example has more holes than a slice of swiss cheese. Write something better :-)
You could use a cron job or set it up to run in a bash script. These would have the advantage of not being tied to your application.
When running unit and functional tests using rake, on a rails application, I notice that there is a seed value which is specified on the command line: --seed x
$ rake test
(in /code/blah)
Loaded suite /../ruby-1.9.2-p180/gems/rake-0.8.7/lib/rake/rake_test_loader
Started
.
Finished in 0.12345 seconds.
1 tests, 1 assertions, 0 failures, 0 errors, 0 skips
Test run options: --seed 20290
I assume it is possible to use this value in the tests, but I can't figure out how.
I've tried Google, Rails Guides et al. but can't seem to find the answer.
EDIT:
Could this seed value be the option that is used by Minitest to randomize the execution order of tests?
I found this online about MiniTest: http://www.mikeperham.com/2012/09/25/minitest-ruby-1-9s-test-framework/
Turns out, you are right. It is about randomizing the execution order of tests. You can explicitly use them like this:
rake TESTOPTS="--seed=1261"
(according to the above link).
The answer from MrDanA is correct. This solution also works and is slightly shorter and easier to remember.
SEED=20290 rake test
I'm using Rails 3, Thinking-sphinx 2.0.2 and Cucumber 0.10.2. I would like to test my thinking-sphinx search statements in my integration tests using Cucumber, but it seems very slow.
I've added to my env.rb file:
require 'cucumber/thinking_sphinx/external_world'
Cucumber::ThinkingSphinx::ExternalWorld.new
Cucumber::Rails::World.use_transactional_fixtures = false
As soon as I add these lines and run 'time rake cucumber' on a fresh project with no tests written yet it takes 48seconds (on a i7-930 with 12GB ram and an Intel SSD). If I take out the thinking_sphinx lines it takes 10 seconds.
Is this normal?
If so can I limit the loading of sphinx for anything with tag #slow?
What happens if you remove the last line (ie. set transactional fixtures back to true). The TS docs are a little out of date - refer to my blog post instead.
That said, the second line could be the cause of the slowness - as it'll automatically configure, index and start Sphinx - and then stop it once the task is finished.
>rails -v
Rails 1.2.6
>ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]
When I run a test fixture (that tests a rails model class) like this, it takes 20-30 secs to start executing these tests (show the "Loaded suite..."). What gives?
>ruby test\unit\category_test.rb
require File.dirname(__FILE__) + '/../test_helper'
class CategoryTest < Test::Unit::TestCase
def setup
Category.delete_all
end
def test_create
obCategoryEntry = Category.new({:name=>'Apparel'})
assert obCategoryEntry.save, obCategoryEntry.errors.full_messages.join(', ')
assert_equal 1, Category.count
assert_not_nil Category.find(:all, :conditions=>"name='Apparel'")
end
#.. 1 more test here
end
This one is Rails using a MySql DB with no fixtures. This time it clocked 30secs+ to startup.
Take a look at this Rails Test Server.
A quote from the author:
"Every time you run a test in a Rails
application, the whole environment is
loaded, including libraries that don’t
change between two consecutive runs.
That can take a considerable amount of
time. What if we could load the
environment once, and only reload the
changing parts before each run?
Introducing RailsTestServing.
With RailsTestServing, the run time of
a single test file has gone from 8
seconds down to .2 of a second on my
computer. That’s a x40 speed
improvement. Now, I don’t think twice
before hitting ⌘R in TextMate. It
feels liberating!"
(This was featured on the Rails Envy Podcast this past week which is where I found this.)
When starting any tests, Rails first loads any fixtures you have (in test/fixtures) and recreates the database with them.
20-30 seconds sounds very slow though. Do you have a lot of fixtures that need to be loaded before your tests run, or is your database running slow?
Ruby's gem tool follows a path discovery algorithm which, apparently, is not Windows (as I see from your ruby -v) friendly.
You can get a clear picture if you trace, for example, a Rails application loading with ProcMon. Every (I really mean every) require starts a scan over all directories in Ruby's path plus all gem directories. A typical require takes 20 ms on an average machine. Since Rails makes hundreds of requires, those 20 ms easily sum up to seconds every time you launch the Rails environment. Take in the time to initialize the fixtures in the database and you get a better idea of why it takes so much time to just begin running the test-cases.
Perhaps because of each file-system architecture and implementation (path caching etc.), this is less of a problem in Linux than in Windows. I don't know who you should blame, though. It looks like the NTFS file-system could be improved with a better path caching implementation, but clearly the gem tool could implement the caching itself and have its performance not so dependent on the platform.
It seems like Test::Unit is the simplest, but also one of the slowest ways to do unit testing with Ruby. One of alternatives is ZenTest.
Test unit startup isn't particularly slow, and nowhere near 20 seconds.
(11:39) ~/tmp $ cat test_unit.rb
require 'test/unit'
class MyTest < Test::Unit::TestCase
def test_test
assert_equal("this", "that")
end
end
(11:39) ~/tmp $ time ruby test_unit.rb
Loaded suite test_unit
Started
F
Finished in 0.007338 seconds.
1) Failure:
test_test(MyTest) [test_unit.rb:4]:
<"this"> expected but was
<"that">.
1 tests, 1 assertions, 1 failures, 0 errors
real 0m0.041s
user 0m0.027s
sys 0m0.012s
It's probably something you're doing in your tests. Are you doing anything complicated? Setting up a database? Retrieving something from the internet?
Complete shot in the dark, but the majority of the time I see long startup times on things, it is usually due to some sort of reverse DNS lookup happening with some TCP socket communication somewhere along the way.
Try adding:
require 'socket'
Socket.do_not_reverse_lookup = true
at the top of your test file after your other require line.
What does your test_helper.rb look like? Are you using instantiated fixtures?
self.use_instantiated_fixtures = true
[edit]
If this is set to true try setting it to false.