Is there a way to get a report of how long each test is taking to run on a Ruby on Rails project? I have a a small set of tests, 2 or 3, which take about 50% of the time and I'd like to find out which ones are.
Ideas of the top of my mind:
test_benchmark: http://github.com/timocratic/test_benchmark
test_benchmarker: http://github.com/myronmarston/test_benchmarker/
If you are running rspec, you can use the command --format profile will give you the top 10 slowest tests
You could write data ... somewhere during setup and teardown of a test.
Related
When running minutest tests, is it possible to peek at the information about the errors that has happened?
For example, this test suite takes ten minutes to complete. But I would like some more info about the letter E appearing in the tests result.
I don't want to wait ten minutes.
*** Running FRONTEND component engine specs
Run options: --seed 29704
# Running:
......................................................................................................................................................................................E...........
That's E for "error", so one of your tests is failing. Normally you get output that explains more. Once you identify which test is failing you can run that test in a more focused capacity, like:
ruby test/unit/broken_test.rb --name=test_that_is_broken
Where that is the path to your test script and the name of the testing method that's failing.
You may need to make your tests self-contained, able to be run this way, by using:
require_relative '../test_helper'
Or whatever the helper stub is that kicks off the testing framework. Some skeleton files contain things like require 'test_helper' which won't be found in your current $LOAD_PATH.
I am writing Rails tests using the standard Test::Unit/TestCase.
Is there any way to somehow filter what gets printed to the log, so that you only print the stack for specific test cases.
I have a functional test file with many test cases in it, and I'm really only interested in debugging one test case. Printing my own log statements still requires searching through a few thousand lines of generated log. Or something similar to the RSpec 'pending' functionality.
Run from a command line ruby test/unit/my_model.rb to run one test suite. You can also use a debugger, such as (wrapped by) RubyMine or pry, to stop on a specific test case and look at the log.
But if a sledge-hammer does not solve the problem, you can use tweezers: config.logger.level = Logger::WARN in your test.rb, from Set logging levels in Ruby on Rails
It is probably better if instead of strangling the output to log/test.log, you become familiar with a command such as grep. Grep allows you to run very advanced search queries through files or directories, as long as your running on some flavor of *nix. The simplest use would be
grep search_term file_name
The reason I say you shouldn't constrict the log output is because someday that could bit you in the **s. Hope this helps.
I'm using Rails 3, Thinking-sphinx 2.0.2 and Cucumber 0.10.2. I would like to test my thinking-sphinx search statements in my integration tests using Cucumber, but it seems very slow.
I've added to my env.rb file:
require 'cucumber/thinking_sphinx/external_world'
Cucumber::ThinkingSphinx::ExternalWorld.new
Cucumber::Rails::World.use_transactional_fixtures = false
As soon as I add these lines and run 'time rake cucumber' on a fresh project with no tests written yet it takes 48seconds (on a i7-930 with 12GB ram and an Intel SSD). If I take out the thinking_sphinx lines it takes 10 seconds.
Is this normal?
If so can I limit the loading of sphinx for anything with tag #slow?
What happens if you remove the last line (ie. set transactional fixtures back to true). The TS docs are a little out of date - refer to my blog post instead.
That said, the second line could be the cause of the slowness - as it'll automatically configure, index and start Sphinx - and then stop it once the task is finished.
I'm having trouble debugging cucumber steps due to unique conditions of the testing environment. I wish there was a step that could pause a selenium test and let me take over.
E.g.
Scenario: I want to take over here
Given: A bunch of steps have already run
When: I'm stuck on an error
Then: I want to take control of the mouse
At that point I could interact with the application exactly as if I had done all the previous steps myself after running rails server -e test
Does such a step exist, or is there a way to make it happen?
You can integrate ruby-debug into your Cucumber tests. Nathaniel Ritmeyer has directions here and here which worked for me. You essentially require ruby-debug, start the debugger in your environment file, and then put "breakpoint" where ever you want to see what's going on. You can both interact with the browser/application and see the values of your ruby variables in the test. (I'm not sure whether it'll let you see the variables in your rails application itself - I'm not testing against a rails app to check that).
I came up with the idea to dump the database. It doesn't let you continue work from the same page, but if you have the app running during the test, you can immediately act on the current state of things in another browser (not the one controlled by Selenium).
Here is the step:
When /I want to take control/i do
exec "mysqldump -u root --password=* test > #{Rails.root}/support/snapshot.sql"
end
Because it is called by exec, DatabaseCleaner has no chance to truncate tables, so actually it's irrelevant that the command is a database dump. You don't have to import the sql to use the app in its current state, but it's there if you need it.
My teammate has done this using selenium, firebug a hook (#selenium_with_firebug)
Everything he learned came from this blogpost:
http://www.allenwei.cn/tips-add-firebug-extension-to-capybara/
Add the step
And show me the page
Where you want to interact with it
Scenario: I want to take over here
Given: A bunch of steps have already run
When: I'm stuck on an error
Then show me the page
use http://www.natontesting.com/2009/11/09/debugging-cucumber-tests-with-ruby-debug/
Big thank you to #Reed G. Law for the idea of dumping the database. Then loading it into development allowed me to determine exactly why my cucumber feature was not impacting database state as I had expected. Here's my minor tweak to his suggestion:
When /Dump the database/i do
`MYSQL_PWD=password mysqldump -u root my_test > #{Rails.root}/snapshot.sql`
# To replicate state in development run:
# `MYSQL_PWD=password mysql -u root my_development < snapshot.sql`
end
You can also use the following in feature/support/debugging.rb to let you step through the feature one step at a time:
# `STEP=1 cucumber` to pause after each step
AfterStep do |scenario|
next unless ENV['STEP']
unless defined?(#counter)
puts "Stepping through #{scenario.title}"
#counter = 0
end
#counter += 1
print "At step ##{#counter} of #{scenario.steps.count}. Press Return to"\
' execute...'
STDIN.getc
end
>rails -v
Rails 1.2.6
>ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]
When I run a test fixture (that tests a rails model class) like this, it takes 20-30 secs to start executing these tests (show the "Loaded suite..."). What gives?
>ruby test\unit\category_test.rb
require File.dirname(__FILE__) + '/../test_helper'
class CategoryTest < Test::Unit::TestCase
def setup
Category.delete_all
end
def test_create
obCategoryEntry = Category.new({:name=>'Apparel'})
assert obCategoryEntry.save, obCategoryEntry.errors.full_messages.join(', ')
assert_equal 1, Category.count
assert_not_nil Category.find(:all, :conditions=>"name='Apparel'")
end
#.. 1 more test here
end
This one is Rails using a MySql DB with no fixtures. This time it clocked 30secs+ to startup.
Take a look at this Rails Test Server.
A quote from the author:
"Every time you run a test in a Rails
application, the whole environment is
loaded, including libraries that don’t
change between two consecutive runs.
That can take a considerable amount of
time. What if we could load the
environment once, and only reload the
changing parts before each run?
Introducing RailsTestServing.
With RailsTestServing, the run time of
a single test file has gone from 8
seconds down to .2 of a second on my
computer. That’s a x40 speed
improvement. Now, I don’t think twice
before hitting ⌘R in TextMate. It
feels liberating!"
(This was featured on the Rails Envy Podcast this past week which is where I found this.)
When starting any tests, Rails first loads any fixtures you have (in test/fixtures) and recreates the database with them.
20-30 seconds sounds very slow though. Do you have a lot of fixtures that need to be loaded before your tests run, or is your database running slow?
Ruby's gem tool follows a path discovery algorithm which, apparently, is not Windows (as I see from your ruby -v) friendly.
You can get a clear picture if you trace, for example, a Rails application loading with ProcMon. Every (I really mean every) require starts a scan over all directories in Ruby's path plus all gem directories. A typical require takes 20 ms on an average machine. Since Rails makes hundreds of requires, those 20 ms easily sum up to seconds every time you launch the Rails environment. Take in the time to initialize the fixtures in the database and you get a better idea of why it takes so much time to just begin running the test-cases.
Perhaps because of each file-system architecture and implementation (path caching etc.), this is less of a problem in Linux than in Windows. I don't know who you should blame, though. It looks like the NTFS file-system could be improved with a better path caching implementation, but clearly the gem tool could implement the caching itself and have its performance not so dependent on the platform.
It seems like Test::Unit is the simplest, but also one of the slowest ways to do unit testing with Ruby. One of alternatives is ZenTest.
Test unit startup isn't particularly slow, and nowhere near 20 seconds.
(11:39) ~/tmp $ cat test_unit.rb
require 'test/unit'
class MyTest < Test::Unit::TestCase
def test_test
assert_equal("this", "that")
end
end
(11:39) ~/tmp $ time ruby test_unit.rb
Loaded suite test_unit
Started
F
Finished in 0.007338 seconds.
1) Failure:
test_test(MyTest) [test_unit.rb:4]:
<"this"> expected but was
<"that">.
1 tests, 1 assertions, 1 failures, 0 errors
real 0m0.041s
user 0m0.027s
sys 0m0.012s
It's probably something you're doing in your tests. Are you doing anything complicated? Setting up a database? Retrieving something from the internet?
Complete shot in the dark, but the majority of the time I see long startup times on things, it is usually due to some sort of reverse DNS lookup happening with some TCP socket communication somewhere along the way.
Try adding:
require 'socket'
Socket.do_not_reverse_lookup = true
at the top of your test file after your other require line.
What does your test_helper.rb look like? Are you using instantiated fixtures?
self.use_instantiated_fixtures = true
[edit]
If this is set to true try setting it to false.