I'm using SimpleCov with my rails app. Everything is working, but there seems to be a bug or i'm doing something wrong.
My test coverage when I run rake test comes back at 90.77%.
Then rake test TEST=test/path/to/file_test which returns 66%.
Finally, rake test and the coverage returned is 66%.
Is there some caching issue i'm missing? Also, I've notice all the sudden on the low 66% it seems to be counting all the blank lines as failed(see img).
It's caused by this bug at the merge process: https://github.com/colszowka/simplecov/issues/444. The suggested workaround worked for me:
At lib/simplecov/merge_helpers.rb replace if element.nil? && new_array[i].nil? with if element.nil? || new_array[i].nil?
All credit goes to ksmurphy.
Related
Minitest crashes every time a test fails when I run rails test (Rails 5 and Ruby 2.4.2). For example, I forced a simple test to fail by switching the assert to assert_not:
Failure: TransactionTest#test_transaction_should_be_valid [/home/.../test/models/transaction_test.rb:11]: Expected true to be nil or false
/home/.../.rvm/gems/ruby-2.4.2/gems/railties-5.1.4/lib/rails/test_unit/reporter.rb:70:in `method': undefined method `test_transaction_should_be_valid' for class `Minitest::Result' (NameError)
If the test passes, then the entire report is successful and I get the report summary. If only one test fails I get the error report above. I defined test_transaction_should_be_valid in my transaction_test.rb file and the proof of it is that it runs smoothly when the test passes.
I'm stuck on this now. Any ideas as to what might be causing this?
Apparently Rails is not yet quite compatible with Minitest 5.10/5.11: https://github.com/seattlerb/minitest/issues/730 They have posted a monkeypatch to tide us over.
Rails has apparently also fixed this in edge. https://github.com/rails/rails/pull/31624
This error disappeared for me when I pulled from the master branch. Put this in your Gemfile:
gem 'rails', git: 'https://github.com/rails/rails.git'
edit: promoting from comments (thanks coco9nyc):
or you can try downgrading minitest:
gem 'minitest', '5.10.3'
I'm struggling with this for quite a while now: I'm trying to upgrade an app from Rails 3.2 to Rails 4. While on Rails 3.2 all specs are passing, they fail under certain conditions in Rails 4.
Some specs are passing in isolation while failing when run together with other specs.
Example Video
https://www.wingolf.org/ak-internet-files/Spec_Behaviour.mp4 (4 mins)
This example video shows:
Running 3 specs using :focus–––green.
Running them together with another spec–––two specs passing before now fail.
Running the 3 specs, but inserting two empty lines–––one spec fails.
Undo does not help when using guard.
focus/unfocus does not help.
Restarting guard does not help.
Running all specs and then running the 3 specs again does help and make them green again. But adding the other task makes two specs fail, again.
As one can see, some specs are red when run together with other specs. Even entering blank lines can make a difference.
More Observations
For some specs, passing or failing occurs randomly when run several times.
The behavior is not specific to one development machine but can be reproduced on travis.
To delete the database completely between the specs using database_cleaner does not help.
To Rails.cache.clear between the specs does not help.
Wrapping each spec in an ActiveRecord::Base.transaction does not help.
This does occur in Rails 4.0.0 as well as in Rails 4.1.1.
Using this minimal spec_helper.rb without spring or anything does not help.
Using guard vs. using bundle exec rspec some_spec.rb:123 directly doesn't make a difference.
This behavior goes for model specs, thus doesn't have to do anything with parallel database connections for features specs.
I've already tried to keep as many gems at the same version as in the (green) Rails-3.2 branch, including guard, rspec, factory_girl, etc.–––does not help.
Update: Observations Based on Comments & Answers
Thanks to engineerDave, I've inserted require 'pry'; binding.pry; into one of the concerning specs. Using the cd and show-source of pry, it was ingeniously easy and fun to narrow down the problem: Apparently, this has_many :through relation does not return objects when run together with other specs, even when called with (true).
has_many(:groups,
-> { where('dag_links.descendant_type' => 'User').uniq },
through: :memberships,
source: :ancestor, source_type: 'Group'
)
If I call groups directly, I get an empty result. But if I go through the memberships, the correct groups are returned:
#user.groups # => []
#user.groups(true) # => []
#user.memberships.collect { |m| m.group } # returns the correct groups
Has Rails changed the has many through behavior in Rails 4 in a way that could be responsible? (Remember: The spec works in isolation.)
Any help, insights and experiences are appreciated. Thanks very much in advance!
Code
Current master branch on Rails 3.2––all green.
Rails-4 branch––strange behavior.
The file/commit seen in the video––strange behavior.
All specs passing on travis for Rails 3.2.
Diff of the Gemfile.lock (or use git diff master..sf/rails4-minimal-update Gemfile.lock |grep rspec)
How to Reproduce
This is how one can check if the issue still exists:
Preparation
git clone git#github.com:fiedl/wingolfsplattform.git
cd wingolfsplattform
git checkout sf/rails4-minimal-update
bundle install
# please create `config/database.yml` to your needs.
bundle exec rake db:create db:migrate db:test:prepare
Run the specs
bundle exec rspec ./vendor/engines/your_platform/spec/models/user_group_membership_spec.rb
bundle exec rspec ./vendor/engines/your_platform/spec/models/user_group_membership_spec.rb:213
The problem still exists, if the spec :213 is green in the second call but is red when run together with the other specs in the first call.
Based on that you're using:
the should syntax
that you indicate you've upgraded recently (perhaps a bundle update?)
that your failure messages indicate a NilObject error.
Is something like this perhaps what is causing it?
https://stackoverflow.com/a/16427072/793330
Are you are calling an object in your test which hasn't been instantiated yet?
I think this might be an rspec 3 upgrade issue where should is deprecated.
Have you ruled out an rspec gem upgrade to the new rspec 3 syntax (2.99 or 3.0.0+) as the culprit?
"Remove support for deprecated expect(...).should. (Myron Marston)"
IMO this behavior would not be caused by a Rails 4 update as its centered around your test suite.
Update (with pry debug):
Also you could use the pry gem to get a window into what is going on in your specs.
Essentially you can put a big "stop" sign (similar to a debug break) right before the spec executes to get a handle on the environment at that point.
it {
require 'pry'; binding.pry
should == something
}
Although beaware sometimes these pry calls wreck havoc on guard's threading and you have to kill it with CTRL+Z and then kill -9 PID that shows.
Update #2: Looking at updated answer.
You might be running up against FactoryGirl issues based on your has_many issue
You may need to trigger a before action in your Factory to pre-populate the associated record. Although this could get messy, i.e. here be monsters, you can trigger after and before callbacks in your factory that will bring these objects into being.
after(:create) do |instance|
do stuff here
end
I'm trying to test minitest files like this:
COVERAGE=true ruby -Itest test/views/info_pages_test.rb
COVERAGE=true ruby -Itest test/views/errors_test.rb
now my info_pages_test has 97% coverage and my errors_test has 75% coverage. Together they should cover 100%.. but each time I run the above commands, I get one result or the other 75% or 97%. Is there a way to combine the results of the two test files into one coverage report?
Help would be greatly appreciated!
Here is the top of my minitest_helper.rb file
## SimpleCOV
require 'simplecov'
if ENV["COVERAGE"]
SimpleCov.start('rails') do
add_filter "/test/"
end
puts "Started SimpleCOV"
end
I also have a .simplecov file in the application root but using it gives me unpredictable results.. I get 100% coverage once in a while and it is unpredictable..
.simplecov file
SimpleCov.use_merging true
SimpleCov.merge_timeout 3600
The problem you are bumping into is that each of those "test suites" will overwrite each other because the suite name (configurable via SimpleCov.command_name 'xyz'), as opposed to when merging for example cucumber and rspec results.
Preferred solution: Generate the coverage report by running the whole test suite at once, using rake test or some other, similar facility.
If you insist on running individual test files you can trick SimpleCov into merging those results instead of overwriting them by supplying a pseudo-random command name, i.e. SimpleCov.command_name "MiniTest #{Time.now}", or (depending on your setup) using ARGV, i.e. SimpleCov.command_name "Minitest #{File.basename(ARGV[1])}". The latter has the advantage of not duplicating results on re-runs of the same test file since those will be overwritten on merge, but may fail when you run all your tests and do not check for the presence of ARGV correctly, or your test framework tampers with ARGV before you can grab it.
Although you can make this work for individual test runs, in general I'd recommend to base coverage reports off full test suite runs only, as the other approaches leave room for error.
We're using Spork with Rspec and if we run Spork, our tests pass, but if we don't start spork and run the test with:
bundle exec rspec spec
Several failures occur, and all of them are the ones using the should.have syntax like:
inactive_user.received_messages.should have(1).message
1) Message introduction messages to active users should be created as messages to both users
Failure/Error: initiator.sent_messages.should have(1).message
expected 1 message, got 6
What's interesting about the number is that that's how many messages are in the database total, so :
initiator.sent_messages.should have(1) == Message.count
Without Spork, if I modify the test like:
inactive_user.received_messages.count.should == 1
everything works fine. So it seems like the matching method is looking at the wrong count. Any idea why this would be?
I have the same problem. I am using shoulda gem to test relationship. I have Instance class that has many WeeklyStatistics and the first time I run :
should have_many(:weekly_statistics)
with spork the test was red indicating that I should set up foreigh key instance_id to the weekly stats but after I have done it the test still failed with the same error - missing association. Then I stopped spork and the test was green.
When running unit and functional tests using rake, on a rails application, I notice that there is a seed value which is specified on the command line: --seed x
$ rake test
(in /code/blah)
Loaded suite /../ruby-1.9.2-p180/gems/rake-0.8.7/lib/rake/rake_test_loader
Started
.
Finished in 0.12345 seconds.
1 tests, 1 assertions, 0 failures, 0 errors, 0 skips
Test run options: --seed 20290
I assume it is possible to use this value in the tests, but I can't figure out how.
I've tried Google, Rails Guides et al. but can't seem to find the answer.
EDIT:
Could this seed value be the option that is used by Minitest to randomize the execution order of tests?
I found this online about MiniTest: http://www.mikeperham.com/2012/09/25/minitest-ruby-1-9s-test-framework/
Turns out, you are right. It is about randomizing the execution order of tests. You can explicitly use them like this:
rake TESTOPTS="--seed=1261"
(according to the above link).
The answer from MrDanA is correct. This solution also works and is slightly shorter and easier to remember.
SEED=20290 rake test