TL;DR: How can I stop rspec aborting all specs when it encounters an error?
When running rails' built in testing suite, I get output similar to the following:
...F..F.EE...E
Finished in 0.64396 seconds
14 examples, 2 failures, 3 errors
The . represents a passing test, the F a failing test and the E an erroneous test. A failing test means the code of your rails application is failing, and an erroneous test means the code of your actual test is failing. Erroneous tests are never good.
What I do like about this though, is that when the program encounters an E, it keeps on going through all the other tests.
I'm using rpsec, and while I do like it, I sort of hate how, when it encounters an erroneous spec, it completely gives up the ghost, and quits all specs, like this:
12:39:37 - INFO - Running: spec
/usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/validations.rb:57:in `save!': Validation failed: Email has already been taken (ActiveRecord::RecordInvalid)
from /usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/attribute_methods/dirty.rb:41:in `save!'
from /usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/transactions.rb:275:in `block in save!'
from /usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/transactions.rb:326:in `block in with_transaction_returning_status'
While I do like the error reporting, I really want it to carry on running all specs when it encounters an erroneous spec. Rather than the above output, I really want something along the lines of the following:
12:40:01 - INFO - Running: spec
...F..F..E
Failures:
1) User A new user can be created email should == "awesomedog#hotmail.co.uk"
Failure/Error: its(:email) { should == "awesomedog#hotmail.co.uk" }
expected: "awesomedog#hotmail.co.uk"
got: "awesomedozg#hotmail.co.uk" (using ==)
# ./spec/model/user_spec.rb:11:in `block (3 levels) in <top (required)>'
2) User A new user can be created should not be valid
Failure/Error: it { should_not be_valid }
expected valid? to return false, got true
# ./spec/model/user_spec.rb:20:in `block (3 levels) in <top (required)>'
Errors:
1) `save!': Validation failed: Email has already been taken (ActiveRecord::RecordInvalid)
Finished in 0.64396 seconds
10 examples, 2 failures, 1 error
Failed examples:
rspec ./spec/model/user_spec.rb:11 # User A new user can be created email should == "awesomedog#hotmail.co.uk"
rspec ./spec/model/user_spec.rb:20 # User A new user can be created should not be valid
I'm using factory-girl and factories in my specs, and data-base cleaner to clean my database between each spec. I'm using rspec-guard to run all specs on the event of any project file (apart from those in tmp,log or db) being saved. Because rpsec keeps wimping out if it hits an error, I'm getting this error:
How can I clean my database between erroneous rspec specs?
Basically, database-cleaner is configured to clean my database when a spec begins and when a spec finishes. Because rspec quits in the middle of a spec when it hits an error, database-cleaner doesn't detect that the spec has finished, and so never cleans my database. This means I have to manually empty it with my database shell.
I'd also prefer to see the state of all my other specs, even if one is erroneous! Rspec is really silly in this regard methinks!
Here are my factories and my spec:
spec/model/user_spec.rb:
require 'spec_helper'
describe User do
context "Valid user" do
user = FactoryGirl.create(:user_with_all_valid)
subject { user }
its(:first_name) { should == "Jimmy" }
its(:last_name) { should == "Thehat" }
its(:profile_name) { should == "Jimbohatboy893" }
its(:email) { should == "awesomedog#hotmail.co.uk" }
its(:password) { should == "thisisasupersecretpassword12234234" }
its(:password_confirmation) { should == "thisisasupersecretpassword12234234" }
end
end
spec/factories.rb:
FactoryGirl.define do
factory :user_with_all_valid, class: User do
first_name "Jimmy"
last_name "Thehat"
profile_name "Jimbohatboy893"
email "awesomedog#hotmail.co.uk"
password "thisisasupersecretpassword12234234"
password_confirmation "thisisasupersecretpassword12234234"
end
end
Your issue is this line:
context "Valid user" do
user = FactoryGirl.create(:user_with_all_valid) # Right here!
...
Test setup should always be done in a before block; variable definition is done in a let block. Doing it in your test definition will cause rspec to fail as described. In general, you should only ever have rspec directives (before, after, let, subject, it, its, etc) in a context block. If you're running test setup, application code, whatever, you're going to open yourself up to this kind of problem. What you want is:
context "Valid user" do
let(:user) { FactoryGirl.create(:user_with_all_valid) }
or:
context "Valid user" do
before { #user = FactoryGirl.create(:user_with_all_valid) }
The suite will then run the whole thing as expected, and report failures for the tests that fail to do their defined setup (the user definition) properly, rather than bombing out of the whole suite.
You need to have a unique email each time due to the validation on the model.
btw the other 7 or 8 are passing (dots).
Try adding:
new_user_time = Time.new
to save the current time.
Then pass that in to the create user factory.
Then use it to check that the users email is correct.
Use sequences as detailed at: https://github.com/thoughtbot/factory_girl/blob/master/GETTING_STARTED.md#sequences
Unique values in a specific format (for example, e-mail addresses) can be generated using sequences. Sequences are defined by calling sequence in a definition block, and values in a sequence are generated by calling FactoryGirl.generate:
# Defines a new sequence
FactoryGirl.define do
sequence :email do |n|
"person#{n}#example.com"
end
end
FactoryGirl.generate :email
# => "person1#example.com"
FactoryGirl.generate :email
# => "person2#example.com"
...
The just have your spec check that part of the email address is correct, e.g. the #something.com or use a regular expression to compare, e.g. match(/person.*#something\.com/)
Related
Testing Rails model validations with RSpec, without testing AR itself
Lets as setup we have model User:
class User < ActiveRecord::Base
validate :name, presence: true, uniqueness: { case_sensitive: false }, on: :create
validate :password, presence: true, format: { with: /\A[a-zA-z]*\z/ }
end
A see several ways to test this:
it { expect(user).to validate_presence_of(:name).on(:create) }
or
it do
user = User.create(name: '')
expect(user.errors[:name]).to be_present
end
My main question is which of the approaches is better and why? Can suggest me different approach?
Additional questions:
How much should I test? As an example, I can write so many tests for the regex, but it will be hell for maintenance.
How much you think will be full test coverage in this example?
The functionalities of:
Rails being able to validate the presence of an arbitrary value on your model
errors being added to an object for an attribute that is missing when a validation for it is configured
are covered in the tests for Rails itself (specifically, in the ActiveModel tests).
That leaves needing to write the tests for the config that covers the business logic of your app eg validating the presence of the specific name attribute on your specific User class etc. In my opinion, the matchers from the shoulda-matchers gem should have you covered:
RSpec.describe User, type: :model do
subject(:user) { build(:user) } # assuming you're using FactoryGirl
describe 'validations' do
specify 'for name' do
expect(user).to validate_presence_of(:name).on(:create)
# NOTE: saving here is needed to test uniqueness amongst users in
# the database
user.save
expect(user).to validate_uniqueness_of(:name)
end
specify 'for password' do
expect(user).to validate_presence_of(:password)
expect(user).to allow_value('abcd').for(:password)
expect(user).to_not allow_value('1234').for(:password)
end
end
end
I think that unless you have specific custom error messages for your errors that you want to test for (ie you've overridden the default Rails ones), then tests like expect(user.errors[:name]).to be_present can be removed (even if you have custom errors, I still think they're of dubious value since those messages will become locale-dependent if you internationalise your app, so I'd test for the display of some kind of error on the page in a feature spec instead).
I can write so many tests for the regex, but it will be hell for maintenance.
I don't think you can really get around this when testing validations for format, so I'd suggest just write some representative test cases and then add/remove those cases as you discover any issues you may have missed, for example:
# use a `let` or extract out into a test helper method
let(:valid_passwords) do
['abcd', 'ABCD', 'AbCd'] # etc etc
end
describe 'validations' do
specify 'for password' do
valid_passwords.each do |password|
expect(user).to allow_value(password).for(:password)
end
end
end
How much you think will be full test coverage in this example?
I've gotten 100% code coverage from reports like SimpleCov when writing unit specs as described above.
These 2 of them should be used, because:
it { expect(user).to validate_presence_of(:name).on(:create) }
=> You are expecting the validate_presence_of should be run on create, this should be the test case for model
it do
user = User.create(name: '')
expect(user.errors[:name]).to be_present
end
=> You are expecting a side effect when creating user with your input, so this should be the test case for controller
Why you shouldn't remove 1 of them:
Remove the 1st test case: what happens if you do database validation level instead, you expect an active record level validation
Remove the 2nd test case: what happens on controller actually creates a new User, how do you expect the error returning!
I have a users factory:
FactoryGirl.define do
factory :user, :aliases => [:assignee] do
sequence(:email) { |n| "foo#{n}#example.com" }
sequence(:username) { |n| "foo#{n}#example.com" }
profile
end
end
And a jobs factory:
FactoryGirl.define do
factory :job do
association :assignee, factory: :user
description "MyText"
completion_comment "MyText"
job_type "MyString"
...
end
end
Which are loaded into my jobs_feature_spec_helper:
def add_valid_job
click_job
within("#add_job") do
select 'foo1#example.com', from: 'Assignee'
fill_in_html("job__textarea", {:with => job[:description]})
click_button "Save"
end
end
My feature page spec passes in RSpec on my machine, but my employer sent me an email saying that the spec failed in Jenkins. Here's the message he sent:
Unable to find option "foo1#example.com <mailto:foo1#example.com>"
./spec/support/job_feature_spec_helpers.rb:11:in `block in add_valid_job'
./spec/support/job_feature_spec_helpers.rb:10:in `add_valid_job'
./spec/features/jobs/edit_job_line_spec.rb:8:in `block (2 levels) in <top (required)>'
So, it appears that when Jenkins runs the spec, it's not finding the assignee from the drop-down list, yet RSpec is. I have never personally used Jenkins, so if anyone has any advice that may help, I'd appreciate hearing it. Thanks!
The problem is about the sequence number. You can't use hardcorded 'foo1#example.com' in test.
There is no guarantee that the sequence number will start from 1. I have no time to figure out the reason, but just know the fact. In my tests I often see it from tens after running several tests.
I would suggest you to get the email from an existing user in db, created by FactoryGirl.
Although this was not your issue, one possible cause of getting different results between jenkins and test environments is using different DBs on each.
In my case I was using sqlite on jenkins and postgres on test, and that was generating different results.
Locally you can run
RAILS_ENV="jenkins" bundle exec rspec
to run specs on jenkins environment.
I'm testing some methods that involve email, and am trying to use a mock mailer object. I'm obviously going about it the wrong way, because the test works the first time but the very same test fails subsequently. I'd appreciate it if anyone can explain what is going on here. Thanks.
describe SentMessage do
before(:each) do
Notifier ||= mock('Notifier', :send_generic => true)
end
it "Sends an email" do
Notifier.should_receive(:send_generic).with(['a#contact.com'], 'test')
Notifier.send_generic(['a#contact.com'], 'test').should_not be_nil
end
it "Sends an email" do
Notifier.should_receive(:send_generic).with(['a#contact.com'], 'test')
Notifier.send_generic(['a#contact.com'], 'test').should_not be_nil
end
end
Result:
Failures:
1) SentMessage Sends an email
Failure/Error: Notifier.send_generic(['a#contact.com'], 'test').should_not be_nil
expected: not nil
got: nil
# ./spec/models/test.rb:14:in `block (2 levels) in <top (required)>'
Rspec inserts setup/teardown stuff for mocks and expectations. This does stuff like verify that should_receive expectations have been met and clear up mocks set in objects that endure beyond a single spec. For example if you had stubbed User.find in one spec, you wouldn't expect that stub to exist in another spec.
So at the end of the first spec, rspec is removing the stub setup in the before each. Because you're doing ||=, Notifier isn't recreated and neither is the stub. This in turn means that when you call should_receive in the second spec you're setting up a new stub. Since this new stub has no specified return value, nil is returned.
I've very green to this TDD business, so any help would be fantastic!
So, I've got a factory with the following:
FactoryGirl.define do
factory :account do
email "example#example.com"
url "teststore"
end
end
And an Rspec test with:
it "fails validation without unique email" do
account1 = FactoryGirl.create(:account)
account2 = FactoryGirl.create(:account)
account2.should have(1).error_on(:email)
end
I get a failure with the following message:
1) Account fails validation without unique email
Failure/Error: account2 = FactoryGirl.create(:account)
ActiveRecord::RecordInvalid:
Validation failed: Email taken, please choose another, Url taken, please choose another
# ./spec/models/account_spec.rb:11:in `block (2 levels) in <top (required)>'
Is this the correct way to create new factories? Any ideas what I'm doing wrong here (I have no doubt I'm doing something totally incorrect!)
EDIT: I'm thinking of instead of using 'create' on the second account, I may want to use .build and then .save instead?
Save yourself the database interactions and use the build method for situations like this.
it "fails validation without unique email" do
account1 = create(:account)
account2 = build(:account)
account2.should_not be_valid
account2.should have(1).error_on(:email)
end
You don't need to try and create an account for valid? to return false. You have access to the errors object on the account even when it's just built in memory. This will decrease database interactions and thus making your tests much faster.
Have you considered using sequences in your factories? I don't know how far along you are with your RSpec / FactoryGirl experience, but you will find that things like the following are very useful.
factories.rb
factory :account do
sequence(:email) { |n| "user#{n}#example.com" }
url "teststore"
end
Every time you call build or create on the account factory, you will get unique emails.
Remember that you can always specify values for the attributes on the factory using the options hash. So when testing your uniqueness validation on the account, you would do something like this.
it "fails validation without unique email" do
account1 = create(:account, :email => "foo#bar.com")
account2 = build(:account, :email => "foo#bar.com")
account2.should_not be_valid
account2.should have(1).error_on(:email)
end
Try this:
FactoryGirl.create(:account)
lambda {
FactoryGirl.create(:account)
}.should raise_error(ActiveRecord::RecordInvalid)
This will help - with similar syntax to what you're doing.
However, searching for "rspec validate_uniqueness_of" will find you some more elegant approaches instead of using factory girl like this!
I'm working on a goal application, where a user can check a box next to a goal to mark it as complete. I wrote the test for this functionality, but something isn't quite right, because it keeps failing. I eventually got fed up and just wrote the code, which works, but the test still keeps failing. I'm still learning Rspec, so any advice you could give me would be helpful.
The Test
# #user has already been signed in
describe "completing a goal" do
before(:each) do
#goal = Factory(:goal, :user => #user)
end
it "should set 'completed' attribute to true" do
visit edit_goal_path(#goal)
check('goal_completed')
click_button('goal_submit')
#goal.completed.should be_true
end
end
The Result
Failures:
1) Goals completing a goal should set 'completed' attribute to true
Failure/Error: #goal.completed.should be_true
expected nil to be true
# ./spec/requests/goals_spec.rb:81:in `block (3 levels) in <top (required)>'
If you want this to be an integration test, it would be better for your assertion to be about the contents of the page. Something like:
assert_text "Goal met on 1/1/2011"
If you want to stick with model assertions, I'm pretty sure you just need to reload:
#goal.reload.completed.should be_true