I like the way Cucumber connects user specifications to integration testing. The Cucumber part is more or less clear to me.
I struggle when it comes to what should be tested with rspec (non-integration testing) and what shouldn't.
Is it right to unit-test with rspec something that has already been tested with Cucumber (e.g. unit-test will 100% fail if Cucumber test fails, and unit-test will 100% succeed if Cucumber test succeeds)?
To be specific, I have three examples I'd like to resolve.
Here is a case from the RSpec book.
They have the following Cucumber scenario:
Given I am not yet playing
When I start a new game
Then I should see "Welcome to Codebreaker!"
And I should see "Enter guess:"
They build two rspec-tests right after:
describe "#start" do
it "sends a welcome message" do
end
it "prompts for the first guess" do
end
end
Another example is testing routing or action redirect, while there is the following scenario:
Given I am at the login page
When I fill in the right username and password
Then I should be at the index page
Sometimes we test helpers that are already tested with Cucumber:
Given Mike has spent 283 minutes online
When I go to the Mike's profile page
Then I should see "4:43" for "Time online:"
I should probably test helper that breaks 283 minutes into "4:43", but it turns out that it is already tested with Cucumber.
It may not be the best examples, but it illustrates what I am talking about.
To me those tests are duplicates.
Could you please comment on the examples above?
Is there any principles or guidelines on what should be tested with rspec, when there are Cucumber tests already?
All this is only my personal opinion on this broad and open topic. Some people might disagree and funny enough they can.
As of general guidelines you should use Cucumber to test entire application stack, something that is called user experience. This application stack might be composed of many smaller independent objects but, same as user using your application wouldn't be interested in those details, your cucumber test shouldn't care for them and focus instead on outer layer of your application.
RSpec( in your setup!) should on the other hand put a main focus on those small objects, building blocks of your application.
There is a big problem with small application examples from the books: They are to small!
Boarder between outer layer of your application and its interiors is to blurred. Entire application is build with two objects! It is hard to distinguished what should test what. As your application grow in size its getting more obvious what is a user experience test(cucumber) and what is object - state test/message expectation test(RSpec).
Using your second example:
With this Cucumber story:
Given I am at the login page
When I fill in the right username and password
Then I should be at the index page
Rspec:
You will probably have some sort of User model:
test user name syntax(must start with capital for example)
test password must be 7 characters have numbers etc...
You can have some sort of authentication object:
test for valid and invalid logins etc
test for exceptions being thrown ....
What if your Authentication database is on different server?
mock connection and authentication database and test if database receive an request from authentication object.
And yada yada yada... Forever your cucumber test will guard general purpose of your application user login. Even when you add or change behaviour, under the hood, as long as this test pass you can be confident that user can login.
you should test things once(in one place)
object its responsible for testing its incoming massage(object public interface) for state(return value)
when your object depends on other object(sends its a message) don't test it for state(return value) that object is responsible for it.
when your object depends on other object(sends its a message) mock that second object and test if it receive your message.
Generally your question is too broad and you wont find a single answer, different people will have different ideas. What you should focus on is to test, as good as you can and with time you will definitely find right way for you. Not a great test suit is much better then none.
Because good design helps to write good test I would recommend Practical Object-Oriented Design in Ruby: An Agile Primer by Sandi Metz(which is one of a best book I've read)
Related
Actually i am newbie to rails testing
I want to know what are all the step covered in the rails testing.?
Could anyone help me knowing the steps involve in Professional Testing
Check out the rails guide Gavin linked you. Can also check out this book: http://pragprog.com/book/achbd/the-rspec-book. It covers how to use rspec, and covers testing various parts of the application.
I also rather enjoyed Rails 3 in action, which teaches the basics and various topics of rails, while doing inside-out testing throughout the entire book. Rails 4 in action is a work in progress edition as well.
As for my answer:
Generally testing first before you write code makes the process better & easier. Writing tests after the fact isn't as nice, even though it does give you the coverage. When it's part of your regular workflow to add features, it's quite nice. If you have a new feature that you want to add, such as user sign up that sends a confirmation email. You can write the integration test first. It describes the user as going through the sign up form, filling it properly, and then checking if an email was sent.
From there you can test internals of the user model, making sure methods you build for this to work function as expected. Then when you have your tests green, add another scenario to the integration tests where the login was invalid.
I want to start diving into BDD. I have never used TDD before and am
not sure if I should start by learning RSpec and then jump to Cucumber
or just go straight to using Cucumber.
I have been reading on the internet about both and it seems to me that
Cucumber could be a 'replacement' for RSpec. Am I right or should be
one used for certain things and the other one for others?
Cucumber and RSpec are both used for BDD but have different uses.
Think of Cucumber as describing what happens from the user's perspective, through interaction with the web browser. So you can have steps like:
Given I'm not logged in
When I login
Then I should be on the user dashboard page
Pretty broad, but there's a lot going on under the hood there. Cucumber is good for making sure all these sort of high-level features and functionality are covered (e.g., that when your user logs in, they're taken to the right page). But it's not a good tool for testing lower-level code. That's where RSpec comes in.
Take the login example above. Your user may be logging in with an email address or username. You probably want to ensure the email address is valid or that the username is a certain length...or that the thing they're using to login with is unique. You'd do this in your User model with a validation.
It's a trivial example, but this is the kind of thing you'd test using RSpec (in your spec/models/user_spec.rb file). This is not something you'd test using Cucumber.
So bottom line is:
Cucumber for higher-level tests describing broad functionality as viewed from the user's perspective
RSpec for lower-level tests describing details for how your classes, methods, models, controller, etc should actually work.
This post actually does a really good job of explaining when to transition from one tool to another:
http://www.sarahmei.com/blog/2010/05/29/outside-in-bdd/
I also recommend "The RSpec Book" and "Rails Test Prescriptions" for good resources on both tools and testing in general.
P.S. - Not to confuse things, but you can actually use RSpec for the high-level stuff too. But some of that decision is a matter of which tool you prefer or maybe whether or not you're working with a non-technical client who would benefit more from Cucumber's user-friendly syntax for describing scenarios.
Yes, cucumber and rspec are both used for BDD.
I personally prefer Cucumber, but some people find it offputting, and
prefer
their tests to be less english, more code. Both are great tools, though.
Kenton did a great job with this answer. Here is how the authors of RSpec and Cucumber see it:
We use Cucumber to describe the behavior of applications and use RSpec to describe the behavior of objects.
Although we use Cucumber to focus on high-level behavior and use RSpec
to focus on more granular behavior, each can be used for either
purpose.
For example if I write Add A customer feature which might go like this:
Scenario: Add A customer
Given I am on the customer page
When I enter login name
then I press Add
And I should see the newly added customer confirmation message
I start to write the code in watin to open the browser and go to the customer page.
Which doesn't exist at this point.
The questions are:
1) do I then jump into unit tests and write a unit test for the page that doesn't exist? in MVC this would be a controller but in asp.net webforms it's the same test that is in the step definition.
2) How does the unit test tie in with the step definition? Let's say at the end of the project I have a load of features and a load of unit tests. Then one of the features starts failing if I looked at that how would I know which unit tests correspond to it? Or does this even matter?
I'm not sure if this is a best practice question or it's whatever people feel is right.
Thanks in advance.
One of the cool things about specflow (and cucumber...) is that you can write the test before any page exists. So what I'd do is exactly what you planned to do.
1) Write a specflow test, which describes how you want to work with the page. Looks like you have that already.
2) Run the test. The first line will fail (Given I am on the customer page).
Now you have a choice. Do you just wan to write the code that fires up watin and navigates to the customer page? Fine, do that; then write the customer page (just enough so that the first line of the test passes).
Or, recognize that your test requires a customer page. There's lots here to test: do all the data fields exist? Does the page do input validation correctly? If I post to the server, do I get the correct response? If I post to the server, does the DB get updated correctly? And so on. Some of these sound like unit tests (business logic that takes the form data and stores it in the DB); some sound like UI tests that would be good with SpecFlow (page details, validation) and some sound like integration tests (probably good with specflow).
Once you've written all those and they pass, go back to your original test, and get the second step to pass (I click...)
And so on.
To answer your second question, hopefully you're running the unit tests and specflow tests often enough that if a test fails, it's due to something you've done in the last few minutes. It shouldn't be too hard to see what failed and match it to what you just did.
If you're not running your tests that often, then well you should be.
So, I am starting to build a new application, my first big one in rails since moving from .net.
I really want to BDD the entire application so I have cucumber all set up and ready to go.
I have written more then a couple of test for the simple stuff (deleting, adding, updating) and so on but now I am stuck trying to figure out how to test this scenario.
I have these models
User, Account, Plan
each account has a plan, the plans are billable each month (on the same day the account was created).
I want to test selecting a plan, creating an account, check the billing process (even mock, without PayPal at this point).
I would appreciate help with this, just to emphasize, I am not looking for complete code, just an explanation of how (in concept) would you go about and test this.
Also, plan is upgradable and downgrade-able so I want to test this as well.
Thanks in advance for your help
I like to practice outside-in development, where we begin by writing acceptance tests, then drop into unit tests to handle domain logic. Let's take creating an account as an example.
Start by writing a Cucumber story for the desired feature. For example:
Feature: Create an account
In order to use the application
As a user
I want to create an account
Scenario: Create an account from home page
Given I am on the home page
When I follow "Sign up"
And I fill in "Username" with "bob"
And I fill in "Password" with "test123"
And I press "Create"
Then I should see "You have successfully signed up! You may now sign in."
When we run our Cucumber features using the cucumber features command, the first step in the scenario will fail because the home page does not yet exist. To create it, we may consider it a separate feature. Therefore, we might write another Cucumber feature like:
Scenario: Visitor visits the home page
When I go to the home page
Then I should see "Welcome to the Website of Awesomeness"
Running this feature, we will discover that there is no root route defined in the Rails app. Once we fix that issue, we'll need a controller, view, and the text in the view. So far, we have only written Cucumber tests.
Once all of these features are passing, we realize a username should be required. We can write a Cucumber step to test this case:
Scenario: Username must be filled out
Given I am on the home page
When I follow "Sign up"
And I fill in "Password" with "test123"
And I press "Create"
Then I should see "Username cannot be blank."
To implement this, we must add a validation to our model that will validate the inclusion of a username. Now we will drop to unit testing because we are modifying domain logic. As a general rule, when you modify a model, you should drop into RSpec or Test::Unit and test that modification directly. For example, using RSpec we would add a spec to ensure usernames must be present (and unique, etc). Once this test passes, our scenario should also begin to pass.
This was long-winded, but it should help you start practicing BDD in a very real way. For more information, see the RSpec book (which includes a wealth of information about outside-in practices using Cucumber and RSpec): http://www.pragprog.com/titles/achbd/the-rspec-book
I'm currently reading through the beta version of The Rspec Book: http://www.pragprog.com/titles/achbd/the-rspec-book
It describes The Behaviour Driven Development Cycle (red, green, refactor) as taking small steps during the development process. This means adding features one at a time.
My question is:
If I were to describe a single feature of my software (eg: a successful user login scenario in a cucumber test) AND if I were using a modular component (such as Devise) which has many features (scenarios). How could I possibly follow Behaviour Driven techniques? Once my first step passes, I must reverse engineer my other tests to reflect the functionality of the software component I am using thus violating the principal of BDD!
Edit (for clarity):
My first scenario is passing after implementing Devise. But now I must factor all my subsequent end to end tests (that I have not yet written) around the behaviour of Devise rather than to my stake holder's requirements. So the BDD cycle can no longer be applied. I must reverse engineer Devise in my tests to make them pass or not write the tests.
The BDD cycle involves creating scenarios, then having conversations around those scenarios to discover any more which are missing, any misunderstandings, etc.
If you use a BDD tool like Cucumber, then you can capture the scenarios you've discussed.
Ideally, the scenarios will be in terms of high-level steps, focused on the capabilities of the system and the value they provide to the users. They won't be about logging in or failing to authenticate.
BDD isn't really about testing. It's about learning, and keeping things easy to change. If you're never going to change the mechanisms for logging in then it's enough to manually verify authentication. Focus instead on the things which make your application different to all the other application out there. How does your software actually provide value? How does it communicate with other systems, applications and users? How is money made, or lives saved, or fun had?
If you can answer those questions and keep the steps focused on those, you'll have failing scenarios still, even with Devise. Your scenarios will look more like:
Given I have registered an account
When I <do this differentiating thing>
Then I <achieve this differentiating outcome>
Logging in will be implicit, and called as a lower level step as part of the first Given.
So I can understand better: You have an existing system and it includes tests but does not include login functionality. You decide to add login:
Given a visitor is not logged in
When a visitor goes to the admin page
Then the visitor should see the login page
Ok, so then you decide how you want to implement the login because this scenario is failing. Right? So you choose Devise and the scenario passes but all the tests or specs that relied on a security-free system now fail. Am I still on track? If that's the case, you've encountered a case where you are adding a feature that is so pervasive in your application that you need to touch a number of tests/specs to get them all running. You are still testing first because either the code is broken or the test doesn't recognize the new feature. In any case, you know where the work needs to be done.
Helpful?