I am learning RSpec and Cucumber, so that I can write tests for our Rails website at work.
While learning, I used a book called "The RSpec Book: BDD with RSpec, Cucumber and Friends". During this, RSpec was used for granular tests and Cucumber was used for overall usability tests.
The book used a simple CLI game as an example.
Now I am heading into a web environment, wondering what is the mentality of testing in a web environment.
I already know that I will be writing spec tests for my models and controllers.
I assume that where Cucumber comes in is full feature tests which will use both controller and model to accomplish.
Please correct me if I'm wrong, but let's say there are User, Post and Likes models.
Each of these would have a model spec + however many controller specs.
Then I would write a single feature test as:
Feature: liking a post
As a user of abc.com
I want to like a post
So that I can show my friends
Where I would log in as someone, find a post and like it?
Anyone with experience mind sharing the angle of approach on testing with these tools on a Rails application?
Let's say I had to implement the situation above, would you start by fleshing out the feature test, write the model specs, then write the controller specs and then write code?
[Repeating some of what others said for the sake of completeness:]
A good approach known as BDD (see bdd) works outside-in by
test-driving a feature with an acceptance test (which in the Ruby world is usually written as a Cucumber feature or RSpec feature spec)
test-driving details with unit tests to express detailed requirements if necessary.
Using this method to test-drive a feature of a Rails project, you'd first write an acceptance test and then implement it, which would bring into existence the route(s), view(s), controller(s) and model(s) needed for the feature. Since you're test-driving, you haven't written any code that isn't fully tested by the acceptance test and you don't need any controller, model, etc. specs yet. You might then start over with another acceptance test, or you might want to test-drive some detailed requirements using lower-level specs.
Suppose that your acceptance test tested that a user's full name is shown on the page and you want to be sure that it looks correct when a user hasn't entered their last name. If User#full_name returns the user's full name, you could just write a model spec for that method rather than writing another acceptance test.
In my experience well-factored Rails applications written using BDD need few or no controller specs, since controllers should delegate to models and other classes as much as possible and so are completely tested by acceptance tests. Lots of details are best tested in model specs.
Regarding how to write acceptance tests, I need to disagree with others' statements about Cucumber, or at least give a warning: I've seen Cucumber result in a readable, maintainable acceptance test suite for a large production application, and RSpec feature specs result in an bloated, unmaintainable acceptance test suite, so don't dismiss Cucumber, or the needs that it addresses:
Don't be fooled by the idea that human-readable acceptance tests are only for your product owner or customer; they're absolutely valuable for engineers, allowing you to think at a high level when appropriate instead of getting lost in the noise of code. The larger the project, the more you'll care.
Cucumber nicely separates acceptance-level descriptions of steps in tests from implementations of those steps (CSS selectors, ActiveRecord queries, etc.), almost automatically making those steps reusable. With RSpec feature specs, it's all up to you; be prepared to come up with your own system of extracting detail into methods whose names make it clear what the user-level point of each step is. (I don't mean page objects; they're a level of detail below what I mean, and are useful in both Cucumber and RSpec feature specs.)
The role of cucumber here is as an acceptance specs. They test the behavior of your application as a whole from the view point of a user. They cover the entire stack from routes to controllers, models and views.
In BDD you often use acceptance testing not just as frosting on top of the cake as in TDD but rather as a driving force in what is called outside in development.
In outside in development you would create a failing specification which details the user story before setting up the routes, controllers or any of the underpinnings.
Compare this to the traditional TDD approach where you start by breaking things down into their smallest units to get a fast red - yellow - green cycle. And then you would start stacking the block on top each other with tests that gradually cover more and more of the stack.
However this does not mean that acceptance specs are the only useful form of specs - they have a rather large overhead and it can be difficult to test edge cases. Thus using just Cucumber is not a very attractive solution.
First this is more opinion than anything else, but here goes. Generally, with TDD, I try to work from the outside in (outside-in-development). In other words, write feature specs, then controller specs (if desired), and then model specs. The rationale here is that your feature tests are the user-facing aspect of your application. Focusing on high-level user behavior up-front helps mitigate time wasting and building things you don't need. Additionally, if you start with feature tests, you can mock/stub lower levels of your application, such as model interactions in your controller, as necessary, which yields faster tests that are well isolated.
Finally, and again, this is more of an opinion -- I think it's overkill to use both Cucumber and RSpec. RSpec works fine, handles unit testing better, and is easier to maintain than Cucumber. Every time I've created a project using both, I eventually scrap Cucumber and just stick to RSpec.
Specifically, for testing user login, I'd recommend including both using the Devise login helpers which makes logging in users faster and easier https://github.com/plataformatec/devise#test-helpers.
Related
When writing specs for a simple Rails app, is the following a correct approach for full test coverage?
Write feature specs for all user stories
Write controller specs to ensure that individual action responses are correct and all required variables are set
Write model specs to ensure all methods, validations,e tc. are working as intended
Write mailer specs
Write routing specs
Is this enough, too much (e.g. can I skip some lower-level specs if I've written feature specs), or not enough? Why?
You don't need to write specs for every object in every layer either to get 100% test coverage or to test-drive (require you to implement) all of the important behavior in your application. Instead, as behavior-driven development (BDD) advises, write specs outside in, and write lower-level specs only as necessary.
The most important measure of test completeness is requirement coverage: it's helpful for each user story, and each detail of each story that requires new code, to be represented in at least one test. If you're following typical agile practices (mentioning user stories suggests that you are) your tests are probably the only place where you record your requirements, so you probably can't put a number on this kind of coverage. It's also helpful to have
line coverage (what most people mean when they say test coverage), meaning that every line of code is exercised by at least one test, and
integration coverage, meaning that every method call from one class to another is exercised by at least one test.
For each story,
Write only the feature specs that will test-drive all of the story's distinct happy paths.
Write additional feature specs to ensure integration coverage of architecturally interesting minor variations of happy paths and of sad paths. For example, I often write three feature specs for a story that involves a form: one where the user fills in every possible field and succeeds, another where the user fills in as little information as possible and still succeeds (ensuring that unspecified values and defaults work as intended), and one where the user makes a mistake, fails, corrects the mistake and succeeds.
At this point you've already test-driven every layer (controllers, models, views, helpers, mailers, etc.) into existence, with only feature specs.
Write model and helper specs to drive out detailed requirements which live entirely in those classes. For example, once you've written a single sad-path feature spec that establishes that entering one particular invalid attribute sends the user back to edit their form submission and displays a message, you can handle other invalid attributes entirely by writing more examples in that model's spec that test that model attributes are validated, and let the architecture that you've already test-driven propagate the errors back to the user.
Note that although your feature specs already test the happy paths through model and helper methods, as soon as you start writing examples for a method for minor or error cases, you'll probably want to write the happy-path example or examples for that method too, so you can see the entire description of the method in one place, and so you can test the method fully just by running all its examples and not also have to run any feature specs.
You might not need some kinds of specs at all:
Well-factored controller actions are short and have few or no conditionals, so you often won't need any controller specs at all. Write them only when needed, and stub out model, mailer, etc. behavior to keep them simple and fast.
Similarly, views and mailers should have few or no conditionals (complex code should be refactored into helper and model methods), so you often won't need view or mailer specs at all.
Your feature specs will have test-driven all the routes you need, so you probably won't need routing specs. I've only ever gotten use out of routing specs when I had to do a major refactor of routes, as when upgrading from one major version of Rails to the next.
As long as you always write a test before you write new code, you'll always have 100% line coverage.
That testing strategy sounds really comprehensive. If you had all of these tests in place you would have great test coverage. However it would take you longer to deliver your project. You would also not be agile as someone who is doing more limited testing. Testing has to suit the project. Don't over test. Over testing can cost time and money. Don't under test. Under testing can cost time and money.
There are right ways to do unit testing. There are right ways to do integration testing. The glove has to fit. If your application is largely front end facing then perhaps it's best to start with integration tests. If your writing a back end application or perhaps an API then unit tests maybe a better place to start. I think approaching with one style of testing and then expanding to different styles is a better start than to try and test every layer of your application.
Why not start with simple unit tests? They are easy to write. Write these tests and then track how many bugs you ship. Are you letting in too many bugs? Are you having a lot of regression issues? Are there bugs that are getting through to production that your suite is not picking up? If the answer is yes then maybe it's time to write some higher level tests. Remember the higher level a test is the more development cost you will have to pay.
If your not shipping bugs then you have no reason to write any more tests. Remember the end goal here. We want to ship bug free code. If we can write one test and one test alone that will ensure we are doing this then there is no reason to test any further.
We now have a considerable size rails(2.3.9) application with 0 rails test cases. (What???). Yes. It is possible:).
Moving on, if we want to base our testing on one framework, which one should we pick, cucumber, rspec1 or rails built in testing framework?. The reason I am saying only one framework is because of the complexity in learning and managing multiple frameworks for the same purpose.
Q1: Some of the y2009 SO
questions recommend using rspec
for unit testing
models/controllers and cucumber for
testing views. Is that (as of 23 June
2011) still a valid recommendation?
Q2: Has anybody managed to build all
the test cases they needed on a single
framework(say just Cucumber)?, if
yes,which one?
Q1: It depends on your developers. Basically you can test almost the same things with all frameworks. I would personally go with rspec1. You will find many resources for rspec especially for rails 2.3.x. On the long run I would try to migrate the project to rails 3. The longer you wait the more painful it gets.
Q2: No, AFAIK cucumber is used for integration testing. You could just use rspec or testunit but then you would lack integration testing. If you go only for integration testing you would lack unit testing but you need unit (or behavior) tests. Furthermore you should invest some time in mocking frameworks as they will save you much time.
It's somewhat a matter of preference, but of the three I would pick rspec or Test::Unit. RSpec has been criticized for being over-engineered lately, but I really like:
the matcher syntax (value.should == 1 vs. assert_equal(value, 1))
test names not confined to method names (it "does something awesome" { ... } vs def test_it_does_something_awesome; ...; end)
You can achieve both of these with TestUnit with some supporting libraries with relative ease... but I just prefer to use RSpec.
When you write code and test, usually (always?) it's better to start from the outside in: IE, write an integration test (a logged in user clicks a button to add $5.00 item to cart, then checks out, and pays. His credit card should be charged $5.00). So rather than testing specific class behaviors (as you do with unit tests), you'll want to be creating objects in the database and making get / post / etc. calls that interact with the site in a similar way that the users web browser would, making your test penetrate the entire stack.
Lately I have liked using Steak inside of RSpec for my integration tests (using Capybara to facilitate the web interaction)
Once you build up your outer perimeter, then I would recommend you start focusing on the unit tests.
Unit tests can be fantastic for documenting a class's behavior. I group them around methods and describe what can be passed to the method, and what I expect to be returned.
describe Object do
describe "#method" do
it "returns 4 when passed 2" do
...
end
end
end
For controller tests, Rails recommends to check for things like HTTP response, authentication, assigns, session and flash messages. However in the app I'm working with right now I see a lot of Rspec tests with response.body.should have_tag(), which, as far as I can tell, would ideally be suited for view tests.
What I'm wondering about is:
Is there any considerable performance/other-kind-of penalty related to such unideal way of testing?
If you are rendering views in your controller this will make your controller tests take more time. It all depends on whether you think it's worthwhile to test for the presence of view elements (which tends to make your tests more brittle). If you do, then doing it in the controller specs makes it easier since you don't have to write out a separate view spec file.
If you want to test a lot of things on your views, then you'll probably want to write the separate view specs. Separating out view specs will probably make the overall time for your test suite to run increase. However, for debugging purposes, it'll be clear whether something is wrong in a view vs. a controller with things separated out.
I suspect most Rails programmers don't write view specs. Instead they are probably relying on their integration tests (Capybara +/- Cucumber) to test their views. However integration tests take more time than unit tests. The RSpec book gives the following argument for writing separate view specs:
View specs provide us with an opportunity to discover APIs that we need from the controllers and models. This is not that valuable when the APIs are following the most standard conventions. The value increases, however, as we stray from them....
The only way to really get a feel for the benefits of them [view tests] is to learn to write them well. And only once you really understand how they fit in the flow are you going to be able to make well-grounded decisions about if and when to use them.
Just want to make sure I understand things.
From what I gather so far, Cucumber is simply a 'wrapper' or a nice way to organize your tests by categorizing things into features and steps, where the actual unit tests are in the steps phase.
It allows to organize your tests into how things will work.
Is that correct?
Sort of.
It is a way to organize tests, but it's more than that. It behaves like the original Rails integration tests were supposed to, but is much easier to use. The big win here being that your session is maintained transparently across the entire Scenario.
Another thing that's going on with Cucumber is that you're (supposed to be) testing from the point-of-view of a browser or client using your code. If you want you can use steps to build objects and set up state, but usually you want your steps to go through the motions required to achieve that state in the first place.
For example you could do this:
Given /I have a user account/ do
#user = Factory.create(:user)
# ... more user set up
end
But you should probably do something like this instead:
Given /I have a user account/ do
visit new_user_url
fill_in "user_login", :with => "some name"
fill_in "user_password", :with => "foo"
# ... whatever else your sign up page needs
end
And of course you can pass in arguments to either of those steps to give your calling code some more fine-grained control over the step's behavior.
What I have generally been doing with my tests is as follows.
I use shoulda, but rspec is fine too.
I write my negative-auth tests (i.e. Access denied is expected) as Rails functional tests.
I write my positive-auth tests (i.e. users doing what they are supposed to be doing) as Cucumber features.
And, of course, I still write Rails unit tests for my models, libraries, helpers, etc.
I like Cucumber because it describes what the behavior is without describing how it is implemented. Cucumber basically decouples the spec (which is also an executable acceptance/regression test) from the implementation. The step definitions are the adapter between the spec and the system under test, which enables the decoupling.
You can also argue that Cucumber gives you a more human-readable spec than RSpec or other Ruby-based syntaxes, which can be understood (even written) by a non-programmer client.
I'd say you understanding is incorrect.
Unit test and Cucumber steps are completely different and largely unrelated things.
Cucumber Tests are combination integration/user acceptance tests and as such are supposed to be independent from the actual implementation in code. So theoretically you could completely rewrite your app in a different language and your cucumber tests and possibly steps should still be useful assuming you haven't changed your applications behaviour. Unit tests on the other hand are intimately related to the code, testing things like return values of specific method calls, edge cases related to data structures, etc... Change a method and you have to change your unit test.
There are some good screencasts out there that talk about using Cucumber and RSpec to build/test your app from the outside-in. Here are few that should help you get started improving your understanding of how integration tests and unit tests can fit together:
http://blog.josephwilk.net/ruby/outside-in-development-with-cucumber-and-rspec.html
http://rubyconf2008.confreaks.com/rspec-and-cucumber.html
http://confreaks.net/videos/72-mwrc2009-bdd-with-cucumber
My own personal use of Cucumber is a bit heretical in that I often use just cucumber on smaller projects as it replaces the monotony of running and rerunning scripts that are just not important enough to do full unit test driven development on. This is however definitely not the community consensus (at least in public) on how to do things.
Cucumber is not a wrapper around RSpec. RSpec is a tool used to drive you code at a unit (one class at a time) level while cucumber is a tool to define features of your system that are in turn automated.
RSpec specs are typically written one at a time as the codebase evolves. We specify the next small behavior of the class we are currently writing.
Contrast that to Cucumber where we write a set of Scenarios that are nothing more than an example of how the system will be used by an end user. Typically most of the Scenarios are written prior to beginning of development and they serve the purpose of defining what functionality the feature should contain. In an Agile environment a feature is considered functionally complete when all of the Scenarios pass.
In smaller shops the developers write both the Cucumber Features/Scenarios as well as the RSpec specs. In larger shops these activities are divided between developers writing the specs while the product owners and testers write and automate the scenarios.
A Rails/tool specific version of: How deep are your unit tests?
Right now, I currently write:
Cucumber features (integration tests) - these test against the HTML/JS that is returned by our app, but sometimes also tests other things, like calls to third-party services.
RSpec controller tests (functional tests), originally only if the controllers have any meaningful logic, but now more and more.
RSpec model tests (unit tests)
Sometimes this is entirely necessary; it is necessary to test behavior in the model that is not entirely obvious or visible to the end-user. When models are complex, they should definitely be tested. But other times, it seems to me the tests are redundant. For instance, do you test method foo if it is only called by bar, and bar is tested? What if bar is a simple helper method on a model that is used by and easily testable in a Cucumber feature? Do you test the method in rspec as well as Cucumber? I find myself struggling with this, as writing more tests take time and maintaining multiple "versions" of what is effectively the same behaviors, which makes maintaining the test suite more time intensive, which in turn makes changes more expensive.
In short, do you believe there is there a time when writing only Cucumber features is enough? Or should you always test at every level? If you think there is a grey area, what is your threshold for "this needs a functional/unit test." In practical terms, what do you do currently, and why (or why not) do you think it's sufficient?
EDIT: Here's an example of what might be "test overkill." Admittedly, I was able to write this pretty quickly, but it was completely hypothetical.
Good question, one I've grappled with recently while working on a Rails app, also using Cucumber/RSpec. I try to test as much as possible at every level, however, I've also found that as the codebase grows, I sometimes feel I'm repeating myself needlessly.
Using "Outside-in" testing, my process usually goes something like: Cucumber Scenario -> Controller Spec -> Model Spec. More and more I find myself skipping over the controller specs as the cucumber scenarios cover much of their functionality. I usually go back and add the controller specs, but it can feel like a bit of a chore.
One step I take regularly is to run rcov on my cucumber features with rake cucumber:rcov and look for notable gaps in coverage. These are areas of the code I make sure to focus on so they have decent coverage, be it unit or integration tests.
I believe models/libs should be unit tested extensively, right off the bat, as it is the core business logic. It needs to work in isolation, outside of the normal web request/response process. For example, if I'm interacting with my app through the Rails console, I'm working directly with the business logic and I want the reassurance that methods I call on my models/classes are well tested.
At the end of the day, every app is different and I think it's down to the developer(s) to determine how much test coverage should be devoted to different parts of the codebase and find the right balance so that your test suite doesn't bog you down as your app grows.
Here's an interesting article I dug up from my bookmarks that is worth reading:
http://webmozarts.com/2010/03/15/why-not-to-want-100-code-coverage/
Rails has a well-tested codebase, so I'd avoid re-testing stuff that is covered in those steps.
For example, unless it is custom code, it is pointless to test the results of validations at unit and functional levels. I'd test them at the integration level though. Cucumber features act as specifications for your project, so it is good to specify that you need a validation for x and y, even if the implementation is a single line declaration in the model.
You usually don't want to have both Cucumber stories and RSpec controller specs/integration tests. Pick one (generally Cucumber is the better choice, except for certain special cases). Then use RSpec for your models, and that's all you need.
I test complex model/lib methods with rspec then the main business logic in web with cucumber, so I'm sure that the main features of the web will work 100%, then if I got more time and resources I test everything else.
Its easier to write simple specs for simple methods. Its much easier than writing cukes.
If you keep your methods simple - and keep your specs simple - by testing only the logic inside that method - you will find joy in unit testing.
If anything is redundant its cucumber tests. If you have well tested models and lib your software should work.
The purpose of Cucumber is not to run integration tests. Cucumber, an in general BDD, works as a communication platform, a way to improve the "talk" inside a big team of developers an non-developers that are developing big and complex software. BDD is very useful to communicate a model an its domain at the same level to everybody in the team, even if they don't know anything about computers.
If that is not your scenario, don't use cucumber, because you don't need it. Use rspec and capybara to test your JS code and your integration tests.