I'm extremely new to behaviour-driven development, and I'm trying to wrap my head around something fundamental: the difference between test systems and spec systems.
So far I've been using the Ruby-based minitest/test library where a typical unit test would look something like this:
require "minitest/test"
class TestMeme < Minitest::Test
def test_that_kitty_can_eat
assert_equal "OHAI!", #meme.i_can_has_cheezburger?
end
end
On the other hand, a spec from minitest/spec would look something like this:
require "minitest/autorun"
describe Meme do
describe "when asked about cheeseburgers" do
it "must respond positively" do
#meme.i_can_has_cheezburger?.must_equal "OHAI!"
end
end
end
So what exactly is the difference between tests and specs? Should I choose, or can I use both?
What you are showing is merely the style of tests. You can either use the minitest API to create tests, or you can use the spec DSL to create the tests. As you point out, they are functionally equivalent. The difference is a matter of preference.
The larger question is TDD vs. BDD. Both advocate for driving the design of your software by writing tests. First you write a test that fails, then you write only the code needed to make it pass, then you refactor your code to make it right while keeping your tests passing. This is called the TDD cycle.
BDD goes a bit farther and specifies how developers can interact with users to define the stories that developers work off of. And it advocates for a different role for mocks than most TDD practitioners use. But those are small differences compared to the very large area that both methodologies agree.
You can practice TDD with spec-style tests, and you can practice BDD with test-style tests. You can mix and match, with spec-style describe and it blocks with assert_* assertions, or test style class and method definitions with must_* expectations. The differences between the test-style and spec-style syntax is superficial.
The BDD test is a business-analyst friendly unit test, meanwhile a unit test is just developer-friendly. BDD defines a language to interact between developers and BAs.
Also ideally BA could produce a human oriented but automatically executable documentation, well integrated with agile, formatted as User Stories.
I think you just need to read more about BDD, wiki has good article.
Related
I happen to be a subscriber of Ruby Inside, since I'm particularly interested in Rails. Yesterday, the creator of Rails, David Heinemeier Hansson, pretty much said that he's just using test/unit. I would understand that, since it's Rails internal, but he seems to have given a strong opinion. He believes that RSpec and Cucumber are needlessly complicated.
I would normally not pay much attention, but it depends on who says something. I respect Hansson a lot and his opinion got me thinking. When I started with Rails, I never really looked into test/unit. Just RSpec and Cucumber.
And that is why I want your insight. Do you think that RSpec is indeed complicated for not much added value? Does writing test/unit take less time and effort?
My recommendation would be to use either Shoulda (extends Test::Unit) or RSpec with Capybara, and -no- Cucumber.
I think that the use of either RSpec or Shoulda for nested contexts is definitely worth doing. RSpec is definitely heavy-weight (perhaps overweight) though, and I'm on the fence with it for that reason.
Cucumber, I've finally come to understand, is waaay more cumbersome than it's usually worth. You can accomplish what you need more simply and robustly with plain ol' integration tests and Capybara. Remember -- Capybara != Cucumber, and Capybara is quite capable all on its own.
Shoulda is nice, because it simply adds conveniences to the standard Test::Unit framework, and is therefore much lighter-weight than RSpec (technically, each solves a different set of problems, but they both provide nested-context capabilities). RSpec has the advantage of making assertions read more naturally, and also generating more helpful failure messages in many cases, without the need for writing message arguments on the assertions.
Also, remember that Cucumber does not actually require RSpec, so if you want to keep using Cucumber, you can do that with just Test::Unit. Choices abound.
It's all about semantics. RSpec and Test::Unit are similar functionally. Personally I've always preferred RSpec because I found it more natural to write tests using it. I also like the simplicity of writing custom matchers, and the default matchers provided are useful.
Cucumber is a different beast entirely. Yes it's fairly cumbersome and becomes hard to maintain if you don't organise your step definitions properly but it does have one very strong use case and that is when your client is writing your specifications.
I've worked on projects where the client has been writing Cucumber scenarios together with one of our QA team. As a non-technical person it's an extremely approachable and natural way to specify user stories in the code. Cucumber really helped us walk the walk when it came to following our agile practices. The quality of the end product benefitted from that but no I do not like Cucumber as a developer :)
It's a matter of personal taste.
I like to write easy cucumber tests without worrying about details. Just testing the "happy" paths of my app. (portable, understandable, slow)
I leave the details to Test/Unit. (easy, fast)
It takes more time to understand:
get :products, :session => #session_id_for_product_banana
assert_select "table" do
assert_select "td#name", "Banana"
end
instead of
When I go to the banana page
Then I should see "Banana"
Sure those tests are not equal but who cares whether "Banana" is in a div or a table or doesn't have the right html-id.
I dont' like functional tests because after refactoring, the id's could be gone, the session expectation could be changed. If that's the case you will need to refactor your code AND tests. If you'd use cucumber you wouldn't have to change your scenario's.
For integration tests of my Rails web app I use Steak (something like Cucumber). The specs of Steak are in a folder named spec/acceptance. Are Steak/Cucumber now for integration or acceptance testing? I always thought that this is something different.
First, a note on the terminology: the term integration test is a bit vague in the TDD community. Depending whether you come from Java or Rails (with Test::Unit), you might understand different things by it. In Rails (with Test::Unit) integration tests are the tests that test your full stack, while functional tests would be the ones testing your controller. Most people in the Java community (at least by my observation) would think it is the other way around. I personally prefer to call the end-to-end tests acceptance tests, while tests that hit several layers of the system (but not everything) -- integration tests. All in all, that is pretty dependent on the culture your are in.
As for Cucumber and Steak -- both are frameworks that allow a development style known as Behavior-Driven Development (or BDD for short). The point is that you have two levels of tests:
End-to-end tests, which test your through the full stack -- they simulate a browser, go through your controllers and hit the database. Cucumber and Steak fit this niche.
Unit tests, which test a small bit of functionality in isolation (usually a single class, mocking its collaborators). This is where RSpec fits.
In BDD, you start with a failing end-to-end test (lovingly know as the "upper gear"), and then you start implementing functionality test-first with RSpec (the "lower gear"), until you get the end-to-end test passing. This way the end-to-end test is driving your unit tests, which in turn are driving your implementation. The main benefit is avoiding scope creep -- you don't end up implementing user-visible functionality that you don't need (since you don't write an end-to-end test for it).
If you want more information on this, I've heard that the Behavior Driven Development Wikipedia article is surprisingly good. Also, the RSpec book.
So, both Cucumber and Steak are frameworks that allow you to write tests in the "upper gear". The difference is in the style -- Cucumber has you writing your tests in natural language. This has several benefits.
Tests are readable by the business people -- while you cannot expect non-programmers to write them, they do a great job in communicating what you intend to do. You can write the feature (the Cucumber test first) and show it to the customer to get some feedback on whether this is what they actually want. I've found this very useful.
Cucumber features communicate intent better -- since you get to use the full power of the English language (or any, really), you can communicate why this feature is relevant and how the users accomplish their goal on a level that Ruby won't allow you to.
Cucumber helps discovering the ubiquitous language -- the domain includes a lot of terms that fly around in the conversations with the customers. Cucumber allows you to discover and capture them before you start implementing the feature. And it's all test-driven.
Cucumber features are somewhat higher-level, which makes the features (but not the step definitions) more independent of the interface. This way if the interface needs changing, you won't have to rework the features.
The downsides include that it is a bit tricky to learn how to apply it nicely and that you have to write a bit more (both features and step definitions). I've found that the second is not really a problem if you have been doing it for a while, since you get a body of reusable steps that allow you to write the next features faster.
Steak, on the other hand is simpler and it's Ruby. You loose all the benefits of using English, but you can write less and it will execute faster (somewhat).
In the bottom line, you use both to write the end-to-end tests that drive development.
Disclaimer: I have not used RoR, and I have not generated tests. But, I will still dare to post this question.
Quality Assurance is theoretically impossible to get 100% right in general (Undecidable problem ;), and it is hard in practice.
So many developers do not understand that writing good automated tests is an art, and it is hard.
When I hear that RoR generates the tests for you, I get very skeptical. It cannot be that easy.
Testing is a general concept; it applies across languages. So does the concept of code contracts, it is similar for languages that support it. Code contracts do not generate themselves. The programmer must add the requirements and the promises manually, after doing some thinking about the algorithm / function. If a human gets it wrong, then the tools will propagate the error. Similarly with testing - it takes human judgement about what should happen. Tests do not write themselves, and we are far from the day when a business analyst can just have a conversation with a computer and tell it informally what the requirements are and have the computer do all the work.
There is no magic ... how can RoR generate good tests for you?
Please shed some light on this. Opinions are ok, for this is a community wiki. Thanks!
Where have you read that Rails generates tests? It doesn't. What it generates is a simple default test stub for every model or controller that you create with the command line tool.
Let's say that you create a model for a blog post:
script/generate model post
Rails automatically generates the file:
test/unit/post_test.rb
That has the following content:
require 'test_helper'
class PostTest < ActiveSupport::TestCase
# Replace this with your real tests.
def test_truth
assert true
end
end
As you can see, the content of the test is still up to you.
Here's more info on Rails testing: http://guides.rubyonrails.org/testing.html
At present, my development process flows like this:
I describe the expected behaviour as an integration test using using WebRat
I write the Ruby on Rails code to provide that behaviour, so passing the test
I refactor, ensuring the tests still pass at the end of the process
I write the next integration test
It seems to me that by definition, my integration tests are testing every model, controller and view that I can create. In reality, am I missing anything by not writing unit tests too?
I'm actually pretty sympathetic to your point of view here. I love Cucumber and I love RSpec -- and I use them both, but not always on the same code. For instance, I rarely write RSpec examples for Rails controllers these days, and I almost never write view specs. Most of my controllers are very similar and don't deviate much from the "stock" controller pattern -- which is already well-tested by Rails's own unit tests. Verifying the same behavior again doesn't gain much for the time it takes and the hassle of mocking all the models. With Cucumber at an integration level I can skip that mocking and get the essential verification I'm looking for. View testing is handled much more transparently in Cucumber most of the time as well. (Then I should see "foo" and so forth.)
But that isn't to say I don't use RSpec extensively in Rails. I use it for the places where my logic lives: models, controller filters, and view helpers. I also have a couple of projects that are almost all business logic, e.g. libraries or API adapters against complex third-party interfaces. For those I usually find myself using RSpec exclusively and skipping Cucumber.
As a heuristic, I'd suggest that you should strongly consider writing a unit test any time any of the following questions can be answered "Yes":
Is the code I'm writing more than trivially complicated?
Does this code exist primarily to give answers to other code?
Is this existing code that I'm refactoring (that doesn't already have a unit test)?
Have I found a bug in this code? (If so, write a unit test before fixing it so it never sneaks in again.)
Do I have to think for more than ten seconds about the most elegant way to implement this code?
Is my Spidey Sense tingling?
If none of the above is true, then maybe you can get away with just doing integration testing. Again, there are a lot of cases where that's reasonable. But if you do run into problems later, be prepared to pay the price -- and that price should include writing unit tests at any moment if they seem called for.
Integration tests are useful to verify that different parts of code are well integrated. They may involve all layers and cover all code but... when an integration test fails, will it tell you where the bug is located? I may be wrong but I don't think so. This will just tell you that there is a problem somewhere. On the other hand, when a real unit tests (written in isolation using mocks or stubs) fails, you know exactly in which unit the problem is located (this is actually the purpose of unit testing, verifying that a unit implements the expected behavior). In other words, unit tests and integration tests are both useful but they have different purposes.
Do you have any rake tasks? Custom capistrano code? Cron methods? An API? Monkeypatches? How about flex or iPhone app integration? A job runner?
In a typical Rails application, there's lots of code that isn't exercised by the HTML UI. So no, unless your app is incredibly simple, webrat tests won't be sufficient.
I am a little confused by the sheer number of testing frameworks available for Ruby/ROR.
I have recently watched the Cucumber Railscasts and found them very interesting. So I started having a play and then struggled to see conceptually where I would put various tests.
It would seem to be quite possible to do everything that can be done in unit tests within Cucumber, so do I need to write unit tests or should I just write my feature definitions and concentrate on providing as good a coverage as I can get using that.
Should I create my Unit tests using Rspec or Test:Unit? When I'm testing Ajax functionality should I use Selenium or Watir?
There seem to be so many options here I am struggling to see which tools to use and where the boundaries are.
What are other peoples experiences of Cucumber and where to draw the line between writing Cucumber Integration tests and Test:Unit and/or Rspec based unit and functional tests. Is anyone aware of a good write-up on this subject suggesting where to draw lines between testing methods and the strengths and weaknesses of the various tool.
I appreciate that some of this is subjective but common approaches on how to attack this issue would be welcomed.
Use Cucumber at a high level to describe what a user should be able to see and do. Use RSpec, Test:Unit, Shoulda, etc. to write unit tests. Straight from the horse's mouth:
When you decide you want to add a new feature or fix a bug, start by writing a new feature or scenario that describes how the feature should work. Don’t write any code (yet).
...
This is when you start writing code. Start by writing a couple of lines of code to address the failure you got from Cucumber. Run cucumber again. Repeat and rinse until you’re happy with your feature. When you get down to nitty gritty details, drop down one abstraction level and use RSpec, or any Ruby testing framework, to write some specs/tests for your classes.
Cucumber is made to test your whole stack, together, as opposed to 'units'.
You need to decide where to draw the line, but a lot of under the hood stuff probably wouldn't be covered in a cucumber test. Say when signing up, I fill out a form, with my name, email, phone number, etc. A unit test might check to see that a new User will also create a new TelephoneNumber. From the user's perspective, they don't really care that it creates a new TelephoneNumber, they care that once they've signed up, they have an account and can see their telephone number.
I don't have too much experience writing cucumber tests (not yet), but I hope this helps a bit.
When a unit test fails (I mean a real unit test that tests a method in isolation using mocks), it tells you what "unit" has a problem. When an acceptance test fails, it tells you what "feature" has a problem, not where the problem is located.
When you create a rails app you get functional, interation, and unit tests by default. Cucumber is an additional test it is a way to also test the experience that your user will have. When they click the button labeled "go" they should see "success" rendered rather than a 404. This will make sure that nothing you do accidentally messes up the user experience, and that from the top to bottom your app works for the most common use cases you can think of. The other tests are meant to insure that nothing goes wrong, and that you have inspected ever model and method with a microscope. It may be possible to replicate unit tests in their entirety with cucumber, but it would be painful (and crazy slow to execute, especially if you're using selenium). The best time to write tests is when you're developing code, and the quickiest and easiest way to do that, is by using in the built-in rails testing and maybe some additional help such as shoulda, rspec, also i'm a huge fan of factory-girl. If you haven't already checked it out www.railscasts.com has a great intro to cucumber, and rspec, and factory-girl, ... I know this question has already been answered (it's no) but this is my two cents. Good luck coding!!
I have thought/struggled with this question much, and here's where I've arrived.
Cucumber first and Cucumber last. Cucumber will provide the primary test coverage.
The core model methods that do the real business work of the application should also be developed/covered with rspec/unit tests.
Why the unit tests as well?
1) The unit tests will test run much faster.
2) This core business logic may (will probably) be used in several ways beyond the current views (which Cucumber tests through). These methods ought to be hammered with all types of possible inputs and outputs directly calling the method in the test.
Why not unit test the rest of the models, and the controllers and views?
1) Cucumber already has it covered once.
2) I find that the views-controller-some-model-methods all work together to get things done (think everything exercised to log in); so I like to test them together.
I've been practicing Cucumber/RSpec for the past half year or so doing BDD.
First of all BDD is not easy to get into, it will feel unnatural at the beginning.
But once you get into it, there's no other way to do programming.
To answer your question. To test Javascript you'll need a javascript driver that can be used by Capybara which is used by Cucumber.
capybara-webkit is what all the cool kids use now these days
There's one important thing to note.
Integration tests are slow.
And unit tests are fast, but can be slow, so it's important you use the right database cleaner and you write good tests that have good isolation.
My test setup which I'm extremely happy with:
Guard for loading spork
Spork for faster tests
Cucumber for integration testing
capybara-webkit for javascript testing
RSpec for unit testing
I don't do view tests and controller tests as these are redundant in my opinion as good knowledge of XPATH willl have you writing remarkable tests that even cover your page layout and structure.
Personally I don't think that you should stop writing unit tests. As an acceptance testing tool, Cucumber should replace your functional tests and, if you writing, view tests.
Cucumber features are supposed to be simple and coupled to the real user's value a given feature has.
From my experience, Cucumber and Rspec have different appeal. Rspec appeals to me from a developer perspective because its easy to write and provides very quick feedback when something breaks. Cucumber does not appeal to me as a developer because it does not run as quickly as Rspec. However, Cucumber does appeal to me as a business stakeholder since it provides full coverage of entire features.
Do yourself a favor and keep writing unit tests.