I have been watching Ryan Bates' RailsCasts for years now, and I'm also a customer of the pay portion of the site. I learned BDD from watching Cucumber episodes in the past.
Now I have learned about TestUnit, RSpec, Capybara, and MiniTest. I am beginning to get confused about what is what.
What is the difference between these 4 items? I know obviously Cucumber executes plain text features, and I guess that could be considered integration testing.
But now I also see that recent versions of Cucumber require MiniTest? Is Cucumber simply a DSL that sits on top of a testing framework?
I know also that RSpec has its own syntactical sugar for doing assertions, the "describe" blocks. And it appears that MiniTest also supports this syntax.
I know that Capybara is used for looking at the web page content that is generated, I think.
Here is my question:
If I am creating a new Rails 3.2 application, what combination of these testing programs should I use? What would be extra helpful is a list that explains how these gems and their associated processes complement each other where applicable, like:
Cucumber is a DSL for driving BDD
Cucumber is for integration tests and is based on creating user stories that are customer-readable
It implements its tests behind the scenes via MiniTest
MiniTest is a new testing framework that comes with Ruby 1.9 and is very fast.
MiniTest is also used for unit testing, such as testing controllers and models
It does not yet have as many features as RSpec
Cucumber also uses Capybara to access DOM elements from a javascript-enabled browser simulator such as Selenium
When you test in Rails, you have the ability to do the following kinds of tests: controllers, views, models, and integration (MVC together)
Some people just do integration and model testing, because they feel that integration testing handles enough of the controller and view testing itself, and anything too complex can simply be moved up to the model
Thank you so much for any help you can offer to clear these ideas up for me.
Okay, so let me try and explain, based on my own experience
Cucumber, is a ATDD (or BDD) tool that lets you write tests in a business-oriented domain language. It's primary use is as a conversation tool with you product owners. Instead of writing detailed requirements, you express those requirements as examples of the system under test. Each cucumber test, in effect, becomes a business requirement that must be satisfied.
The Cucumber tool itself, translates these plain-text statements into a little module for you to execute ruby code in. What Ruby Library you use inside of the step definitions depends entirely on the project you are working on.
The safest way to describe Cucumber is that it's a testing framework with an emphasis on communication between IT and Business Partners, which is why it's becoming so popular.
Rspec and Minitest are other frameworks with other strengths, but lack this business readability factor, in that they are mostly code and not nearly as readable to a non-technical person. That isn't necessarily a bad thing, especially if your product owner is a little more hands off.
How does this tie in with something like Capybara? Capybara is an integration test automation library that drives a headless browser on the Rack::Test framework for very fast tests, and has a highly readable DSL. The only downside is Rack::Test doesn't support javascript, so it provides a way to fail over to Selenium if you a running a javascript test. Rspec and Cucumber both have mechanisms for triggering this failover.
There are lots of other Ruby automation libraries. For instance, if you are doing testing against a non-rails Web App, your cucumber scenarios might be defined with Watir-Webdriver which will drive a full web browser like Capybara does when in Javascript mode. The main difference being that WW has a lot more robust set of selctors than Capybara does, so it's a bit easier to write especially if your code isn't super clean (Capybara only supports selecting by ID, Value and Text, whereas WW supports selecting by almost anything)
So chances are you are going to want to use RSpec or Minitest for your unit testing )or the default Test::Unit), and Cucumber for Integration Testing IF you have an interested product owner or a need for a common language for test scenarios. If you do not, you can write your integration tests as Rspec examples or the minitest equivalent without losing much.
Related
I am using Ruby on Rails 3.0.10 and I am planning to switch to the versioni 3.1 as soon as possible. With the new RoR version I am planning also to add automated tests in my application.
What do you advice in order to test my application?
Do you advice some book to read and\or software to use?
If I start to write tests for RoR 3.0.10, those will be available and compatible in RoR 3.1?
In few words, I would like to know all about testing that I can, should and "must" use to improve my applications (also to test jQuery-JavaScript code).
What do you mean by automated tests?
If you mean unit/functional/integration tests, then you should have those in place before you upgrade to Rails 3.1. They will be very helpful in finding where things break during the upgrade, and when they all pass you can have some degree of confidence that your application works again.
If by automated tests, you mean something more high level, like UI testing (which is what I interpret it as), I would suggest either Watir or Selenium. Tests like those should still function and wouldn't be affected by the underlying implementation (whether it's Rails 3.0.10 or 3.1). As long as the app still works those tests would still work.
Use Rspec 2
Use Capybara
Use Spork
Use Guard
Guard + Spork (+ Rspec + Capybara) = Automated Testing.
I don't recommend using Cucumber. Everyone loves Cucumber, but it's a lot more work than it's worth. It's a regular expression layer on top of what you can do very easily with Rspec + Capybara with the new acceptance testing feature (it was a separate gem, now it's included in rspec).
When you build your app, I recommend this:
As you're conceptualizing your models, write Rspec models test descriptions. Things like it "should save a :start_date". Start building your models first by writing tests for the attributes (which start_date values it accepts, how to handle invalid values, etc.). Then as needed, write integration tests between models. They're model tests, not unit tests. When you're writing tests for the model attributes, those are unit tests (isolated class tests). When you're testing the functionality between models (when I do user.posts.create(:title => "My first blog post"), I expect a post is created for that user), those are in my opinion "integration" tests. In Rspec, these are "model" tests.
After you get some basics of your models in, write view tests. In order to implement the assertions in your view tests, you'll need to have Spork, Guard, and Capybara setup. And then you'll need to write your Haml… and then your controller code… In the end, you'll be writing the code to make the view test pass. The idea of having both controller and view tests is overkill. With Rspec + Capybara acceptance tests, you can wrap everything into the spec/acceptance directory. Saves you time and lots of thought. In the end you test your models thoroughly, and the requests/html thoroughly.
As your app gets larger, the tests will take much longer to run. It can easily take 10-20 minutes to run all tests! That'll quickly turn you off, especially if all you want is to just fix a typo before re-pushing. To avoid this, figure out clever ways to use seed data (data you don't need to recreate between each test), or clever ways to use mock data, and setup Spork. By not having to reload the Rails server between test runs, you cut out a TON of time. And with Guard + Spork, you don't need Autotest (Autotest is outdated).
If you want to go a step further, you can then test your javascript. This is still a very new area. There are two approaches right now, 1) using Capybara with a javascript-enabled driver (selenium is the only one that really works as of this post), or 2) using jasmine. I'd wait on this one, if your app is not a "purely javascript" app.
For Javascript tests, I'm thinking building some sort of node.js/websocket layer that allows you to run jasmine tests locally against your Rails + Spork test server, on your Mac and Windows (via Parallels or VMWare Fusion or whatever), is a much better way to do view + javascript testing, because you get the whole app as it really is in production.
That's all you'll need as of now to build a solid Rails app. Hope that helps, Lance.
First of all tests must be written before code.
I suggest cover you application by tests and then switch to Rails 3.1 in your case.
I prefer use rspec and cucumber together.
This is two different approach of testing TDD(rspec) and BDD(cucumber).
I would like to write some automatic tests for my Rails 3 application.
I wonder how to start with that.
I've heard about Selenium/RSpec/Cucumber, and I guess there are many more options.
What are the advantages/disadvantages of these testing frameworks ? Which of them has the best documentation ? Which one is the most popular in the Ruby world ? And in the industry at all ?
I have the general knowledge of how to write tests. I just want to learn the appropriate testing framework(s) for testing Rails applications.
Please help me to decide with which testing framework to start.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
I wouldn't say there is any one best set of testing tools out there; and the community definitely has not decided on any being standard in any way.
I highly recommend The RSpec Book from the Pragmatic Programmers-- about half of the book is on testing with Rails with Cucumber, RSpec, and browser simulators like Selenium. It's a comprehensive overview of the different situations you will find yourself in with Rails and what tools you might want to use in each situation.
Selenium is a framework to automate testing of user interface. Selenium script launches a browser window, goes to a web-page and manipulates page elements.
RSpec and Cucumber are tools for what is called "behavior driven development (BDD)". BDD is a development process where you have description of a separate feature, and some script (can be a Selenium script) that checks this feature. RSpec and Cucumber basically connect these two, so that when you run a test script, you see the list of features written in plain language and their testing status.
Selenium scripts can be written in a number of languages, while Cucumber scripts use Ruby. Documentation is OK for all of them, some knowledge of programming language is required though.
Starting links:
Cucumber;
Selenium;
Testing rails applications
The easiest way to get started is using TestUnit. When you generate a model, view, controller using scaffolding it will automatically generate a testing folder for you containing a test environment setup file and tests, or you can add tests to existing models with a rails generator.
It is then a matter of reading the documentation on TestUnit, of which there is a fair amount.
For a next step, my personal preference is to use RSpec over Cucumber - I've found Cucumber hard to maintain, and whilst it is wonderful to have tests that are readable by your boss, I've found that most bosses don't really want to read or contribute to tests, and it makes for time-consuming development and is very 'bitty' - lots of small bits of text scattered in various files.
If you want to jump right in and begin using RSpec for Rails, then I'd recommend Rspec-Rails, which includes many helpers for asserting the correct behaviour of your application.
I could go on, but you've asked for 'where do I start' - there's two good places.
There are a bunch of frameworks that help you test your ruby/rails code... the great thing is the dynamic nature of Ruby gives you a lot of flexibility.
I like to use RSpec to test the internals of my code, and Cucumber to test the application's behavior. So generally, RSpec will test models and methods, and Cucumber tests user interaction through the browser.
I highly recommend this pragmatic programmers ebook on BDD using RSpec on Cucumber.
Railscast on getting started in cucumber (also called cukes)
Cucumber project
Rails project
For integration tests of my Rails web app I use Steak (something like Cucumber). The specs of Steak are in a folder named spec/acceptance. Are Steak/Cucumber now for integration or acceptance testing? I always thought that this is something different.
First, a note on the terminology: the term integration test is a bit vague in the TDD community. Depending whether you come from Java or Rails (with Test::Unit), you might understand different things by it. In Rails (with Test::Unit) integration tests are the tests that test your full stack, while functional tests would be the ones testing your controller. Most people in the Java community (at least by my observation) would think it is the other way around. I personally prefer to call the end-to-end tests acceptance tests, while tests that hit several layers of the system (but not everything) -- integration tests. All in all, that is pretty dependent on the culture your are in.
As for Cucumber and Steak -- both are frameworks that allow a development style known as Behavior-Driven Development (or BDD for short). The point is that you have two levels of tests:
End-to-end tests, which test your through the full stack -- they simulate a browser, go through your controllers and hit the database. Cucumber and Steak fit this niche.
Unit tests, which test a small bit of functionality in isolation (usually a single class, mocking its collaborators). This is where RSpec fits.
In BDD, you start with a failing end-to-end test (lovingly know as the "upper gear"), and then you start implementing functionality test-first with RSpec (the "lower gear"), until you get the end-to-end test passing. This way the end-to-end test is driving your unit tests, which in turn are driving your implementation. The main benefit is avoiding scope creep -- you don't end up implementing user-visible functionality that you don't need (since you don't write an end-to-end test for it).
If you want more information on this, I've heard that the Behavior Driven Development Wikipedia article is surprisingly good. Also, the RSpec book.
So, both Cucumber and Steak are frameworks that allow you to write tests in the "upper gear". The difference is in the style -- Cucumber has you writing your tests in natural language. This has several benefits.
Tests are readable by the business people -- while you cannot expect non-programmers to write them, they do a great job in communicating what you intend to do. You can write the feature (the Cucumber test first) and show it to the customer to get some feedback on whether this is what they actually want. I've found this very useful.
Cucumber features communicate intent better -- since you get to use the full power of the English language (or any, really), you can communicate why this feature is relevant and how the users accomplish their goal on a level that Ruby won't allow you to.
Cucumber helps discovering the ubiquitous language -- the domain includes a lot of terms that fly around in the conversations with the customers. Cucumber allows you to discover and capture them before you start implementing the feature. And it's all test-driven.
Cucumber features are somewhat higher-level, which makes the features (but not the step definitions) more independent of the interface. This way if the interface needs changing, you won't have to rework the features.
The downsides include that it is a bit tricky to learn how to apply it nicely and that you have to write a bit more (both features and step definitions). I've found that the second is not really a problem if you have been doing it for a while, since you get a body of reusable steps that allow you to write the next features faster.
Steak, on the other hand is simpler and it's Ruby. You loose all the benefits of using English, but you can write less and it will execute faster (somewhat).
In the bottom line, you use both to write the end-to-end tests that drive development.
For a new application I want to start dabbling in BDD and I'm trying to decide between using RSpec or Thoughtbot's Shoulda. I like the macros that Shoulda uses, and the fact that it doesn't seem to reinvent the way Ruby/Rails does testing, but simply provides an add-on. On the other hand, the macros seem like a bit too much "magic" instead of being explicit about what you're testing (however I know from dabbling that it's annoying to write a dozen "should be invalid without xxx" two-liners on a model). To be honest I find writing specifications/tests for models to be trivially and almost boringly easy, but I find writing them for controllers to be insanely difficult because I'm never sure exactly what I should be testing or how to write it.
I'm iffy on the subject of mocking and stubbing since I think they give you false assumptions (since you can just tell it to think it has whatever data you need or to pretend that Method X was called) and I know that RSpec makes heavy use of both of them. I like the documentation that RSPec produces but I'm creating an application for sale, not to give to a client so the pretty documentation isn't that useful. I like Cucumber but it seems like overkill (and yes I know it can be used with Shoulda).
At this point is the Rails community in favor of RSpec or Shoulda?
Regarding mocks and stubs (and fakes, doubles and whatnot) - when you're testing at the unit level, either with TDD or after the fact, the whole point is telling it to think it has the data you need, using a Stub. And you write a test for the real object to ensure that it actually produces that data. The intent is to check the internal behaviour of the class under test, not that its upstream connections are behaving properly. That's at the unit level - you will test the end-to-end behaviour in your integration or feature/story/acceptance tests (or whatever flavour of higher-level test name you prefer).
A mock object is, to my mind, more about the downstream - you want to check that the class under test has made the appropriate call - you're not concerned that anything actually happens, just that the right method was called with the right arguments. Mocks are really good for that. Rspec has its own mocking framework, but Mocha and FlexMock are also widely used.
There's been a lot of discussion/explanation/debate/flame-warring about nomenclature here, BTW. Martin Fowler (who is better-qualified than most to pronounce on the subject) wrote a seminal blog post to clarify it and I think it makes sense. Here's another article, with a few examples.
The rails community is in favor of both RSpec and Shoulda. It depends of the developer.
If you prefer Shoulda, use it.
If you prefer RSpec, use it ;)
They're both different library with a similar goal. It doesn't mean every developer has to be for or against it. It only means that you can use either of them.
It's up to you to make your choice depending of your preferences (and the other developers you're working with).
You can use shoulda macros in RSpec. It is definitely less common, but a great option: http://robots.thoughtbot.com/post/159805987/speculating-with-shoulda.
But as Radar says, ultimately you should try them different libraries and decide.
I use Shoulda matchers with RSpec. Best of both worlds: big community behind RSpec, fast development and lots of coverage with Shoulda matchers.
Shoulda watchers: 758.
RSpec watchers: 1279.
Ultimately, it's up to you to decide which one you prefer.
As far as i can tell, since you mention BDD, there seems to be a more natural match between cucumber and RSpec. The thing i like most about shoulda are its validation-macro's. There are two options to solve that in RSpec:
use the shoulda macro's in RSpec, a great option, answered before
use rspec-validations-expectations plugin, small and hardly known, but which fixes just that (easy ActiveRecord validations testing).
You should definitely go with which library feels most natural to you (how tests are expressed). For me, with the previously mentioned options, it was easier to discard the shoulda option (on its own at least), and i went for rspec and cucumber.
My current stack of testing tools is:
Steak for acceptance testing
Capybara for Browser simulation with drivers: Selenium & Akephalos
Machinist for stubs
Rspec for unit testing
A lot of the Rails developers out in the world use RSpec, and some of those use Shoulda. DHH, the lead developer of Rails prefers Test::Unit and Minitest. Thoughbot's Shoulda builds on both Test::Unit (And of course Minitest) and RSpec.
Ultimately, as a Rails Hotline volunteer, you'll get more community support out of RSpec and there are a ton of additional gems out there specifically for improving RSpec. With that said Minitest and Test::Unite are core to Ruby.
[OPINION] I tend to use RSpec if the software requires a "behavior flow" and Minitest if it requires pure unit functionality (mostly because Minitest's benchmark library is really simple).
I am a little confused by the sheer number of testing frameworks available for Ruby/ROR.
I have recently watched the Cucumber Railscasts and found them very interesting. So I started having a play and then struggled to see conceptually where I would put various tests.
It would seem to be quite possible to do everything that can be done in unit tests within Cucumber, so do I need to write unit tests or should I just write my feature definitions and concentrate on providing as good a coverage as I can get using that.
Should I create my Unit tests using Rspec or Test:Unit? When I'm testing Ajax functionality should I use Selenium or Watir?
There seem to be so many options here I am struggling to see which tools to use and where the boundaries are.
What are other peoples experiences of Cucumber and where to draw the line between writing Cucumber Integration tests and Test:Unit and/or Rspec based unit and functional tests. Is anyone aware of a good write-up on this subject suggesting where to draw lines between testing methods and the strengths and weaknesses of the various tool.
I appreciate that some of this is subjective but common approaches on how to attack this issue would be welcomed.
Use Cucumber at a high level to describe what a user should be able to see and do. Use RSpec, Test:Unit, Shoulda, etc. to write unit tests. Straight from the horse's mouth:
When you decide you want to add a new feature or fix a bug, start by writing a new feature or scenario that describes how the feature should work. Don’t write any code (yet).
...
This is when you start writing code. Start by writing a couple of lines of code to address the failure you got from Cucumber. Run cucumber again. Repeat and rinse until you’re happy with your feature. When you get down to nitty gritty details, drop down one abstraction level and use RSpec, or any Ruby testing framework, to write some specs/tests for your classes.
Cucumber is made to test your whole stack, together, as opposed to 'units'.
You need to decide where to draw the line, but a lot of under the hood stuff probably wouldn't be covered in a cucumber test. Say when signing up, I fill out a form, with my name, email, phone number, etc. A unit test might check to see that a new User will also create a new TelephoneNumber. From the user's perspective, they don't really care that it creates a new TelephoneNumber, they care that once they've signed up, they have an account and can see their telephone number.
I don't have too much experience writing cucumber tests (not yet), but I hope this helps a bit.
When a unit test fails (I mean a real unit test that tests a method in isolation using mocks), it tells you what "unit" has a problem. When an acceptance test fails, it tells you what "feature" has a problem, not where the problem is located.
When you create a rails app you get functional, interation, and unit tests by default. Cucumber is an additional test it is a way to also test the experience that your user will have. When they click the button labeled "go" they should see "success" rendered rather than a 404. This will make sure that nothing you do accidentally messes up the user experience, and that from the top to bottom your app works for the most common use cases you can think of. The other tests are meant to insure that nothing goes wrong, and that you have inspected ever model and method with a microscope. It may be possible to replicate unit tests in their entirety with cucumber, but it would be painful (and crazy slow to execute, especially if you're using selenium). The best time to write tests is when you're developing code, and the quickiest and easiest way to do that, is by using in the built-in rails testing and maybe some additional help such as shoulda, rspec, also i'm a huge fan of factory-girl. If you haven't already checked it out www.railscasts.com has a great intro to cucumber, and rspec, and factory-girl, ... I know this question has already been answered (it's no) but this is my two cents. Good luck coding!!
I have thought/struggled with this question much, and here's where I've arrived.
Cucumber first and Cucumber last. Cucumber will provide the primary test coverage.
The core model methods that do the real business work of the application should also be developed/covered with rspec/unit tests.
Why the unit tests as well?
1) The unit tests will test run much faster.
2) This core business logic may (will probably) be used in several ways beyond the current views (which Cucumber tests through). These methods ought to be hammered with all types of possible inputs and outputs directly calling the method in the test.
Why not unit test the rest of the models, and the controllers and views?
1) Cucumber already has it covered once.
2) I find that the views-controller-some-model-methods all work together to get things done (think everything exercised to log in); so I like to test them together.
I've been practicing Cucumber/RSpec for the past half year or so doing BDD.
First of all BDD is not easy to get into, it will feel unnatural at the beginning.
But once you get into it, there's no other way to do programming.
To answer your question. To test Javascript you'll need a javascript driver that can be used by Capybara which is used by Cucumber.
capybara-webkit is what all the cool kids use now these days
There's one important thing to note.
Integration tests are slow.
And unit tests are fast, but can be slow, so it's important you use the right database cleaner and you write good tests that have good isolation.
My test setup which I'm extremely happy with:
Guard for loading spork
Spork for faster tests
Cucumber for integration testing
capybara-webkit for javascript testing
RSpec for unit testing
I don't do view tests and controller tests as these are redundant in my opinion as good knowledge of XPATH willl have you writing remarkable tests that even cover your page layout and structure.
Personally I don't think that you should stop writing unit tests. As an acceptance testing tool, Cucumber should replace your functional tests and, if you writing, view tests.
Cucumber features are supposed to be simple and coupled to the real user's value a given feature has.
From my experience, Cucumber and Rspec have different appeal. Rspec appeals to me from a developer perspective because its easy to write and provides very quick feedback when something breaks. Cucumber does not appeal to me as a developer because it does not run as quickly as Rspec. However, Cucumber does appeal to me as a business stakeholder since it provides full coverage of entire features.
Do yourself a favor and keep writing unit tests.