I'm beginning the planning phase of creating a testing suite for my rails 3.0.8 application. I'm trying to decide on which testing framework/gems to use. Normally I prefer to stick to Rails convention as much as possible. However, this means using TestUnit. There are many competing test frameworks to choose from that were created as an alternative to TestUnit. Has TestUnit gotten better over the years, or is it not a very good contender?
I've also heard of a lot of good things about rspec. Are rspec and TestUnit close in terms of functionality, or does rspec blow TestUnit out of the water?
Whatever framework I choose, I'd prefer it to have a good support base(lots of users and documentation), ease of use/simplicity, and a lasting future.
Both Test::Unit and Rspec are very nice solutions. As for me I prefer to use RSpec in my Rails project - because it has many automatic solutions like autotest, highlighting syntax, shortcut commands like rake spec and other stuff that make using test easer. But in my plain ruby scripts I use Test::Unit - it is very lightweight and staightforward when you don't need complex testing (integrational and others)
The answer is matter of taste, use both in one project and find yourself what do you prefer. (I do Test::Unit, RSpec has to much sugar for me).
Update 2014: As Ruby 2.x has MiniTest built-in I would suggest to check MiniTest::Spec eventually which is simple BDD extension. You can mix "describe" and "it" blocks to nicely structure test contexts with. It also provides "must_be.." matchers if you want to more than assert.
I prefer and use these gems for testing
Rspec
Faker (for creating random and proper data for database)
Fabrication (for simple creating instances)
To start with rspec take a look at this presentation. It contains all the necessary examples:
http://kerryb.github.com/iprug-rspec-presentation
Also read through these articles on rspec:
http://relishapp.com/rspec/rspec-rails
This Ruby Inside article, especially if you follow the links and read the comments, provides quite a thorough discussion of the relative merits of Test::Unit and RSpec. A key differentiator seems to be that:
people who have non-technical clients are more likely to view RSpec and Cucumber more favourably;
others are more likely to prefer more direct systems like Test::Unit.
Also see this part of Michael Hartl's Rails Tutorial for a comparison of RSpec and Cucumber.
You could use rspec with cucumber and webrat gems which makes a great combination, if you have a large website. It will be cumbersome at first but later on it will be a huge benefit.
You could watch the following railscasts episodes to have more info about those gems:
155 Beginning with Cucumber
156 Webrat
157 RSpec Matchers & Macros
159 More on Cucumber
There are also some other gems like capybara which is relatively new and promising, but honestly I didnt try it.
If you're beginning your journey with testing Rails application I would recommend you start doing it with TestUnit. You can find very nice course about best testing rails app and best practices on http://www.codeschool.com/courses/rails-testing-for-zombies
If you're more experienced when it comes to testing you can try RSpec.
Related
I have been watching Ryan Bates' RailsCasts for years now, and I'm also a customer of the pay portion of the site. I learned BDD from watching Cucumber episodes in the past.
Now I have learned about TestUnit, RSpec, Capybara, and MiniTest. I am beginning to get confused about what is what.
What is the difference between these 4 items? I know obviously Cucumber executes plain text features, and I guess that could be considered integration testing.
But now I also see that recent versions of Cucumber require MiniTest? Is Cucumber simply a DSL that sits on top of a testing framework?
I know also that RSpec has its own syntactical sugar for doing assertions, the "describe" blocks. And it appears that MiniTest also supports this syntax.
I know that Capybara is used for looking at the web page content that is generated, I think.
Here is my question:
If I am creating a new Rails 3.2 application, what combination of these testing programs should I use? What would be extra helpful is a list that explains how these gems and their associated processes complement each other where applicable, like:
Cucumber is a DSL for driving BDD
Cucumber is for integration tests and is based on creating user stories that are customer-readable
It implements its tests behind the scenes via MiniTest
MiniTest is a new testing framework that comes with Ruby 1.9 and is very fast.
MiniTest is also used for unit testing, such as testing controllers and models
It does not yet have as many features as RSpec
Cucumber also uses Capybara to access DOM elements from a javascript-enabled browser simulator such as Selenium
When you test in Rails, you have the ability to do the following kinds of tests: controllers, views, models, and integration (MVC together)
Some people just do integration and model testing, because they feel that integration testing handles enough of the controller and view testing itself, and anything too complex can simply be moved up to the model
Thank you so much for any help you can offer to clear these ideas up for me.
Okay, so let me try and explain, based on my own experience
Cucumber, is a ATDD (or BDD) tool that lets you write tests in a business-oriented domain language. It's primary use is as a conversation tool with you product owners. Instead of writing detailed requirements, you express those requirements as examples of the system under test. Each cucumber test, in effect, becomes a business requirement that must be satisfied.
The Cucumber tool itself, translates these plain-text statements into a little module for you to execute ruby code in. What Ruby Library you use inside of the step definitions depends entirely on the project you are working on.
The safest way to describe Cucumber is that it's a testing framework with an emphasis on communication between IT and Business Partners, which is why it's becoming so popular.
Rspec and Minitest are other frameworks with other strengths, but lack this business readability factor, in that they are mostly code and not nearly as readable to a non-technical person. That isn't necessarily a bad thing, especially if your product owner is a little more hands off.
How does this tie in with something like Capybara? Capybara is an integration test automation library that drives a headless browser on the Rack::Test framework for very fast tests, and has a highly readable DSL. The only downside is Rack::Test doesn't support javascript, so it provides a way to fail over to Selenium if you a running a javascript test. Rspec and Cucumber both have mechanisms for triggering this failover.
There are lots of other Ruby automation libraries. For instance, if you are doing testing against a non-rails Web App, your cucumber scenarios might be defined with Watir-Webdriver which will drive a full web browser like Capybara does when in Javascript mode. The main difference being that WW has a lot more robust set of selctors than Capybara does, so it's a bit easier to write especially if your code isn't super clean (Capybara only supports selecting by ID, Value and Text, whereas WW supports selecting by almost anything)
So chances are you are going to want to use RSpec or Minitest for your unit testing )or the default Test::Unit), and Cucumber for Integration Testing IF you have an interested product owner or a need for a common language for test scenarios. If you do not, you can write your integration tests as Rspec examples or the minitest equivalent without losing much.
I've been using RSpec for testing my Rails application, but I find that testing views, partials and helpers that it isn't fully optimized for that. Can someone suggest a better approach to testing these things?
Many users will pair RSpec with Cucumber. Cucumber provides higher level testing and will allow you to test most aspects of your views efficiently.
Edit:
A few resources:
http://railscasts.com/episodes/155-beginning-with-cucumber
http://railscasts.com/episodes/159-more-on-cucumber
http://railscasts.com/episodes/186-pickle-with-cucumber
http://pragprog.com/book/achbd/the-rspec-book
Helpers are really easy to test with RSpec, but views and partials can be tricky since you really are doing an integration test. For these I use Cucumber since it allows you to describe the behavior in plain english and then use a nice DSL like Capybara to interact with the application. Much bigger learning curve for Cucumber than RSpec (IMHO), but the payoff has big rewards.
Yeah. Cucumbers all the way. Also see:
http://pragprog.com/book/hwcuc/the-cucumber-book
http://peepcode.com/products/cucumber
An alternative to Cucumber is RSpec request specs. The syntax is less natural than Cucumber, but still very readable.
So yeah, I'm gettin' into this testing thing!
But I don't know what to use =/
Rspec + Shoulda?
Rspec + Steak?
Minitest?
Cucumber?
Capybara?
Coulda? (Cucumber with Shoulda mixup)
Mini_shoulda? (Minitest with Shoulda mixup)
Argh, so many choices! I'm confused <.< I do know I want something small and simple to test my future gems and sinatra & rails apps. What do you people use and why?
If you are just starting out, you should go with the defaults and follow the official guides. The important thing is that you practice testing. It's less important which testing library/framework you use - that's a stylistic choice that's secondary to actually practicing testing in the first place - and the easiest way to get starting testing your Rails app is to follow the "Golden Path" that the official guides lay out.
Once you get more comfortable with testing your Rails app, you will be in a much better position to evaluate the options out there. I personally like rspec and cucumber (at the moment - I may decide to change my preference), but I was only able to come up with this preference after already being somewhat familiar with testing and after already being somewhat familiar with Rails.
I just came across this blog post which states:
It seems that the accepted way to test in Ruby is to use Rspec for unit tests and to use Cucumber for acceptance tests (higher level functional testing).
It then goes on to ask: "if we agree that BDD is good, why don’t we write our unit tests in a format that is more amenable to BDD, that being the Cucumber format of tests?" and compares rspec unit tests against unit tests in Cucumber.
At work we use the pretty standard RSpec and Cucumber combination. I don't know how they decided on this specific combination, it was already there when I started and worked for us, so there was no need to change. Also it's a quite widely use combination so finding examples with Google etc. isn't too hard.
For my 1.9 based private projects I think I will use Minitest from now on. No external dependencies, a simple BDD DSL (require 'minitest/spec') and some other niceties like randomization. Here's a nice quick intro:
http://www.bootspring.com/2010/09/22/minitest-rubys-test-framework/
For someone just getting started I recommend just using RSpec. It doesn't do everything but it will allow you to build up reasonable sets of unit and integration tests. When you run into the limits of what RSpec handles easily then let that need guide you in choosing additional tools.
Can't offer anything more sensible than Justice, but before you ignore that wisdom :-) do check out this video (minitest author Ryan Davis at Cascadia 2011 ruby conf):
http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-t-matter
slides: http://www.zenspider.com/~ryan/presentations/CascadiaRubyConf_2011_-_Size_Doesn%27t_Matter.pdf
Being able to run all your tests very quickly is a very good thing.
For a new application I want to start dabbling in BDD and I'm trying to decide between using RSpec or Thoughtbot's Shoulda. I like the macros that Shoulda uses, and the fact that it doesn't seem to reinvent the way Ruby/Rails does testing, but simply provides an add-on. On the other hand, the macros seem like a bit too much "magic" instead of being explicit about what you're testing (however I know from dabbling that it's annoying to write a dozen "should be invalid without xxx" two-liners on a model). To be honest I find writing specifications/tests for models to be trivially and almost boringly easy, but I find writing them for controllers to be insanely difficult because I'm never sure exactly what I should be testing or how to write it.
I'm iffy on the subject of mocking and stubbing since I think they give you false assumptions (since you can just tell it to think it has whatever data you need or to pretend that Method X was called) and I know that RSpec makes heavy use of both of them. I like the documentation that RSPec produces but I'm creating an application for sale, not to give to a client so the pretty documentation isn't that useful. I like Cucumber but it seems like overkill (and yes I know it can be used with Shoulda).
At this point is the Rails community in favor of RSpec or Shoulda?
Regarding mocks and stubs (and fakes, doubles and whatnot) - when you're testing at the unit level, either with TDD or after the fact, the whole point is telling it to think it has the data you need, using a Stub. And you write a test for the real object to ensure that it actually produces that data. The intent is to check the internal behaviour of the class under test, not that its upstream connections are behaving properly. That's at the unit level - you will test the end-to-end behaviour in your integration or feature/story/acceptance tests (or whatever flavour of higher-level test name you prefer).
A mock object is, to my mind, more about the downstream - you want to check that the class under test has made the appropriate call - you're not concerned that anything actually happens, just that the right method was called with the right arguments. Mocks are really good for that. Rspec has its own mocking framework, but Mocha and FlexMock are also widely used.
There's been a lot of discussion/explanation/debate/flame-warring about nomenclature here, BTW. Martin Fowler (who is better-qualified than most to pronounce on the subject) wrote a seminal blog post to clarify it and I think it makes sense. Here's another article, with a few examples.
The rails community is in favor of both RSpec and Shoulda. It depends of the developer.
If you prefer Shoulda, use it.
If you prefer RSpec, use it ;)
They're both different library with a similar goal. It doesn't mean every developer has to be for or against it. It only means that you can use either of them.
It's up to you to make your choice depending of your preferences (and the other developers you're working with).
You can use shoulda macros in RSpec. It is definitely less common, but a great option: http://robots.thoughtbot.com/post/159805987/speculating-with-shoulda.
But as Radar says, ultimately you should try them different libraries and decide.
I use Shoulda matchers with RSpec. Best of both worlds: big community behind RSpec, fast development and lots of coverage with Shoulda matchers.
Shoulda watchers: 758.
RSpec watchers: 1279.
Ultimately, it's up to you to decide which one you prefer.
As far as i can tell, since you mention BDD, there seems to be a more natural match between cucumber and RSpec. The thing i like most about shoulda are its validation-macro's. There are two options to solve that in RSpec:
use the shoulda macro's in RSpec, a great option, answered before
use rspec-validations-expectations plugin, small and hardly known, but which fixes just that (easy ActiveRecord validations testing).
You should definitely go with which library feels most natural to you (how tests are expressed). For me, with the previously mentioned options, it was easier to discard the shoulda option (on its own at least), and i went for rspec and cucumber.
My current stack of testing tools is:
Steak for acceptance testing
Capybara for Browser simulation with drivers: Selenium & Akephalos
Machinist for stubs
Rspec for unit testing
A lot of the Rails developers out in the world use RSpec, and some of those use Shoulda. DHH, the lead developer of Rails prefers Test::Unit and Minitest. Thoughbot's Shoulda builds on both Test::Unit (And of course Minitest) and RSpec.
Ultimately, as a Rails Hotline volunteer, you'll get more community support out of RSpec and there are a ton of additional gems out there specifically for improving RSpec. With that said Minitest and Test::Unite are core to Ruby.
[OPINION] I tend to use RSpec if the software requires a "behavior flow" and Minitest if it requires pure unit functionality (mostly because Minitest's benchmark library is really simple).
I am setting up a rails app and I just finished making some unit tests and my friend said that apparently fixtures are no longer cool and people are now using RSpec or shoulda. I was wondering what the actual benefits are to use these other toolkits. Any information at all is appreciated.
-fREW
I personally prefer Shoulda to RSpec. I find that Shoulda has less magic syntax than RSpec. My problem with RSpec is that yeah it's very readable when I read it aloud, but when I get to writing it, hmmmm, I'm never sure how a given assertion should be written. Prag Dave explains the problem better than me. He also likes Shoulda and has a few examples.
RSpec and similar frameworks are tooling designed to aid in Behavior Driven Development. They're not just a prettier way to write tests, though they do help with that.
There is plenty of information on BDD here: http://behaviour-driven.org/
And wikipedia: http://en.wikipedia.org/wiki/Behavior_Driven_Development
There are too many benefits to list here, so I'd recommend browsing that site a little.
There are two different things here:
The first thing is what framework to use for writing tests/specs. Here you can choose between Test::Unit, RSpec, Shoulda and so on. The choice is whether you want to do traditional TDD (Test::Unit) or whether you prefer the alternative ways of thinking about specifiying behaviour advocated by developers like David Chemlinsky (RSpec and to some extent Shoulda).
The second thing is how to handle test data. There are Rails fixtures and alternatives desgined with other goals such as the FixtureReplacement plugin. Before Rails 2.0 fixtures had significant and well-documented pratical problems. Many of the practical issues were fixed in Rails 2.0. However fixtures can lead to inadvertent test coupling and some of the alternatives try to avoid this.
RSpec is way more powerful because it's far easier to both read and write tests in. It's also very elegant when using mocks and stubs, a concept which will become extremely useful once you start using them in your tests. Try it in a simple test app (NON RAILS!) and you'll see how elegant your specs are versus the equivalent standard testing.
Check out Josh Susser's The Great Test Framework Dance-off for a comparison of the popular Ruby testing frameworks.
If you are building a large application and don't have a team that are all really good at writing decoupled code that can be well-tested with black box tests and are prepared to fully embrace using/debugging lots of mocks & stubs, don't go down the Factory road.
Wherever you read about how Awesome Factories Are you'll see a little caveat about how factories might not be feasible in a large application because they are a little slower than fixtures.
But "a little slower" is really orders of magnitude slower.
Factories are not significantly easier to code than fixtures that use labels for ids, so long as you keep the fixtures organized. And in some cases factories are harder to debug.
Just tonight I converted a single factory to fixtures, and the runtime of the test file that used it went from 65 seconds to 15 seconds, even though only about 15% of the tests in that test file use that factory.
If you use minitest you can run your tests in random order; this will quickly reveal any data coupling between tests. (not sure if rspec has the option to randomize test order)
Test::Unit is good for small applications. But there are a lot of benefits to use testing frameworks like Shoulda or RSpec, e. g. contexts!!
I don't see Shoulda and RSpec in an either-or-relation. I use Shoulda as a substitute for RSpec when it comes to single-assertion testing. I really like the Shoulda one-liners, but writing matchers is much easier in RSpec. So my advise is to use the different testing tools where they fit best.
You may use testing framework like Cucumber which is even more faster than RSpec..