I happen to be a subscriber of Ruby Inside, since I'm particularly interested in Rails. Yesterday, the creator of Rails, David Heinemeier Hansson, pretty much said that he's just using test/unit. I would understand that, since it's Rails internal, but he seems to have given a strong opinion. He believes that RSpec and Cucumber are needlessly complicated.
I would normally not pay much attention, but it depends on who says something. I respect Hansson a lot and his opinion got me thinking. When I started with Rails, I never really looked into test/unit. Just RSpec and Cucumber.
And that is why I want your insight. Do you think that RSpec is indeed complicated for not much added value? Does writing test/unit take less time and effort?
My recommendation would be to use either Shoulda (extends Test::Unit) or RSpec with Capybara, and -no- Cucumber.
I think that the use of either RSpec or Shoulda for nested contexts is definitely worth doing. RSpec is definitely heavy-weight (perhaps overweight) though, and I'm on the fence with it for that reason.
Cucumber, I've finally come to understand, is waaay more cumbersome than it's usually worth. You can accomplish what you need more simply and robustly with plain ol' integration tests and Capybara. Remember -- Capybara != Cucumber, and Capybara is quite capable all on its own.
Shoulda is nice, because it simply adds conveniences to the standard Test::Unit framework, and is therefore much lighter-weight than RSpec (technically, each solves a different set of problems, but they both provide nested-context capabilities). RSpec has the advantage of making assertions read more naturally, and also generating more helpful failure messages in many cases, without the need for writing message arguments on the assertions.
Also, remember that Cucumber does not actually require RSpec, so if you want to keep using Cucumber, you can do that with just Test::Unit. Choices abound.
It's all about semantics. RSpec and Test::Unit are similar functionally. Personally I've always preferred RSpec because I found it more natural to write tests using it. I also like the simplicity of writing custom matchers, and the default matchers provided are useful.
Cucumber is a different beast entirely. Yes it's fairly cumbersome and becomes hard to maintain if you don't organise your step definitions properly but it does have one very strong use case and that is when your client is writing your specifications.
I've worked on projects where the client has been writing Cucumber scenarios together with one of our QA team. As a non-technical person it's an extremely approachable and natural way to specify user stories in the code. Cucumber really helped us walk the walk when it came to following our agile practices. The quality of the end product benefitted from that but no I do not like Cucumber as a developer :)
It's a matter of personal taste.
I like to write easy cucumber tests without worrying about details. Just testing the "happy" paths of my app. (portable, understandable, slow)
I leave the details to Test/Unit. (easy, fast)
It takes more time to understand:
get :products, :session => #session_id_for_product_banana
assert_select "table" do
assert_select "td#name", "Banana"
end
instead of
When I go to the banana page
Then I should see "Banana"
Sure those tests are not equal but who cares whether "Banana" is in a div or a table or doesn't have the right html-id.
I dont' like functional tests because after refactoring, the id's could be gone, the session expectation could be changed. If that's the case you will need to refactor your code AND tests. If you'd use cucumber you wouldn't have to change your scenario's.
Related
So yeah, I'm gettin' into this testing thing!
But I don't know what to use =/
Rspec + Shoulda?
Rspec + Steak?
Minitest?
Cucumber?
Capybara?
Coulda? (Cucumber with Shoulda mixup)
Mini_shoulda? (Minitest with Shoulda mixup)
Argh, so many choices! I'm confused <.< I do know I want something small and simple to test my future gems and sinatra & rails apps. What do you people use and why?
If you are just starting out, you should go with the defaults and follow the official guides. The important thing is that you practice testing. It's less important which testing library/framework you use - that's a stylistic choice that's secondary to actually practicing testing in the first place - and the easiest way to get starting testing your Rails app is to follow the "Golden Path" that the official guides lay out.
Once you get more comfortable with testing your Rails app, you will be in a much better position to evaluate the options out there. I personally like rspec and cucumber (at the moment - I may decide to change my preference), but I was only able to come up with this preference after already being somewhat familiar with testing and after already being somewhat familiar with Rails.
I just came across this blog post which states:
It seems that the accepted way to test in Ruby is to use Rspec for unit tests and to use Cucumber for acceptance tests (higher level functional testing).
It then goes on to ask: "if we agree that BDD is good, why don’t we write our unit tests in a format that is more amenable to BDD, that being the Cucumber format of tests?" and compares rspec unit tests against unit tests in Cucumber.
At work we use the pretty standard RSpec and Cucumber combination. I don't know how they decided on this specific combination, it was already there when I started and worked for us, so there was no need to change. Also it's a quite widely use combination so finding examples with Google etc. isn't too hard.
For my 1.9 based private projects I think I will use Minitest from now on. No external dependencies, a simple BDD DSL (require 'minitest/spec') and some other niceties like randomization. Here's a nice quick intro:
http://www.bootspring.com/2010/09/22/minitest-rubys-test-framework/
For someone just getting started I recommend just using RSpec. It doesn't do everything but it will allow you to build up reasonable sets of unit and integration tests. When you run into the limits of what RSpec handles easily then let that need guide you in choosing additional tools.
Can't offer anything more sensible than Justice, but before you ignore that wisdom :-) do check out this video (minitest author Ryan Davis at Cascadia 2011 ruby conf):
http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-t-matter
slides: http://www.zenspider.com/~ryan/presentations/CascadiaRubyConf_2011_-_Size_Doesn%27t_Matter.pdf
Being able to run all your tests very quickly is a very good thing.
For a new application I want to start dabbling in BDD and I'm trying to decide between using RSpec or Thoughtbot's Shoulda. I like the macros that Shoulda uses, and the fact that it doesn't seem to reinvent the way Ruby/Rails does testing, but simply provides an add-on. On the other hand, the macros seem like a bit too much "magic" instead of being explicit about what you're testing (however I know from dabbling that it's annoying to write a dozen "should be invalid without xxx" two-liners on a model). To be honest I find writing specifications/tests for models to be trivially and almost boringly easy, but I find writing them for controllers to be insanely difficult because I'm never sure exactly what I should be testing or how to write it.
I'm iffy on the subject of mocking and stubbing since I think they give you false assumptions (since you can just tell it to think it has whatever data you need or to pretend that Method X was called) and I know that RSpec makes heavy use of both of them. I like the documentation that RSPec produces but I'm creating an application for sale, not to give to a client so the pretty documentation isn't that useful. I like Cucumber but it seems like overkill (and yes I know it can be used with Shoulda).
At this point is the Rails community in favor of RSpec or Shoulda?
Regarding mocks and stubs (and fakes, doubles and whatnot) - when you're testing at the unit level, either with TDD or after the fact, the whole point is telling it to think it has the data you need, using a Stub. And you write a test for the real object to ensure that it actually produces that data. The intent is to check the internal behaviour of the class under test, not that its upstream connections are behaving properly. That's at the unit level - you will test the end-to-end behaviour in your integration or feature/story/acceptance tests (or whatever flavour of higher-level test name you prefer).
A mock object is, to my mind, more about the downstream - you want to check that the class under test has made the appropriate call - you're not concerned that anything actually happens, just that the right method was called with the right arguments. Mocks are really good for that. Rspec has its own mocking framework, but Mocha and FlexMock are also widely used.
There's been a lot of discussion/explanation/debate/flame-warring about nomenclature here, BTW. Martin Fowler (who is better-qualified than most to pronounce on the subject) wrote a seminal blog post to clarify it and I think it makes sense. Here's another article, with a few examples.
The rails community is in favor of both RSpec and Shoulda. It depends of the developer.
If you prefer Shoulda, use it.
If you prefer RSpec, use it ;)
They're both different library with a similar goal. It doesn't mean every developer has to be for or against it. It only means that you can use either of them.
It's up to you to make your choice depending of your preferences (and the other developers you're working with).
You can use shoulda macros in RSpec. It is definitely less common, but a great option: http://robots.thoughtbot.com/post/159805987/speculating-with-shoulda.
But as Radar says, ultimately you should try them different libraries and decide.
I use Shoulda matchers with RSpec. Best of both worlds: big community behind RSpec, fast development and lots of coverage with Shoulda matchers.
Shoulda watchers: 758.
RSpec watchers: 1279.
Ultimately, it's up to you to decide which one you prefer.
As far as i can tell, since you mention BDD, there seems to be a more natural match between cucumber and RSpec. The thing i like most about shoulda are its validation-macro's. There are two options to solve that in RSpec:
use the shoulda macro's in RSpec, a great option, answered before
use rspec-validations-expectations plugin, small and hardly known, but which fixes just that (easy ActiveRecord validations testing).
You should definitely go with which library feels most natural to you (how tests are expressed). For me, with the previously mentioned options, it was easier to discard the shoulda option (on its own at least), and i went for rspec and cucumber.
My current stack of testing tools is:
Steak for acceptance testing
Capybara for Browser simulation with drivers: Selenium & Akephalos
Machinist for stubs
Rspec for unit testing
A lot of the Rails developers out in the world use RSpec, and some of those use Shoulda. DHH, the lead developer of Rails prefers Test::Unit and Minitest. Thoughbot's Shoulda builds on both Test::Unit (And of course Minitest) and RSpec.
Ultimately, as a Rails Hotline volunteer, you'll get more community support out of RSpec and there are a ton of additional gems out there specifically for improving RSpec. With that said Minitest and Test::Unite are core to Ruby.
[OPINION] I tend to use RSpec if the software requires a "behavior flow" and Minitest if it requires pure unit functionality (mostly because Minitest's benchmark library is really simple).
I am a little confused by the sheer number of testing frameworks available for Ruby/ROR.
I have recently watched the Cucumber Railscasts and found them very interesting. So I started having a play and then struggled to see conceptually where I would put various tests.
It would seem to be quite possible to do everything that can be done in unit tests within Cucumber, so do I need to write unit tests or should I just write my feature definitions and concentrate on providing as good a coverage as I can get using that.
Should I create my Unit tests using Rspec or Test:Unit? When I'm testing Ajax functionality should I use Selenium or Watir?
There seem to be so many options here I am struggling to see which tools to use and where the boundaries are.
What are other peoples experiences of Cucumber and where to draw the line between writing Cucumber Integration tests and Test:Unit and/or Rspec based unit and functional tests. Is anyone aware of a good write-up on this subject suggesting where to draw lines between testing methods and the strengths and weaknesses of the various tool.
I appreciate that some of this is subjective but common approaches on how to attack this issue would be welcomed.
Use Cucumber at a high level to describe what a user should be able to see and do. Use RSpec, Test:Unit, Shoulda, etc. to write unit tests. Straight from the horse's mouth:
When you decide you want to add a new feature or fix a bug, start by writing a new feature or scenario that describes how the feature should work. Don’t write any code (yet).
...
This is when you start writing code. Start by writing a couple of lines of code to address the failure you got from Cucumber. Run cucumber again. Repeat and rinse until you’re happy with your feature. When you get down to nitty gritty details, drop down one abstraction level and use RSpec, or any Ruby testing framework, to write some specs/tests for your classes.
Cucumber is made to test your whole stack, together, as opposed to 'units'.
You need to decide where to draw the line, but a lot of under the hood stuff probably wouldn't be covered in a cucumber test. Say when signing up, I fill out a form, with my name, email, phone number, etc. A unit test might check to see that a new User will also create a new TelephoneNumber. From the user's perspective, they don't really care that it creates a new TelephoneNumber, they care that once they've signed up, they have an account and can see their telephone number.
I don't have too much experience writing cucumber tests (not yet), but I hope this helps a bit.
When a unit test fails (I mean a real unit test that tests a method in isolation using mocks), it tells you what "unit" has a problem. When an acceptance test fails, it tells you what "feature" has a problem, not where the problem is located.
When you create a rails app you get functional, interation, and unit tests by default. Cucumber is an additional test it is a way to also test the experience that your user will have. When they click the button labeled "go" they should see "success" rendered rather than a 404. This will make sure that nothing you do accidentally messes up the user experience, and that from the top to bottom your app works for the most common use cases you can think of. The other tests are meant to insure that nothing goes wrong, and that you have inspected ever model and method with a microscope. It may be possible to replicate unit tests in their entirety with cucumber, but it would be painful (and crazy slow to execute, especially if you're using selenium). The best time to write tests is when you're developing code, and the quickiest and easiest way to do that, is by using in the built-in rails testing and maybe some additional help such as shoulda, rspec, also i'm a huge fan of factory-girl. If you haven't already checked it out www.railscasts.com has a great intro to cucumber, and rspec, and factory-girl, ... I know this question has already been answered (it's no) but this is my two cents. Good luck coding!!
I have thought/struggled with this question much, and here's where I've arrived.
Cucumber first and Cucumber last. Cucumber will provide the primary test coverage.
The core model methods that do the real business work of the application should also be developed/covered with rspec/unit tests.
Why the unit tests as well?
1) The unit tests will test run much faster.
2) This core business logic may (will probably) be used in several ways beyond the current views (which Cucumber tests through). These methods ought to be hammered with all types of possible inputs and outputs directly calling the method in the test.
Why not unit test the rest of the models, and the controllers and views?
1) Cucumber already has it covered once.
2) I find that the views-controller-some-model-methods all work together to get things done (think everything exercised to log in); so I like to test them together.
I've been practicing Cucumber/RSpec for the past half year or so doing BDD.
First of all BDD is not easy to get into, it will feel unnatural at the beginning.
But once you get into it, there's no other way to do programming.
To answer your question. To test Javascript you'll need a javascript driver that can be used by Capybara which is used by Cucumber.
capybara-webkit is what all the cool kids use now these days
There's one important thing to note.
Integration tests are slow.
And unit tests are fast, but can be slow, so it's important you use the right database cleaner and you write good tests that have good isolation.
My test setup which I'm extremely happy with:
Guard for loading spork
Spork for faster tests
Cucumber for integration testing
capybara-webkit for javascript testing
RSpec for unit testing
I don't do view tests and controller tests as these are redundant in my opinion as good knowledge of XPATH willl have you writing remarkable tests that even cover your page layout and structure.
Personally I don't think that you should stop writing unit tests. As an acceptance testing tool, Cucumber should replace your functional tests and, if you writing, view tests.
Cucumber features are supposed to be simple and coupled to the real user's value a given feature has.
From my experience, Cucumber and Rspec have different appeal. Rspec appeals to me from a developer perspective because its easy to write and provides very quick feedback when something breaks. Cucumber does not appeal to me as a developer because it does not run as quickly as Rspec. However, Cucumber does appeal to me as a business stakeholder since it provides full coverage of entire features.
Do yourself a favor and keep writing unit tests.
Everyone is talking about TDD (BDD) in Rails (and not just Rails) development world today. It's easy to find plenty of good general information about it, there are quite a few tools you can use for this purpose and there are many (good) examples of how to use them.
Now, I'm already on the train. I like the idea (never did TDD before) and I decided to go for Shoulda (because it seems to me quite easy to understand). I read plenty of examples and have done some first tests. My problem is this: I'm just not sure if I'm doing it right. As I said, there are many tiny examples all over the net, but I haven't found any comprehensive guide covering overall workflow of TDD with Shoulda.
Here are my questions:
What is your way of practising TDD with Shoulda? (What is your usual workflow throughout one iteration?)
What other tools do you use (except of Shoulda itself)?
Any other tips?
I'm using RSpec instead of shoulda, (although I believe that shoulda can be used with RSpec) and the biggest improvement to my workflow has come from using Autotest and then RSpactor.
These tools automatically monitor your source tree for changes and execute the relevant tests if update either the test or implementation file. A pretty small change to the testing practice but the immediate feed back has increased my discipline in writing tests for every piece of functionality.
+1 for Autotest.
If you have a large test suite and only want to run the test you are working on (as opposed to the whole test suite) check out this monkey patch for Autotest.
As for other tools you should check out Factory Girl as a fixtures replacement.
Most of what I know about using Shoulda came by reading others test code. Check out some of Thoughtbot's projects on github, such as Clearance and Pacecar.
I totally agree that there is a significant gap in the rails BDD documentaion. Lots of little examples but no big ones giving an overall picture of how to do it. Even the books that have chapters on testing are guilt of this. The only full examples I have seen are peepcode's three part screencast on rspec and a video on confreaks about flex mock.
I'd love to hear about other good resources on BDD that go beyond minor examples of syntax and I'd really like see a book on the topic.
I am setting up a rails app and I just finished making some unit tests and my friend said that apparently fixtures are no longer cool and people are now using RSpec or shoulda. I was wondering what the actual benefits are to use these other toolkits. Any information at all is appreciated.
-fREW
I personally prefer Shoulda to RSpec. I find that Shoulda has less magic syntax than RSpec. My problem with RSpec is that yeah it's very readable when I read it aloud, but when I get to writing it, hmmmm, I'm never sure how a given assertion should be written. Prag Dave explains the problem better than me. He also likes Shoulda and has a few examples.
RSpec and similar frameworks are tooling designed to aid in Behavior Driven Development. They're not just a prettier way to write tests, though they do help with that.
There is plenty of information on BDD here: http://behaviour-driven.org/
And wikipedia: http://en.wikipedia.org/wiki/Behavior_Driven_Development
There are too many benefits to list here, so I'd recommend browsing that site a little.
There are two different things here:
The first thing is what framework to use for writing tests/specs. Here you can choose between Test::Unit, RSpec, Shoulda and so on. The choice is whether you want to do traditional TDD (Test::Unit) or whether you prefer the alternative ways of thinking about specifiying behaviour advocated by developers like David Chemlinsky (RSpec and to some extent Shoulda).
The second thing is how to handle test data. There are Rails fixtures and alternatives desgined with other goals such as the FixtureReplacement plugin. Before Rails 2.0 fixtures had significant and well-documented pratical problems. Many of the practical issues were fixed in Rails 2.0. However fixtures can lead to inadvertent test coupling and some of the alternatives try to avoid this.
RSpec is way more powerful because it's far easier to both read and write tests in. It's also very elegant when using mocks and stubs, a concept which will become extremely useful once you start using them in your tests. Try it in a simple test app (NON RAILS!) and you'll see how elegant your specs are versus the equivalent standard testing.
Check out Josh Susser's The Great Test Framework Dance-off for a comparison of the popular Ruby testing frameworks.
If you are building a large application and don't have a team that are all really good at writing decoupled code that can be well-tested with black box tests and are prepared to fully embrace using/debugging lots of mocks & stubs, don't go down the Factory road.
Wherever you read about how Awesome Factories Are you'll see a little caveat about how factories might not be feasible in a large application because they are a little slower than fixtures.
But "a little slower" is really orders of magnitude slower.
Factories are not significantly easier to code than fixtures that use labels for ids, so long as you keep the fixtures organized. And in some cases factories are harder to debug.
Just tonight I converted a single factory to fixtures, and the runtime of the test file that used it went from 65 seconds to 15 seconds, even though only about 15% of the tests in that test file use that factory.
If you use minitest you can run your tests in random order; this will quickly reveal any data coupling between tests. (not sure if rspec has the option to randomize test order)
Test::Unit is good for small applications. But there are a lot of benefits to use testing frameworks like Shoulda or RSpec, e. g. contexts!!
I don't see Shoulda and RSpec in an either-or-relation. I use Shoulda as a substitute for RSpec when it comes to single-assertion testing. I really like the Shoulda one-liners, but writing matchers is much easier in RSpec. So my advise is to use the different testing tools where they fit best.
You may use testing framework like Cucumber which is even more faster than RSpec..