I would like to write some automatic tests for my Rails 3 application.
I wonder how to start with that.
I've heard about Selenium/RSpec/Cucumber, and I guess there are many more options.
What are the advantages/disadvantages of these testing frameworks ? Which of them has the best documentation ? Which one is the most popular in the Ruby world ? And in the industry at all ?
I have the general knowledge of how to write tests. I just want to learn the appropriate testing framework(s) for testing Rails applications.
Please help me to decide with which testing framework to start.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
I wouldn't say there is any one best set of testing tools out there; and the community definitely has not decided on any being standard in any way.
I highly recommend The RSpec Book from the Pragmatic Programmers-- about half of the book is on testing with Rails with Cucumber, RSpec, and browser simulators like Selenium. It's a comprehensive overview of the different situations you will find yourself in with Rails and what tools you might want to use in each situation.
Selenium is a framework to automate testing of user interface. Selenium script launches a browser window, goes to a web-page and manipulates page elements.
RSpec and Cucumber are tools for what is called "behavior driven development (BDD)". BDD is a development process where you have description of a separate feature, and some script (can be a Selenium script) that checks this feature. RSpec and Cucumber basically connect these two, so that when you run a test script, you see the list of features written in plain language and their testing status.
Selenium scripts can be written in a number of languages, while Cucumber scripts use Ruby. Documentation is OK for all of them, some knowledge of programming language is required though.
Starting links:
Cucumber;
Selenium;
Testing rails applications
The easiest way to get started is using TestUnit. When you generate a model, view, controller using scaffolding it will automatically generate a testing folder for you containing a test environment setup file and tests, or you can add tests to existing models with a rails generator.
It is then a matter of reading the documentation on TestUnit, of which there is a fair amount.
For a next step, my personal preference is to use RSpec over Cucumber - I've found Cucumber hard to maintain, and whilst it is wonderful to have tests that are readable by your boss, I've found that most bosses don't really want to read or contribute to tests, and it makes for time-consuming development and is very 'bitty' - lots of small bits of text scattered in various files.
If you want to jump right in and begin using RSpec for Rails, then I'd recommend Rspec-Rails, which includes many helpers for asserting the correct behaviour of your application.
I could go on, but you've asked for 'where do I start' - there's two good places.
There are a bunch of frameworks that help you test your ruby/rails code... the great thing is the dynamic nature of Ruby gives you a lot of flexibility.
I like to use RSpec to test the internals of my code, and Cucumber to test the application's behavior. So generally, RSpec will test models and methods, and Cucumber tests user interaction through the browser.
I highly recommend this pragmatic programmers ebook on BDD using RSpec on Cucumber.
Railscast on getting started in cucumber (also called cukes)
Cucumber project
Rails project
Related
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
I have been watching Ryan Bates' RailsCasts for years now, and I'm also a customer of the pay portion of the site. I learned BDD from watching Cucumber episodes in the past.
Now I have learned about TestUnit, RSpec, Capybara, and MiniTest. I am beginning to get confused about what is what.
What is the difference between these 4 items? I know obviously Cucumber executes plain text features, and I guess that could be considered integration testing.
But now I also see that recent versions of Cucumber require MiniTest? Is Cucumber simply a DSL that sits on top of a testing framework?
I know also that RSpec has its own syntactical sugar for doing assertions, the "describe" blocks. And it appears that MiniTest also supports this syntax.
I know that Capybara is used for looking at the web page content that is generated, I think.
Here is my question:
If I am creating a new Rails 3.2 application, what combination of these testing programs should I use? What would be extra helpful is a list that explains how these gems and their associated processes complement each other where applicable, like:
Cucumber is a DSL for driving BDD
Cucumber is for integration tests and is based on creating user stories that are customer-readable
It implements its tests behind the scenes via MiniTest
MiniTest is a new testing framework that comes with Ruby 1.9 and is very fast.
MiniTest is also used for unit testing, such as testing controllers and models
It does not yet have as many features as RSpec
Cucumber also uses Capybara to access DOM elements from a javascript-enabled browser simulator such as Selenium
When you test in Rails, you have the ability to do the following kinds of tests: controllers, views, models, and integration (MVC together)
Some people just do integration and model testing, because they feel that integration testing handles enough of the controller and view testing itself, and anything too complex can simply be moved up to the model
Thank you so much for any help you can offer to clear these ideas up for me.
Okay, so let me try and explain, based on my own experience
Cucumber, is a ATDD (or BDD) tool that lets you write tests in a business-oriented domain language. It's primary use is as a conversation tool with you product owners. Instead of writing detailed requirements, you express those requirements as examples of the system under test. Each cucumber test, in effect, becomes a business requirement that must be satisfied.
The Cucumber tool itself, translates these plain-text statements into a little module for you to execute ruby code in. What Ruby Library you use inside of the step definitions depends entirely on the project you are working on.
The safest way to describe Cucumber is that it's a testing framework with an emphasis on communication between IT and Business Partners, which is why it's becoming so popular.
Rspec and Minitest are other frameworks with other strengths, but lack this business readability factor, in that they are mostly code and not nearly as readable to a non-technical person. That isn't necessarily a bad thing, especially if your product owner is a little more hands off.
How does this tie in with something like Capybara? Capybara is an integration test automation library that drives a headless browser on the Rack::Test framework for very fast tests, and has a highly readable DSL. The only downside is Rack::Test doesn't support javascript, so it provides a way to fail over to Selenium if you a running a javascript test. Rspec and Cucumber both have mechanisms for triggering this failover.
There are lots of other Ruby automation libraries. For instance, if you are doing testing against a non-rails Web App, your cucumber scenarios might be defined with Watir-Webdriver which will drive a full web browser like Capybara does when in Javascript mode. The main difference being that WW has a lot more robust set of selctors than Capybara does, so it's a bit easier to write especially if your code isn't super clean (Capybara only supports selecting by ID, Value and Text, whereas WW supports selecting by almost anything)
So chances are you are going to want to use RSpec or Minitest for your unit testing )or the default Test::Unit), and Cucumber for Integration Testing IF you have an interested product owner or a need for a common language for test scenarios. If you do not, you can write your integration tests as Rspec examples or the minitest equivalent without losing much.
What are some general guidelines, procedures, or practices for testing large, complicated Rails applications? Are there certain pieces of an application(ie. user auth, search) that should be tested over others? When and how should tests be run in an ongoing development cycle?
I'm fairly new to Rails and large-scale development in general, but I have a decent grasp of the Rails framework through online guides and personal tinkering. I don't, however, know how to approach the task of testing an app that's already in development, with many models and controllers. My ultimate goal is to develop a testing harness for this application, but for now I'm trying to learn how Rails developers go about testing their (big) applications. Any resources or advice on related topics is greatly appreciated.
Every line below is debatable; this is just what works for me:
Let your unit tests be true unit tests, not functional tests. Isolate them with mocks and stubs, they make tests easier to write, read and change.
To give yourself the freedom to do true unit tests, supplement them with integration tests. At least visit each of your routes and check the response code.
The debugger gives you wings.
FactoryGirl, or something else that isn't fixtures.
Rcov / SimpleCov. Train yourself to feel naked when uncovered.
Rspec, Shoulda, Webrat. Take the time to make your specs scan in English. Read the book. Write lots of spec helpers and custom matchers.
Ixnay on Cucumber, too verbose, adds little.
Jenkins. Integrate with a coverage tool & flog. Never deploy without a green light.
I'm new to rails, and I read recently on the internet (so it must to be true) that the TDD library that comes with rails is incompatible with RSpec, but also I read that RSpec is the right tool to do test.
So, my question is, if this is truth, what is the right tool to make test with rails: the rails TDD or RSpec? Or are this 2 tools total different purposes?
Thank for the clarifications!!!
TDD means test driven development. It's a methodology, not a library.
Rails ships with Ruby's Test::Unit. It is easily replacable with other libraries such as Rspec, should you wish to do so.
There is no "right way" when it comes to which tool to use. It's all down to preference. I prefer Rspec personally...
RSpec is pure Ruby and a very good way to write tests for your code. It is easy to integrate using Gems and is straightforward to work with.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
This is more of a general question and some sort of best practice discussion.
How would one test a Rails application?
There are mantras like BDD and TDD and frameworks like RSpec and Cucumber but how much is enough and what is the best way to go?
Is it enough to use Cucumber as integration tests?
Are you guys writing additional unit tests, too?
So what and how is your Rails testing strategy?
Looking forward to good opinions...
How would one test a Rails application?
Thoroughly, aiming for Eighty percent coverage and no less!
The actual decision as to "how" is easy, but "how much" can be a difficult to answer.
I have a couple of small (almost but not quite legacy) projects with next to no tests, and the tests tend to be low-level unit tests of crucial components in the code. On occasion I wish they had more tests, but in general they are thrown together with quite small surface areas that make debugging and manual testing pretty straight forward.
In my day job, we're using rSpec, Cucumber and Selenium on a fairly large Rails project (10+ developers, several years in the making).
rSpec provides unit coverage for all our models, controllers, helpers and other classes.
Cucumber provides higher-level functional and integration tests.
Selenium is used to excercise javascript-intensive areas of our UI with javascript (running through Capybara and Cucumber). We also have a suite of regression tests in Selenium used by our Test and QA team (versions released to QA are green-lit through our CI build).
My "bare-minimum" setup these days would be unit-level coverage using rSpec on the critical and/or complex areas and a full suite of functional Cucumber tests on the critical paths through the application.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?