This is more of a general question and some sort of best practice discussion.
How would one test a Rails application?
There are mantras like BDD and TDD and frameworks like RSpec and Cucumber but how much is enough and what is the best way to go?
Is it enough to use Cucumber as integration tests?
Are you guys writing additional unit tests, too?
So what and how is your Rails testing strategy?
Looking forward to good opinions...
How would one test a Rails application?
Thoroughly, aiming for Eighty percent coverage and no less!
The actual decision as to "how" is easy, but "how much" can be a difficult to answer.
I have a couple of small (almost but not quite legacy) projects with next to no tests, and the tests tend to be low-level unit tests of crucial components in the code. On occasion I wish they had more tests, but in general they are thrown together with quite small surface areas that make debugging and manual testing pretty straight forward.
In my day job, we're using rSpec, Cucumber and Selenium on a fairly large Rails project (10+ developers, several years in the making).
rSpec provides unit coverage for all our models, controllers, helpers and other classes.
Cucumber provides higher-level functional and integration tests.
Selenium is used to excercise javascript-intensive areas of our UI with javascript (running through Capybara and Cucumber). We also have a suite of regression tests in Selenium used by our Test and QA team (versions released to QA are green-lit through our CI build).
My "bare-minimum" setup these days would be unit-level coverage using rSpec on the critical and/or complex areas and a full suite of functional Cucumber tests on the critical paths through the application.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
Related
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
What are some general guidelines, procedures, or practices for testing large, complicated Rails applications? Are there certain pieces of an application(ie. user auth, search) that should be tested over others? When and how should tests be run in an ongoing development cycle?
I'm fairly new to Rails and large-scale development in general, but I have a decent grasp of the Rails framework through online guides and personal tinkering. I don't, however, know how to approach the task of testing an app that's already in development, with many models and controllers. My ultimate goal is to develop a testing harness for this application, but for now I'm trying to learn how Rails developers go about testing their (big) applications. Any resources or advice on related topics is greatly appreciated.
Every line below is debatable; this is just what works for me:
Let your unit tests be true unit tests, not functional tests. Isolate them with mocks and stubs, they make tests easier to write, read and change.
To give yourself the freedom to do true unit tests, supplement them with integration tests. At least visit each of your routes and check the response code.
The debugger gives you wings.
FactoryGirl, or something else that isn't fixtures.
Rcov / SimpleCov. Train yourself to feel naked when uncovered.
Rspec, Shoulda, Webrat. Take the time to make your specs scan in English. Read the book. Write lots of spec helpers and custom matchers.
Ixnay on Cucumber, too verbose, adds little.
Jenkins. Integrate with a coverage tool & flog. Never deploy without a green light.
I would like to write some automatic tests for my Rails 3 application.
I wonder how to start with that.
I've heard about Selenium/RSpec/Cucumber, and I guess there are many more options.
What are the advantages/disadvantages of these testing frameworks ? Which of them has the best documentation ? Which one is the most popular in the Ruby world ? And in the industry at all ?
I have the general knowledge of how to write tests. I just want to learn the appropriate testing framework(s) for testing Rails applications.
Please help me to decide with which testing framework to start.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
I wouldn't say there is any one best set of testing tools out there; and the community definitely has not decided on any being standard in any way.
I highly recommend The RSpec Book from the Pragmatic Programmers-- about half of the book is on testing with Rails with Cucumber, RSpec, and browser simulators like Selenium. It's a comprehensive overview of the different situations you will find yourself in with Rails and what tools you might want to use in each situation.
Selenium is a framework to automate testing of user interface. Selenium script launches a browser window, goes to a web-page and manipulates page elements.
RSpec and Cucumber are tools for what is called "behavior driven development (BDD)". BDD is a development process where you have description of a separate feature, and some script (can be a Selenium script) that checks this feature. RSpec and Cucumber basically connect these two, so that when you run a test script, you see the list of features written in plain language and their testing status.
Selenium scripts can be written in a number of languages, while Cucumber scripts use Ruby. Documentation is OK for all of them, some knowledge of programming language is required though.
Starting links:
Cucumber;
Selenium;
Testing rails applications
The easiest way to get started is using TestUnit. When you generate a model, view, controller using scaffolding it will automatically generate a testing folder for you containing a test environment setup file and tests, or you can add tests to existing models with a rails generator.
It is then a matter of reading the documentation on TestUnit, of which there is a fair amount.
For a next step, my personal preference is to use RSpec over Cucumber - I've found Cucumber hard to maintain, and whilst it is wonderful to have tests that are readable by your boss, I've found that most bosses don't really want to read or contribute to tests, and it makes for time-consuming development and is very 'bitty' - lots of small bits of text scattered in various files.
If you want to jump right in and begin using RSpec for Rails, then I'd recommend Rspec-Rails, which includes many helpers for asserting the correct behaviour of your application.
I could go on, but you've asked for 'where do I start' - there's two good places.
There are a bunch of frameworks that help you test your ruby/rails code... the great thing is the dynamic nature of Ruby gives you a lot of flexibility.
I like to use RSpec to test the internals of my code, and Cucumber to test the application's behavior. So generally, RSpec will test models and methods, and Cucumber tests user interaction through the browser.
I highly recommend this pragmatic programmers ebook on BDD using RSpec on Cucumber.
Railscast on getting started in cucumber (also called cukes)
Cucumber project
Rails project
I'm new to rails, and I read recently on the internet (so it must to be true) that the TDD library that comes with rails is incompatible with RSpec, but also I read that RSpec is the right tool to do test.
So, my question is, if this is truth, what is the right tool to make test with rails: the rails TDD or RSpec? Or are this 2 tools total different purposes?
Thank for the clarifications!!!
TDD means test driven development. It's a methodology, not a library.
Rails ships with Ruby's Test::Unit. It is easily replacable with other libraries such as Rspec, should you wish to do so.
There is no "right way" when it comes to which tool to use. It's all down to preference. I prefer Rspec personally...
RSpec is pure Ruby and a very good way to write tests for your code. It is easy to integrate using Gems and is straightforward to work with.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
For integration tests of my Rails web app I use Steak (something like Cucumber). The specs of Steak are in a folder named spec/acceptance. Are Steak/Cucumber now for integration or acceptance testing? I always thought that this is something different.
First, a note on the terminology: the term integration test is a bit vague in the TDD community. Depending whether you come from Java or Rails (with Test::Unit), you might understand different things by it. In Rails (with Test::Unit) integration tests are the tests that test your full stack, while functional tests would be the ones testing your controller. Most people in the Java community (at least by my observation) would think it is the other way around. I personally prefer to call the end-to-end tests acceptance tests, while tests that hit several layers of the system (but not everything) -- integration tests. All in all, that is pretty dependent on the culture your are in.
As for Cucumber and Steak -- both are frameworks that allow a development style known as Behavior-Driven Development (or BDD for short). The point is that you have two levels of tests:
End-to-end tests, which test your through the full stack -- they simulate a browser, go through your controllers and hit the database. Cucumber and Steak fit this niche.
Unit tests, which test a small bit of functionality in isolation (usually a single class, mocking its collaborators). This is where RSpec fits.
In BDD, you start with a failing end-to-end test (lovingly know as the "upper gear"), and then you start implementing functionality test-first with RSpec (the "lower gear"), until you get the end-to-end test passing. This way the end-to-end test is driving your unit tests, which in turn are driving your implementation. The main benefit is avoiding scope creep -- you don't end up implementing user-visible functionality that you don't need (since you don't write an end-to-end test for it).
If you want more information on this, I've heard that the Behavior Driven Development Wikipedia article is surprisingly good. Also, the RSpec book.
So, both Cucumber and Steak are frameworks that allow you to write tests in the "upper gear". The difference is in the style -- Cucumber has you writing your tests in natural language. This has several benefits.
Tests are readable by the business people -- while you cannot expect non-programmers to write them, they do a great job in communicating what you intend to do. You can write the feature (the Cucumber test first) and show it to the customer to get some feedback on whether this is what they actually want. I've found this very useful.
Cucumber features communicate intent better -- since you get to use the full power of the English language (or any, really), you can communicate why this feature is relevant and how the users accomplish their goal on a level that Ruby won't allow you to.
Cucumber helps discovering the ubiquitous language -- the domain includes a lot of terms that fly around in the conversations with the customers. Cucumber allows you to discover and capture them before you start implementing the feature. And it's all test-driven.
Cucumber features are somewhat higher-level, which makes the features (but not the step definitions) more independent of the interface. This way if the interface needs changing, you won't have to rework the features.
The downsides include that it is a bit tricky to learn how to apply it nicely and that you have to write a bit more (both features and step definitions). I've found that the second is not really a problem if you have been doing it for a while, since you get a body of reusable steps that allow you to write the next features faster.
Steak, on the other hand is simpler and it's Ruby. You loose all the benefits of using English, but you can write less and it will execute faster (somewhat).
In the bottom line, you use both to write the end-to-end tests that drive development.