Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I am newbie to rails testing I google and found few testing framework like
1. rspec
2. cucumber
3. capybara
4. factorygirl
5. jasmin
and default rails has its own testing tools.
I need to know what to use and when to use for
functional, unit and
integration testing
And how we can automate the testing so before deploy it will run all our test cases?
Definitely read the Rails guides on testing best practices; there's a ton of books, blog posts, and Youtube videos out there on how to test. But part of your question has been a particular pain point for me so I'll give a brief answer here too: Which of these complex tools are worth using, and for what kinds of tests?
The Rails community is huge on tests, but personally I think that it's possible to go too far, focus too much energy on your test suite at the expense of actually writing code. Some high-profile folks have even advocated that developers try out not writing tests once or twice, just so that we can get firsthand experience with the effect it has on our code quality, number of bugs, etc. (Spoiler: quality goes down, bugs go up, customer satisfaction goes down. But it's different for you to learn this firsthand than for me to just tell you.)
So my personal testing habits for Rails apps are:
I prefer Minitest over Rspec. The former encourages plain, straightforward unit tests; the latter encourages verbose specs and a larger testing infrastructure. Minitest (or Rspec) is more or less "global" to your test suite, in that all of your specs will be written using that same framework. Which is good because there's less tools to manage and less to learn.
FactoryGirl is useful in quickly setting up a specific database scenario for a given test. I use it in model, controller, and integration tests.
Capybara is a framework for writing highly readable end-user acceptance / integration tests. It's appealing but it's easy to overuse, leading to verbose tests that get brittle and hard to maintain. I write (thorough) controller specs first, then write judicious integration tests afterwards.
Cucumber is a layer on top of Capybara (for integration tests) or for any other kind of test; it lets you write out your tests in a human-readable format and forces you to think in terms of acceptance tests. But it gets easy to become too verbose, and on the tiny-team projects I've worked on so far, I haven't yet found it useful. I avoid it.
Jasmine is like Rspec for Javascript: a language for writing readable and well-organized specs of your JS code. This may be useful if you have tons of custom JS code / logic, but I haven't yet used it or envied it.
Model unit tests: I write at least one unit test for each custom method.
Controller unit tests: I write at least one unit test for each logical path available to each controller action, and enable rendering views so the template generation gets exercised here too.
Integration / acceptance tests: I go light on these, only adding very minimal paths to cover the essential use cases of the website. I find most bugs aren't caught at this level; the controller tests catch a lot more, since they render every single view template and execute every bit of model logic that the user can possibly trigger.
Related
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
For integration tests of my Rails web app I use Steak (something like Cucumber). The specs of Steak are in a folder named spec/acceptance. Are Steak/Cucumber now for integration or acceptance testing? I always thought that this is something different.
First, a note on the terminology: the term integration test is a bit vague in the TDD community. Depending whether you come from Java or Rails (with Test::Unit), you might understand different things by it. In Rails (with Test::Unit) integration tests are the tests that test your full stack, while functional tests would be the ones testing your controller. Most people in the Java community (at least by my observation) would think it is the other way around. I personally prefer to call the end-to-end tests acceptance tests, while tests that hit several layers of the system (but not everything) -- integration tests. All in all, that is pretty dependent on the culture your are in.
As for Cucumber and Steak -- both are frameworks that allow a development style known as Behavior-Driven Development (or BDD for short). The point is that you have two levels of tests:
End-to-end tests, which test your through the full stack -- they simulate a browser, go through your controllers and hit the database. Cucumber and Steak fit this niche.
Unit tests, which test a small bit of functionality in isolation (usually a single class, mocking its collaborators). This is where RSpec fits.
In BDD, you start with a failing end-to-end test (lovingly know as the "upper gear"), and then you start implementing functionality test-first with RSpec (the "lower gear"), until you get the end-to-end test passing. This way the end-to-end test is driving your unit tests, which in turn are driving your implementation. The main benefit is avoiding scope creep -- you don't end up implementing user-visible functionality that you don't need (since you don't write an end-to-end test for it).
If you want more information on this, I've heard that the Behavior Driven Development Wikipedia article is surprisingly good. Also, the RSpec book.
So, both Cucumber and Steak are frameworks that allow you to write tests in the "upper gear". The difference is in the style -- Cucumber has you writing your tests in natural language. This has several benefits.
Tests are readable by the business people -- while you cannot expect non-programmers to write them, they do a great job in communicating what you intend to do. You can write the feature (the Cucumber test first) and show it to the customer to get some feedback on whether this is what they actually want. I've found this very useful.
Cucumber features communicate intent better -- since you get to use the full power of the English language (or any, really), you can communicate why this feature is relevant and how the users accomplish their goal on a level that Ruby won't allow you to.
Cucumber helps discovering the ubiquitous language -- the domain includes a lot of terms that fly around in the conversations with the customers. Cucumber allows you to discover and capture them before you start implementing the feature. And it's all test-driven.
Cucumber features are somewhat higher-level, which makes the features (but not the step definitions) more independent of the interface. This way if the interface needs changing, you won't have to rework the features.
The downsides include that it is a bit tricky to learn how to apply it nicely and that you have to write a bit more (both features and step definitions). I've found that the second is not really a problem if you have been doing it for a while, since you get a body of reusable steps that allow you to write the next features faster.
Steak, on the other hand is simpler and it's Ruby. You loose all the benefits of using English, but you can write less and it will execute faster (somewhat).
In the bottom line, you use both to write the end-to-end tests that drive development.
This is more of a general question and some sort of best practice discussion.
How would one test a Rails application?
There are mantras like BDD and TDD and frameworks like RSpec and Cucumber but how much is enough and what is the best way to go?
Is it enough to use Cucumber as integration tests?
Are you guys writing additional unit tests, too?
So what and how is your Rails testing strategy?
Looking forward to good opinions...
How would one test a Rails application?
Thoroughly, aiming for Eighty percent coverage and no less!
The actual decision as to "how" is easy, but "how much" can be a difficult to answer.
I have a couple of small (almost but not quite legacy) projects with next to no tests, and the tests tend to be low-level unit tests of crucial components in the code. On occasion I wish they had more tests, but in general they are thrown together with quite small surface areas that make debugging and manual testing pretty straight forward.
In my day job, we're using rSpec, Cucumber and Selenium on a fairly large Rails project (10+ developers, several years in the making).
rSpec provides unit coverage for all our models, controllers, helpers and other classes.
Cucumber provides higher-level functional and integration tests.
Selenium is used to excercise javascript-intensive areas of our UI with javascript (running through Capybara and Cucumber). We also have a suite of regression tests in Selenium used by our Test and QA team (versions released to QA are green-lit through our CI build).
My "bare-minimum" setup these days would be unit-level coverage using rSpec on the critical and/or complex areas and a full suite of functional Cucumber tests on the critical paths through the application.
There are many tools for testing rails and other webapps from many different aspects. But if you are new to testing I highly recommend you start with learning Rails own testing framework before start using other tools.
Learning, and later mastering, one testing framework makes it easier in the future to understand pros/cons with other framework and make them work in unison.
You could start with testing the following things:
Unit Testing your Models
Functional Tests for Your Controllers
Learning about Fixtures and how to load test data
I have seen many failed testing efforts, but I never saw them fail because they choose the wrong tool/framework. They fail because they don't know how to master the tools they use, and learn enough about the basics about testing.
Read more about Rails testing here.
http://guides.rubyonrails.org/testing.html
Manual Exploratory Testing
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Read more about mixing Automated and Manual Testing in another of my answers here:
What test methods do you use for developing websites?
At present, my development process flows like this:
I describe the expected behaviour as an integration test using using WebRat
I write the Ruby on Rails code to provide that behaviour, so passing the test
I refactor, ensuring the tests still pass at the end of the process
I write the next integration test
It seems to me that by definition, my integration tests are testing every model, controller and view that I can create. In reality, am I missing anything by not writing unit tests too?
I'm actually pretty sympathetic to your point of view here. I love Cucumber and I love RSpec -- and I use them both, but not always on the same code. For instance, I rarely write RSpec examples for Rails controllers these days, and I almost never write view specs. Most of my controllers are very similar and don't deviate much from the "stock" controller pattern -- which is already well-tested by Rails's own unit tests. Verifying the same behavior again doesn't gain much for the time it takes and the hassle of mocking all the models. With Cucumber at an integration level I can skip that mocking and get the essential verification I'm looking for. View testing is handled much more transparently in Cucumber most of the time as well. (Then I should see "foo" and so forth.)
But that isn't to say I don't use RSpec extensively in Rails. I use it for the places where my logic lives: models, controller filters, and view helpers. I also have a couple of projects that are almost all business logic, e.g. libraries or API adapters against complex third-party interfaces. For those I usually find myself using RSpec exclusively and skipping Cucumber.
As a heuristic, I'd suggest that you should strongly consider writing a unit test any time any of the following questions can be answered "Yes":
Is the code I'm writing more than trivially complicated?
Does this code exist primarily to give answers to other code?
Is this existing code that I'm refactoring (that doesn't already have a unit test)?
Have I found a bug in this code? (If so, write a unit test before fixing it so it never sneaks in again.)
Do I have to think for more than ten seconds about the most elegant way to implement this code?
Is my Spidey Sense tingling?
If none of the above is true, then maybe you can get away with just doing integration testing. Again, there are a lot of cases where that's reasonable. But if you do run into problems later, be prepared to pay the price -- and that price should include writing unit tests at any moment if they seem called for.
Integration tests are useful to verify that different parts of code are well integrated. They may involve all layers and cover all code but... when an integration test fails, will it tell you where the bug is located? I may be wrong but I don't think so. This will just tell you that there is a problem somewhere. On the other hand, when a real unit tests (written in isolation using mocks or stubs) fails, you know exactly in which unit the problem is located (this is actually the purpose of unit testing, verifying that a unit implements the expected behavior). In other words, unit tests and integration tests are both useful but they have different purposes.
Do you have any rake tasks? Custom capistrano code? Cron methods? An API? Monkeypatches? How about flex or iPhone app integration? A job runner?
In a typical Rails application, there's lots of code that isn't exercised by the HTML UI. So no, unless your app is incredibly simple, webrat tests won't be sufficient.
I am a little confused by the sheer number of testing frameworks available for Ruby/ROR.
I have recently watched the Cucumber Railscasts and found them very interesting. So I started having a play and then struggled to see conceptually where I would put various tests.
It would seem to be quite possible to do everything that can be done in unit tests within Cucumber, so do I need to write unit tests or should I just write my feature definitions and concentrate on providing as good a coverage as I can get using that.
Should I create my Unit tests using Rspec or Test:Unit? When I'm testing Ajax functionality should I use Selenium or Watir?
There seem to be so many options here I am struggling to see which tools to use and where the boundaries are.
What are other peoples experiences of Cucumber and where to draw the line between writing Cucumber Integration tests and Test:Unit and/or Rspec based unit and functional tests. Is anyone aware of a good write-up on this subject suggesting where to draw lines between testing methods and the strengths and weaknesses of the various tool.
I appreciate that some of this is subjective but common approaches on how to attack this issue would be welcomed.
Use Cucumber at a high level to describe what a user should be able to see and do. Use RSpec, Test:Unit, Shoulda, etc. to write unit tests. Straight from the horse's mouth:
When you decide you want to add a new feature or fix a bug, start by writing a new feature or scenario that describes how the feature should work. Don’t write any code (yet).
...
This is when you start writing code. Start by writing a couple of lines of code to address the failure you got from Cucumber. Run cucumber again. Repeat and rinse until you’re happy with your feature. When you get down to nitty gritty details, drop down one abstraction level and use RSpec, or any Ruby testing framework, to write some specs/tests for your classes.
Cucumber is made to test your whole stack, together, as opposed to 'units'.
You need to decide where to draw the line, but a lot of under the hood stuff probably wouldn't be covered in a cucumber test. Say when signing up, I fill out a form, with my name, email, phone number, etc. A unit test might check to see that a new User will also create a new TelephoneNumber. From the user's perspective, they don't really care that it creates a new TelephoneNumber, they care that once they've signed up, they have an account and can see their telephone number.
I don't have too much experience writing cucumber tests (not yet), but I hope this helps a bit.
When a unit test fails (I mean a real unit test that tests a method in isolation using mocks), it tells you what "unit" has a problem. When an acceptance test fails, it tells you what "feature" has a problem, not where the problem is located.
When you create a rails app you get functional, interation, and unit tests by default. Cucumber is an additional test it is a way to also test the experience that your user will have. When they click the button labeled "go" they should see "success" rendered rather than a 404. This will make sure that nothing you do accidentally messes up the user experience, and that from the top to bottom your app works for the most common use cases you can think of. The other tests are meant to insure that nothing goes wrong, and that you have inspected ever model and method with a microscope. It may be possible to replicate unit tests in their entirety with cucumber, but it would be painful (and crazy slow to execute, especially if you're using selenium). The best time to write tests is when you're developing code, and the quickiest and easiest way to do that, is by using in the built-in rails testing and maybe some additional help such as shoulda, rspec, also i'm a huge fan of factory-girl. If you haven't already checked it out www.railscasts.com has a great intro to cucumber, and rspec, and factory-girl, ... I know this question has already been answered (it's no) but this is my two cents. Good luck coding!!
I have thought/struggled with this question much, and here's where I've arrived.
Cucumber first and Cucumber last. Cucumber will provide the primary test coverage.
The core model methods that do the real business work of the application should also be developed/covered with rspec/unit tests.
Why the unit tests as well?
1) The unit tests will test run much faster.
2) This core business logic may (will probably) be used in several ways beyond the current views (which Cucumber tests through). These methods ought to be hammered with all types of possible inputs and outputs directly calling the method in the test.
Why not unit test the rest of the models, and the controllers and views?
1) Cucumber already has it covered once.
2) I find that the views-controller-some-model-methods all work together to get things done (think everything exercised to log in); so I like to test them together.
I've been practicing Cucumber/RSpec for the past half year or so doing BDD.
First of all BDD is not easy to get into, it will feel unnatural at the beginning.
But once you get into it, there's no other way to do programming.
To answer your question. To test Javascript you'll need a javascript driver that can be used by Capybara which is used by Cucumber.
capybara-webkit is what all the cool kids use now these days
There's one important thing to note.
Integration tests are slow.
And unit tests are fast, but can be slow, so it's important you use the right database cleaner and you write good tests that have good isolation.
My test setup which I'm extremely happy with:
Guard for loading spork
Spork for faster tests
Cucumber for integration testing
capybara-webkit for javascript testing
RSpec for unit testing
I don't do view tests and controller tests as these are redundant in my opinion as good knowledge of XPATH willl have you writing remarkable tests that even cover your page layout and structure.
Personally I don't think that you should stop writing unit tests. As an acceptance testing tool, Cucumber should replace your functional tests and, if you writing, view tests.
Cucumber features are supposed to be simple and coupled to the real user's value a given feature has.
From my experience, Cucumber and Rspec have different appeal. Rspec appeals to me from a developer perspective because its easy to write and provides very quick feedback when something breaks. Cucumber does not appeal to me as a developer because it does not run as quickly as Rspec. However, Cucumber does appeal to me as a business stakeholder since it provides full coverage of entire features.
Do yourself a favor and keep writing unit tests.