I learned Rails to create a website and the basic version is up and running. Unfortunately, I only wrote a couple of tests for my code. What should I do now to get test coverage for my code? It will be difficult to go back now and write tests for all my previous code. Would it make sense to use a recording tool like Selenium to visit the sites and record tests? Is there a specific recording tool for Rails?
(In short, how can one catch up to get test coverage on code that doesn't have enough tests?)
First, quantitatively measure your test coverage with a tool like rcov.
You're right that integration tests will provide more coverage, so they are the right choice for your goal. However, they provide less meaningful feedback when they fail, so you'll want to augment or replace them with more isolated tests in the future.
I expect that recording tests will result in a test whose source is hard for humans to read. So, you might want to consider capybara-webkit instead (but don't use capybara 2.1 yet).
Finally, because your current testbase is small, you have an opportunity to consider rspec or shoulda-context.
Truth is, you're going to just have to go back and right all the tests.
First of, no kind of "recording" tool would help with the unit and controller tests.
Second, if you did use something to "record" feature/integration tests, what exactly would you be testing? If you aren't specific to what functionally you expect to be there, how can you be sure your app is even doing now what it's supposed to be doing?
Related
I'm really getting frustrated with learning how to properly develop software using TDD. It seems that everyone does it differently and in a different order. At this point, I'd just like to know what are all the considerations? This much is what I've come up with: I should use rspec, and capybara. With that said, what are all the different types of test I need to write, to have a well built and tested application. I'm looking for a list that comprises the area of my application being tested, the framework needed to test it, and any dependencies.
For example, it seems that people advise to start by unit testing your models, but when I watch tutorials on TDD it seems like they only write integration test. Am I missing something?
Well, the theme "how do you TDD" is as much out there in the open as the theme "how do you properly test?". In Ruby, and more specifically in Rails, rspec should be the tool to start with, but not be done with. RSpec allows you to write Unit Tests for your components, to test them separately. In the Rails context, that means:
test your models
test your controllers
test your views
test your helpers
test your routes
It is a very good tool not exactly rails-bound, it is also used to test other frameworks.
After you're done with RSpec, you should jump to cucumber. Cucumber (http://cukes.info/) is the most used tool (again, for the Rails environment) to write integration tests. You can then integrate capybara on cucumber.
After you're done with cucumber, you'll be done with having tested your application backend and (part of) its HTML output. That's when you should also test your javascript code. How to do that? First, you'll have to Unit test it. Jasmine (http://pivotal.github.com/jasmine/) is one of the tools you might use for the job.
Then you'll have to test its integration in your structure. How to do that? You'll come back to cucumber and integrate selenium (http://seleniumhq.org/) with your cucumber framework, and you'll be able to test your integration "live" in the browser, having access to your javascript magic and testing it on the spot.
So, after you're done with these steps, you'll have covered most of the necessary steps to have a well-integrated test environment. Are we done? Not really. You should also set a coverage tool (one available: https://github.com/colszowka/simplecov) to check if your code is being really well tested and no loose ends are left.
After you're done with these morose steps, you should also do one last thing, in case you are not developing it all alone and the team is big enough to make it still unmanageable by itself: you'll set a test server, which will do nothing other than run all the previous steps regularly and deliver notifications about its results.
So, all of this sets a good TDD environment for the interested developer. I only named the most used frameworks in the ruby/rails community for the different types of testing, but that doesn't mean there aren't other frameworks as or more suitable for your job. It still doesn't teach you how to test properly. For that there's more theory involved, and a lot of subdebates.
In case I forgot something, please write it in a comment below.
Besides that, you should approach how you test properly. Namely, are you going for the declarative or imperative approach?
Start simple and add more tools and techniques as you need them. There are many way to TDD an app because every app is different. One way to do that is to start with an end-to-end test with Rspec and Capybara (or Cucumber and Capybara) and then add more fine-grained tests as you need them.
You know you need more fine-grained tests when it takes more than a few minutes to make a Capybara test pass.
Also, if the domain of your application is non-trivial it might be more fruitful for you to start testing the domain first.
It depends! Try different approaches and see what works for you.
End-to-end development of real-world applications with TDD is an underdocumented activity indeed. It's true that you'll mostly find schoolbook examples, katas and theoretical articles out there. However, a few books take a more comprehensive and practical approach to TDD - GOOS for instance (highly recommended), and, to a lesser extent, Beck's Test Driven Development by Example, although they don't address RoR specifically.
The approach described in GOOS starts with writing end-to-end acceptance tests (integration tests, which may amount to RSpec tests in your case) but within that loop, you code as many TDD unit tests as you need to design your lower-level objects. When writing those you can basically start where you want -from the outer layers, the inner layers or just the parts of your application that are most convenient to you. As long as you mock out any dependency, they'll remain unit tests anyway.
I also have the same question when I started learning rails, there're so many tools or methods to make the test better but after spending to much time on that, I finally realized that you could simply forget the rule that you must do something or not, test something that you think it might have problem first, then somewhere else. Well ,it needs time.
that's just my point of view.
I'm new to unit/func testing in Rails 3. So I'm starting now, better late than never.
I have a method in /lib/mailingjob.rb called find_reply(body)
Right now I'm using cucumber to test this but given that this is all backend, no web interface to this, should I not be using cucumber?
I'm finding when to use RPSEC vs cucumber confusing.
Thanks
In my opinion, you need to use both. Rspec is very good for unit testing, that is testing models, controllers, views. On the other hand, cucumber is a very nice tool to check full scenarios like a user logs in, clicks a link and he is supposed to view this.
I highly advise that you take a look at the cucumber railscast from railscasts.com. Also, make sure that you use webrat and maybe something to auto load your specs like watchr(which i prefer).
I'd advise you to never use Cucumber unless you are specifically employed as a business analyst. That makes your life much easier: all your tests will be in Rspec.
Cucumber takes a toll on your productivity, and your sanity. Running parallel test environments, adding layers of abstraction and breaking your editor with its odd syntax make for lost time. I've written about this extensively on my blog:
Why Bother With Cucumber Testing?
If I didn't already have a Stack Overflow account, I would have made one JUST to write this post. I urge you to re-read #Jack Kinsella's answer
I got in to BDD about 3 months ago, fell in love with it, and quickly started using Cucumber for everything. I forced it in to every project, no matter what
Recently, I've had to learn a yet another language, and my first step was to get the testing environment set up. I came across these 3 articles:
http://robots.thoughtbot.com/post/25650434584/writing-better-cucumber-scenarios-or-why-were
http://robots.thoughtbot.com/post/25653308407/describe-the-users-perspective-ddd-acceptance
http://37signals.com/svn/posts/3159-testing-like-the-tsa
Honestly, their content isn't that important; what's important is that they hinted at an overall idea. I began to suspect that I'd been using Cucumber wrong all along, and I went hunting for answers. This morning I found your SO question, and #Jack Kinsella's blog post
#Jack comes right out and says the idea I believe those articles were tentatively circling. He also gives me the language I was looking for. I now consider his article to be the final word on the subject :)
According to him, what we've ACTUALLY been using Cucumber for is "integration testing". I never understood what that meant before
Unit testing:
How the code works internally. Math.Add(1,1) should be 2, but the website user doesn't care. Just give me a web page!
-> Use RSpec or an equivalent
Integration testing:
How different branches of the code work together to make the website. I type my name and click Log In, and should be taken to the Home page
-> ALSO use RSpec!
(Inside of RSpec, add whatever you need to handle multiple-technologies-touching-eachother. For the code-to-webBrowsers example: Capybara, Watir, etc)
Acceptance testing:
Many of us don't need it. Someone signed a contract with you saying "I'm going to write 'I can add subpages to my micro-site.' in a text file". You write me code that makes it turn green
-> Use Cucumber. Only if you have to do this kind of testing. Which you almost certainly don't
What an elegant solution. No 2nd testing environment. No steps directory and swarm of extra files!
I had a moment of "But I love how Cucumber seperates English from the code". Regular bdd does that too. "rspec --format nested" or Jasmine test results
#Jack is right. Cucumber isn't adding anything; not the way we've been using it. And it's costing a lot. To quote him:
Why not admit to yourself that you don’t do acceptance testing and that you do not need it in your projects? Swap Cucumber for pure integration tests using Capybara, and you’ll be surprised by how much more productive you can be
To complete SpyrosP's answer, there is an awesome blog post from Sarah Mei which describes a scenario where you use both Rspec and Cucumber. It is called oustide-in Behavior Driven Development and you can find it here.
However, I think most of Jack's problem can be solved by a good cucumber editor. So the problem is not Cucumber. The problem is that there wasn't a capable editor when he wrote this two years ago. But today RubyMine has full cucumber editing capability. While it is not free, it can solve most of Jack's problem: Auto completing, pointing out errors, refactoring. So you probably want to go back and check the points he raised, and see if they are still valid today.
My take so this issue many years later is very simple: If you do not know what to use now, just start with RSpec, because it is the central stone for all your testing that you will always need.
With time you will discover that two big testing needs:
Is my code still working?
Can my users still perform the actions already implemented?
As a developer I tend to think that (1) is for every single code change and (2) when I finish a feature and/or deploy.
Having two test suites is nice because during a deployment, you can sit and run a very generic test suite: Are my user features still working? Can I finally deploy this?
Indeed, it should be so generic that if something breaks in (2) you need to do some heavy research (one row missing in your tables) while a fail in (1) should pin-point the exact class with the problem.
Normally I find myself in the situation where I want separate integration+acceptance testing without the Cucumber burden. First I use RSpec tags to filter out these tests. Whenever I need more I like going for Steak+Capybara since it enables Cucumber like objectives on top of something I already know quite well.
Definitely use both. Use cucumber for user experience tests(views), use rspec for testing internals(models, etc.). You may write controller rspec tests but I think they are not necessary(try to move the logic to models).
Whatever your system is, I strongly advise you to use Cucumber because scenarios will let you and your stakeholders know the functions of your system in a clear written fashion.
You will also have your documentation ready and you'll always know where you are and where you are heading. You'll also relate scenarios with support issues in the future.
Cuke them right by the way don't use variables in scenarios, they should be understandable.
And if you want them to run fast as rspecs use poltergeist(phantomjs).
I've been tempted not use cucumber because in many cases it's not simple as rspec. However every time I understood that I needed scenarios to take full control on the project.
I am starting a new project for a client today.
I have done some rails projects before but never bothered writing tests for them.
I'd like to change that starting with this new project.
I am aware there are several testing tools, but am a bit confused as to which I should be using.
I heard of RSpec, Mocha, Webrat, and Cucumber.
Please keep in mind I never really wrote any regular tests, so my knowledge of testing in general is quite limited.
How would you suggest I get started?
Thanks!
Thank you for all the responses! I posted a related question that may interest those who view this question in the future. You can find it here.
I prefer using Rspec and Cucumber in conjunction. I also prefer Capybara over Webrat.
Rspec is wonderful for it's rails integration and structure. It's technically behavior testing, but in practice it ends up feeling like unit testing. It also allows you to group and define "specs", or rspec tests, in any structure you like with 'describe' blocks.
Cucumber is higher level. It is what you will translate your user stories into. For instance, you may decide that your users should be able to change their password, so you'll write something like this:
Scenario: Change password
Given I am logged in
And I am on the change password page
When I fill in "p4ssw0rd" for "old_password"
And I fill in "newp4ssw0rd$$" for "new_password"
Then my password should be "newp4ssw0rd$$"
If you are working with a QA dept, you can show your tests to them, and they'll instantly know the status of any given feature, and how the feature is supposed to work. Additionally, if you have enterprising testers, they can write tests themselves, even if they are not programmers in general.
They all serve different function and if you really care about the quality of your software, you would use them all.
RSpec is more for unit testing. Unit Testing is testing the smallest part, usually your models methods.
WebRat is more for functionality testing as end user would test it from a browser. You are actually testing the interaction and test whether it returns the correct view/data.
Cucumber is more like for behaviour driven testing or end-to-end testing. It is more to test the business logic according to many different scenarios or data.
Of course you could use all of them together in conjunction. For example, I can use Cucumber to do end-to-end testing and it would involve functional testing using WebRat and Rspec as well. Hopefully this will get the idea for you to start off.
You can have a look at the most popular testing frameworks here http://ruby-toolbox.com/
Cheers
I am a little confused by the sheer number of testing frameworks available for Ruby/ROR.
I have recently watched the Cucumber Railscasts and found them very interesting. So I started having a play and then struggled to see conceptually where I would put various tests.
It would seem to be quite possible to do everything that can be done in unit tests within Cucumber, so do I need to write unit tests or should I just write my feature definitions and concentrate on providing as good a coverage as I can get using that.
Should I create my Unit tests using Rspec or Test:Unit? When I'm testing Ajax functionality should I use Selenium or Watir?
There seem to be so many options here I am struggling to see which tools to use and where the boundaries are.
What are other peoples experiences of Cucumber and where to draw the line between writing Cucumber Integration tests and Test:Unit and/or Rspec based unit and functional tests. Is anyone aware of a good write-up on this subject suggesting where to draw lines between testing methods and the strengths and weaknesses of the various tool.
I appreciate that some of this is subjective but common approaches on how to attack this issue would be welcomed.
Use Cucumber at a high level to describe what a user should be able to see and do. Use RSpec, Test:Unit, Shoulda, etc. to write unit tests. Straight from the horse's mouth:
When you decide you want to add a new feature or fix a bug, start by writing a new feature or scenario that describes how the feature should work. Don’t write any code (yet).
...
This is when you start writing code. Start by writing a couple of lines of code to address the failure you got from Cucumber. Run cucumber again. Repeat and rinse until you’re happy with your feature. When you get down to nitty gritty details, drop down one abstraction level and use RSpec, or any Ruby testing framework, to write some specs/tests for your classes.
Cucumber is made to test your whole stack, together, as opposed to 'units'.
You need to decide where to draw the line, but a lot of under the hood stuff probably wouldn't be covered in a cucumber test. Say when signing up, I fill out a form, with my name, email, phone number, etc. A unit test might check to see that a new User will also create a new TelephoneNumber. From the user's perspective, they don't really care that it creates a new TelephoneNumber, they care that once they've signed up, they have an account and can see their telephone number.
I don't have too much experience writing cucumber tests (not yet), but I hope this helps a bit.
When a unit test fails (I mean a real unit test that tests a method in isolation using mocks), it tells you what "unit" has a problem. When an acceptance test fails, it tells you what "feature" has a problem, not where the problem is located.
When you create a rails app you get functional, interation, and unit tests by default. Cucumber is an additional test it is a way to also test the experience that your user will have. When they click the button labeled "go" they should see "success" rendered rather than a 404. This will make sure that nothing you do accidentally messes up the user experience, and that from the top to bottom your app works for the most common use cases you can think of. The other tests are meant to insure that nothing goes wrong, and that you have inspected ever model and method with a microscope. It may be possible to replicate unit tests in their entirety with cucumber, but it would be painful (and crazy slow to execute, especially if you're using selenium). The best time to write tests is when you're developing code, and the quickiest and easiest way to do that, is by using in the built-in rails testing and maybe some additional help such as shoulda, rspec, also i'm a huge fan of factory-girl. If you haven't already checked it out www.railscasts.com has a great intro to cucumber, and rspec, and factory-girl, ... I know this question has already been answered (it's no) but this is my two cents. Good luck coding!!
I have thought/struggled with this question much, and here's where I've arrived.
Cucumber first and Cucumber last. Cucumber will provide the primary test coverage.
The core model methods that do the real business work of the application should also be developed/covered with rspec/unit tests.
Why the unit tests as well?
1) The unit tests will test run much faster.
2) This core business logic may (will probably) be used in several ways beyond the current views (which Cucumber tests through). These methods ought to be hammered with all types of possible inputs and outputs directly calling the method in the test.
Why not unit test the rest of the models, and the controllers and views?
1) Cucumber already has it covered once.
2) I find that the views-controller-some-model-methods all work together to get things done (think everything exercised to log in); so I like to test them together.
I've been practicing Cucumber/RSpec for the past half year or so doing BDD.
First of all BDD is not easy to get into, it will feel unnatural at the beginning.
But once you get into it, there's no other way to do programming.
To answer your question. To test Javascript you'll need a javascript driver that can be used by Capybara which is used by Cucumber.
capybara-webkit is what all the cool kids use now these days
There's one important thing to note.
Integration tests are slow.
And unit tests are fast, but can be slow, so it's important you use the right database cleaner and you write good tests that have good isolation.
My test setup which I'm extremely happy with:
Guard for loading spork
Spork for faster tests
Cucumber for integration testing
capybara-webkit for javascript testing
RSpec for unit testing
I don't do view tests and controller tests as these are redundant in my opinion as good knowledge of XPATH willl have you writing remarkable tests that even cover your page layout and structure.
Personally I don't think that you should stop writing unit tests. As an acceptance testing tool, Cucumber should replace your functional tests and, if you writing, view tests.
Cucumber features are supposed to be simple and coupled to the real user's value a given feature has.
From my experience, Cucumber and Rspec have different appeal. Rspec appeals to me from a developer perspective because its easy to write and provides very quick feedback when something breaks. Cucumber does not appeal to me as a developer because it does not run as quickly as Rspec. However, Cucumber does appeal to me as a business stakeholder since it provides full coverage of entire features.
Do yourself a favor and keep writing unit tests.
I have a Rails project which I neglected to build tests for (for shame!) and the code base has gotten pretty large. A friend of mine said that RSpec was a pain to use unless you use it from the beginning. Is this true? What would make him say that?
So, considering the available tests suites and the fact that the code base is already there, what would be my best course of action for getting this thing testable? Is it really that much different than doing it from the beginning?
This question came up recently on the RSpec mailing list, and the advice we generally gave was:
Don't bother trying to retro-fit specs to existing, working, code unless you're going to change it - it's exhausting and, unless the code needs to be changed, rather pointless.
Start writing specs for any changes you make from now on. Bug fixes are an especially good opportunity for this.
Try to train yourself into the discipline that before you touch the code, first of all write a failing example (=spec) to drive out the change.
You may find that the design of code which wasn't driven out by code examples or unit tests makes it awkward to write tests or specs for. This is perhaps what your friend was alluding to. You will almost certainly need to learn a few key refactoring techniques to break up dependencies so that you can exercise each class in isolation from your specs. Michael Feathers' excellent book, Working Effectively With Legacy Code has some great material to help you learn this delicate skill.
I'd also encourage you to use the built-in spec:rcov rake task to generate code coverage stats. It's extremely rewarding to watch these numbers go up as you start to get your codebase under test.
Maybe start with the models? They should be testable in isolation, which ought to make them the lowest-hanging fruit.
Then pick a model and start writing tests that say what it does. As you go along, think about other ways to test the code - are there edge cases that maybe you're not sure about? Write the tests and see how the model behaves. As you develop the tests, you may see areas in the code that aren't as clean and de-duplicated (DRY) as they might be. Now you have tests, you can refactor the code, since you know that you're not affecting behaviour. Try not to start improving design until you have tests in place - that way lies madness.
Once you have the models pinned down, move up.
That's one way. Alternatives might be starting with views or controllers, but you may find it easier to start with end-to-end transaction tests andwork your way into smaller and smaller pieces as you go along.
The accepted answer is good advice - although not practical in some instances. I recently was faced with this problem on a few apps of mine because I NEEDED tests for existing code. There simply was no other way around it.
I started off doing all unit tests, then moved onto functionals.
Get in the habit of writing failing tests for any new code, or whenever you're going to change a part of the system. I've found this has helped me gain more knowledge of testing as I go.
Use rcov to measure your progress.
Good luck!
Writing tests for existing code may reveal bugs in your code. These tests will force you to look at the existing code so you can see what test you need to write in order to get it to pass and you may see some code that could possibly be written better, or is now useless.
Another tip is to write a test when you encounter a bug so it should never re-occur, this is called regressional testing.
Retrofitting specs is not inevitably a bad idea. You go from working code to working code with known properties which allows you to understand whether any future change breaks anything. At the moment if you need to make a change how can you know what it will affect?
What people mean when they say that it is hard to add tests/specs to exisitng code is that code which is hard to test is often highly coupled which makes it hard to write low-level isolated tests.
One idea would be to start with full-stack tests using something like the RSpec story runner. You can then work from the 'outside in' isolating what you can in low-level isolated tests and gradually untangle the harder code bit by bit.
You can start writing "characterization tests". With this,you might what to try out the pretentious gem here:
It is still a work in progress though.