I have a Rails project which I neglected to build tests for (for shame!) and the code base has gotten pretty large. A friend of mine said that RSpec was a pain to use unless you use it from the beginning. Is this true? What would make him say that?
So, considering the available tests suites and the fact that the code base is already there, what would be my best course of action for getting this thing testable? Is it really that much different than doing it from the beginning?
This question came up recently on the RSpec mailing list, and the advice we generally gave was:
Don't bother trying to retro-fit specs to existing, working, code unless you're going to change it - it's exhausting and, unless the code needs to be changed, rather pointless.
Start writing specs for any changes you make from now on. Bug fixes are an especially good opportunity for this.
Try to train yourself into the discipline that before you touch the code, first of all write a failing example (=spec) to drive out the change.
You may find that the design of code which wasn't driven out by code examples or unit tests makes it awkward to write tests or specs for. This is perhaps what your friend was alluding to. You will almost certainly need to learn a few key refactoring techniques to break up dependencies so that you can exercise each class in isolation from your specs. Michael Feathers' excellent book, Working Effectively With Legacy Code has some great material to help you learn this delicate skill.
I'd also encourage you to use the built-in spec:rcov rake task to generate code coverage stats. It's extremely rewarding to watch these numbers go up as you start to get your codebase under test.
Maybe start with the models? They should be testable in isolation, which ought to make them the lowest-hanging fruit.
Then pick a model and start writing tests that say what it does. As you go along, think about other ways to test the code - are there edge cases that maybe you're not sure about? Write the tests and see how the model behaves. As you develop the tests, you may see areas in the code that aren't as clean and de-duplicated (DRY) as they might be. Now you have tests, you can refactor the code, since you know that you're not affecting behaviour. Try not to start improving design until you have tests in place - that way lies madness.
Once you have the models pinned down, move up.
That's one way. Alternatives might be starting with views or controllers, but you may find it easier to start with end-to-end transaction tests andwork your way into smaller and smaller pieces as you go along.
The accepted answer is good advice - although not practical in some instances. I recently was faced with this problem on a few apps of mine because I NEEDED tests for existing code. There simply was no other way around it.
I started off doing all unit tests, then moved onto functionals.
Get in the habit of writing failing tests for any new code, or whenever you're going to change a part of the system. I've found this has helped me gain more knowledge of testing as I go.
Use rcov to measure your progress.
Good luck!
Writing tests for existing code may reveal bugs in your code. These tests will force you to look at the existing code so you can see what test you need to write in order to get it to pass and you may see some code that could possibly be written better, or is now useless.
Another tip is to write a test when you encounter a bug so it should never re-occur, this is called regressional testing.
Retrofitting specs is not inevitably a bad idea. You go from working code to working code with known properties which allows you to understand whether any future change breaks anything. At the moment if you need to make a change how can you know what it will affect?
What people mean when they say that it is hard to add tests/specs to exisitng code is that code which is hard to test is often highly coupled which makes it hard to write low-level isolated tests.
One idea would be to start with full-stack tests using something like the RSpec story runner. You can then work from the 'outside in' isolating what you can in low-level isolated tests and gradually untangle the harder code bit by bit.
You can start writing "characterization tests". With this,you might what to try out the pretentious gem here:
It is still a work in progress though.
Related
I have to preface all my posts about rails: I'm a novice.
Is it necessary to write tests for my app to work properly or is it strictly for finding breaks?
Testing your app is not necessary for it to work, but it is highly recommended and a very good practice. Testing your application will help you develop a better piece of software and a much more solid application. In the Rails world BDD(Behavior Driven Development) is very used as a testing and development technique.
I recommend you two diferent testing suites:
The first one is Rspec that will help you with all your controllers, and models unit testing
Cucumber is a testing suite that will test your application as a whole(integration test), this one is great for a more "real life" testing approach
I encourage you to check both Rspec and Cucumber, there are also other great testing suites like Test Unit.
Remember, testing your application will give you great benefits!
NOTE: Rspec and Cucumber are not mutually exclusive, actually they are recommended to use in conjunction
Writing tests is not necessary in the sense that your application won't operate without them, but they are not only for keeping bugs away. If you're a novice, writing the tests should also help you to understand how everything works.
There's no reason not to write tests. Just write good tests, and don't waste time testing things that don't need to be tested (like generated attr_accessors).
It's not necessary at all but its considered by almost every ruby developer I know to be standard procedure.
I did a couple "rails apps" without testing, but as soon as I needed some real backend logic in ruby, testing helped me to understand what I was doing.
No, writing tests is not required for your application to run. It is a good practice, though, so if you are not used to writing tests I'd recommend that you start learning. It's easy and it will save you a lot of headaches on whatever platform you are using.
Just to chime in with everyone else, I'd suggest you find yourself an article about BDD and test first development. And then read up on mocking and stubbing. Wrapping your head around the why and how of it will probably convince you it's worth the time and effort.
When I first dove into the XP & RoR world I live in now I was daunted by what felt to me like test-mania but it really pays off in spades.
The first time someone had me write the test first THEN write the code we were testing I was mind blown. But I've never gone back to my previous evil ways.
There is an existing project that is already deployed in production. We want to add some tests on it (the sooner the better) and I have to choose between going the BDD way (rspec/cucumber) or the TDD way (TestUnit). I am really starting with BDD and I am wondering what could be the best decision to take? I am affraid that using rspec/cucumber on an existing rails project (which was deployed this week and requires really fast iterations) will be quite hard to do (especially that it is not supposed to be used this way, I mean we are supposed to write stories/features first and iterate from there).
TestUnit could be more reasonable, may be.
Do you have any thoughts on that? An experience to share? Some advices?
I believe the easiest way to get coverage for an existing application is to use cucumber. This will allow to describe and document how to the website/application should work (and will keep working).
Because it works from the outside in, this also has the advantage you do not need to comprehend the inner workings completely yet. At the same time, you test all layers of the application (model-view-controller) in one test.
When you start actually changing code, then I would start adding the unit-tests for the code you are changing, using your favourite testing framework. I personally favour rspec, but as you know this is a personal choice :)
I'm new to unit/func testing in Rails 3. So I'm starting now, better late than never.
I have a method in /lib/mailingjob.rb called find_reply(body)
Right now I'm using cucumber to test this but given that this is all backend, no web interface to this, should I not be using cucumber?
I'm finding when to use RPSEC vs cucumber confusing.
Thanks
In my opinion, you need to use both. Rspec is very good for unit testing, that is testing models, controllers, views. On the other hand, cucumber is a very nice tool to check full scenarios like a user logs in, clicks a link and he is supposed to view this.
I highly advise that you take a look at the cucumber railscast from railscasts.com. Also, make sure that you use webrat and maybe something to auto load your specs like watchr(which i prefer).
I'd advise you to never use Cucumber unless you are specifically employed as a business analyst. That makes your life much easier: all your tests will be in Rspec.
Cucumber takes a toll on your productivity, and your sanity. Running parallel test environments, adding layers of abstraction and breaking your editor with its odd syntax make for lost time. I've written about this extensively on my blog:
Why Bother With Cucumber Testing?
If I didn't already have a Stack Overflow account, I would have made one JUST to write this post. I urge you to re-read #Jack Kinsella's answer
I got in to BDD about 3 months ago, fell in love with it, and quickly started using Cucumber for everything. I forced it in to every project, no matter what
Recently, I've had to learn a yet another language, and my first step was to get the testing environment set up. I came across these 3 articles:
http://robots.thoughtbot.com/post/25650434584/writing-better-cucumber-scenarios-or-why-were
http://robots.thoughtbot.com/post/25653308407/describe-the-users-perspective-ddd-acceptance
http://37signals.com/svn/posts/3159-testing-like-the-tsa
Honestly, their content isn't that important; what's important is that they hinted at an overall idea. I began to suspect that I'd been using Cucumber wrong all along, and I went hunting for answers. This morning I found your SO question, and #Jack Kinsella's blog post
#Jack comes right out and says the idea I believe those articles were tentatively circling. He also gives me the language I was looking for. I now consider his article to be the final word on the subject :)
According to him, what we've ACTUALLY been using Cucumber for is "integration testing". I never understood what that meant before
Unit testing:
How the code works internally. Math.Add(1,1) should be 2, but the website user doesn't care. Just give me a web page!
-> Use RSpec or an equivalent
Integration testing:
How different branches of the code work together to make the website. I type my name and click Log In, and should be taken to the Home page
-> ALSO use RSpec!
(Inside of RSpec, add whatever you need to handle multiple-technologies-touching-eachother. For the code-to-webBrowsers example: Capybara, Watir, etc)
Acceptance testing:
Many of us don't need it. Someone signed a contract with you saying "I'm going to write 'I can add subpages to my micro-site.' in a text file". You write me code that makes it turn green
-> Use Cucumber. Only if you have to do this kind of testing. Which you almost certainly don't
What an elegant solution. No 2nd testing environment. No steps directory and swarm of extra files!
I had a moment of "But I love how Cucumber seperates English from the code". Regular bdd does that too. "rspec --format nested" or Jasmine test results
#Jack is right. Cucumber isn't adding anything; not the way we've been using it. And it's costing a lot. To quote him:
Why not admit to yourself that you don’t do acceptance testing and that you do not need it in your projects? Swap Cucumber for pure integration tests using Capybara, and you’ll be surprised by how much more productive you can be
To complete SpyrosP's answer, there is an awesome blog post from Sarah Mei which describes a scenario where you use both Rspec and Cucumber. It is called oustide-in Behavior Driven Development and you can find it here.
However, I think most of Jack's problem can be solved by a good cucumber editor. So the problem is not Cucumber. The problem is that there wasn't a capable editor when he wrote this two years ago. But today RubyMine has full cucumber editing capability. While it is not free, it can solve most of Jack's problem: Auto completing, pointing out errors, refactoring. So you probably want to go back and check the points he raised, and see if they are still valid today.
My take so this issue many years later is very simple: If you do not know what to use now, just start with RSpec, because it is the central stone for all your testing that you will always need.
With time you will discover that two big testing needs:
Is my code still working?
Can my users still perform the actions already implemented?
As a developer I tend to think that (1) is for every single code change and (2) when I finish a feature and/or deploy.
Having two test suites is nice because during a deployment, you can sit and run a very generic test suite: Are my user features still working? Can I finally deploy this?
Indeed, it should be so generic that if something breaks in (2) you need to do some heavy research (one row missing in your tables) while a fail in (1) should pin-point the exact class with the problem.
Normally I find myself in the situation where I want separate integration+acceptance testing without the Cucumber burden. First I use RSpec tags to filter out these tests. Whenever I need more I like going for Steak+Capybara since it enables Cucumber like objectives on top of something I already know quite well.
Definitely use both. Use cucumber for user experience tests(views), use rspec for testing internals(models, etc.). You may write controller rspec tests but I think they are not necessary(try to move the logic to models).
Whatever your system is, I strongly advise you to use Cucumber because scenarios will let you and your stakeholders know the functions of your system in a clear written fashion.
You will also have your documentation ready and you'll always know where you are and where you are heading. You'll also relate scenarios with support issues in the future.
Cuke them right by the way don't use variables in scenarios, they should be understandable.
And if you want them to run fast as rspecs use poltergeist(phantomjs).
I've been tempted not use cucumber because in many cases it's not simple as rspec. However every time I understood that I needed scenarios to take full control on the project.
I'm just starting to grasp BDD and RSpec and one thing I'm really having trouble with is figuring out how thorough I should be with my testing.
I'm just not understanding how fine-grained my testing should be to still be useful but not double development time.
Is it just a matter of preference? Or is there some general standard for what should be tested?
There's a few factors to consider here.
Spec coverage should be greatest for the most important features and the things most likely to break.
Specs should express developer intention. Don't write specs for trivial things you or someone else might change later.
Test behavior not implementation. You should be able to change the internal implementation of a class and still pass the specs. This makes refactoring easier.
Any time you fix bugs add specs to prevent the bug from regressing.
I find it helpful to think of Tests/Specs as a safety net. I want a spec to fail if another dev (or myself) breaks something that I spent time to make work. I don't want them to have to spend a lot of time changing specs for things that don't really matter. I also don't want to inhibit them from improving the application because of specs that test things that weren't important to me when writing the code.
I'm confused about what the various testing appliances in Ruby on Rails are for. I have been using the framework for about 6 months but I've never understood the testing part of it. The only testing I've used is JUnit3 in Java and that only briefly.
Everything I've read about it just shows testing validations. Shouldn't the validations in rails just work? It seems more like testing the framework than testing the your code. Why would you need to test validations?
Furthermore, the tests seem super fragile to any change in your code. So if you change anything in your models, you have to change your tests and fixtures to match. Doesn't this violate the DRY principle?
Third, writing test code seems to take alot of time. Is that normal? Wouldn't it just be faster to refresh my browser and see if it worked? I already have to play with my application just to see if it flows correctly and make sure my CSS hasn't exploded. Why wouldn't manual testing be enough?
I've asked these questions before and I haven't gotten more than "automated testing is automated". I am smart enough to figure out the advantages of automating a task. My problem is that costs of writing tests seem absurdly high compared to the benefits. That said, any detailed response is welcome because I probably missed a benefit or two.
Shouldn't
the validations in rails just work? It
seems more like testing the framework
than testing the your code. Why would
you need to test validations?
The validations in Rails do work -- in fact, there are unit tests in the Rails codebase to ensure it. When you test a model's validation, you're testing the specifics of the validation: the length, the accepted values, etc. You're making sure the code was written as intended. Some validations are simple helpers and you may opt not to test them on the notion that "no one can mess up a validates_numericality_of call." Is that true? Does every developer always remember to write it in the first place? Does every developer never accidentally delete a line on a bad copy paste? In my personal opinion, you don't need to test every last combination of values for a Rails' validation helper, but you need a line to test that it's there with the right values passed, just in case some punk changes it in the future without proper forethought.
Further, other validations are more complex, requiring lots of custom code -- they may warrant more thorough testing.
Furthermore, the tests seem super
fragile to any change in your code. So
if you change anything in your models,
you have to change your tests and
fixtures to match. Doesn't this
violate the DRY principle?
I don't believe it violates DRY. They're communicating (that's what programming is, communication) two very different things. The test says the code should do something. The code says what it actually does. Testing is extremely important when there is a disconnect between those things.
Test code and application code are intimately linked, obviously. I think of them as two sides of a coin. You wouldn't want a front without a back, or a back without a front. Good test code reinforces good application code, and vice versa. The two together are used to understand the whole problem that you're trying to solve. And well written test code is documentation -- it shows how the application code should be used.
Third, writing test code seems to take
alot of time. Is that normal? Wouldn't
it just be faster to refresh my
browser and see if it worked? I
already have to play with my
application just to see if it flows
correctly and make sure my CSS hasn't
exploded. Why wouldn't manual testing
be enough?
You've only worked on very small projects, for which that testing is arguably sufficient. However, when you work on a project with several developers, thousands or tens of thousands of lines of code, integration points with web services, third party libraries, multiple databases, months of development and requirements changes, etc, there are a lot of other factors in play. Manual testing is simply not enough. In a project of any real complexity, changes in one place can often have unforeseen results in others. Proper architecture helps mitigate this problem, but automated testing helps as well (and helps identify points where the architecture can be improved) by identifying when a change in one place breaks another.
My problem is that
costs of writing tests seem absurdly
high compared to the benefits. That
said, any detailed response is welcome
because I probably missed a benefit or
two.
I'll list a few more benefits.
If you test first (Test Driven Development) your code will probably be better. I haven't met a programmer who gave it a solid shot for whom this wasn't the case. Testing first forces you to think about the problem and actually design your solution, versus hacking it out. Further, it forces you to understand the problem domain well enough to where if you do have to hack it out, you know your code works within the limitations you've defined.
If you have full test coverage, you can refactor with NO RISK. If a software problem is very complicated (again, real world projects that last for months tend to be complicated) then you may wish to simplify code that has previously been written. So, you can write new code to replace the old code, and if it passes all of your tests, you're done. It does exactly what the old code did with respect to the tests. For a project that plans to use an agile development method, refactoring is absolutely essential. Changes will always need to be made.
To sum up, automated testing, especially test driven development, is basically a method of managing the complexity of software development. If your project isn't very complex, the cost may outweigh the benefits (although I doubt it). However, real world projects tend to be very complex, and the results of testing and TDD speak for themselves: they work.
(If you're curious, I find Dan North's article on Behavior Driven Development to be very helpful in understanding a lot of the value in testing: http://dannorth.net/introducing-bdd)
I haven't really used Rails much, but I would think that these automated tests would be useful as smoke tests to be sure that the thing you just did doesn't break something that you did last week. This will become increasingly important as your project grows.
Also, writing the tests before you write the code (using the Test-Driven-Development model) will help you write the code better and faster, since the tests force you to fully think the problem through. It will also help you to know where to break up complex methods into smaller methods that you can test individually.
You are right, writing and maintaining tests takes a lot of time. Sometimes more time than the code itself. However, it can save you time in bug fixing and refactoring for the reasons above.
Tests should validate your application logic. Personally, I think my most important tests are the ones I run in Selenium. They check that what shows up in the browser is actually what I expect to see. However, if that's all I had, then I would find it hard to debug - it helps to have lower level tests as well and integration, functional, and unit tests are all useful tools. Unit tests let you check that the model behaves the way you expect it to (and that means every method, not just validatins). Validatins will certainly Just Work, but only if you get them right. If you get them wrong, they will Just Work, but not the way you expected. Writing a couple of lines of test is quicker than debugging later on.
A simple example like the one at http://wiseheartdesign.com/2006/01/16/testing-rails-validations just checks validations in a unit test. The O'Reilly article at http://www.oreillynet.com/pub/a/ruby/2007/06/07/rails-testing-not-just-for-the-paranoid.html?page=1 is a bit more complete (though still fairly basic).
Automated testing is particularly useful in regression testing where you change something and run a suite of tests to check that you didn't break anything else.
Tests are a form of repetition, but they don't violate DRY because they express things in a different way. A test says "I did X so Y should happen". Code says "X happened, so now I need to do Z, which happens to cause Y to happen". i.e. a test stimulates a cause and checks an effect, while code responds to a cause, and effects something.
A lot of the testing tutorials and the sample tests created by the Rails generators are pretty lame and IMHO that can give the mistaken impression that you're supposed to test stupid stuff like the built in Rails methods, etc.
Since Rails has it's own test suite, there's no point in you writing or running tests that only test built in Rails functionality. Your tests should exercise the code you're writing! :-)
As for the relative merit of running tests vs just refreshing in your browser.. The larger your app gets, the more of a pain in the ass it is to have to manually run through numerous scenarios and edge cases to make sure nothing in your application has broken. Eventually, you'll stop testing your entire application after each change and just start "spot testing" the areas you think should have been affected. Inevitably, you'll find something that used to work months ago that is now completely broken, and you have no certainty when it broke or which changes broke it. After that happens enough times... you'll come to value automated testing.... :-)
For example:
I work on a 25000+ lines project (yes, in rails 1.2) and last monday I was told if I could make Users dissapear from every list except admin ones if they had "leave_date" attribute set to the past.
You can rewrite every list action (50+) to put a
#users.reject!{|u| Date.today > u.leave_date}
Or you can override the "find" method (DRY;-), but only if you have tests (on everything that finds users!) you will know you didn't break anything by overriding User#find !!
Everything I've read about it just shows testing validations. Shouldn't the validations in rails just work? It seems more like testing the framework than testing the your code. Why would you need to test validations?
There's a good Railscast showing one way to test controllers.