I have started writing tests using Rspec for a really old project. The models which i am testing are all ActiveRecords(backend is Oracle).
I have read some blogs that say we should use mocking and stubbing/fixtures/factory girl over firing an actual sql.
I am confused. I will have to stub a lot of methods and create lot of objects. Is this a good practice ?
When you are testing a model, it's good to let your tests integrate against the database. That is, don't try to mock out the ActiveRecord stuff, and just use a model object. FactoryGirl and Fabrication are both just convenient shortcuts to building real model objects, and they are best practises when it comes to testing ActiveRecord models.
Since this is legacy code, I would suggest not mocking or stubbing too much in the old code, because isolation only works if each component is tested in isolation.
However, when writing code with TDD, mocking and stubbing has many benefits:
Gives you fine grained tests (if a method breaks, your tests tell you which one)
Your tests run much faster, and thus your TDD cycle is shorter
Lets you make assertions about how your code interacts with other objects
If you have to stub and mock other models excessively to isolate one model, it is usually a good sign that your code is too highly coupled and deserves a refactor
Normally you would use stubs, then you can run tests without loading AR, and they will run a lot faster.
Personally I believe that testing fake data is no test... Maybe in a development environment doing TDD... Maybe..., but for a test environment it should be as close to the real thing as possible.
This is of course my opinion and may not be the current school of thought.
Related
To test rails/mongoid DB models until now, I used to have a real model with few data (3 .. 5), trying to cover every case for real.
I'm already testing outside in BDD with cucumber, but I usually stop earlier, using rspec very little and without fixture and mockup, prefearing real cases and printing the fields values with rails logger via Rails.logger.info
But now I fed up with that, when the models get more complicated, it's too overwhelming.
What's the best way to test mongoid models in rails with tdd/bdd ?
Where can I start ?
I haven't worked with Mongoid before, but is there any reason why testing those models would be any different than one using ActiveRecord? I personally rely on RSpec for unit testing, which should only be testing the external API (behavior) of your models and should not rely much on fixtures or testing internal data or private methods. You can use mocks & stubs to avoid DB persistence in a lot of cases too.
If your models are getting too complicated, you might want to consider taking a more OOP approach and extracting some logic to other classes which do not rely on Mongoid. You have a lot of options in this case and I highly recommend Refactoring Ruby if you're interested in learning more.
A Rails/tool specific version of: How deep are your unit tests?
Right now, I currently write:
Cucumber features (integration tests) - these test against the HTML/JS that is returned by our app, but sometimes also tests other things, like calls to third-party services.
RSpec controller tests (functional tests), originally only if the controllers have any meaningful logic, but now more and more.
RSpec model tests (unit tests)
Sometimes this is entirely necessary; it is necessary to test behavior in the model that is not entirely obvious or visible to the end-user. When models are complex, they should definitely be tested. But other times, it seems to me the tests are redundant. For instance, do you test method foo if it is only called by bar, and bar is tested? What if bar is a simple helper method on a model that is used by and easily testable in a Cucumber feature? Do you test the method in rspec as well as Cucumber? I find myself struggling with this, as writing more tests take time and maintaining multiple "versions" of what is effectively the same behaviors, which makes maintaining the test suite more time intensive, which in turn makes changes more expensive.
In short, do you believe there is there a time when writing only Cucumber features is enough? Or should you always test at every level? If you think there is a grey area, what is your threshold for "this needs a functional/unit test." In practical terms, what do you do currently, and why (or why not) do you think it's sufficient?
EDIT: Here's an example of what might be "test overkill." Admittedly, I was able to write this pretty quickly, but it was completely hypothetical.
Good question, one I've grappled with recently while working on a Rails app, also using Cucumber/RSpec. I try to test as much as possible at every level, however, I've also found that as the codebase grows, I sometimes feel I'm repeating myself needlessly.
Using "Outside-in" testing, my process usually goes something like: Cucumber Scenario -> Controller Spec -> Model Spec. More and more I find myself skipping over the controller specs as the cucumber scenarios cover much of their functionality. I usually go back and add the controller specs, but it can feel like a bit of a chore.
One step I take regularly is to run rcov on my cucumber features with rake cucumber:rcov and look for notable gaps in coverage. These are areas of the code I make sure to focus on so they have decent coverage, be it unit or integration tests.
I believe models/libs should be unit tested extensively, right off the bat, as it is the core business logic. It needs to work in isolation, outside of the normal web request/response process. For example, if I'm interacting with my app through the Rails console, I'm working directly with the business logic and I want the reassurance that methods I call on my models/classes are well tested.
At the end of the day, every app is different and I think it's down to the developer(s) to determine how much test coverage should be devoted to different parts of the codebase and find the right balance so that your test suite doesn't bog you down as your app grows.
Here's an interesting article I dug up from my bookmarks that is worth reading:
http://webmozarts.com/2010/03/15/why-not-to-want-100-code-coverage/
Rails has a well-tested codebase, so I'd avoid re-testing stuff that is covered in those steps.
For example, unless it is custom code, it is pointless to test the results of validations at unit and functional levels. I'd test them at the integration level though. Cucumber features act as specifications for your project, so it is good to specify that you need a validation for x and y, even if the implementation is a single line declaration in the model.
You usually don't want to have both Cucumber stories and RSpec controller specs/integration tests. Pick one (generally Cucumber is the better choice, except for certain special cases). Then use RSpec for your models, and that's all you need.
I test complex model/lib methods with rspec then the main business logic in web with cucumber, so I'm sure that the main features of the web will work 100%, then if I got more time and resources I test everything else.
Its easier to write simple specs for simple methods. Its much easier than writing cukes.
If you keep your methods simple - and keep your specs simple - by testing only the logic inside that method - you will find joy in unit testing.
If anything is redundant its cucumber tests. If you have well tested models and lib your software should work.
The purpose of Cucumber is not to run integration tests. Cucumber, an in general BDD, works as a communication platform, a way to improve the "talk" inside a big team of developers an non-developers that are developing big and complex software. BDD is very useful to communicate a model an its domain at the same level to everybody in the team, even if they don't know anything about computers.
If that is not your scenario, don't use cucumber, because you don't need it. Use rspec and capybara to test your JS code and your integration tests.
I am finding holes in my coverage because I have been mocking my models in controller examples. When I remove a model's method upon which a controller depends, I do not get a failure.
Coming from TDD in statically typed languages, I would always mock dependencies to the object under test that hit the database to increase speed. I would still get failures in the above example, since my mocks subclassed the original object. I am looking for best practices in a dynamic language.
Thanks.
UPDATE:
After getting a lot of conflicting opinions on this, it seems it boils down to which philosophy you buy into.
The Rspec community appears to embrace heavily stubbing dependencies to achieve isolation of the object under test. Acceptance tests (traditionally known as integration tests ;) are used to ensure your objects work with their runtime dependencies.
The shoulda / Test::Unit community appears to stay away from stubbing as much as possible. This allows your tests to confirm your object under test actually works with its dependencies.
This video summarizes this nicely: http://vimeo.com/3296561
Yes, in your controller examples, mock your models. In your model examples, test your models.
If you're using Mocha, the following should do it.
Mocha::Configuration.prevent(:stubbing_non_existent_method)
While writing unit tests, the whole aim should be test that unit only. Consider Model as one unit and cover it separately. Change in Model should not directly impact unit test coverage of controller.
Besides the fact that Rails incorporates the database layer into the unit tests (which then is strictly not a unit test), what if I test for model interdependencies, e.g. check if has_many/belongs_to with :dependent => :destroy really destroys the associated items or items from join models to not leave orphans around?
Should such a test maybe better be placed in the function or integration tests?
Or asked another way: Is there a guide with examples which kind of tests should go where and why? I didn't find anything really useful.
As a general rule of thumb models are unit tested, controllers are functionally tested and integration tests are done with views. If you're using RSpec you'll have model, controller and view specs - but essentially testing the same stuff.
You can always mock actions (with gems like Mocha or Flexmock) to avoid having calls to methods actually do anything - but you want to make sure the expected behavior is happening. (E.g. a call is made to MyClass.create or MyClass.destroy). It is especially true of controller/view tests where I often will mock a call to a model.
In my experience though I don't usually have problems with models and associations using the database. You might want to look at Factory Girl or some alternative to fixtures, which make it easier to manage test objects/data.
A unit in a unit-test isnt always a single class or object. Testing interdependencies shouldn't be a problem. I havent found any hard and fast rules for defining units in unit tests but I often use ideas from domain driven design to guide me. Tests that test behaviour of a single aggregate are usually pretty maintainable as unit tests. Test that depend on more aggregates or even aggregates and services together are integration tests.
I'm currently looking at a hefty Rails test suite. It's nothing I can get into specifics about, but the run time for the entire suite (unit/functional/some integration) can run upward of 5 minutes.
We're completely reliant on fixtures and are we're not mocking and stubbing as much as we should be.
Our next few sprints are going to be completely focused on the test suite, both improving coverage, writing better tests and most importantly writing more efficient tests.
So aside from more mocking and stubbing within our tests, we're considering replacing our fixtures with most likely Factory Girl. I see a lot of happy folks doing similar situations but haven't been able to find a good resource on any minuses of moving to a factory. I have seen some slower benchmarks when using benchmarks from various resources but cannot find a definitive this why factories are good and this is why you might not want to use them.
Can anyone educate me on why or why I shouldn't be using factories?
Thanks!
Oleg's answer is great, but let me offer the perspective of someone who is using both.
Fixtures have sort of been the whipping boy of the Rails community for a while. Everyone understands the drawbacks of fixtures, but no one is really championing their strengths. In my experience, factories by themselves can easily become just as difficult to maintain as fixtures (it really depends on the schema, but I digress). The real strength of factories is in selective replacement of fixture-based pain. Let's talk about a couple specifics.
The first issue is performance. If you can test most of your app without hitting the database then you will see a significant speed up, but for most applications I don't think it's wise to test without hitting the database entirely. At some point you want to test the whole stack. Every time you mock or stub you are making an assumption about an interface that may contain subtle bugs. So, assuming that you need to hit the database on some significant percentage of tests, transactional fixtures (you are using transactional fixtures right?) could well be much much faster than instantiating a whole environment for every test.
I'd say, with the size of your test suite that you really need to look towards Continuous Integration to scale your development to the next level. No matter how much you speed them up, it's still a long time for developers to wait. Maybe look at autotest as well to help at the individual level. But ultimately CI is going to allow you to maintain testing discipline without sacrificing developer agility.
The place where fixtures really shine is in functional/integration testing. The way I look at it is that the fixtures should set up a healthy base state for the app to be tested. Most unit tests don't really need this. You can get very good unit coverage using factories. However when it comes to functional testing, any given page may be hitting dozens of models. I don't want to set up all that stuff in each test. As I construct ever more complex scenarios, I'm getting closer and closer to recreating a global data state which is exactly what fixtures were designed to do in the first place.
One controversial belief I hold is that all else being equal, I prefer one functional test to 20 unit tests (using Rails parlance). Why? Because the functional test proves that the end result that is sent to the user is correct. The unit tests are great for getting at nuances of functionality, but at the end of the day, you could still have a bug along an interface that breaks your entire site. Functional tests are what give me the confidence hitting deploy without actually loading up the page in my browser. I know that I could stub everything out and test both interfaces and get the same coverage, but if I can test the whole stack in one simple test at the expense of a little CPU, I'd much rather do that.
So what are my best practices for fixtures?
Set up a handful for every model to cover the broadest categories of data
When adding a major new feature that cuts across many models and controllers, add some new fixtures to represent the major states
Avoid editing old fixtures except for adding/removing fields
Use factories for more smaller/more localized variations
Use factories for testing pagination or other mass creation that is only needed for a few tests
Also, let me recommend Jay Fields' blog for really good pragmatic testing advice. The thing I like most about Jay's blog is that he always acknowledges that testing is very project-specific, and what works for one project does not necessarily work for another. He's short on dogma and long on pragmatism.
There could be some issues with setting up all dependencies between entities for good test suite. Anyway, it's still much easier than maintaing a lot of fixtures.
Fixtures:
hard to maintain relationships (especially many-to-many);
test suite runtime is usually slower due more DB hits;
tests are very sensitive to changes in schema.
Factories:
you stub everything you don't test at current unit test;
you prepare entities that you are testing with factories. This is where factories show their real advantage — it's easy to set up new test cases, as you don't need to maintain a ton of YAML-files for that;
you concentrate on testing. If tests require changing scenario, you don't shift your mindset. As long as stubs are reasonable and factories are easily customized, you should be fine.
So, factories seem a good way to go. The only possible drawbacks I see, are:
time you spent migrating from fixtures;
keeping a sane set of scenarios can require some effort.