I really like using Factory Girl to setup my tests. I can build chains of associations in a single line. For example:
Factory.create(:manuscript)
Automatically creates a journal, a journal owner, a manuscript author, etc. It allows me to to keep my setup blocks really simple, and that's fantastic.
However, there's a cost of course. Creating several objects in the background means my unit tests are sometimes as long as 0.8 seconds. That's fine when your app is small, but now I've got a few hundred tests and my specs take over a minute to run (not including the time it takes for the app to spin up). It's starting to feel painful.
I'm not especially interested in anything too drastic, like mocking everything. At least while my app is relatively small, I'd like to maintain my factory girl abstractions. I just want to figure out a way to make them work a bit faster.
Any suggestions?
If you're testing object behavior and don't need to actually save them to a database, you can use Factory.build(:model). It basically instantiates the object and it's associations, but does not write it to the DB. This will be much faster than creating and storing all those objects. If you still want to write some or most objects to the DB, you can setup an SQlite memory test database. Here's an example
Not sure there's a good solution to this problem. As Beerlington suggested, you can save some time using Factory.build rather than Factory.create. But even that's not nearly as fast as testing a plain old ruby objects. The fact, it seems, is that factory girl is not a good choice if you're very concerned with speed.
That said, I was able to make some fairly significant speed improvements by reading through my entire suite and making liberal use of the rspec-set gem. This allows you to run your setup once -- and only once -- for the entire group of tests. It's similar to using before(:all) except that it takes advantage of transactions to reset the state of objects between each spec.
Related
I love Faker, I use it in my seeds.rb all the time to populate my dev environment with real-ish looking data.
I've also just started using Factory Girl which also saves a lot of time - but when i sleuth around the web for code examples I don't see much evidence of people combining the two.
Q. Is there a good reason why people don't use faker in a factory?
My feeling is that by doing so I'd increase the robustness of my tests by seeding random - but predictable - data each time, which hopefully would increase the chances of a bug popping up.
But perhaps that's incorrect and there is either no benefit over hard coding a factory or I'm not seeing a potential pitfall. Is there a good reason why these two gems should or shouldn't be combined?
Some people argue against it, as here.
DO NOT USE RANDOM ATTRIBUTE VALUES
One common pattern is to use a fake data library (like Faker or Forgery) to generate random values on
the fly. This may seem attractive for names, email addresses or
telephone numbers, but it serves no real purpose. Creating unique
values is simple enough with sequences:
FactoryGirl.define do
sequence(:title) { |n| "Example title #{n}" }
factory :post do
title
end
end
FactoryGirl.create(:post).title # => 'Example title 1'
Your randomised
data might at some stage trigger unexpected results in your tests,
making your factories frustrating to work with. Any value that might
affect your test outcome in some way would have to be overridden,
meaning:
Over time, you will discover new attributes that cause your test to
fail sometimes. This is a frustrating process, since tests might fail
only once in every ten or hundred runs – depending on how many
attributes and possible values there are, and which combination
triggers the bug. You will have to list every such random attribute in
every test to override it, which is silly. So, you create non-random
factories, thereby negating any benefit of the original randomness.
One might argue, as Henrik Nyh does, that random values help you
discover bugs. While possible, that obviously means you have a bigger
problem: holes in your test suite. In the worst case scenario the bug
still goes undetected; in the best case scenario you get a cryptic
error message that disappears the next time you run the test, making
it hard to debug. True, a cryptic error is better than no error, but
randomised factories remain a poor substitute for proper unit tests,
code review and TDD to prevent these problems.
Randomised factories are therefore not only not worth the effort, they
even give you false confidence in your tests, which is worse than
having no tests at all.
But there's nothing stopping you from doing it if you want to, just do it.
Oh, and there is an even easier way to inline a sequence in recent FactoryGirl, that quote was written for an older version.
It's up to you.
In my opinion is a very good idea to have random data in tests and it always helped me to discover bugs and corner cases I didn't think about.
I never regret to have random data. All the points described by #jrochkind would be correct (and you should read the other answer before reading this one) but it's also true that you can (and should) write that in your spec_helper.rb
config.before(:all) { Faker::Config.random = Random.new(config.seed) }
this will make so that you have repeatable tests with repeatable data as well. If you don't do that then you have all the problems described in the other answer.
I like to use Faker and usually do so when working with larger code bases. I see the following advantages and disadvantages when using Faker with Factory Girl:
Possible disadvantages:
A bit harder to reproduce the exact same test scenario (at least RSpec works around this by displaying the random number generator seed every time and allows you to reproduce the exact same test with it)
Generating data wastes a bit of performance
Possible advantages:
Makes data displayed usually more humanly comprehensible. When creating test-data manually, people tend to all kinds of short-cuts to avoid the tediousness.
Building factories with Faker for tests at the same time provides you with the means of generating nice demo data for presentations.
You could randomly discover edge case bugs when running the tests a lot
I'm new to testing in rails and I don't understand why or when I should use fixtures (or factory) rather than just seeding my test db and querying it to run the tests?
In many cases, it should be faster and easier to have the same data in dev and test env.
For example, if I want to test an index page, should I create 100 records via a factory or should I seed the db with 100 records?
I someone could clarify this, it would be great.
Thanks!
This is actually a deeper question of how to test efficiently, and you will find a lot of different opinions.
The reason to avoid a database in your unit tests is merely speed. Database operations are slow. It might not seem slow with one test, but if you have continuous integration going (as you should) or when you made a quick change and just want to see what happens, those delays add up. So prefer mocks to truly unit test code.
Then your own integration tests should hit an in-memory database rather than your real database--for the same reason, speed. These will be slower than your mocked tests, but still faster than hitting the real database. When you're developing, the build-test-deploy cycle needs to be as fast as possible. Note that some people call these unit tests as well. I wouldn't, but I guess it is just semantics.
These first two kinds of tests are by developers for developers.
Then the testers will hit the real database, which will be populated with test data defined by the testers and subject-matter experts. There are lots of clever ways to speed this up as well, but this will be the place where they test the integration of your code with the production-like database. If all your in-memory database tests passed and something goes wrong here, then you know it has to do with something like database configuration, vendor-specific SQL, etc. rather than something fundamentally bad. You will also get your first taste of what the performance is like.
Note that everything I've said here is a matter of debate. But hopefully it clarifies what you should consider about when to do certain things and why.
I recently learned, Rspec + Capybara (I'm new to testing things). And I've been testing some new models I've created. Now, I was thinking of adding the title column to my Post model. It's not a hard task, and I've done it many times before. But I realized that it's going to take me some work updating the spec files (e.g. post, post pages, user, user pages, factories, sample_data, etc.).
Is testing in this case, a waste of time? Or it is a good practice to test absolutely everything?
I also wonder if testing gems is a waste of time as well? Since they've been probably already tested by their creators (the same for testing something taken from a reliable tutorial).
While the ruby community is known by its focus on testing, every case should be analyzed in context to figure out what the most cost-effective decisions are.
It is useful to think of tests as a technique to manage risk. If you don't think that displaying titles/adding them to the database is a task that is error-prone, complex to perform, or that the cost of adding those tests is higher than the risk of eventually breaking something, them it might be wise to not do it.
That said, since you said that you are new to testing, I would recommend (if you are not time-constrained, of course) to try to take the idea to the extreme of testing absolutely everything, so that you can have an idea of how much of it adds value, and how much is waste of time.
You might want to take a look at this blog post from DHH, the creator of Rails, where he talks about that: Testing like the TSA
About testing gems, it is generally well-accepted that they shouldn't be tested by your application tests, but by their creators. Most of the time you can simply assume that they work the ways they promise.
one of the main concepts of testing is: "It's impossible to test everything"
It seems, that there are two totally different approaches to testing, and I would like to cite both of them.
The thing is, that those opinions were stated 5 years ago (2007), and I am interested, what has changed since then and which way should I go.
Brandon Keepers:
The theory is that tests are supposed to be agnostic of the
implementation. This leads to less brittle tests and actually tests
the outcome (or behavior).
With RSpec, I feel like the common approach of completely mocking your
models to test your controllers ends up forcing you to look too much
into the implementation of your controller.
This by itself is not too bad, but the problem is that it peers too
much into the controller to dictate how the model is used. Why does it
matter if my controller calls Thing.new? What if my controller decides
to take the Thing.create! and rescue route? What if my model has a
special initializer method, like Thing.build_with_foo? My spec for
behavior should not fail if I change the implementation.
This problem gets even worse when you have nested resources and are
creating multiple models per controller. Some of my setup methods end
up being 15 or more lines long and VERY fragile.
RSpec’s intention is to completely isolate your controller logic from
your models, which sounds good in theory, but almost runs against the
grain for an integrated stack like Rails. Especially if you practice
the skinny controller/fat model discipline, the amount of logic in the
controller becomes very small, and the setup becomes huge.
So what’s a BDD-wannabe to do? Taking a step back, the behavior that I
really want to test is not that my controller calls Thing.new, but
that given parameters X, it creates a new thing and redirects to it.
David Chelimsky:
It’s all about trade-offs.
The fact that AR chooses inheritance rather than delegation puts us in
a testing bind – we have to be coupled to the database OR we have to
be more intimate with the implementation. We accept this design choice
because we reap benefits in expressiveness and DRY-ness.
In grappling with the dilemma, I chose faster tests at the cost of
slightly more brittle. You’re choosing less brittle tests at the cost
of them running slightly slower. It’s a trade-off either way.
In practice, I run the tests hundreds, if not thousands, of times a
day (I use autotest and take very granular steps) and I change whether
I use “new” or “create” almost never. Also due to granular steps, new
models that appear are quite volatile at first. The valid_thing_attrs
approach minimizes the pain from this a bit, but it still means that
every new required field means that I have to change
valid_thing_attrs.
But if your approach is working for you in practice, then its good! In
fact, I’d strongly recommend that you publish a plugin with generators
that produce the examples the way you like them. I’m sure that a lot
of people would benefit from that.
Ryan Bates:
Out of curiosity, how often do you use mocks in your tests/specs?
Perhaps I'm doing something wrong, but I'm finding it severely
limiting. Since switching to rSpec over a month ago, I've been doing
what they recommend in the docs where the controller and view layers
do not hit the database at all and the models are completely mocked
out. This gives you a nice speed boost and makes some things easier,
but I'm finding the cons of doing this far outweigh the pros. Since
using mocks, my specs have turned into a maintenance nightmare. Specs
are meant to test the behavior, not the implementation. I don't care
if a method was called I just want to make sure the resulting output
is correct. Because mocking makes specs picky about the
implementation, it makes simple refactorings (that don't change the
behavior) impossible to do without having to constantly go back and
"fix" the specs. I'm very opinionated about what a spec/tests should
cover. A test should only break when the app breaks. This is one
reason why I hardly test the view layer because I find it too rigid.
It often leads to tests breaking without the app breaking when
changing little things in the view. I'm finding the same problem with
mocks. On top of all this, I just realized today that mocking/stubbing
a class method (sometimes) sticks around between specs. Specs should
be self contained and not influenced by other specs. This breaks that
rule and leads to tricky bugs. What have I learned from all this? Be
careful where you use mocking. Stubbing is not as bad, but still has
some of the same issues.
I took the past few hours and removed nearly all mocks from my specs.
I also merged the controller and view specs into one using
"integrate_views" in the controller spec. I am also loading all
fixtures for each controller spec so there's some test data to fill
the views. The end result? My specs are shorter, simpler, more
consistent, less rigid, and they test the entire stack together
(model, view, controller) so no bugs can slip through the cracks. I'm
not saying this is the "right" way for everyone. If your project
requires a very strict spec case then it may not be for you, but in my
case this is worlds better than what I had before using mocks. I still
think stubbing is a good solution in a few spots so I'm still doing
that.
I think all three opinions are still completely valid. Ryan and I were struggling with the maintainability of mocking, while David felt the maintenance tradeoff was worth it for the increase in speed.
But these tradeoffs are symptoms of a deeper problem, which David alluded to in 2007: ActiveRecord. The design of ActiveRecord encourages you to create god objects that do too much, know too much about the rest of the system, and have too much surface area. This leads to tests that have too much to test, know too much about the rest of the system, and are either too slow or brittle.
So what's the solution? Separate as much of your application from the framework as possible. Write lots of small classes that model your domain and don't inherit from anything. Each object should have limited surface area (no more than a few methods) and explicit dependencies passed in through the constructor.
With this approach, I've only been writing two types of tests: isolated unit tests, and full-stack system tests. In the isolation tests, I mock or stub everything that is not the object under test. These tests are insanely fast and often don't even require loading the whole Rails environment. The full stack tests exercise the whole system. They are painfully slow and give useless feedback when they fail. I write as few as necessary, but enough to give me confidence that all my well-tested objects integrate well.
Unfortunately, I can't point you to an example project that does this well (yet). I talk a little about it in my presentation on Why Our Code Smells, watch Corey Haines' presentation on Fast Rails Tests, and I highly recommend reading Growing Object Oriented Software Guided by Tests.
Thanks for compiling the quotes from 2007. It is fun to look back.
My current testing approach is covered in this RailsCasts episode which I have been quite happy with. In summary I have two levels of tests.
High level: I use request specs in RSpec, Capybara, and VCR. Tests can be flagged to execute JavaScript as necessary. Mocking is avoided here because the goal is to test the entire stack. Each controller action is tested at least once, maybe a few times.
Low level: This is where all complex logic is tested - primarily models and helpers. I avoid mocking here as well. The tests hit the database or surrounding objects when necessary.
Notice there are no controller or view specs. I feel these are adequately covered in request specs.
Since there is little mocking, how do I keep the tests fast? Here are some tips.
Avoid excessive branching logic in the high level tests. Any complex logic should be moved to the lower level.
When generating records (such as with Factory Girl), use build first and only switch to create when necessary.
Use Guard with Spork to skip the Rails startup time. The relevant tests are often done within a few seconds after saving the file. Use a :focus tag in RSpec to limit which tests run when working on a specific area. If it's a large test suite, set all_after_pass: false, all_on_start: false in the Guardfile to only run them all when needed.
I use multiple assertions per test. Executing the same setup code for each assertion will greatly increase the test time. RSpec will print out the line that failed so it is easy to locate it.
I find mocking adds brittleness to the tests which is why I avoid it. True, it can be great as an aid for OO design, but in the structure of a Rails app this doesn't feel as effective. Instead I rely heavily on refactoring and let the code itself tell me how the design should go.
This approach works best on small-medium size Rails applications without extensive, complex domain logic.
Great questions and great discussion. #ryanb and #bkeepers mention that they only write two types of tests. I take a similar approach, but have a third type of test:
Unit tests: isolated tests, typically, but not always, against plain ruby objects. My unit tests don't involve the DB, 3rd party API calls, or any other external stuff.
Integration tests: these are still focused on testing one class; the differences is that they integrate that class with the external stuff I avoid in my unit tests. My models will often have both unit tests and integration tests, where the unit tests focus in the pure logic that can be tested w/o involving the DB, and the integration tests will involve the DB. In addition, I tend to test 3rd party API wrappers with integration tests, using VCR to keep the tests fast and deterministic, but letting my CI builds make the HTTP requests for real (to catch any API changes).
Acceptance tests: end-to-end tests, for an entire feature. This isn't just about UI testing via capybara; I do the same in my gems, which may not have an HTML UI at all. In those cases, this exercises whatever the gem does end-to-end. I also tend to use VCR in these tests (if they make external HTTP requests), and like in my integration tests, my CI build is setup to make the HTTP requests for real.
As far as mocking goes, I don't have a "one size fits all" approach. I've definitely overmocked in the past, but I still find it to be a very useful technique, especially when using something like rspec-fire. In general, I mock collaborators playing roles freely (particularly if I own them, and they are service objects) and try to avoid it in most other cases.
Probably the biggest change to my testing over the last year or so has been inspired by DAS: whereas I used to have a spec_helper.rb that loads the entire environment, now I explicitly load just the class-under test (and any dependencies). Besides the improved test speed (which does make a huge difference!) it helps me identify when my class-under-test is pulling in too many dependencies.
I'm currently looking at a hefty Rails test suite. It's nothing I can get into specifics about, but the run time for the entire suite (unit/functional/some integration) can run upward of 5 minutes.
We're completely reliant on fixtures and are we're not mocking and stubbing as much as we should be.
Our next few sprints are going to be completely focused on the test suite, both improving coverage, writing better tests and most importantly writing more efficient tests.
So aside from more mocking and stubbing within our tests, we're considering replacing our fixtures with most likely Factory Girl. I see a lot of happy folks doing similar situations but haven't been able to find a good resource on any minuses of moving to a factory. I have seen some slower benchmarks when using benchmarks from various resources but cannot find a definitive this why factories are good and this is why you might not want to use them.
Can anyone educate me on why or why I shouldn't be using factories?
Thanks!
Oleg's answer is great, but let me offer the perspective of someone who is using both.
Fixtures have sort of been the whipping boy of the Rails community for a while. Everyone understands the drawbacks of fixtures, but no one is really championing their strengths. In my experience, factories by themselves can easily become just as difficult to maintain as fixtures (it really depends on the schema, but I digress). The real strength of factories is in selective replacement of fixture-based pain. Let's talk about a couple specifics.
The first issue is performance. If you can test most of your app without hitting the database then you will see a significant speed up, but for most applications I don't think it's wise to test without hitting the database entirely. At some point you want to test the whole stack. Every time you mock or stub you are making an assumption about an interface that may contain subtle bugs. So, assuming that you need to hit the database on some significant percentage of tests, transactional fixtures (you are using transactional fixtures right?) could well be much much faster than instantiating a whole environment for every test.
I'd say, with the size of your test suite that you really need to look towards Continuous Integration to scale your development to the next level. No matter how much you speed them up, it's still a long time for developers to wait. Maybe look at autotest as well to help at the individual level. But ultimately CI is going to allow you to maintain testing discipline without sacrificing developer agility.
The place where fixtures really shine is in functional/integration testing. The way I look at it is that the fixtures should set up a healthy base state for the app to be tested. Most unit tests don't really need this. You can get very good unit coverage using factories. However when it comes to functional testing, any given page may be hitting dozens of models. I don't want to set up all that stuff in each test. As I construct ever more complex scenarios, I'm getting closer and closer to recreating a global data state which is exactly what fixtures were designed to do in the first place.
One controversial belief I hold is that all else being equal, I prefer one functional test to 20 unit tests (using Rails parlance). Why? Because the functional test proves that the end result that is sent to the user is correct. The unit tests are great for getting at nuances of functionality, but at the end of the day, you could still have a bug along an interface that breaks your entire site. Functional tests are what give me the confidence hitting deploy without actually loading up the page in my browser. I know that I could stub everything out and test both interfaces and get the same coverage, but if I can test the whole stack in one simple test at the expense of a little CPU, I'd much rather do that.
So what are my best practices for fixtures?
Set up a handful for every model to cover the broadest categories of data
When adding a major new feature that cuts across many models and controllers, add some new fixtures to represent the major states
Avoid editing old fixtures except for adding/removing fields
Use factories for more smaller/more localized variations
Use factories for testing pagination or other mass creation that is only needed for a few tests
Also, let me recommend Jay Fields' blog for really good pragmatic testing advice. The thing I like most about Jay's blog is that he always acknowledges that testing is very project-specific, and what works for one project does not necessarily work for another. He's short on dogma and long on pragmatism.
There could be some issues with setting up all dependencies between entities for good test suite. Anyway, it's still much easier than maintaing a lot of fixtures.
Fixtures:
hard to maintain relationships (especially many-to-many);
test suite runtime is usually slower due more DB hits;
tests are very sensitive to changes in schema.
Factories:
you stub everything you don't test at current unit test;
you prepare entities that you are testing with factories. This is where factories show their real advantage — it's easy to set up new test cases, as you don't need to maintain a ton of YAML-files for that;
you concentrate on testing. If tests require changing scenario, you don't shift your mindset. As long as stubs are reasonable and factories are easily customized, you should be fine.
So, factories seem a good way to go. The only possible drawbacks I see, are:
time you spent migrating from fixtures;
keeping a sane set of scenarios can require some effort.