I would like to have a varying number of records saved to the database for different tests. The purpose of one test would be to see a method's behavior with only one record in the database while another might be to see how the method behaves with ten records. I understand Fixtures are the preferred way to load data into the testing database, but as far as I know, Fixtures are static and can't be molded for a specific test. I am therefore, using CRUD operations in some of my tests. Most of these tests begin by deleting all the records from the table being tested then adding records that suit the specific test.
The tests that rely on CRUD operations and do not use Fixture data I have placed at the end of my tests so they don't interfere with the tests that use Fixtures. I am new to testing in Rails and would like to know if bypassing Fixtures and using CRUD operations in unit tests is a good/bad practice?
You'll see all kinds of answers, from using fixtures to factories to POROs ("Plain Old Ruby Objects"), but I think in general you want to minimize or eliminate database interaction as much as possible when working at the unit testing level, while leaving that more for integration or acceptance tests.
While unit testing, it might be helpful to think about inputs and outputs, and how/what messages are sent to and from each "unit" under test.
There are plenty of books, articles, and posts out there on this topic, but in the end I'd recommend a balance between practicality and the ideal worldview some of these present.
You might find this interesting:
https://www.youtube.com/watch?v=qPfQM4w4I04
Related
TLDR: What is the best approach to seed global data with FactoryBot?
Currently, we have a fairly large application and using FactoryBot for most tests. However, we are still using fixtures to seed some default data in a database, for CRUD tests that need to talk to the DB, and can't be "stubbed".
Using fixtures to seed data, lets us shave off 50% of the time during setup/teardown for things that are common for almost all CRUD tests.
Also for some business rules validations, we need to use actual data, not 'faked' from Faker.
Examples of the seed model/fixtures:
ShippingMethod (FedEx, UPS, USPS, etc. )
User (user record is used in almost every other model)
Warehouse (we only have 2)
Setting ( some global app settings)
Actual Question
What is the best practice to use inside FactoryBot to move away from using fixtures completely, so that we maintain test data in one spot, or this a good use-case for fixtures to be used with factories?
Keep in mind that since tests are run in parallel, seed data needs to be available to all tests throughout the run of the test suite.
Thank you for any suggestions.
When writing specs for a simple Rails app, is the following a correct approach for full test coverage?
Write feature specs for all user stories
Write controller specs to ensure that individual action responses are correct and all required variables are set
Write model specs to ensure all methods, validations,e tc. are working as intended
Write mailer specs
Write routing specs
Is this enough, too much (e.g. can I skip some lower-level specs if I've written feature specs), or not enough? Why?
You don't need to write specs for every object in every layer either to get 100% test coverage or to test-drive (require you to implement) all of the important behavior in your application. Instead, as behavior-driven development (BDD) advises, write specs outside in, and write lower-level specs only as necessary.
The most important measure of test completeness is requirement coverage: it's helpful for each user story, and each detail of each story that requires new code, to be represented in at least one test. If you're following typical agile practices (mentioning user stories suggests that you are) your tests are probably the only place where you record your requirements, so you probably can't put a number on this kind of coverage. It's also helpful to have
line coverage (what most people mean when they say test coverage), meaning that every line of code is exercised by at least one test, and
integration coverage, meaning that every method call from one class to another is exercised by at least one test.
For each story,
Write only the feature specs that will test-drive all of the story's distinct happy paths.
Write additional feature specs to ensure integration coverage of architecturally interesting minor variations of happy paths and of sad paths. For example, I often write three feature specs for a story that involves a form: one where the user fills in every possible field and succeeds, another where the user fills in as little information as possible and still succeeds (ensuring that unspecified values and defaults work as intended), and one where the user makes a mistake, fails, corrects the mistake and succeeds.
At this point you've already test-driven every layer (controllers, models, views, helpers, mailers, etc.) into existence, with only feature specs.
Write model and helper specs to drive out detailed requirements which live entirely in those classes. For example, once you've written a single sad-path feature spec that establishes that entering one particular invalid attribute sends the user back to edit their form submission and displays a message, you can handle other invalid attributes entirely by writing more examples in that model's spec that test that model attributes are validated, and let the architecture that you've already test-driven propagate the errors back to the user.
Note that although your feature specs already test the happy paths through model and helper methods, as soon as you start writing examples for a method for minor or error cases, you'll probably want to write the happy-path example or examples for that method too, so you can see the entire description of the method in one place, and so you can test the method fully just by running all its examples and not also have to run any feature specs.
You might not need some kinds of specs at all:
Well-factored controller actions are short and have few or no conditionals, so you often won't need any controller specs at all. Write them only when needed, and stub out model, mailer, etc. behavior to keep them simple and fast.
Similarly, views and mailers should have few or no conditionals (complex code should be refactored into helper and model methods), so you often won't need view or mailer specs at all.
Your feature specs will have test-driven all the routes you need, so you probably won't need routing specs. I've only ever gotten use out of routing specs when I had to do a major refactor of routes, as when upgrading from one major version of Rails to the next.
As long as you always write a test before you write new code, you'll always have 100% line coverage.
That testing strategy sounds really comprehensive. If you had all of these tests in place you would have great test coverage. However it would take you longer to deliver your project. You would also not be agile as someone who is doing more limited testing. Testing has to suit the project. Don't over test. Over testing can cost time and money. Don't under test. Under testing can cost time and money.
There are right ways to do unit testing. There are right ways to do integration testing. The glove has to fit. If your application is largely front end facing then perhaps it's best to start with integration tests. If your writing a back end application or perhaps an API then unit tests maybe a better place to start. I think approaching with one style of testing and then expanding to different styles is a better start than to try and test every layer of your application.
Why not start with simple unit tests? They are easy to write. Write these tests and then track how many bugs you ship. Are you letting in too many bugs? Are you having a lot of regression issues? Are there bugs that are getting through to production that your suite is not picking up? If the answer is yes then maybe it's time to write some higher level tests. Remember the higher level a test is the more development cost you will have to pay.
If your not shipping bugs then you have no reason to write any more tests. Remember the end goal here. We want to ship bug free code. If we can write one test and one test alone that will ensure we are doing this then there is no reason to test any further.
I'm new to testing in rails and I don't understand why or when I should use fixtures (or factory) rather than just seeding my test db and querying it to run the tests?
In many cases, it should be faster and easier to have the same data in dev and test env.
For example, if I want to test an index page, should I create 100 records via a factory or should I seed the db with 100 records?
I someone could clarify this, it would be great.
Thanks!
This is actually a deeper question of how to test efficiently, and you will find a lot of different opinions.
The reason to avoid a database in your unit tests is merely speed. Database operations are slow. It might not seem slow with one test, but if you have continuous integration going (as you should) or when you made a quick change and just want to see what happens, those delays add up. So prefer mocks to truly unit test code.
Then your own integration tests should hit an in-memory database rather than your real database--for the same reason, speed. These will be slower than your mocked tests, but still faster than hitting the real database. When you're developing, the build-test-deploy cycle needs to be as fast as possible. Note that some people call these unit tests as well. I wouldn't, but I guess it is just semantics.
These first two kinds of tests are by developers for developers.
Then the testers will hit the real database, which will be populated with test data defined by the testers and subject-matter experts. There are lots of clever ways to speed this up as well, but this will be the place where they test the integration of your code with the production-like database. If all your in-memory database tests passed and something goes wrong here, then you know it has to do with something like database configuration, vendor-specific SQL, etc. rather than something fundamentally bad. You will also get your first taste of what the performance is like.
Note that everything I've said here is a matter of debate. But hopefully it clarifies what you should consider about when to do certain things and why.
Besides the fact that Rails incorporates the database layer into the unit tests (which then is strictly not a unit test), what if I test for model interdependencies, e.g. check if has_many/belongs_to with :dependent => :destroy really destroys the associated items or items from join models to not leave orphans around?
Should such a test maybe better be placed in the function or integration tests?
Or asked another way: Is there a guide with examples which kind of tests should go where and why? I didn't find anything really useful.
As a general rule of thumb models are unit tested, controllers are functionally tested and integration tests are done with views. If you're using RSpec you'll have model, controller and view specs - but essentially testing the same stuff.
You can always mock actions (with gems like Mocha or Flexmock) to avoid having calls to methods actually do anything - but you want to make sure the expected behavior is happening. (E.g. a call is made to MyClass.create or MyClass.destroy). It is especially true of controller/view tests where I often will mock a call to a model.
In my experience though I don't usually have problems with models and associations using the database. You might want to look at Factory Girl or some alternative to fixtures, which make it easier to manage test objects/data.
A unit in a unit-test isnt always a single class or object. Testing interdependencies shouldn't be a problem. I havent found any hard and fast rules for defining units in unit tests but I often use ideas from domain driven design to guide me. Tests that test behaviour of a single aggregate are usually pretty maintainable as unit tests. Test that depend on more aggregates or even aggregates and services together are integration tests.
I'm currently looking at a hefty Rails test suite. It's nothing I can get into specifics about, but the run time for the entire suite (unit/functional/some integration) can run upward of 5 minutes.
We're completely reliant on fixtures and are we're not mocking and stubbing as much as we should be.
Our next few sprints are going to be completely focused on the test suite, both improving coverage, writing better tests and most importantly writing more efficient tests.
So aside from more mocking and stubbing within our tests, we're considering replacing our fixtures with most likely Factory Girl. I see a lot of happy folks doing similar situations but haven't been able to find a good resource on any minuses of moving to a factory. I have seen some slower benchmarks when using benchmarks from various resources but cannot find a definitive this why factories are good and this is why you might not want to use them.
Can anyone educate me on why or why I shouldn't be using factories?
Thanks!
Oleg's answer is great, but let me offer the perspective of someone who is using both.
Fixtures have sort of been the whipping boy of the Rails community for a while. Everyone understands the drawbacks of fixtures, but no one is really championing their strengths. In my experience, factories by themselves can easily become just as difficult to maintain as fixtures (it really depends on the schema, but I digress). The real strength of factories is in selective replacement of fixture-based pain. Let's talk about a couple specifics.
The first issue is performance. If you can test most of your app without hitting the database then you will see a significant speed up, but for most applications I don't think it's wise to test without hitting the database entirely. At some point you want to test the whole stack. Every time you mock or stub you are making an assumption about an interface that may contain subtle bugs. So, assuming that you need to hit the database on some significant percentage of tests, transactional fixtures (you are using transactional fixtures right?) could well be much much faster than instantiating a whole environment for every test.
I'd say, with the size of your test suite that you really need to look towards Continuous Integration to scale your development to the next level. No matter how much you speed them up, it's still a long time for developers to wait. Maybe look at autotest as well to help at the individual level. But ultimately CI is going to allow you to maintain testing discipline without sacrificing developer agility.
The place where fixtures really shine is in functional/integration testing. The way I look at it is that the fixtures should set up a healthy base state for the app to be tested. Most unit tests don't really need this. You can get very good unit coverage using factories. However when it comes to functional testing, any given page may be hitting dozens of models. I don't want to set up all that stuff in each test. As I construct ever more complex scenarios, I'm getting closer and closer to recreating a global data state which is exactly what fixtures were designed to do in the first place.
One controversial belief I hold is that all else being equal, I prefer one functional test to 20 unit tests (using Rails parlance). Why? Because the functional test proves that the end result that is sent to the user is correct. The unit tests are great for getting at nuances of functionality, but at the end of the day, you could still have a bug along an interface that breaks your entire site. Functional tests are what give me the confidence hitting deploy without actually loading up the page in my browser. I know that I could stub everything out and test both interfaces and get the same coverage, but if I can test the whole stack in one simple test at the expense of a little CPU, I'd much rather do that.
So what are my best practices for fixtures?
Set up a handful for every model to cover the broadest categories of data
When adding a major new feature that cuts across many models and controllers, add some new fixtures to represent the major states
Avoid editing old fixtures except for adding/removing fields
Use factories for more smaller/more localized variations
Use factories for testing pagination or other mass creation that is only needed for a few tests
Also, let me recommend Jay Fields' blog for really good pragmatic testing advice. The thing I like most about Jay's blog is that he always acknowledges that testing is very project-specific, and what works for one project does not necessarily work for another. He's short on dogma and long on pragmatism.
There could be some issues with setting up all dependencies between entities for good test suite. Anyway, it's still much easier than maintaing a lot of fixtures.
Fixtures:
hard to maintain relationships (especially many-to-many);
test suite runtime is usually slower due more DB hits;
tests are very sensitive to changes in schema.
Factories:
you stub everything you don't test at current unit test;
you prepare entities that you are testing with factories. This is where factories show their real advantage — it's easy to set up new test cases, as you don't need to maintain a ton of YAML-files for that;
you concentrate on testing. If tests require changing scenario, you don't shift your mindset. As long as stubs are reasonable and factories are easily customized, you should be fine.
So, factories seem a good way to go. The only possible drawbacks I see, are:
time you spent migrating from fixtures;
keeping a sane set of scenarios can require some effort.