For my tests I'm using the combination of the latest Rspec, FactoryGirl, Guard against a local SQLite database. When I'm developing BDD-style, I will often write a failing test that has the likelihood of leaving some garbage data behind in the DB, I think especially if there's an interpreter error.
I'm often forced to delete the DB and re-migrate in order to restore it to a clean state, which is time consuming and a pain. It seems that most of the times Rspec's own rollback mechanism does a fine job, but sometimes it's not enough.
What is the proper way of handling this situation? Should the DB be wiped every single time a single test is being executed? Is database_cleaner going to do a better job than Rspec?
Thanks!
Three ways spring to mind.
One, wrap your tests in a transaction, and rollback at the end.
Two, use an in-memory DB that can be created/torn down very quickly
Three, Mock out your DB connection entirely and perform validation based on DB calls that are made.
Related
I've got a fairly big scary legacy app with no tests that I'm trying to build some tests into. My problem is that the schema's rather large, and to drop the database and reload it takes 56 seconds. to run all my tests (so far) takes 2. I'm using transactional fixtures, it runs each test without reloading the db, I don't see why I shouldn't be able to load the test environment once, build the db, then run tests over and over without needing to drop the db and rebuild? a 1 minute test cycle doesn't sound like much, but it really adds up. transactional fixtures should ensure the db doesn't get muddy ya?
Trying to figure it out, I didn't see anything that would do what I wanted, and before diving into the rake gem to try and modify the rake test task, I figured I'd ask, since i get the feeling I'm doing it wrong.
Thanks!
Don't drop the database; you're shouldn't be testing database creation, so doing so is often a TDD anti-pattern. Instead, truncate your tables. You can do this in a number of ways, but a very common way is with the DatabaseCleaner gem.
This is my env.rb
require 'cucumber/rails'
ActionController::Base.allow_rescue = false
Cucumber::Rails::World.use_transactional_fixtures = false
After running cucumber Its cleaning data from database. How to prevent data erasing
You want cucumber scenarios to run with some sort of database cleaning otherwise every scenario becomes dependent on previously run scenarios, and you scenarios will start to fail in all sorts of strange ways that are very hard to debug.
If you are writing scenarios that are dependent on previous scenarios, you should stop that right now. If you are trying to pre-populate data for a test, try doing this in a Given, e.g.
Given the database is pre-populated
If that is too slow you have a couple of choices
Improve how you prepoulate the data, e.g. load a SQL dump of a saved database
Write scenarios that a less dependent on pre-populated data
Write fewer scenarios that are dependent on pre-populated data (perhaps you can unit test lower level components).
Its a fundamental part of all automated testing, that you start each test/scenario from a stable, constistent, repeatable starting point.
We're using Grails as our web framework and we have integration tests that we want to have isolated. One idea that was brought up is to drop the database after every test case. What are alternative and more applicable ways to achieve test isolation.
we were doing two things:
rollback transaction. this deals with most of the problems (DML) but not all (for example sequences). also sometimes you need to commit a few transactions to make a proper tests. that's why we:
reset the database. you can achieve it in many ways: use frameworks that cleans data (db-unit), do manual reverts of inserted data, drop db or, as we did, truncate all tables and run custom script (if you need to reset sequences or something else). it's much faster than dropping and recreating db. it worked really good
You probably don't need something that aggressive. Each integration test is run in a transaction that's rolled back at the end of the test, so you shouldn't see anything from previous tests. There are cases where you want to commit a transaction though, so you can disable the automatic transaction and rollback and manage things yourself. That will increase the likelihood of data pollution between tests but it's easy enough to fix.
Since the database is in-memory initially, it's relatively inexpensive to drop the whole database and re-create all of the tables and objects, but this will be significantly slower if you move to a "real" database, e.g. a local install of Postgres to ensure that tests are running as close to how the code will run in production as possible.
I have some code that uses an Informix 11.5 database that I want to run some tests against.
If the tests fail they will often leave the database in an inconsistent state that needs to be manually resolved before the tests can be run again.
I would like to automate this so that the tests do not require manual intervention before running the tests again.
My current solution is to write some code that does the cleanup, but this means the code must be maintained whenever potential new inconsistent states can occur in new features.
The code runs a lot of stored procedures, which themselves often use transactions. As Informix does not support nested transactions I can't just wrap up all the work in one big transaction.
Is there another way to create a checkpoint which I can restore the database back to?
You could create a virtual machine with an undo disk and after you run the test you can close the virtual machine without saving the changes. It's equivalent to like you never ran the tests!
If this is a development only server, how about taking a Level 0 ontape system archive before the test? I think this can be done via the sysadmin functions too (not sure though), so it can be automated. After the tests you just restore the archive.
Changing database state - and resetting it back to a known state - is one of the reasons that the Unit Test community spends time and effort avoiding testing against databases. It is a tough problem.
Informix 11.50 does support savepoints; however, it does not support one BEGIN WORK after another without an intervening COMMIT or ROLLBACK.
To the extent possible, have the tests create and load a set of tables with the known data. One way of achieving that is to create a whole new database for the test. However, this is only borderline feasible if you need to test with high volumes of data.
I don't think this issue is in any way unique to Informix - it is a general problem with testing DBMS operations.
What data do you use with Selenium tests on Rails apps? Do you load from fixtures? Use an existing dev db? Use a separate (non-fixture) db?
I'm considering my options here. I have a Rails app with a large Selenium test suite that runs on a modified version of Selenium Grid. Part of the process, right now, is loading a large set of fixtures, once, before the test suite runs. It's a LOT of data. Most of it is reporting info exported from our production db. When I set it up originally, I exported the data to yaml from Oracle.
Now there's been a schema change in some of the reporting tables, so of course I have to regenerate the fixture data. There is so much of it that it's not worthwhile to edit the files by hand. But it seems inefficient to have to regenerate for every little schema change - not to mention that it's yet another step to remember to do. Is there a better way?
EDIT: I originally intended to load the fixtures before each test and unload them after each test, like regular Rails tests. But it takes about 15 minutes to load the fixtures due to this reporting data. There are 200+ tests, and the suite runs every 12 hours. I cannae bend spacetime captain!
EDIT 2: I also agree that having this big set of fixtures is a bad smell. I'm not sure how to pare it down, though, because the reports aggregate a lot of data and much of the value of the selenium tests is that they test the reports.
Even if it's a small set of data, though...it's still another set to keep co-ordinated with schema changes. (We have a separate, smaller set for unit, functional, and [Rails] integration tests.)
Which brings me back to my original question - are there other options besides doing it by hand, or remembering to regenerate them each time?
If you can, the best possible thing to do is have a system in which each Selenium test gets it's own data state (ie: DB tables dropped and recreated, bootstrap data re-inserted, and caches cleared). This is easier said than done and usually is only possible if the project planned for it from the start.
The next best thing is to have a consistent DB state for each test suite/run. This is not as nice since there is now a strong chance that some tests will depend on the success of previously run tests, making it more difficult identify true failures vs. false negatives.
The worst case, IMO, is to use a static DB in which each test run mutates the date. This almost always leads to problems and is usually a "project smell". The key to doing it the "right way" (again, IMO) is to be vigilant about any state/schema change and capture it as part of the automated test/build process.
Rails does a good job with this already with Migrations, so take advantage of them! Without knowing your situation, I'd generally question the need to run Selenium tests against a snap of the full DB. Most DBs can (or should) be distilled down to less than 1MB for automated testing, making automated schema migrations and data reset much more efficient.
The only time I've seen a "valid" reason for massive DBs for Selenium tests is when the DB itself contains large chunks of "logic data" in which the data affects the application flow (think: data-driven UI).
I think you're asking two questions here that are intertwined so if I'm to break it down:
You want to get test data into and out of your DB quickly and fixtures aren't doing it for you.
You've been burnt by a schema change and you want to make sure that whatever you do doesn't require eight iterations themed "fiddling with the test data...still" :)
You've got a couple of alternatives here which I've hashed out below. Because you've mentioned Oracle I'm using Oracle technologies here but the same thing is true for other DB platforms (e.g. Postgresql):
Rake tesks that call PL/SQL scripts to generate the data, nasty horrible evil idea, don't do it unless there's no other option. I did it on one project that needed to load in billions of rows for some infrastructure architecture tests. I still sulk about it.
Get your DB into a dump format. For speedy binary dumps check out the exp/imp and data pump utilities. This will allow you quick setup and teardown of your DB. Certainly on a rails project I worked on we used rake tasks to exp/imp a database which had around 300k records in under a minute. Also check SQLLoader which is the logical dump utility, as its logical its slower and requires you to have control scripts to help SQLLoader understand the dumps. However, the benefit of the logical dump is that you can run transformation scripts over them to massage the data into the latest format. Sadly though just like fixtures all these tools are pretty sensitive to change in the schema.
Use a plugin such as Machinist or Factory Girl to make the generation of the data nicer. You still incur the penalty of using ActiveRecord to setup the DB but these fake object generators will help you stay close to you migrations and are a lot less hassle to maintain than fixtures.
Combine approaches 2 and 3. What happens here is that you make some test data with say Machinst. You export that test data to a dump and then reload the dump during each test run. When the schema changes update the Machinist config and re-export.
Hope that helps.
I'm currently on a project with an enormous Selenium test suite--actually, the one Selenium Grid was written for--and our tests use a small amount of reference data (though we don't use Rails YAML fixtures) and object factories for one-off data needed for particular tests.
Alternatively, on many of the ThoughtWorks Rails projects I've been on we've written checkin scripts that incorporate a number of pre-commit hooks--for example, running the tests before allowing a commit. One thing you might consider trying is writing (or customizing) a similar checkin script that will check for schema changes and reload the reference data as needed.
See e.g. Paul Gross's rake commit tasks on Github.