I'm writing a lot of request specs right now, and I'm spending a lot of time building up factories. It's pretty cumbersome to make a change to a factory, run the specs, and see if I forgot about any major dependencies in my data. Over and over and over...
It makes me want to set up some sort of sandboxed environment, where I could browse the site and refresh the database from my factories at will. Has anyone ever done this?
EDIT:
I'm running spork and rspec-guard to make this easier, but I still lose a lot of time.
A large part of that time is spent waiting for Capybara/FireFox to spin up. These are request specs, and quite often there are some JavaScript components that need to be exercised as well.
You might look at a couple of solutions first:
You can run specific test files rather than the whole suite with something like rspec spec/request/foo_spec.rb. You can run a specific test with the -e option, or by appending :lineno to the filename, where lineno is the line number the test starts on.
You can use something like guard-rspec, watchr, or autotest to automatically run tests when their associated files change.
Tools like spork and zeus can be used to preload the app environment so that test suite runs take less time to run. However, I don't think this will reload factories so they may not apply here.
See this answer for ways that you can improve Rails' boot time. This makes running the suite substantially less painful.
Related
I've got a fairly big scary legacy app with no tests that I'm trying to build some tests into. My problem is that the schema's rather large, and to drop the database and reload it takes 56 seconds. to run all my tests (so far) takes 2. I'm using transactional fixtures, it runs each test without reloading the db, I don't see why I shouldn't be able to load the test environment once, build the db, then run tests over and over without needing to drop the db and rebuild? a 1 minute test cycle doesn't sound like much, but it really adds up. transactional fixtures should ensure the db doesn't get muddy ya?
Trying to figure it out, I didn't see anything that would do what I wanted, and before diving into the rake gem to try and modify the rake test task, I figured I'd ask, since i get the feeling I'm doing it wrong.
Thanks!
Don't drop the database; you're shouldn't be testing database creation, so doing so is often a TDD anti-pattern. Instead, truncate your tables. You can do this in a number of ways, but a very common way is with the DatabaseCleaner gem.
I have test suite for acceptance tests in my rails app that uses pure capybara (no cucumber).
It has 220 examples and it takes 21 minutes to finish. My non-js driver is rack_test and my js_driver is capybara-webkit instead of selenium.
I would like to improve test times, but i have no idea if there is a common bottle-neck in this kind of testing.
Some ideas i have/had:
Change capybara server. It was using mongrel as a fallback. The default is thin. I installed thin but i didn't get any speed improvement. Seem like thins advantage is concurrency, an tests dont have it.
Since I am cleaning the database between tests, before each example of a private part of my app (MOST of the examples are like this) I need to login. That mean it loggin the app 200 times. There is a way to mantain session between examples in order to avoid loggin again and again?
there are two things that come to my mind:
parallel_tests can improve your test-speed if you run multicore https://github.com/grosser/parallel_tests
providing a backdoor-login-route for your test-login can improve test-speed by bypassing the login-step
in general acceptance-tests are slow. that's why i use them only for testing critical user workflows. i try to keep my whole test-suite within a 5 minute range. i really think that it's critical for your application test suite to be fast. that's why i try to put a lot of logic outside of rails tests so that a test-run completes within a second or less.
I have a couple of complicated objects to stub out (instances of gems I use). Where can I centralize these stubs to make them available to all tests?
How can I programatically clear the DB between tests without rake:test? I want to quickly run individual tests through textmate, but doing so will error out since it doesn't clear the DB between tests
The tests run slow since it has to spin up a Rails instance. How to speed up the tests? Especially while writing the tests and wanting to quickly run changes
1) You can either put them in test_helper.rb to make them available to all tests or you could write your own module which contains those methods and then include that module in the tests that require those stubs.
2) You could put Model.destroy_all (or .delete_all if appropriate which would be quicker) in your test setup method to strip out those models that you are no longer interested in.
However, if you are running tests in transactions (and your database supports transactions) then you shouldn't need to clear out any data because the creation of the data and the test will run in a transaction which will then be rolled back clearing the data automatically.
3) Not so sure on this one. I had this problem a lot developing on Windows but not so much on *nix. You could look into some kind of continuous testing but there's still going to be a delay on feedback. It might be worth investigating what is causing the rails environment to be so slow starting - it might be something you can skip in your testing environment.
I am looking for a system to parallelise a large suite of tests in a Ruby on Rails app (using rspec, cucumber) that works using JRuby. Cucumber is actually not too bad, but the full rSpec suite currently takes nearly 20 minutes to run.
The systems I can find (hydra, parallel-test) look like they use forking, which isn't the ideal solution for the JRuby environment.
We don't have a good answer for this kind of application right now. Just recently I worked on a fork of spork that allows you to keep a process running and re-run specs or features in it, provided you're using an app framework that supports code reloading (like Rails). Take a look at the jrubyhub application for an example of how I use Spork.
You might be able to spawn a spork instance for your application and then send multiple, threaded requests to it to run different specs. But then you're relying on RSpec internals to be thread-safe, and unfortunately I'm pretty sure they're not.
Maybe you could take my code as a starting point and build a cluster of spork instances, and then have a client that can distribute your test suite across them. It's not going to save memory and will still take a long time to start up, but if you start them all once and just re-use them for repeated runs, you might make some gains in efficiency.
Feel free to stop by user#jruby.codehaus.org or #jruby on freenode for more ideas.
What data do you use with Selenium tests on Rails apps? Do you load from fixtures? Use an existing dev db? Use a separate (non-fixture) db?
I'm considering my options here. I have a Rails app with a large Selenium test suite that runs on a modified version of Selenium Grid. Part of the process, right now, is loading a large set of fixtures, once, before the test suite runs. It's a LOT of data. Most of it is reporting info exported from our production db. When I set it up originally, I exported the data to yaml from Oracle.
Now there's been a schema change in some of the reporting tables, so of course I have to regenerate the fixture data. There is so much of it that it's not worthwhile to edit the files by hand. But it seems inefficient to have to regenerate for every little schema change - not to mention that it's yet another step to remember to do. Is there a better way?
EDIT: I originally intended to load the fixtures before each test and unload them after each test, like regular Rails tests. But it takes about 15 minutes to load the fixtures due to this reporting data. There are 200+ tests, and the suite runs every 12 hours. I cannae bend spacetime captain!
EDIT 2: I also agree that having this big set of fixtures is a bad smell. I'm not sure how to pare it down, though, because the reports aggregate a lot of data and much of the value of the selenium tests is that they test the reports.
Even if it's a small set of data, though...it's still another set to keep co-ordinated with schema changes. (We have a separate, smaller set for unit, functional, and [Rails] integration tests.)
Which brings me back to my original question - are there other options besides doing it by hand, or remembering to regenerate them each time?
If you can, the best possible thing to do is have a system in which each Selenium test gets it's own data state (ie: DB tables dropped and recreated, bootstrap data re-inserted, and caches cleared). This is easier said than done and usually is only possible if the project planned for it from the start.
The next best thing is to have a consistent DB state for each test suite/run. This is not as nice since there is now a strong chance that some tests will depend on the success of previously run tests, making it more difficult identify true failures vs. false negatives.
The worst case, IMO, is to use a static DB in which each test run mutates the date. This almost always leads to problems and is usually a "project smell". The key to doing it the "right way" (again, IMO) is to be vigilant about any state/schema change and capture it as part of the automated test/build process.
Rails does a good job with this already with Migrations, so take advantage of them! Without knowing your situation, I'd generally question the need to run Selenium tests against a snap of the full DB. Most DBs can (or should) be distilled down to less than 1MB for automated testing, making automated schema migrations and data reset much more efficient.
The only time I've seen a "valid" reason for massive DBs for Selenium tests is when the DB itself contains large chunks of "logic data" in which the data affects the application flow (think: data-driven UI).
I think you're asking two questions here that are intertwined so if I'm to break it down:
You want to get test data into and out of your DB quickly and fixtures aren't doing it for you.
You've been burnt by a schema change and you want to make sure that whatever you do doesn't require eight iterations themed "fiddling with the test data...still" :)
You've got a couple of alternatives here which I've hashed out below. Because you've mentioned Oracle I'm using Oracle technologies here but the same thing is true for other DB platforms (e.g. Postgresql):
Rake tesks that call PL/SQL scripts to generate the data, nasty horrible evil idea, don't do it unless there's no other option. I did it on one project that needed to load in billions of rows for some infrastructure architecture tests. I still sulk about it.
Get your DB into a dump format. For speedy binary dumps check out the exp/imp and data pump utilities. This will allow you quick setup and teardown of your DB. Certainly on a rails project I worked on we used rake tasks to exp/imp a database which had around 300k records in under a minute. Also check SQLLoader which is the logical dump utility, as its logical its slower and requires you to have control scripts to help SQLLoader understand the dumps. However, the benefit of the logical dump is that you can run transformation scripts over them to massage the data into the latest format. Sadly though just like fixtures all these tools are pretty sensitive to change in the schema.
Use a plugin such as Machinist or Factory Girl to make the generation of the data nicer. You still incur the penalty of using ActiveRecord to setup the DB but these fake object generators will help you stay close to you migrations and are a lot less hassle to maintain than fixtures.
Combine approaches 2 and 3. What happens here is that you make some test data with say Machinst. You export that test data to a dump and then reload the dump during each test run. When the schema changes update the Machinist config and re-export.
Hope that helps.
I'm currently on a project with an enormous Selenium test suite--actually, the one Selenium Grid was written for--and our tests use a small amount of reference data (though we don't use Rails YAML fixtures) and object factories for one-off data needed for particular tests.
Alternatively, on many of the ThoughtWorks Rails projects I've been on we've written checkin scripts that incorporate a number of pre-commit hooks--for example, running the tests before allowing a commit. One thing you might consider trying is writing (or customizing) a similar checkin script that will check for schema changes and reload the reference data as needed.
See e.g. Paul Gross's rake commit tasks on Github.