I've had a weird event of having rake test and rake test:functionals producing different results on a Rails 3 app that uses fixtures to insert test database records. The first group was failing on one of the controller tests, while the other one was successfully passing.
I also tried to run the failing test with ruby -I test path/to/file and zeus test path/to/file, but those runs were successful. The issue was reproduced on different machines. The test was failing because of the state of the test database, which seemed impossible to me when I looked at the fixtures.
Strangely enough, the issue disappeared the next day and I no longer can reproduce it. What could be the cause of such a problem and how to avoid it?
Related
Is anyone able to explain what this actually means? The documentation seems to say that it is similar to setting an order (--seed 123 # same as --order rand:123), but from what I can tell doesn't seem to go into it any further. I'm assuming it's not related to database seed data, but I could well be wrong.
In the default configuration, RSpec runs its tests in random order.
This is considered a good practice because tests should be independent of each other. Running them in a random order helps to find tests that only pass when they are run in a specific order and fail in another order.
But the problem is: When RSpec runs the test in random order and then fails, how can you re-run the test in the exact same order again to debug the issue?
That can be done by telling RSpec to use the same seed for its randomness as it was used before.
RSpec tells you this seed when it is starting:
$ rspec spec
Randomized with seed 48111
.....*.........
To re-run the specs in the exact same order run:
$ rspec spec --seed 48111
Reading about Random#seed might be interesting in this context.
The first couple times I run rspec spec, I receive failures and if I run it again and thereafter, it passes. Why is this happening and how do I fix it?
This appears to be related to uniqueness code to prevent adding a record with a name that has already been. Below is the test that consistently fails the first time:
it { should validate_uniqueness_of(:name).case_insensitive.with_message(/has already been taken. Please use a different name./) }
The full repo can be found here: https://github.com/melissajstudent/koth
You are right in thinking that the uniquess is a good indicator of what’s happening. Likely your test database has some remaining records from the last time you ran the tests and is trying to creat records with the same factories.
To that end, there are a few ways to clean your database in between runs of your test suites. Sometimes it’s helpful just to run bundle exec rake db:test:prepare Before running your test suite. In other cases you can implement some more robust database cleaning throughout your run with the database cleaner gem: https://github.com/DatabaseCleaner/database_cleaner
I start with a test database containing the schema but with no data in the tables. I run a test like so
cd test/
ruby unit/directive_test.rb
I get failures indicating that the code found no data in the data tables. However, I look at the tables after running that test and the data is now in the the table. In fact, if I immediately run the test again I get no failures.
So it appears that the fixture is being loaded into the table too late for one of my modules to find it.
When are the fixtures loaded? After or before the app/model/*.rb files are executed?
If it is after the models are executed is there a way to delay the loading?
This issue is also relevant when running rake test:units since that task clears the test data after it finished.
first of all see this thread and see if it can help you.
if you run the rail task rake test:units it will for sure load all fixtures before you run your code. if you are running just the test, and your unit test has no reference to the test_help.rb probably it is not loading the fixtures. You should try to run it through the rake tasks.
Another tip that i give you is that you forget the fixtures and use factories (here i recommend factory_girl). It takes sometime to get used, but it worth. Fixtures are too hard to manage, update and etc.
there is another post explaing little about the concept behind factories.
When creating a record in a unit test, I'm getting an 'ActiveRecord::RecordInvalid' when it's run with rake test:units. I don't get this error when I run the unit test manually with ruby ('ruby unit/blah_test.rb'). I've narrowed the issue down to a validation that exists in my model. It is an inclusion validation which is actually performing a find against a different model to get the valid values. I'm thinking this is related to the fixtures not being loaded in time, but I do have 'fixtures :all' in my test_helper.rb (I've also tried including 'fixtures :all' in the unit test itself.
Does anyone have any suggestions on how I can try to narrow this down even further?
Thanks.
Certainly what you are seeing would fit with fixtures being missing. With rake test:units the test database schema will be set up (and cleared) so your included model's fixtures may not be populated. With the direct call you'll be using the test database in the state you last left it which probably does include the fixutres for the included model.
Is there another call to fixtures in the test class which may be causing fixtures :all not to kick in?
You could try doing rake db:test:prepare prior to running your test via ruby which would mean you were running on a fresh test database. This would further highlight if the fixtures aren't being loaded for your included model.
When I want to run all my unit tests, I run rake test:units. To run all my functional tests, I run rake test:functionals. If I want to run all the test cases in one file, I run
ruby test/unit/username_test.rb
A few people have been telling me I should run rake instead such as
rake test:units TEST=test/unit/username_test.rb
For running tests, they say I should always run rake. I know I should run rake if I'm testing all my unit tests. But what if it's just one file or one particular test method in a file that I'm testing? Should I still use rake? Is there any difference between the two? Do I get any benefit from running rake over ruby? Is there any disadvantage to running ruby rather than rake?
Sadly, the two are not the same. Running the tests under rake can wind up pulling things from different places than when you run the test directly (more a problem when you have multiple versions of gems, etc. on your system).
The intent is that tests run under rake should be in an environment that matches what rails would produce; I can not attest to how closely they match, but I have seen tests that passed when run directly but failed when run via rake or rails (and visa versa).
Before checking in at the very least I'd recommend running rake to hit everything, in order to be assured that nothing unexpected has broken.
Plain ruby seems ideal for fast testing of single files during iterations.
Be aware that running everything through rake can produce different results to running everything individually, as I found to my confusion recently - I was doing something slightly wrong in one test that worked successfully in isolation but that left a problem lying around for a subsequent test that only showed up when I used rake.
No I dont think so. Rake seems to be a convenient way to run all tests, all unit tests or all functional/controller tests.
For a single file, I use the ruby object_test.rb approach.. shorter and works fine for my rails home project.
They should be identical. if they are not, you're definitely doing something wrong.
As I said in my other comments, if you get tests that pass in one, but fail in the other, you're doing something very wrong; this indicates a poor test setup, and is usually caused by a different test run order between the two test approaches; one of which causes tests to fail.
Usually the cause of this is that you are not using transactions and/or tests are not cleaning up after themselves. For example, not properly requiring the fixtures they later test for, and instead relying on the pre-existing database state.
you are free to use either method. if something breaks, you're doing something wrong in your tests, and you should fix your code.
The two are not the same. Rake will do some preliminary test loading.
The intent is that tests run under rake should be in an environment that matches what rails would produce;
One difference I've noticed is with rake some of the fixture loading happens which could be by-passed with ruby.
I'd recommend using rake, unless you are using the ruby command line to one just one test in the file with the -n option.