I have a large dictionary of 180,000 words that needs to be loaded into the database in order for my app to run and would be useful to test with. Unfortunately, this takes around 30 minutes to seed the database with. Is there anyway to seed the db just once, or to even seed just one table of the db and allowing other tables to be refreshed each run?
EDIT: I ended up using activerecord-import to greatly speed up the seeding process. It now takes 16 seconds and not half an hour. I also noticed that in my /spec/rails_helper.rb file I had the following:
config.before(:suite) do
Rails.application.load_seed # loading seeds
end
I obviously had added it a long time ago and had forgotten about it since this is kind of a template rails_helper file I use. Commenting that out meant I don't run it every time, and if I ever do need to reseed I could, just by uncommenting it.
For some reason I thought incorrectly that rspec just seeded by default, which is not the case.
You can make your seeding more efficient with the new Rails 6 insert_all. This creates multiple records with a single insert and does not instantiate models. OTOH it doesn't do any validation, so be careful.
DictionaryWords.insert_all([
{ word: "foo" },
{ word: "bar" },
])
Alternatively, use activerecord-import.
But it's better not to make 180,000 words at all.
The problem with seeds and fixtures is they're "one size fits all". They must encompass every possible dev and testing situation. They're fragile, one change to the seed might mysteriously break many tests which made assumptions about the fixtures. Seeds will be blown away if you need to reset your databases.
Instead, use a factory and create what you need when you need it. Use a library such as Faker to generate fake but valid data.
For example...
# Assuming you have classes called Dictionary and DictionaryWord
factory :dictionary do
end
factory :dictionary_word do
dictionary
word { Faker::Lorem.unique.word }
end
Then in your tests create words as you need. Here I'm using RSpec.
let(:dictionary) { create(:dictionary) }
let!(:words) { create_list(:dictionary_word, 3, dictionary: dictionary) }
context 'when the word is in the dictionary' do
let(:word) { words.sample }
it 'finds the word' do
expect( dictionary.exists?(word) ).to be_truthy
end
end
context 'when the word is not in the dictionary' do
let(:word) { "septemburary" }
it 'does not find the word' do
expect( dictionary.exists?(word) ).to be_falsey
end
end
And if you need more words for manual testing, open a console and make some.
[1] pry(main)> FactoryBot.create_list(:dictionary_words, 100)
This is not particularly efficient, but you probably don't really need 180,000 words.
I think you can run just once.
first time you should run seed for the test env
RAILS_ENV=test rails db:seed
Now run RSpec
Related
I have a DailyQuestionSet model and a method in it that (is supposed to) go to the database, see if a DailyQuestionSet already exists for that day, if so, return it, or if not, create a new one, save it to the database and return it.
This seems to work when I call it from a Controller but not from a Ruby on Rails automated test of the model. I am not sure if I'm going crazy or missing something.
When
class DailyQuestionSet < ApplicationRecord
def DailyQuestionSet.get_today_dailyquestionset
#dailyquestionset = nil
#questionlist = nil
#dailyquestionset_list = DailyQuestionSet.where('posed_date BETWEEN ? AND ?', DateTime.now.beginning_of_day, DateTime.now.end_of_day).all
if #dailyquestionset_list.empty?
#dailyquestionset = DailyQuestionSet.create!(posed_date: DateTime.now)
#dailyquestionset.save!
else
raise "disaster"
end
return #dailyquestionset
end
end
class DailyQuestionSetTest < ActiveSupport::TestCase
test "make sure today's daily_question_set has been deleted by the setup function" do
# later add function to delete today's daily_question_set so we can create it and then make sure get_today_dailyquestionset has created it once and then we can refer back to the same row
end
test "create daily_question_set and make sure it has questions" do
#dailyquestionset = DailyQuestionSet.get_today_dailyquestionset
....
end
test "create daily_question_set and make sure it has the same questions" do
#dailyquestionset = DailyQuestionSet.get_today_dailyquestionset
....
end
end
What I thought this would do is add a row to the daily_question_sets table in the database every time I run the first test, and then retrieve that row when I run the second test.
But when I look at the test database there's no row in there being created. I think maybe Rails is not committing the transaction to the database?
Or, put more simply, the raise "disaster" exception never gets thrown because get_today_dailyquestionset always returns a new DailyQuestionSet and never gets the one it (should have) created from the database.
I think I might be fundamentally misunderstanding testing in Rails. Should I be messing around with the DB in model tests at all?
Test database is erased on each run and each individual test runs it queries as transactions so you can't share objects from one test to the other.
If you need objects shared between tests use a setup block https://guides.rubyonrails.org/testing.html#putting-it-together or just run both queries one after the other on the same individual test.
It depends on how your tests are set up. Most likely, you have a separate test database for when the test suite is being run. Try running sqlite3 <app-name>_test, from the command line, if you are using the default Rails database solution. If you wanted to view the development database, you would run sqlite3 <app-name>_development.
I am using rails mintiest framework for writing test cases. I want to test the rails activerecord sql queries . To test activerecord queries, I am using rails fixtures that provide sample data for testing.
Initially, there were only two records in my fixture file users.yml as shown below:
user1:
name: 'xxx'
user2:
name: 'xyz'
Now with above data, I have written one test case that counts the number of records in the database are equal to 2 and it works fine. But the issue arises when I add one more user in the users.yml file required for other test case scenario, the count becomes 3 and test case fails.
I want to know, is there any way to create fixtures for a specific test case so that each test case will have its own fixture data?
If the answer is yes, then my test case that counts the number of records in the database won't fail even if I add extra user records in fixture file.
There are a couple of ways to solve this problem - some people might recommend not using fixtures at all, and this may be worth looking into in the future. However, the most immediate solution: do not test the specific state of the database - ever. If you have your test set up as:
it 'should add one' do
User.create(username: 'username')
assert_equal 2, User.count
end
change it instead to simply count that the number of users has gone up by one:
it 'should add one' do
count = User.count
User.create(username: 'username')
assert_equal count + 1, User.count
end
In fact, there is a custom assertion just for this use case:
http://apidock.com/rails/ActiveSupport/Testing/Assertions/assert_difference
it 'should add one' do
assert_difference("User.count", 1) do
User.create(username: 'username')
end
end
It said here https://www.relishapp.com/rspec/rspec-core/v/3-5/docs/helper-methods/let-and-let what variable defined by let is changing across examples.
I've made the same simple test as in the docs but with the AR model:
RSpec.describe Contact, type: :model do
let(:contact) { FactoryGirl.create(:contact) }
it "cached in the same example" do
a = contact
b = contact
expect(a).to eq(b)
expect(Contact.count).to eq(1)
end
it "not cached across examples" do
a = contact
expect(Contact.count).to eq(2)
end
end
First example passed, but second failed (expected 2, got 1). So contacts table is empty again before second example, inspite of docs.
I was using let and was sure it have the same value in each it block, and my test prove it. So suppose I misunderstand docs. Please explain.
P.S. I use DatabaseCleaner
P.P.S I turn it off. Nothing changed.
EDIT
I turned off DatabaseCleaner and transational fixtures and test pass.
As I can understand (new to programming), let is evaluated once for each it block. If I have three examples each calling on contact variable, my test db will grow to three records at the end (I've tested and so it does).
And for right test behevior I should use DatabaseCleaner.
P.S. I use DatabaseCleaner
That's why your database is empty in the second example. Has nothing to do with let.
The behaviour you have shown is the correct behaviour. No example should be dependant on another example in setting up the correct environment! If you did rely on caching then you are just asking for trouble later down the line.
The example in that document is just trying to prove a point about caching using global variables - it's a completely different scenario to unit testing a Rails application - it is not good practice to be reliant on previous examples to having set something up.
Lets, for example, assume you then write 10 other tests that follow on from this, all of which rely on the fact that the previous examples have created objects. Then at some point in the future you delete one of those examples ... BOOM! every test after that will suddenly fail.
Each test should be able to be tested in isolation from any other test!
I have several spec files that look like the following:
describe "My DSL" do
before :each do
#object = prepare_my_object
end
describe "foo" do
before :each do
#result = #object.read_my_dsl_and_store_stuff_in_database__this_is_expensive
end
it "should do this" do
#result.should be_this
end
it "should not do that" do
#result.should_not be_that
end
# ... several more tests about the state of #result
end
# ...
end
These tests take a long time, essentially because the second before :each block runs every time. Using before :all instead does not really help, because it gets called before the outer before :each. Putting all expectations in one single it block would help, but this is considered bad style.
What is best practice to have my expensive method being executed only once?
The fastest way to speed up rspec is to completely decouple the database. The DSL problem is a different problem from the get stuff in to and out of a db problem. If you have one method doing both, is it is possible to break the method into pieces?
Ideally, your DSL would be cached locally, so it wouldn't have to be pulled from the db on every request anyway. It could get loaded once in memory and held there before refreshing.
If you run against a local, in-memory cache, and decouple the db, does that speed things up? If yes, then it's the db call that's slow. If your DSL is completely loaded up in memory and the tests are still slow, then the problem is your DSL itself.
Is there a way to prevent Rails from clearing the test database prior to running the test? I want to use a copy of my production database for testing so I need to prevent rails from clearing out the data every time.
There is quite a bit of data so I would like to avoid using fixtures if possible since I'm assuming it would take a long time to populate the db every time.
Thanks,
Mike
You can avoid it by running the tests manually
ruby -Itest test/unit/some_test.rb
It is the rake task which does the test db recreation (you can run it manually like so)
rake db:test:prepare
But my suggestion is that you're doing it wrong.
The general idea in testing is that you know the state of the database, and therefore know what to expect from a function.
eg.
test "search_by_name" do
expected = User.all.select{|u| u.name =~ /arthur/i}
assert_equal expected, User.search_by_name("Arthur")
end
is a fine test
however, if you don't know the state of the db, how do you know there is an arthur?
The test above would pass in three bad cases;
there are no User records
all Users are called "Arthur"
there are no users called arther.
So its better to create a false reality,
where we know the state of the database.
We want;
at least one user with the name "Arthur"
at least one user with the name "Arthur" as part of another word
at least one non-Arthur user.
a better test, assuming the db is empty, and using factory girl, may be.
test "search_by_name" do
expected = [
Factory.create(:user, :name => "Arthur"),
Factory.create(:user, :name => "MacArthur")
]
not_expected = [Factory.create(:user, :name => "Archer")]
assert_equal expected, User.search_by_name("Arthur")
end
Let me preface this by saying you generally don't want to use production data for testing. That being said...
You could load "seed" data but make sure you don't have any fixtures for it otherwise it will get deleted for every test run.
See this answer for ways to automatically seed data.