I have an After hook in hooks.rb that deletes users created in the last scenario.
I started to notice that when tests run at a specific time of the day, this hook is being executed in the middle of a scenario.
There is a method that executes until a certain line and then the hook executes just before an assert command in that method, which fails because of it.
The tests are run from a batch file ("ruby file_name.rb").
Does anyone have an idea why this might happen or how to solve it?
Thanks!
Don't you run your tests from command line like following?
$ cucumber
I would suggest to use debugger gem. You could add debugger statement just before you think is failing and then use some debugger commands
https://github.com/cldwalker/debugger
Perhaps related to:
https://github.com/cucumber/cucumber/issues/52
Issue 52 is mostly fixed on the master, but I think there are a few remaining tests broken that need to be fixed before release.
Regardless of that, you might instead try using the database_cleaner gem for this purpose in general. We use a clean database before every scenario for testing to ensure we have discrete tests that cannot have false positive/negative results due to results of other tests. We use the following:
begin
# start off entire run with with a full truncation
DatabaseCleaner.clean_with :truncation
# continue with the :transaction strategy to be faster while running tests.
DatabaseCleaner.strategy = :transaction
rescue NameError
raise "You need to add database_cleaner to your Gemfile (in the :test group) if you wish to use it."
end
And we load our test seeds before each run:
Before do |scenario|
load Rails.root.join('db/seeds.rb')
end
Note that our seeds.rb checks which environment it is running in to keep it short. A big seeds file run in this manner would significantly increase test run times, so be careful.
Related
I'm writing a gem for catching flaky tests. The idea is to dump database to locate randomized factory. I'm using system call in the .after hook:
config.after do |example|
system "pg_dump -U postgres -d #{db_name} > #{path}/#{status}.sql"
end
At this moment database contains a record - i've made sure of that by byebug.
However i bump into a problem - as i see, transaction is not closed, so if i try get record via psql - i get nothing, record doesn't exist.
Any ideas how to close transaction during spec? Or maybe i'm wrong somewhere deeply?
Yes, the matter is the transaction was not closed. Database cleaner wrap all rspec execution in begin-rollback. I found two ways to solve my problem:
Use active record connection to dump database content, but it's no so easy - you have to use COPY command to copy (unexpected, huh?) ALL tables to file (.csv for example) and then restore them. Ew, tedious.
Change database cleaner strategie from :transaction to :truncation:
config.before(:each, flaky: true) do
DatabaseCleaner.strategy = :truncation
end
That's the solution.
In Rspec, I'm creating records, e.g. let!(:user) { create(:user) }.
I can access the new user in Rspec and in the main thread of the subject. Both of these return the user:
puts User.all.to_a
puts ActiveRecord::Base.connection.exec_query('select * from users')
However, I can now longer access the user in a new thread:
Thread.new do
puts User.all.to_a
puts ActiveRecord::Base.connection.exec_query('select * from users')
end
How do I fix this? Is this just an Rspec issue? I don't think I can reproduce it in Rails console.
You have probably configured RSpec to run its test within a database transaction
Quote from the RSpec Docs:
>
When you run rails generate rspec:install, the spec/rails_helper.rb file
includes the following configuration:
RSpec.configure do |config|
config.use_transactional_fixtures = true
end
The name of this setting is a bit misleading. What it really means in Rails
is "run every test method within a transaction." In the context of rspec-rails,
it means "run every example within a transaction."
The idea is to start each example with a clean database, create whatever data
is necessary for that example, and then remove that data by simply rolling back
the transaction at the end of the example.
You might want to disable this configuration when your application tests have the requirement to support multiple threads at the same time.
But that means that your test database will not be cleared from test data automatically anymore when a test is done. Instead, you will need to delete the test data manually or use another gem (like database_cleaner) to reset the database after running tests.
Like a lot of non-greenfeild projects, I arrived at my new job to find that most tests are performed using feature specs and not controller/model specs. Naturally this means that the tests take a really long time to execute.
Is there a way I can setup RSpec so that my feature tests run last - and they will only run if all other tests first succeed. I am migrating the application over to controller tests to speed execution, but there is no need to wait the 15mins for the feature tests to complete if there is an issue with model validation or one of my controllers is busted.
Is what I want to do possible?
(I am running the tests in RubyMine at the moment; but I can move to rake spec if need be).
Although I haven't actually tried this myself, reading through the documentation it appears it is possible to do what you wish.
Add the following to your spec_helper.rb to override the global ordering and force rspec to fail on first failure.
RSpec.configure do |config|
config.register_ordering :global do |examples|
feature, other = examples.partition do |example|
example.metadata[:type] == :feature
end
other + feature
end
config.fail_fast = true
end
Running Cucumber in CircleCI with Selenium sometimes the tests fail due to CircleCI's performance. A common failure is a Net::ReadTimeout error, which never seems to happen locally. I want to rescue the steps from that error and try them again, but I do not want to rerun all failed tests.
I could put build a rescue into the specific step(s) that seem to trigger this error, but ideally I would be able to provide Cucumber a list of errors that it rescues once or twice, to rerun that step, before finally letting the error pass through.
Something like:
# support/env.rb
Cucumber.retry_errors = {
# error => number of retries
"Net::ReadTimeoutError" => 2
}
Does anything like that exist?
I would be surprised if you found something like you are looking for in Cucumber.
Re-running a failing step just to make really sure that it is an actual failure and not just a random network glitch is, from my perspective, solving the wrong issue.
My approach would be see if the verification you are looking for is possible to do without a network. I might also consider using other tooling than Cucumber if I really must re-run a few times to make sure an error really is an error. This would, however, lead me down another rabbit hole. How many times should you run, what is the threshold? Should three out of five executions pass for you to declare a passing test? It gets ugly very fast in my eyes.
It looks like this issue is that Selenium takes took long the compile the assets on the first test. Subsequent tests use the compiled assets and do not have an issue. After viewing this Github issue I upped the timeout limit for Selenium.
Capybara.register_driver :chrome do |app|
http_client = Selenium::WebDriver::Remote::Http::Default.new
http_client.timeout = 120 # default is 60 seconds
Capybara::Selenium::Driver.new(app, browser: :chrome, http_client: http_client)
end
I know this doesn't specifically catch retries only of a specific class, but there does exist a clean way to do this in cucumber now, especially since this result comes up in Google when searching for "rerun cucumber"
In your cucumber.yml file, you can do something like this now:
<%
std_opts = "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} --strict --tags 'not #wip'"
std_opts_with_retry = "#{std_opts} --no-strict-flaky --retry 3"
%>
default: <%= std_opts_with_retry %> features
There was a big philosophical debate about whether or not a "flaky" test should be considered a fail. It was agreed if "--strict" is passed, the default should be that "flaky" tests (aka: tests that fail on the first run and pass on a following run) fail the run. So in order to get flaky tests to not "fail" your test run, you can pass the additional --no-strict-flaky along with the --retry 3 option and now any tests that may sometimes take a variable amount of time on your CI platform won't require a rebuild of the entire commit.
One thing to note when doing this: in general, I'd advise bumping your timeout back down to a reasonable limit where the majority of tests can pass without needing a long wait, although I understand in this case it's to accommodate longer compile times.
In rails rspec, I am writing test cases that do this:
after(:each) do
DatabaseCleaner.clean_with(:truncation)
end
I need the database to be cleaned after each test is run. But will this affect the performance of my tests and make them slower?
The "truncation" method has an impact based on the number of tables, because it will execute one query per table to truncate. You can use the "transaction" method instead, which will do a transaction-based rollback instead, which will drastically decrease the amount of resources used to roll back to a clean database state.
If you use ActiveRecord and want to have a clear picture of what happens, you can do something like this in your test:
before do
ActiveRecord::Base.logger = Logger.new(STDOUT)
end
It will have impact on performance depending on the number of tests on which the after hook will be run.
Also, I think it is best to use before instead of after. Depending on the test case, the state of the database is a precondition; the test case requires that precondition. And you should ensure that precondition with the before hook instead of relying on after hooks defined somewhere else.