I am trying to test an app that uses gem devise_token_auth, which basically includes a couple extra DB read/writes on almost every request (to verify and update user access tokens).
Everything is working fine, except when testing a controller action that includes several additional db read/writes. In these cases, the terminal locks up and I'm forced to kill the ruby process via activity monitor.
Sometimes I get error messages like this:
ruby /Users/evan/.rvm/gems/ruby-2.1.1/bin/rspec spec/controllers/api/v1/messages_controller_spec.rb(1245,0x7fff792bf310) malloc: *** error for object 0x7ff15fb73c00: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6
I have no idea how to interpret that. I'm 90% sure the problem is due to this gem and the extra DB activity it causes on each request because I when I revert to my previous, less intensive auth, all the issues go away. I've also gotten things under control by giving postgres some extra time on the offending tests:
after :each do
sleep 2
end
This works fine for all cases except one, which requires a timeout before the expect, otherwise it throws this error:
Failure/Error: expect(#user1.received_messages.first.read?).to eq true
ActiveRecord::StatementInvalid:
PG::UnableToSend: another command is already in progress
: SELECT "messages".* FROM "messages" WHERE "messages"."receiver_id" = $1 ORDER BY "messages"."id" ASC LIMIT 1
which, to me, points to the DB issue again.
Is there anything else I could be doing to track down/control these errors? Any rspec settings I should look into?
If you are running parallel rspec tasks, that could be triggering this. When we've run into issues like this, we have forced those tests to run in single, non-parallel instance of rspec in our CI using tags.
Try something like this:
context 'when both records get updated in one job', non_parallel do
it { is_expected.to eq 2 }
end
And then invoke rspec singularly on the non_parallel tag:
rspec --tag non_parallel
The bulk of your tests (not tagged with non_parallel) can still be run in parallel in your CI solution (e.g. Jenkins) for performance.
Of course, be careful applying this band-aid. It is always better to identify what is not race-safe in your code, since that race could happen in the real world.
Related
Running Cucumber in CircleCI with Selenium sometimes the tests fail due to CircleCI's performance. A common failure is a Net::ReadTimeout error, which never seems to happen locally. I want to rescue the steps from that error and try them again, but I do not want to rerun all failed tests.
I could put build a rescue into the specific step(s) that seem to trigger this error, but ideally I would be able to provide Cucumber a list of errors that it rescues once or twice, to rerun that step, before finally letting the error pass through.
Something like:
# support/env.rb
Cucumber.retry_errors = {
# error => number of retries
"Net::ReadTimeoutError" => 2
}
Does anything like that exist?
I would be surprised if you found something like you are looking for in Cucumber.
Re-running a failing step just to make really sure that it is an actual failure and not just a random network glitch is, from my perspective, solving the wrong issue.
My approach would be see if the verification you are looking for is possible to do without a network. I might also consider using other tooling than Cucumber if I really must re-run a few times to make sure an error really is an error. This would, however, lead me down another rabbit hole. How many times should you run, what is the threshold? Should three out of five executions pass for you to declare a passing test? It gets ugly very fast in my eyes.
It looks like this issue is that Selenium takes took long the compile the assets on the first test. Subsequent tests use the compiled assets and do not have an issue. After viewing this Github issue I upped the timeout limit for Selenium.
Capybara.register_driver :chrome do |app|
http_client = Selenium::WebDriver::Remote::Http::Default.new
http_client.timeout = 120 # default is 60 seconds
Capybara::Selenium::Driver.new(app, browser: :chrome, http_client: http_client)
end
I know this doesn't specifically catch retries only of a specific class, but there does exist a clean way to do this in cucumber now, especially since this result comes up in Google when searching for "rerun cucumber"
In your cucumber.yml file, you can do something like this now:
<%
std_opts = "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} --strict --tags 'not #wip'"
std_opts_with_retry = "#{std_opts} --no-strict-flaky --retry 3"
%>
default: <%= std_opts_with_retry %> features
There was a big philosophical debate about whether or not a "flaky" test should be considered a fail. It was agreed if "--strict" is passed, the default should be that "flaky" tests (aka: tests that fail on the first run and pass on a following run) fail the run. So in order to get flaky tests to not "fail" your test run, you can pass the additional --no-strict-flaky along with the --retry 3 option and now any tests that may sometimes take a variable amount of time on your CI platform won't require a rebuild of the entire commit.
One thing to note when doing this: in general, I'd advise bumping your timeout back down to a reasonable limit where the majority of tests can pass without needing a long wait, although I understand in this case it's to accommodate longer compile times.
I have a fairly complex service that creates and saves a lot of domain instances, and because of the logic, I need to create different instances at different times, and in the middle check for certain conditions, that are not only that each instance is valid, but for example I need to check if certain files exist in the file system, etc.
I'm testing the incorrect cases, where the service throws an exception, and I need to test that no instances where persisted if an exception is thrown.
One specific test case was failing, even if the expected exception was thrown, a domain instance was saved to the DB. Then I read that because the integration test was transactional itself, the rollback really occurs at the end of the test, instead of after using the service, so I can check conditions on the "then" section of the spock test case.
So if the rollback occurs after I can test it, I can't test it :(
Then I read that making the integration test not transaction might help, so I added:
static transactional = false
After this, all my other tests started to fail!
My question is: what is the right way of testing services that should rollback when an exception is thrown? I just need to verify that after an exception occurs, there are no alterations to the database (since this is a healthcare application, data consistency is key)
FYI:
This is the service I need to test: https://github.com/ppazos/cabolabs-ehrserver/blob/master/grails-app/services/com/cabolabs/ehrserver/parsers/XmlService.groovy
This is my current test: https://github.com/ppazos/cabolabs-ehrserver/blob/master/test/integration/com/cabolabs/ehrserver/parsers/XmlServiceIntegrationSpec.groovy
Thanks!
If you can modify your code to rely on the framework, then you don't need to test the framework.
Grails uses Spring and the Spring transaction manager rolls back automatically for RuntimeExceptions but not for checked exceptions. If you throw a RuntimeException for your various validation scenarios then Spring will handle it for you. If you rely on those facts, your test could then stop at verifying that a RuntimeException is thrown.
If you want to use a checked exception (such as javax.xml.bind.ValidationException ) then you can use an annotation to ensure it causes a rollback:
#Transactional(rollbackFor = ValidationException.class)
def processCommit( ....
and then your test need only check
def ex = thrown(ValidationException)
See http://docs.spring.io/spring/docs/current/spring-framework-reference/html/transaction.html#transaction-declarative-attransactional-settings for more information.
I am getting a couple different errors at a particular line of code in one of my models when running in Sidekiq-queued jobs. The code in question is:
#lookup_fields[:asin] ||= self.book_lookups.find_by_name("asin").try(:value)
I either get undefined method 'scope' for #<ActiveRecord::Associations::AssociationScope:0x00000005f20cb0> or undefined method 'aliased_table_for' for #<ActiveRecord::Associations::AliasTracker:0x00000005bc3f90>.
At another line of code in another Sidekiq job, I get the error undefined method 'decrypt_and_verify' for #<ActiveSupport::MessageEncryptor:0x00000007143208>.
All of these errors make no sense, as they are standard methods of the Rails runtime support libraries.
The model in question has a :has_many association defined for the "book_lookups" model, "name" and "value" are fields in the "book_lookups" model. This always happens on the first 1-3 records processed. If I run the same code outside of a Sidekiq job, these errors do not occur.
I cannot reproduce the error on my development machine, only on production which is hosted at Heroku.
I may have "solved" the first set of errors by putting the code `BookLookup.new()' in an initializer, forcing the model to load before Sidekiq creates any threads. Only one night's work to go on, so we'll have to see if the trend continues...
Even if this solves the immediate problem, I don't think it solves the real underlying issue, which is classes not getting fully loaded before being used. Is class-loading an atomic operation? Is it possible for one thread to start loading a class and another to start using the class before it is fully loaded?
I believe that I have discovered the answer: config.threadsafe!, which I had not done. I have now done that and most if not all of the errors have disappeared. References: http://guides.rubyonrails.org/configuring.html, http://m.onkey.org/thread-safety-for-your-rails (especially the section "Ruby's require is not atomic").
I'm having trouble debugging cucumber steps due to unique conditions of the testing environment. I wish there was a step that could pause a selenium test and let me take over.
E.g.
Scenario: I want to take over here
Given: A bunch of steps have already run
When: I'm stuck on an error
Then: I want to take control of the mouse
At that point I could interact with the application exactly as if I had done all the previous steps myself after running rails server -e test
Does such a step exist, or is there a way to make it happen?
You can integrate ruby-debug into your Cucumber tests. Nathaniel Ritmeyer has directions here and here which worked for me. You essentially require ruby-debug, start the debugger in your environment file, and then put "breakpoint" where ever you want to see what's going on. You can both interact with the browser/application and see the values of your ruby variables in the test. (I'm not sure whether it'll let you see the variables in your rails application itself - I'm not testing against a rails app to check that).
I came up with the idea to dump the database. It doesn't let you continue work from the same page, but if you have the app running during the test, you can immediately act on the current state of things in another browser (not the one controlled by Selenium).
Here is the step:
When /I want to take control/i do
exec "mysqldump -u root --password=* test > #{Rails.root}/support/snapshot.sql"
end
Because it is called by exec, DatabaseCleaner has no chance to truncate tables, so actually it's irrelevant that the command is a database dump. You don't have to import the sql to use the app in its current state, but it's there if you need it.
My teammate has done this using selenium, firebug a hook (#selenium_with_firebug)
Everything he learned came from this blogpost:
http://www.allenwei.cn/tips-add-firebug-extension-to-capybara/
Add the step
And show me the page
Where you want to interact with it
Scenario: I want to take over here
Given: A bunch of steps have already run
When: I'm stuck on an error
Then show me the page
use http://www.natontesting.com/2009/11/09/debugging-cucumber-tests-with-ruby-debug/
Big thank you to #Reed G. Law for the idea of dumping the database. Then loading it into development allowed me to determine exactly why my cucumber feature was not impacting database state as I had expected. Here's my minor tweak to his suggestion:
When /Dump the database/i do
`MYSQL_PWD=password mysqldump -u root my_test > #{Rails.root}/snapshot.sql`
# To replicate state in development run:
# `MYSQL_PWD=password mysql -u root my_development < snapshot.sql`
end
You can also use the following in feature/support/debugging.rb to let you step through the feature one step at a time:
# `STEP=1 cucumber` to pause after each step
AfterStep do |scenario|
next unless ENV['STEP']
unless defined?(#counter)
puts "Stepping through #{scenario.title}"
#counter = 0
end
#counter += 1
print "At step ##{#counter} of #{scenario.steps.count}. Press Return to"\
' execute...'
STDIN.getc
end
I've been trying to solve a problem for a few weeks now. I am running rspec tests for my Rails app, and they are working fine except for one error that I can't seem get my head around.
I am using MySQL with the InnoDB engine.
I have set config.use_transactional_fixtures = true in spec_helper.rb
I load my test fixtures manually with the command rake spec:db:fixtures:load.
The rspec test is being written for a BackgrounDRb worker, and it is testing that a record can have its state updated (through the state_machine gem).
Here is my problem:
I have a model called Listings. The rspec test calls the update_sold_items method within a file called listing_worker.rb.
This method calls listing.sell for a particular record, which sets the listing record's 'state' column to 'sold'.
So far, this is all working fine, but when the update_sold_items method finishes, my rspec test fails here:
listing = Listing.find_by_listing_id(listing_id)
listing.state.should == "sold"
expected: "sold",
got: "current" (using ==)
I've been trying to track down why the state change is not persisting, but am pretty much lost. Here is the result of some debugging code that I placed in the update_sold_items method during the test:
pp listing.state # => "current"
listing.sell!
listing.save!
pp listing.state # => "sold"
listing.reload
pp listing.state # => "current"
I cannot understand why it saves perfectly fine, but then reverts back to the original record whenever I call reload, or Listing.find etc.
Thanks for reading this, and please ask any questions if I haven't given enough information.
Thanks for your help,
Nathan B
P.S. I don't have a problem creating new records for other classes, and testing those records. It only seems to be a problem when I am updating records that already exist in the database.
I suspect, like nathan, transaction issues. Try putting a Listing.connection.execute("COMMIT") right before your first save call to break the transaction and see what changes. That will break you out of the transaction so any additional rollback calls will be non-effectual.
Additionally, by running a "COMMIT" command, you could pause the test with a debugger and inspect the database from another client to see what's going on.
The other hypothesis, if the transaction experimentation doesn't yield any results, is that perhaps your model really isn't saving to the database. Check your query logs. (Specifically find the update query).
These kind of issues really stink! Good luck!
If you want to investigate what you have in DB while running tests you might find this helpful...
I have a rspec test where I save #user.save and it works like a charm, but then I wanted to see if it's really saved in the DB.
I opened rails console for test environment
rails c test
ran
User.all
and as expected got nothing
I ran my spec that contains:
user_attr_hash = FactoryGirl.attributes_for(:user)
#user = User.new user_attr_hash
#user.save
binding.pry
I thought that stopping the test after save would mean that it's persisted, but that's not the case. It seems that COMMIT on the connection is fired later (I have no idea when:\ )
So, as #Tim Harper suggests, you have to fire that commit yourself in the pry console:
pry(#<RSpec::Core::ExampleGroup::Nested_1>)> User.connection.execute("COMMIT")
Now, if you run User.all in your rails console you should see it ;)