Today I'm tryng to speed up my tests suite. My application is basically a big integrator between systems so most of my tests are using Savon mocks like this
RSpec.describe MyClass do
describe 'a function which sends a SOAP request'do
before do
savon.mock!
savon.expects(action).returns(File.read("spec/fixtures/somefile.xml"))
end
after { savon.unmock! }
it 'checks something'
it 'checks something else'
it 'checks something more'
it 'checks something different'
end
end
Obviously most of those tests are quite slow as they are loading a file. Moreover sometimes these mocks are inside nested contexts in order to combine multiple shared examples which increases the amount of loads.
Thinking to speed up some of these tests I tried to reduce the number of file loads moving them outside the before block. like this
RSpec.describe MyClass do
describe 'a function which sends a SOAP request'do
the_file = File.read("spec/fixtures/somefile.xml")
before do
savon.mock!
savon.expects(action).returns(the_file)
end
after { savon.unmock! }
it 'checks something'
it 'checks something else'
it 'checks something more'
it 'checks something different'
end
end
Indeed, the speed does not change; I have blocks of 96 tests with multiples, nested contexts and checks and I haven't gained not even 0.01 seconds. So my questions are:
I supposed the before block loads the file for each it, am I
right?
Does Rspec or Savon have some kind of cache?
How can I track the number of times I'm really loading my example file?
Thank you!
Perhaps you should look into the hooks order and specify something that suits you better like before (:suite) or before (:context). Depending the one you use it will be executed
https://relishapp.com/rspec/rspec-core/docs/hooks/before-and-after-hooks
Using let let(:the_file) { File.read("spec/fixtures/somefile.xml") } will solve your problem as let is lazy evaluated
Related
Curious about best practices for testing a particular situation.
I have a model that requires some time consuming operations to set up- reaching out to external services, parsing, kernel stuff, etc. One particular part of the set up is basically optional- I'd like to check that it's been run, but the result won't matter for almost all tests.
This model is used as input to many other classes, so I want to avoid a lengthy test suite and overbearing setup for a relatively unimportant step.
I'd like to know if this covers my bases, or if I'm going about this all wrong.
Currently, I am:
Stubbing out the operation globally
config.before(:each) do
LongOperation.any_instance.stub(:the_operation)
end
Testing that it gets called in my background job
code:
class BackgroundSetupWorker
def perform
LongOperation.the_operation
end
end
and test:
LongOperation.should_receive(:the_operation)
Unit testing the long-running operation
before(:each) do
LongOperation.unstub(:the_operation)
end
it "works preoperly" do
expect(LongOperation.the_operation).to ...
end
I think the ideal thing would be to take the LongOperation class as a param so you can switch it out in the tests however you like.
class BackgroundSetupWorker
def initialize(op_provider = LongOperation)
#op_provider = op_provider
end
def perform
#op_provider.the_operation
end
end
#in spec
describe BackgroundSetupWorker do
let(:op_provider){ double(the_operation: nil) }
subject(:worker){ BackgroundSetupWorker.new(op_provider) }
it 'should call op_provider' do
worker.perform
expect(op_provider).to have_received(:the_operation)
end
end
I have a few slower specs that I would like to optimise.
The example of such spec looks like:
require 'rspec'
class HeavyComputation
def compute_result
sleep 1 # something compute heavy here
"very big string"
end
end
describe HeavyComputation, 'preferred style, but slow' do
subject { described_class.new.compute_result }
it { should include 'big' }
it { should match 'string' }
it { should match /very/ }
# +50 others
end
This is very readable and I'm happy with it generally, except that every additional spec will add at least 1 second to the total run-time. That is not very acceptable.
(Please let's not discuss the optimisation on the HeavyComputation class as it is outside of the scope of this question).
So what I have to resort to is spec like this:
describe HeavyComputation, 'faster, but ugly' do
subject { described_class.new.compute_result }
it 'should have expected result overall' do
should include 'big'
should match 'string'
should match /very/
# +50 others
end
end
This is obviously much better performance wise because the time to run it will always be nearly constant.
The problem is that failures are very hard to track down and it is not very intuitive to read.
So ideally, I would like to have a mix of both. Something along these lines:
describe HeavyComputation, 'what I want ideally' do
with_shared_setup_or_subject_or_something_similar_with do
shared(:result) { described_class.new.compute_result }
subject { result }
it { should include 'big' }
it { should match 'string' }
it { should match /very/ }
# +50 others
end
end
But unfortunately I cannot see where to even start implementing it. There are multiple potential issues with it (should the hooks be called on shared result is among those).
What I want to know if there is an existing solution to this problem.
If no, what would be the best way to tackle it?
You can use a before(:context) hook to achieve this:
describe HeavyComputation, 'what I want ideally' do
before(:context) { #result = described_class.new.compute_result }
subject { #result }
it { should include 'big' }
it { should match 'string' }
it { should match /very/ }
# +50 others
end
Be aware that before(:context) comes with a number of caveats, however:
Warning: before(:context)
It is very tempting to use before(:context) to speed things up, but we
recommend that you avoid this as there are a number of gotchas, as well
as things that simply don't work.
context
before(:context) is run in an example that is generated to provide group
context for the block.
instance variables
Instance variables declared in before(:context) are shared across all the
examples in the group. This means that each example can change the
state of a shared object, resulting in an ordering dependency that can
make it difficult to reason about failures.
unsupported rspec constructs
RSpec has several constructs that reset state between each example
automatically. These are not intended for use from within before(:context):
let declarations
subject declarations
Any mocking, stubbing or test double declaration
other frameworks
Mock object frameworks and database transaction managers (like
ActiveRecord) are typically designed around the idea of setting up
before an example, running that one example, and then tearing down.
This means that mocks and stubs can (sometimes) be declared in
before(:context), but get torn down before the first real example is ever
run.
You can create database-backed model objects in a before(:context) in
rspec-rails, but it will not be wrapped in a transaction for you, so
you are on your own to clean up in an after(:context) block.
(from http://rubydoc.info/gems/rspec-core/RSpec/Core/Hooks:before)
As long as you understand that your before(:context) hook is outside the normal per-example lifecycle of things like test doubles and DB transactions, and manage the necessary setup and teardown yourself explicitly, you'll be fine -- but others who work on your code base may not be aware of these gotchas.
#Myron Marston gave some inspiration, so my first attempt to implement it in a more or less reusable way ended up with the following usage (note the shared_subject):
describe HeavyComputation do
shared_subject { described_class.new.compute_result }
it { should include 'big' }
it { should match 'string' }
it { should match /very/ }
# +50 others
end
The idea is to only render subject once, on the very first spec instead of in the shared blocks.
It makes it pretty much unnecessary to change anything (since all the hooks will be executed).
Of course shared_subject assumes the shared state with all its quirks.
But every new nested context will create a new shared subject and to some extent eliminates a possibility of a state leak.
More importantly, all we need to do in order to deal the state leaks s(should those sneak in) is to replace shared_subject back to subject. Then you're running normal RSpec examples.
I'm sure the implementation has some quirks but should be a pretty good start.
aggregate_failures, added in version 3.3, will do some of what you're asking about. It allows you to have multiple expectations inside of a spec, and RSpec will run each and report all failures instead of stopping at the first one.
The catch is that since you have to put it inside of a single spec, you don't get to name each expectation.
There's a block form:
it 'succeeds' do
aggregate_failures "testing response" do
expect(response.status).to eq(200)
expect(response.body).to eq('{"msg":"success"}')
end
end
And a metadata form, which applies to the whole spec:
it 'succeeds', :aggregate_failures do
expect(response.status).to eq(200)
expect(response.body).to eq('{"msg":"success"}')
end
See: https://www.relishapp.com/rspec/rspec-core/docs/expectation-framework-integration/aggregating-failures
Here's my rspec test:
it "can release an idea" do
james.claim(si_title)
james.release(si_title)
james.ideas.size.should eq 0
si_title.status.should eq "available"
end
Are the two should lines at the end a really bad idea? I read somewhere that you should only test for one thing per it block, but it seems silly to make a whole test just to make sure that the title status changes (the same function does both things in my code).
My interpretation of this isn't so much that there should be precisely one assertion / call to should per spec, but that there should only be one bit of behaviour tested per spec, so for example
it 'should do foo and bar' do
subject.do_foo.should be_true
subject.do_bar.should be_true
end
is bad - you're speccing 2 different behaviours at the same time.
on the other hand if your 2 assertions are just verifying different aspects of one thing then I'm ok with that, for example
it 'should return a prime integer' do
result = subject.do_x
result.should be_a(Integer)
result.foo.should be_prime
end
For me it wouldn't make too much sense to have one spec that checks that it returns an integer and a separate one that it returns a prime.
Of course in this case, the be_prime matcher could easily do both of these checks - perhaps a good rule of thumb is that multiple assertions are ok if you could sensibly reduce them to 1 with a custom matcher (whether actually doing this is actually worthwhile probably depends on your situation)
In your particular case, it could be argued that there are 2 behaviours at play - one is changing the status and other is mutating the ideas collection. I would reword your specs to say what the release method should do -
it 'should change the status to available'
it 'should remove the idea from the claimants ideas'
At the moment those things always happen simultaneously but I would argue they are separate behaviours - you could easily imagine a system where multiple people can claim/release an idea and the status only change when the last person releases the idea.
I have the same problem...
It ought to be one should per it (my boss says) by it makes testing time consuming and, as you said, silly.
Tests require good sense and flexibility or they might end up enslaving you.
Anyway, I agree with your test.
My policy is to always consider multiple assertions to be a sign of a potential problem and worth a second thought, but not necessarily wrong. Probably 1/3 of the specs I write end up having multiple assertions in them for one reason or another.
One of the problems with having multiple assertions is that when one fails, you can't see if the other one passed or not. This can sometimes be worked around by building an array of results and asserting the value of the array.
In the case you are asking about, I don't think that multiple assertions are a problem, but I do see something else that seems an issue to me. It looks like your spec may be overly coupled.
I think that you're trying to assert the behavior of whatever kind of a thing james is, but the test is also depending on behavior of si_title in order to tell us what was done to it (by way of its eventual #status value). What I would usually do instead is make si_title a test-double and use #should_receive to directly specify the messages that it should expect.
I think Frederick Cheung gave a very good answer (+1) especially the why, but I'd also like to give a comparison bit of code for you to look at which uses the its, lets, before and context syntax:
context "Given a si_title" do
let(:james) { James.new } # or whatever
before do
james.claim(si_title)
james.release(si_title)
end
context "That is valid" do
let(:si_title) { Si_title.new } # or whatever
describe "James' ideas" do
subject { james.ideas }
its(:size) { should == 0 }
its(:whatever) { should == "Whatever" }
end
describe "Si title" do
subject { si_title }
its(:status) { should == "available" }
end
end
context "That is invalid" do
# stuff here
end
end
I would even go further and make the expected values lets and then make the examples shared_examples, so that they could then be use to check the different aspects (null arguments, invalid arguments, an incorrect object… ), but I believe that this is a far better way to spec your intentions and still cut down on any repetition. Taking the example from Frederick's answer:
it 'should return a prime integer' do
result = subject.do_x
result.should be_a(Integer)
result.foo.should be_prime
end
Using RSpec's syntax to full effect you get this:
let(:result) { 1 }
subject{ result }
its(:do_x) { should be_a(Integer) }
its(:foo) { should be_prime }
Which means you can check multiple aspects of a subject.
I have several spec files that look like the following:
describe "My DSL" do
before :each do
#object = prepare_my_object
end
describe "foo" do
before :each do
#result = #object.read_my_dsl_and_store_stuff_in_database__this_is_expensive
end
it "should do this" do
#result.should be_this
end
it "should not do that" do
#result.should_not be_that
end
# ... several more tests about the state of #result
end
# ...
end
These tests take a long time, essentially because the second before :each block runs every time. Using before :all instead does not really help, because it gets called before the outer before :each. Putting all expectations in one single it block would help, but this is considered bad style.
What is best practice to have my expensive method being executed only once?
The fastest way to speed up rspec is to completely decouple the database. The DSL problem is a different problem from the get stuff in to and out of a db problem. If you have one method doing both, is it is possible to break the method into pieces?
Ideally, your DSL would be cached locally, so it wouldn't have to be pulled from the db on every request anyway. It could get loaded once in memory and held there before refreshing.
If you run against a local, in-memory cache, and decouple the db, does that speed things up? If yes, then it's the db call that's slow. If your DSL is completely loaded up in memory and the tests are still slow, then the problem is your DSL itself.
My basic logic is to have an infinite loop running somewhere and test it as best as possible. The reason for having an infinite loop is not important (main loop for games, daemon-like logic...) and I'm more asking about best practices regarding a situation like that.
Let's take this code for example:
module Blah
extend self
def run
some_initializer_method
loop do
some_other_method
yet_another_method
end
end
end
I want to test the method Blah.run using Rspec (also I use RR, but plain rspec would be an acceptable answer).
I figure the best way to do it would be to decompose a bit more, like separating the loop into another method or something:
module Blah
extend self
def run
some_initializer_method
do_some_looping
end
def do_some_looping
loop do
some_other_method
yet_another_method
end
end
end
... this allows us to test run and mock the loop... but at some point the code inside the loop needs to be tested.
So what would you do in such a situation?
Simply not testing this logic, meaning test some_other_method & yet_another_method but not do_some_looping ?
Have the loop break at some point via a mock?
... something else?
What might be more practical is to execute the loop in a separate thread, assert that everything is working correctly, and then terminate the thread when it is no longer required.
thread = Thread.new do
Blah.run
end
assert_equal 0, Blah.foo
thread.kill
in rspec 3.3, add this line
allow(subject).to receive(:loop).and_yield
to your before hook will simple yield to the block without any looping
How about having the body of the loop in a separate method, like calculateOneWorldIteration? That way you can spin the loop in the test as needed. And it doesn’t hurt the API, it’s quite a natural method to have in the public interface.
You can not test that something that runs forever.
When faced with a section of code that is difficult (or impossible) to test you should:-
Refactor to isolate the difficult to test part of the code. Make the untestable parts tiny and trivial. Comment to ensure they are not later expanded to become non-trivial
Unit test the other parts, which are now separated from the difficult to test section
The difficult to test part would be covered by an integration or acceptance test
If the main loop in your game only goes around once, this will be immediately obvious when you run it.
What about mocking the loop so that it gets executed only the number of times you specify ?
Module Object
private
def loop
3.times { yield }
end
end
Of course, you mock this only in your specs.
I know this is a little old, but you can also use the yields method to fake a block and pass a single iteration to a loop method. This should allow you to test the methods you're calling within your loop without actually putting it into an infinite loop.
require 'test/unit'
require 'mocha'
class Something
def test_method
puts "test_method"
loop do
puts String.new("frederick")
end
end
end
class LoopTest < Test::Unit::TestCase
def test_loop_yields
something = Something.new
something.expects(:loop).yields.with() do
String.expects(:new).returns("samantha")
end
something.test_method
end
end
# Started
# test_method
# samantha
# .
# Finished in 0.005 seconds.
#
# 1 tests, 2 assertions, 0 failures, 0 errors
I almost always use a catch/throw construct to test infinite loops.
Raising an error may also work, but that's not ideal especially if your loop's block rescue all errors, including Exceptions. If your block doesn't rescue Exception (or some other error class), then you can subclass Exception (or another non-rescued class) and rescue your subclass:
Exception example
Setup
class RspecLoopStop < Exception; end
Test
blah.stub!(:some_initializer_method)
blah.should_receive(:some_other_method)
blah.should_receive(:yet_another_method)
# make sure it repeats
blah.should_receive(:some_other_method).and_raise RspecLoopStop
begin
blah.run
rescue RspecLoopStop
# all done
end
Catch/throw example:
blah.stub!(:some_initializer_method)
blah.should_receive(:some_other_method)
blah.should_receive(:yet_another_method)
blah.should_receive(:some_other_method).and_throw :rspec_loop_stop
catch :rspec_loop_stop
blah.run
end
When I first tried this, I was concerned that using should_receive a second time on :some_other_method would "overwrite" the first one, but this is not the case. If you want to see for yourself, add blocks to should_receive to see if it's called the expected number of times:
blah.should_receive(:some_other_method) { puts 'received some_other_method' }
Our solution to testing a loop that only exits on signals was to stub the exit condition method to return false the first time but true the second time, ensuring the loop is only executed once.
Class with infinite loop:
class Scheduling::Daemon
def run
loop do
if daemon_received_stop_signal?
break
end
# do stuff
end
end
end
spec testing the behaviour of the loop:
describe Scheduling::Daemon do
describe "#run" do
before do
Scheduling::Daemon.should_receive(:daemon_received_stop_signal?).
and_return(false, true) # execute loop once then exit
end
it "does stuff" do
Scheduling::Daemon.run
# assert stuff was done
end
end
end
:) I had this query a few months ago.
The short answer is there is no easy way to test that. You test drive the internals of the loop. Then you plonk it into a loop method & do a manual test that the loop works till the terminating condition occurs.
The easiest solution I found is to yield the loop one time and than return. I've used mocha here.
require 'spec_helper'
require 'blah'
describe Blah do
it 'loops' do
Blah.stubs(:some_initializer_method)
Blah.stubs(:some_other_method)
Blah.stubs(:yet_another_method)
Blah.expects(:loop).yields().then().returns()
Blah.run
end
end
We're expecting that the loop is actually executed and it's ensured it will exit after one iteration.
Nevertheless as stated above it's good practice to keep the looping method as small and stupid as possible.
Hope this helps!