How do I get SpecFlow to expect an exception? - specflow

Question

I'm using SpecFlow, and I'd like to write a scenario such as the following:
Scenario: Pressing add with an empty stack throws an exception
Given I have entered nothing into the calculator
When I press add
Then it should throw an exception
It's calculator.Add() that's going to throw an exception, so how do I handle this in the method marked [Then]?

Answer 1

Great question. I am neither a bdd or specflow expert, however, my first bit of advice would be to take a step back and assess your scenario.
Do you really want to use the terms "throw" and "exception" in this spec? Keep in mind the idea with bdd is to use a ubiquitous language with the business. Ideally, they should be able to read these scenarios and interpret them.
Consider changing your "then" phrase to include something like this:
Scenario: Pressing add with an empty stack displays an error
Given I have entered nothing into the calculator
When I press add
Then the user is presented with an error message
The exception is still thrown in the background but the end result is a simple error message.
Scott Bellware touches this concept in this Herding Code podcast: http://herdingcode.com/?p=176

Answer 2

As a newbie to SpecFlow I won't tell you that this is the way to do it, but one way to do it would be to use the ScenarioContext for storing the exception thrown in the When;
try
{
calculator.Add(1,1);
}
catch (Exception e)
{
ScenarioContext.Current.Add("Exception_CalculatorAdd", e);
}
In your Then you could check the thrown exception and do asserts on it;
var exception = ScenarioContext.Current["Exception_CalculatorAdd"];
Assert.That(exception, Is.Not.Null);
With that said; I agree with scoarescoare when he says that you should formulate the scenario in a bit more 'business-friendly' wordings. However, using SpecFlow to drive the implementation of your domain-model, catching exceptions and doing asserts on them can come in handy.
Btw: Check out Rob Conery's screencast over at TekPub for some really good tips on using SpecFlow: http://tekpub.com/view/concepts/5

Answer 3

BDD can be practiced on feature level behavior or/and on unit level behavior.
SpecFlow is a BDD tool that focuses on feature level behavior.
Exceptions are not something that you should specify/observe on feature level behavior.
Exceptions should be specified/observed on unit-level behavior.
Think of SpecFlow scenarios as a live specification for the non technical stakeholder. You would also not write in the specification that an exception is thrown, but how the system behaves in such a case.
If you do not have any non technical stakeholders, then SpecFlow is the wrong tool for you! Don't waste energy in creating business readable specifications if there is nobody interested in reading them!
There are BDD tools that focus on unit level behavior. In .NET the most popular one is MSpec (http://github.com/machine/machine.specifications).
BDD on unit-level can also easily be practices with standard unit-testing frameworks.
That said, you could still check for an exception in SpecFlow.
Here are some more discussion of bdd on unit-level vs. bdd on feature-level:
SpecFlow/BDD vs Unit Testing
BDD for Acceptance Tests vs. BDD for Unit Tests (or: ATDD vs. TDD)
Also have look at this blog post:
Classifying BDD Tools (Unit-Test-Driven vs. Acceptance Test Driven) and a bit of BDD history

Answer 4

Changing the scenario not to have an exception is a probably good way to have the scenario more user oriented. However, if you still need to have it working, please consider the following:
Catch an exception (I really recommend catching specific exceptions unless you really need to catch all) in the step that invokes an operation and pass it to the scenario context.
[When("I press add")]
public void WhenIPressAdd()
{
try
{
_calc.Add();
}
catch (Exception err)
{
ScenarioContext.Current[("Error")] = err;
}
}
Validate that exception is stored in the scenario context
[Then(#"it should throw an exception")]
public void ThenItShouldThrowAnException()
{
Assert.IsTrue(ScenarioContext.Current.ContainsKey("Error"));
}
P.S. It's very close to one of the existing answers. However, if you try getting value from ScenarioContext using syntax like below:
var err = ScenarioContext.Current["Error"]
it will throw another exception in case if "Error" key doesn't exist (and that will fail all scenarios that perform calculations with correct parameters). So ScenarioContext.Current.ContainsKey may be just more appropriate

Answer 5

My solution involves a couple of items to implement, but at the very end it will look much more elegant:
#CatchException
Scenario: Faulty operation throws exception
Given Some Context
When Some faulty operation invoked
Then Exception thrown with type 'ValidationException' and message 'Validation failed'
To make this work, follow those 3 steps:
Step 1
Mark Scenarios you expect exceptions in with some tag, e.g. #CatchException:
#CatchException
Scenario: ...
Step 2
Define an AfterStep handler to change ScenarioContext.TestStatus to be OK. You may only want ignore errors in for When steps, so you can still fail a test in Then verifying an exception. Had to do this through reflection as TestStatus property is internal:
[AfterStep("CatchException")]
public void CatchException()
{
if (ScenarioContext.Current.StepContext.StepInfo.StepDefinitionType == StepDefinitionType.When)
{
PropertyInfo testStatusProperty = typeof(ScenarioContext).GetProperty("TestStatus", BindingFlags.NonPublic | BindingFlags.Instance);
testStatusProperty.SetValue(ScenarioContext.Current, TestStatus.OK);
}
}
Step 3
Validate TestError the same way you would validate anything within ScenarioContext.
[Then(#"Exception thrown with type '(.*)' and message '(.*)'")]
public void ThenExceptionThrown(string type, string message)
{
Assert.AreEqual(type, ScenarioContext.Current.TestError.GetType().Name);
Assert.AreEqual(message, ScenarioContext.Current.TestError.Message);
}

Answer 6

In case you are testing user interactions I will only advice what has already been said about focusing on the user experience: "Then the user is presented with an error message". But, in case you are testing a level below the UI, I'd like to share my experience:
I'm using SpecFlow to develop a business layer. In my case, I don't care about the UI interactions, but I still find extremely useful the BDD approach and SpecFlow.
In the business layer I don't want specs that say "Then the user is presented with an error message", but actually verifying that the service correctly responds to a wrong input. I've done for a while what has already been said of catching the exception at the "When" and verifying it at the "Then", but I find this option not optimal, because if you reuse the "When" step you could swallow an exception where you didn't expect it.
Currently, I'm using explicit "Then" clauses, some times without the "When", this way:
Scenario: Adding with an empty stack causes an error
Given I have entered nothing into the calculator
Then adding causes an error X
This allows me to specifically code the action and the exception detection in one step. I can reuse it to test as many error cases as I want and it doesn't make me add unrelated code to the non failing "When" steps.

Answer 7

ScenarioContext.Current is deprecated with the latest version of SpecFlow, it is now recommended to add a POCO to the steps test class in the constructor to store/retrieve context between steps, i.e.
public class ExceptionContext
{
public Exception Exception { get; set; }
}
private ExceptionContext _context;
public TestSteps(ExceptionContext context)
{
_context = context;
}
And in your [When] binding....
try
{
// do something
}
catch (MyException ex)
{
_context.Exception = ex;
}
In your [Then] binding, assert that _context.Exception is set and of the exception type you expected.