I'm starting to learn acceptance testing, and I want to test an ASP.NET MVC app using SpecFlow. Currently I'm writing tests for CRUD scenario
I know it's a completely noobie question, but is it practical to test every edge case in my SpecFlow scenarios? I have a form to fill, it has about 15 fields, do I need to test that, for example, "if field A is valid and field B is not, I should see this validation message on the screen"? Or it should be sufficient to write "if form is correctly filled, I should see "Task added" message"? Should I unit-test model validation separately in this case?
Thanks ahead
I tend to look for higher-level behavior in scenarios or acceptance tests than I do with unit tests. Particularly, I'm looking for behavior that's valuable to users or other stakeholders.
In this case, the high-level behavior might be phrased as "the user gets feedback on how to fill in the form". You can then just use one or two examples of how the user gets that feedback, and check that the validation message appears.
You can then put the logic around the specific messages in a unit test.
Acceptance tests, especially if you're using BDD scenarios, aren't really tests. They're examples of how to use the system so that you can use those examples to drive conversation with your business stakeholders, and I would tend to phrase them in the same granularity and terms that the business stakeholder is interested in. BDD isn't a substitute for manual testing, though it can certainly help to reduce the burden.
Related
I am all confused with TDD vs BDD :)
How does TDD and BDD differ in each of below point?
Development: Test case first, development follows next
RestService(HTTP): Don't make rest calls? If so,
a) do we return only hardcoded json using a mock object?
b) how to handle REST call failures? We should have test case for that too?
Especially for item 2, i have googled so many articles, but couldn't find a sample (code) approach on how to handle rest calls.
BDD and TDD are not comparable to each other, although they are both used in test first development.
BDD is more than just writing tests with an English-like syntax, e.g. Kiwi. BDD (also known as ATDD—Acceptance Test Driven Development) starts with developers, QA, and designers (e.g. business, and interaction designers), working together to develop a shared understanding of the proposed solution. It is common to use examples to illustrate the behavior, also known as Specification by Example.
I have found that a useful way to think of abstraction is distinguishing between what you do (abstract, high-level policy), and how you do it (concrete, low-level details). Every concrete detail exists to fulfill a higher-level policy. When you see something concrete, it is beneficial to identify the policy it is serving.
The specification by example can be used to create high-level acceptance tests, which test what the application does, i.e. its behavior.
Unit tests are used to test how the app implements a solution, i.e. test that the appropriate messages are sent to its collaborators/dependencies at the appropriate time.
The phases of the standard TDD cycle are Red, Green, Refactor. During the green phase, your goal is to get the test passing as quickly as possible, by hook or by crook—it is acceptable to write ugly, unorganized code. Once the test passes, you refactor the code to make it more readable/changeable.
Similarly, with a BDD/ATDD cycle, you have Red, Green, Refactor. During the green phase of BDD, just get the acceptance test to pass. All of the code you write can exist within the test itself. During the refactor phase of BDD, you extract test code into production code. You can use TDD to guide the extraction.
So, for a given BDD acceptance test, you might have multiple TDD tests.
Regarding how to test REST calls, let's go back to the premise of abstraction—distinguishing what we do from how we do it.
Calling a REST service is a concrete action. The policy it satisfies may be to provide a list of model objects.
Let's say the use case you are implementing is to invite a friend to lunch. Part of the use case responsibility is to obtain the list of friends from a server; it doesn't care how the server finds the friends.
Your BDD tests would handle getting the list of friends, picking a friend, and completing the invitation. Your BDD tests would not worry about actually making REST calls.
When you use TDD to implement the the class that handles communication with the server, you could have tests that retrieve JSON from a remote data source (i.e. the server), and ensure the JSON is properly parsed into User model objects. You could also have tests to cover the data source responding with an error, etc.
At the point you actually make a REST call, in the implementation of a remote data source that uses REST to communicate with the backend server, I would classify that as an integration test, as you are testing the integration with a component you don't control, i.e. the actual backend server. The integration tests only need to confirm that the server returns JSON data in the format your app expects, or that errors are returned when appropriate.
BDD is actually derived from TDD, so it's not surprising there's a little confusion! BDD is exactly like TDD (or ATDD if you're doing it for a whole system), but without the word "Test". It turns out that can be pretty powerful.
Particularly, it lets developers have conversations with non-technical business people about what the system should do. You can also use it to have conversations about what a class should do, or a module of code should do, even with a technical expert.
So in the example of your REST service, you can imagine that I'm a dev and you're an expert who knows what the REST service should do.
Me: What should it do?
You: It should let me read a record.
Me: Great! Can you give me an example of a record?
You: I have one here...
Me: Is there any context in which someone shouldn't be able to read the record?
You: Sure, if they don't have permissions.
...
Me: Okay, so I've done Read, let's do Update. Can you give me an example of a typical update?
You: Here you go.
Me: Fantastic, and you want it to respond just with success or fail. Is there any scenario in which it should fail?
You: Sure. The record shows when it was last updated. If someone else has already updated it in the meantime, yours should fail when you submit it.
So you see you can use BDD to explore all kinds of scenarios, including those around a REST service. The trick is to ask, "Can you give me an example?" Then you get a concrete example, which you can then automate if you want to. The conversations help us look for other examples and scenarios which we might have missed.
Don't use BDD tools to automate for a technical audience! BDD tools like Cucumber, JBehave etc. work with real English that's a lot harder to refactor than code. Use JUnit, NUnit etc. if you're just doing something like a REST service. You can put "Given, When, Then" in comments, or make a little DSL.
So now you can see that with your REST call failure, if I were coding it, I'd have an example like:
Me: So, this call failure... can you give me an example?
You: Sure, if you access a record that's been deleted it's going to fail.
Me: Give me a typical example of a record that might get deleted?
You: The one we're using before is good.
Me: Okay, is there a situation in which we shouldn't delete a record?
You: Yes, if it's already been published...
Etc.
You can see that throughout, I'm not really using the word "test". Tests are a nice by-product in BDD. It's used more for exploration and specification of requirements. The conversations in BDD are the most important part of it.
The reason it's tricky to find examples of using BDD for REST is first because REST is deliberately simple and doesn't often have a lot of behaviour, and second because BDD's scenarios aren't generally phrased in terms of their implementation, focusing instead on the value of what the service or system provides ("read a record").
TDD and ATDD are exactly the same, if they're done well. It's just easier to have conversations about examples and scenarios than it is to have them about tests.
When writing specs for a simple Rails app, is the following a correct approach for full test coverage?
Write feature specs for all user stories
Write controller specs to ensure that individual action responses are correct and all required variables are set
Write model specs to ensure all methods, validations,e tc. are working as intended
Write mailer specs
Write routing specs
Is this enough, too much (e.g. can I skip some lower-level specs if I've written feature specs), or not enough? Why?
You don't need to write specs for every object in every layer either to get 100% test coverage or to test-drive (require you to implement) all of the important behavior in your application. Instead, as behavior-driven development (BDD) advises, write specs outside in, and write lower-level specs only as necessary.
The most important measure of test completeness is requirement coverage: it's helpful for each user story, and each detail of each story that requires new code, to be represented in at least one test. If you're following typical agile practices (mentioning user stories suggests that you are) your tests are probably the only place where you record your requirements, so you probably can't put a number on this kind of coverage. It's also helpful to have
line coverage (what most people mean when they say test coverage), meaning that every line of code is exercised by at least one test, and
integration coverage, meaning that every method call from one class to another is exercised by at least one test.
For each story,
Write only the feature specs that will test-drive all of the story's distinct happy paths.
Write additional feature specs to ensure integration coverage of architecturally interesting minor variations of happy paths and of sad paths. For example, I often write three feature specs for a story that involves a form: one where the user fills in every possible field and succeeds, another where the user fills in as little information as possible and still succeeds (ensuring that unspecified values and defaults work as intended), and one where the user makes a mistake, fails, corrects the mistake and succeeds.
At this point you've already test-driven every layer (controllers, models, views, helpers, mailers, etc.) into existence, with only feature specs.
Write model and helper specs to drive out detailed requirements which live entirely in those classes. For example, once you've written a single sad-path feature spec that establishes that entering one particular invalid attribute sends the user back to edit their form submission and displays a message, you can handle other invalid attributes entirely by writing more examples in that model's spec that test that model attributes are validated, and let the architecture that you've already test-driven propagate the errors back to the user.
Note that although your feature specs already test the happy paths through model and helper methods, as soon as you start writing examples for a method for minor or error cases, you'll probably want to write the happy-path example or examples for that method too, so you can see the entire description of the method in one place, and so you can test the method fully just by running all its examples and not also have to run any feature specs.
You might not need some kinds of specs at all:
Well-factored controller actions are short and have few or no conditionals, so you often won't need any controller specs at all. Write them only when needed, and stub out model, mailer, etc. behavior to keep them simple and fast.
Similarly, views and mailers should have few or no conditionals (complex code should be refactored into helper and model methods), so you often won't need view or mailer specs at all.
Your feature specs will have test-driven all the routes you need, so you probably won't need routing specs. I've only ever gotten use out of routing specs when I had to do a major refactor of routes, as when upgrading from one major version of Rails to the next.
As long as you always write a test before you write new code, you'll always have 100% line coverage.
That testing strategy sounds really comprehensive. If you had all of these tests in place you would have great test coverage. However it would take you longer to deliver your project. You would also not be agile as someone who is doing more limited testing. Testing has to suit the project. Don't over test. Over testing can cost time and money. Don't under test. Under testing can cost time and money.
There are right ways to do unit testing. There are right ways to do integration testing. The glove has to fit. If your application is largely front end facing then perhaps it's best to start with integration tests. If your writing a back end application or perhaps an API then unit tests maybe a better place to start. I think approaching with one style of testing and then expanding to different styles is a better start than to try and test every layer of your application.
Why not start with simple unit tests? They are easy to write. Write these tests and then track how many bugs you ship. Are you letting in too many bugs? Are you having a lot of regression issues? Are there bugs that are getting through to production that your suite is not picking up? If the answer is yes then maybe it's time to write some higher level tests. Remember the higher level a test is the more development cost you will have to pay.
If your not shipping bugs then you have no reason to write any more tests. Remember the end goal here. We want to ship bug free code. If we can write one test and one test alone that will ensure we are doing this then there is no reason to test any further.
We are running a pretty big website, we have some critical legacy code there we want to have well covered.
At the same time we would like to have a report of the features we are currently supporting and covered. And also we want to be sure we really cover every possible corner case. Some code paths are critical and will need many more tests even after achieving 100% coverage.
As we are already using rspec and rspec has "feature" and "scenario" keywords, we tried to make a list using rspec rather than going for cucumber but I think this question can be applied to any testing tool.
We want something like this:
feature "each advertisement will be shown a specified % of impressions"
scenario "As ..."
This feature is minimal from the point of view of managers but huge in the code. It involves a backend tool, a periodic task, logic in the models and views in backend and front end.
We tried to divide it like this:
feature "each creative will be shown a specified % of impressions"
context "configuration"
context "display"
scenario "..."
context "models"
it "should ..."
context "frontend"
context "display"
scenario "..."
context "models"
it "should ..."
Configuration takes place in another tool, display would contain integration tests and models would contain unit test.
I repeat myself but the idea is sto assure that the feature is really finished(including building the configuration tool) and 100% tested.
But looking at this file, it is not integration, nor unit test not even belong to any particular project.
Definitely there should be a better way of managing this.
Any experiences, resources, ideas you can share to guide us ?
The scenario you're describing is a huge reason why BDD is so popular. It forces you to write code in a way that's easy to test. Having said that, you're obviously not going to go back and rewrite the entire legacy application. There are a few things you should consider though:
As you go through each section of the application, you should ask yourself 'Will it be harder to refactor than to write tests for this?'. Sometimes refactoring before writing tests just cannot be avoided.
Testing isn't about 100% coverage, it's about 100% confidence. As you mentioned, you plan on writing more tests even when you have 100% coverage. This is because you're going for confidence. Once you're confident in a piece of code, move on. You can always come back to it at a later time.
From my experience, Cucumber is easier for tests that cover a large portion of the application. I think the reason for this is that writing out the tests in plain english makes you think of things you wouldn't have otherwise. It also allows you to focus on the behavior instead of the code and can make refactoring a less daunting task.
You don't really get much out of adding tests to existing code if you never touch that code again. Start with testing the code you want to make changes (i.e. refactor) to first.
I also recommend the book Rails Test Prescriptions, specifically one of the last chapters called "Testing a Legacy Application".
What's the point of using Fit/FitNesse instead of xUnit-style integration tests? It has really strange and very unclear syntax in my opinion.
Is it really only to make product owners write tests? They won't! It's too complicated for them. So why should anyone Fit/FitNesse?
Update So it's totally suitable for business-rules tests only?
The whole point is to work with non-programmers, often even completely non-technical people like prospect users of a business application, on what application should do and then put it into tests. While making tests work is certainly too complicated for them, they should be able to discuss tables of sample data filled out in e.g. Word. And the great thing is, unlike traditional specification, those documents live with your application because automated tests force you to update them.
See Introduction To Fit and Fit Workflow by James Shore and follow links to the rest of documentation if you want.
Update: Depends on what you mean by business rules? ;-) Some people would understand it very narrowly (like in business rules engines etc), others---very broadly.
As I see it, Fit is a tool that allows you to write down business (as in domain) use cases with rich realistic examples in a document, which the end users or domain experts (in some domains) can understand, verify and discuss. At the same time these examples are in machine readable form so they can be used to drive automated testing, You neither write the document entirely by yourself, nor requre them to do it. Instead it's a product of callaboration and discussion that reflects growing understanding of what application is going to do, on both sides. Examples get richer as you progress and more corner cases are resolved.
What application will do, not how, is important. It's a form of functional spec. As such it's rather broad and not really organized by modules but rather usage scenarios.
The tests that come out of examples will test external behavior of application in aspects important from business point of view. Yes, you might call it business rules. But lets look at Diego Jancic's example of credit scoring, just with a little twist. What if part of fit document is 1) listing attributes and their scores and then 2) providing client data and checking results, Then which are the actual business rules: scoring table (attributes and their scores) or application logic computing the score for each client (based on scoring table)? And which are tested?
Fit/FitNesse tests seem more suitable for acceptance testing. Other tests (when you don't care about cooperation with clients, users, domain experts, etc., you just want to automate testing) probably will be easier to write and maintain in more traditional ways. xUnit is nice for unit testing and api tests. Each web framework should have some tool for web app/service testing integrated in its modify-build-test-deploy cycle, eg. django has its little test client. You've lots to chose from.
And you always can write your own tool (or preferably tweak some existing) to better fit (pun intended) some testing in your particular domain of interest.
One more general thought. It's often (not always!!!) better to encode your tests, "business rules" and just about anything, in some form of well defined data that is interpreted by some simple, generic piece of code. Then it's easy to use the data in some other way: generate documentation, migrate to new testing framework, port application to new environment/programming language, use to check conformance with some external rules or other system (just use your imagination). It's much harder to pull such information out from code, eg. simple hardcoded unit tests or business rules.
Fit stores test cases as data. In very specific format because of how it's intended to be used, but still. Your domain specific tests may use different formats like simple CSV, JSON or YAML.
The idea is that you (the programmer) defines an easy to understand format, such as an excel sheet. Then, the product owner enters information that is hard to understand for people that is not in the business... and you just validate that your code works as the PO expects running Fit.
The way used in xUnit, can be used for programmers as an input for easy to understand or simple information.
If you're going to need to enter a lot of weird examples with multiple fields in your xUnit test, it will became hard to read.
Imagine a case where you have to decide whether to give a loan to a customer, based on the Age, Married/Single, Amount of Childrens, Wage, Activity, ...
As a programmer, you cannot write that information; and a risk manager cannot write a xUnit test.
Helps reduce redundancy in regression and bug testing. Build manageable repository of test cases. Its like build once and use for ever.
It is very useful during cooperation of the QA and devs teams: QA could show to developer the result of the failed test and a developer will easyly help to solve an environment issue and will understand steps for reproducing a bug.
It is suitable for UI and even for API testing.
Do you need something like Fitnesse, if you have BDD tests?
BDD "tests" exist at multiple different levels of granularity, all the way up to the initial project vision. Most people know about the scenarios. A few people remember that BDD started off with the word "should" as a replacement for JUnit's "test" - as a replacement for TDD. The reason I put "tests" in quotes is because BDD isn't really about testing; it's focused on finding the places where there's a lack or mismatch of understanding.
Because of that focus, the conversations are much more important than the BDD tools.
I'm going to say that again. The conversations are much more important than the BDD tools.
Acceptance testing doesn't actually mandate the conversations, and usually works from the assumption that the tests you're writing are the right tests. In BDD we assume that we don't know what we're doing (and probably don't know that we don't know). This is why we use things like "Given, When, Then" - so that we can have conversations around the scenarios and / or unit-level examples. (Those are the two levels most people are familiar with - the equivalent of acceptance tests and unit tests - but it goes up the scale).
We don't call them "acceptance tests" because you can't ask a business person "Please help me with my acceptance test". They'll look at you with a really weird, squinty gaze and then dismiss you as that geek girl. 93% of you don't want that.
Try "I'd like to talk to you about the scenario where..." instead. Or, "Can you give me an example?" Either of these are good. Calling them "Acceptance tests" starts making people think that you're actually doing testing, which would imply that you know what you're doing and just want to make sure you've done it. At that point, conversations tend to focus on how quickly you can get the wrong thing out, instead of about the fact you're getting out the wrong thing.
And you're getting the wrong thing out. Really, honestly, you are. Even if you think you're not, it's only because you don't understand second-order ignorance. You don't know that you don't know, and that's OK, as long as you've found the places where you could know you don't know. (You won't find all of them. Don't let the categorisation paradox keep you up at night.)
The only way to really get it right is to get all the requirements up front, and you know what happens when you try that. That's right. It's Waterfall. Remember the overtime? The weekend work? The seven years in which not one thing you created made it to production? If you want to avoid that, you only have one chance: assume you're wrong, have some conversations about it to be less wrong, then accept that you're still wrong and go for it anyway. Writing tests too early means you have even more chance to be wrong, and now it's harder to change and everyone thinks you're right and the PM is measuring your velocity and now you're committed to being wrong for another 2 weeks. And - worse - you're about to test that you're wrong, too.
Once again. The conversations are much more important than the BDD tools.
Please, please, don't fixate on the tools. The tools are just a mechanism for capturing the conversations and making sure that they get played into the code. Scenarios are not a replacement for conversations, any more than a 3 x 5 index card is a replacement for requirements.
Having said that, if you must start with a tool, put Slim behind Fitnesse so that it can run lovely Given / When / Thens without having to mess with Fit's tables and fixtures. GivWenZen is based on Slim and either of them rocks. FitSharp is the equivalent for those of you in the .NET space. Or just use Cucumber, or SpecFlow, or knock up a little custom DSL* that will do the job fine for years.
Transparency: *I wrote that one. And bits of JBehave. I wish we had called it "Dont-concentrate-on-BDD-tools-Behave". I might be heavily involved in other bits of BDD. Plus Dan North will buy me a pint if I can get this message out, so it's not exactly impartial advice.
Regardless - have the conversations already. It's just people. Go talk.
I don't know if there's such a thing, strictly speaking, as a "BDD test". BDD is a philosophy that suggests how you can best interact and collaborate with stakeholders to complete a complex project. It doesn't directly make any prescriptions for the best way to write tests. In other words, you'll probably still have all the usual kinds of tests (including acceptance tests) under a BDD-philosophy project.
When you hear of "BDD frameworks", the speaker usually means a framework for writing all your usual kinds of tests but with a BDD twist. For example, in RSpec, you still write unit tests; you just add the BDD flavor to them.
While BDD is larger than the scope of just tests, there are indeed BDD tests. These tests are Unit Tests that follow the BDD language.
Given some initial context (the givens),
When an event occurs,
then ensure some outcomes.
There are a few good BDD frameworks available depending on your language of preference.
JBehave for Java
RSpec for Ruby
NBehave for .NET
I like to draw a distinction between "specs" and "tests."
If I am covering a method called GetAverage(IEnumerable<int> numbers), I am going to write a more or less standard unit test.
If I am covering a method called CalculateSalesTax(decimal amount, State state, Municipality municipality), I am still going to write the unit test, but I'll call it a specification because I'm going to change it up (1) to verify the behaviour of the routine, and (2) because the test itself will document both the routine and its acceptance criteria.
Consider:
[TestFixture]
public class When_told_to_calculate_sales_tax_for_a_given_state_and_municipality() // the name defines the context
{
// set up mocks and expected behaviour
StateSalesTaxWebService stateService
= the_dependency<IStateSalesTaxWebService>;
MunicipalSurchargesWebService municipalService
= the_dependency<IMunicipalSurchargesWebService>;
stateService.Stub(x => x.GetTaxRate(State.Florida))
.Return(0.6);
municipalService.Stub(x => x.GetSurchargeRate(Municipality.OrangeCounty))
.Return(0.05);
// run what's being tested
decimal result = new SalesTaxCalculator().CalculateSalesTax
(10m, State.Florida, Municipality.OrangeCounty);
// verify the expected behaviour (these are the specifications)
[Test]
public void should_check_the_state_sales_tax_rate()
{
stateService.was_told_to(x => x.GetTaxRate(State.Florida)); // extension methods wrap assertions
}
[Test]
public void should_check_the_municipal_surcharge_rate()
{
municipalService.was_told_to(x => x.GetSurchargeRate(Municipality.OrangeCounty));
}
[Test]
public void should_return_the_correct_sales_tax_amount()
{
result.should_be_equal_to(10.65m);
}
}
JBehave (and NBehave recently added the same support) work with regular test files so while many other frameworks add "BDD taste tounit tests" the text based behaviour specifications/examples created with JBehave are suitable for acceptance tests. And no, you don't need fitnesse for that.
To get an idea of how it works I suggest JBehaves 2min tutorial.
For BDD testing in Flex you can try GivWenZen-flex check it out http://bitbucket.org/loomis/givwenzen-flex.
Cheers,
Kris
xBehavior BDD tests implemented well are robo-driven user acceptance criteria.
xSpecification BDD tests are normally unit tests and are unlikely to be acceptable user acceptance criteria.