We are planning to use SpecFlow as our primary requirements tool for a web based project. This application has lot of variations of some common objective forms. These variations are primarily per user country the application serves (Multi-tenancy).
Q1. Is SpecFlow a good choice for such requirements?
We already found that the number of fields in each form is quite large and many of these field needs validations. Some on client side and some on availability of data or other server side validations. This is tedious in SpecFlow although not totally impossible. But this is not what SpecFlow is designed for (Behavior Driven and not data centric)
Q2. Please suggest what best practices should we follow to capture the requirements in SpecFlow and
Q3. In which areas we need to use some other tools. (Where SpecFlow is not able to capture requirements well)
SpecFlow is a .Net implementation of Cucumber, a tool for behavior driven development also referred to as specification by example. Scenarios are described in Cucumber's Gherkin language.
Gherkin lets you specify data in tables either inline or as examples:
Scenario Outline: eating
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then I should have <left> cucumbers
Examples:
| start | eat | left |
| 12 | 5 | 7 |
| 20 | 5 | 15 |
Using tables may help you specify the fields for your forms in an easy to read way.
If this is still unmanagable, another option is to use NUnit which lets you specify multiple values and generate the Combinatorial set of test cases. If you want combinatorial test cases with a Cucumber library you may want to consider switching to TickSpec which supports multiple example tables per scenario.
I don't see any obvious reason why you can't use SpecFlow for this. Just take care to write the specs at a suitably high level of abstraction.
If you post an example of what you think is tedious then it may be possible to give some advice.
Also checkout the companion product 'SpecLog' as a way of visualizing these requirements.
Related
I know BDD and also read Specification by Example the great book by Gojko Adzic; As I understand they are almost the same and have many in common; But I was not able to understand their main difference. I mean Can we use them interchangeably?
General idea
BDD is the format of describing tests in a form of "this functionality does this and that". If requirements are written in the same format, then we end up with the whole team using BDD. Because these requirements can be reused for testing.
Specs by example is.. well, putting a particular example into the requirements.
Testing formats
BDD-for-testing can come in different forms:
Given When Then
Gherkin (a more strict dialect of G/W/T used by Cucumber family of frameworks)
Describe-it (Jasmine, Mocha, etc)
Or something custom. Could be same old JUnit with tests named as "adminHasAccessToProfileOfOthers" instead of older style "testAdminCanAccessProfileOfOthers"
Requirements
As for BDD-for-requirements, here's an example:
If user has sufficient amounts of money, they can buy the product
And here is Specs-by-example:
If user has 10$ and the product is 9$, user can buy that product
See, in the 1st example we used abstract ideas, while in the latter one we used particular values.
Oftentimes when people talk about BDD/Specs-by-example, they would write requirements using Given-When-Then. But as an abstract idea this isn't compulsory - it's just what most testing frameworks support it.
Overall, BDD is a very good idea for testing, especially if it's not a Given-When-Then :) But when it comes to requirements - I haven't seen any project in my experience that really benefited from it.
As for other common misconceptions: BDD and how it does NOT relate to TDD.
I am currently working for a large organisation with about 2k developers working in our IT department. We maintain many things including our e-commerce platform and there are currently about 30 projects currently impacting that.
Recently all of our teams have been instructed to deliver a series of automated tests using Concordion and Selenium Webdriver. For a while this has been going fairly well and many tests have been created but lately maintaining the existing tests while our e-commerce platform constantly changes has been somewhat of a nightmare. We have thousands of test scripts covering many parts of our website but there does not seem to be any facility in Concordion to split scripts into reusable compartments which could then be maintained once, rather than having to make changes to hundreds of HTML files for one change.
How are other people approaching this?
The goal of Concordion is not to implement test scripts as HTML, but rather for the HTML to describe the behaviour that you are testing (what you are trying to achieve). The implementation details (how it is being tested) are implemented as Java code. This code can then be structured with an appropriate level of abstraction so that each change to the system under test only requires a change to one part of the code.
Your HTML specifications should only need to change on the rare occasions that the business rules change.
These concepts are described further on the Hints and Tips tab of the Concordion home page.
Thank you for sharing your experience with us. It’s great to hear / read about large scale application of behavior driven development / specification by example.
One approach that could help you is to focus on key examples (http://gojko.net/2014/05/05/focus-on-key-examples). During specification workshops the entire team is working to get a common understanding of the new user needs and requirements. Then you go on and write specification documents containing key examples. There you should not try to cover everything, but to write only as many examples as necessary to express the common understanding.
Additionally, you should try to identify concepts, on which the examples are based. Are there some examples related to a similar topic – this is probably an underlying concept. It is often easier to understand the examples, if they focus just on one concept (e.g. the validation of a card number). Each concept can be usually described with only a few examples.
Do you have any other types of automated tests (e.g. unit tests)? Are you experiencing the same maintainability challenges with these other tests? Could you use good practices from these other test types to improve your Concordion approach?
Could you tell us more about your setup? How many active specifications have you already created within your company?
We are looking to initiate a bdd-style approach, inspired by Gojko Adzic's specification by example. Implementation is in java and devs are already writing junit tests.
Key requirement is that specifications (acceptance tests) can be written, read and maintained by non-developers. The project will run as an agile team - so it's fine if devs have to instrument the specs. However, I don't want developers, testers or domain experts having to read or write something that looks like code.
So far I've looked at FitNesse, Concordion and various others (e.g. Spock). I've rejected spock and similar tools because they target developers as the primary audience. FitNesse seems to meet most of the requirements.
Concordion is probably current favourite however: specs looks cleaner and simpler.
So my question (actually three):
Any suggestions for other tools I should look at?
Has anyone been successful with using concordion (or another tool) in this way?
Is concordion still actively developed/supported? Difficult to tell from website and most related SO questions are several years old.
Thanks.
I've worked with a number of teams implementing Specification by Example with Concordion. We train our whole team up to write Concordion specifications in HTML. Only a small subset of HTML is required, so we can train a newbie in about 30 minutes. Typically we have the testers writing the HTML specification, with the BA or Scrum Master sometimes writing them.
We've used Eclipse (Web Page Editor) for editing the HTML. This works well, except that Concordion requires valid XHTML, and Eclipse does not allow HTML to be validated as XHTML. This mostly shows with <br> tags being used rather than <br/>. We cover this off in training. We also train the whole team in the use of source control. By using Eclipse, we have a single user interface for editing and source control. We also find that having the team using the same IDE is a step on the journey to a cross-functional team.
I know of another team where the BA is writing the specifications using a Mac-based HTML editor.
Concordion is actively maintained, with rapid responses on the mailing list (Yahoo) and the issues list. The Concordion codebase is stable. The active development over the last year or so has focussed on the extension mechanism, allowing users to add commands and listeners (eg. for capturing screenshots on test failure).
Also take a look at Cucumber and JBehave, both of which allow specifications to be written in plain text.
If you choose to use FitNesse, it may also be worth looking at Slim, which sits behind FitNesse in place of Fit. It provides slightly different table formats to Fit, and I've found it suits BDD much better.
Just to update the topic, you can also consider jnario.
I am trying to get started with BDD and found a view blog posts about MSpec and SpecFlow. I'm currently not quite sure when I would use which and what the advantages/disadvantages of either framework are.
Looking at the documentation it seems that MSpec uses the context specification style whereas SpecFlow uses Given/When/Then style. I don't really mind either but I would like to know if there are any pitfalls to watch out for further down the track when the project/test suite grows.
Basically some real world advice/feedback of someone who uses it in their every day work would be great.
So I've used both.
I like the mspec workflow in away because its an easier sell for me to speak to users and say.
"When logging in"
"I should return to the page I requested"
When I've worked for organisations that have bought more into active collaboration (read agile) I've used the Given When Then pattern. That organisation was used to user stories so they were used to a more rigid style of specification. Also we were using more than one tool to feed the specs into. so the 'text only' feature files could be reused between tools.
In my own projects I use SpecFlow for the 'outside' and 'mspec' for inside of tests.
If I was to give someone advice it would be to use specflow if non technical people are writing the outside specs and mspec if a developer is writing the.
Bad points:
Mspec is class explosion
SpecFlow is a slower workflow
Good points:
Mspec is a more natural language
Specflow is better for reusability for steps.
The bottom line is they work well together.
One disadvantage of mspec is you cannot run in parallel whereas with specflow runner you can. That is a big performance issue.
I am looking at SpecFlow examples, and it's MVC sample contains several alternatives for testing:
Acceptance tests based on validating results generated by controllers;
Integration tests using MvcIntegrationTestFramework;
Automated acceptance tests using Selenium;
Manual acceptance tests when tester is prompted to manually validate results.
I must say I am quite impressed with how well SpecFlow examples are written (and I managed to run them within minutes after download, just had to configure a database and install Selenium Remote Control server). Looking at the test alternatives I can see that most of them complement each other rather than being an alternative. I can think of the following combinations of these tests:
Controllers are tested in TDD style rather than using SpecFlow (I believe Given/When/Then type of tests should be applied on higher, end-to-end level; they should provide good code coverage for respective components;
MvcIntegrationTestFramework is useful when running integration tests during development sessions, these tests are also part of daily builds;
Although Selenium-based tests are automated, they are slow and are mainly to be started during QA sessions, to quickly validate that there are no broken logic in pages and site workflow;
Manual acceptance tests when tester is prompted to confirm result validity are mainly to verify page look and feel.
If you use SpecFlow, Cucumber or other BDD acceptance test framework in you Web development, can you please share your practices regarding choosing between different test types.
Thanks in advance.
It's all behaviour.
Given a particular context, when an event occurs (within a particular scope), then some outcome should happen.
The scope can be a whole application, a part of a system or a single class. Even a function behaves this way, with inputs as context and the output as outcome (you can use BDD for functional language as well!)
I tend to use Unit frameworks (NUnit, JUnit, RSpec, etc.) at a class or integration level, because the audience is technical. Sometimes I document the Given / When / Then in comments.
At a scenario level, I try to find out who actually wants to help read or write the scenarios. Even business stakeholders can read text containing a few dots and brackets, so the main reason for having a natural language framework like MSpec or JBehave is if they want to write scenarios themselves, or show them to people who will really be put off by the dots and brackets.
After that, I look at how the framework will play with the build system, and how we'll give the ability to read or write as appropriate to the interested stakeholders.
Here's an example I wrote to show the kind of thing you can do with scenarios using simple DSLs. This is just written in NUnit.
Here's an example in the same codebase showing Given, When, Then in class-level example comments.
I abstract the steps behind, then I put screens or pages behind those, then in the screens and pages I call whatever automation framework I'm using - which could be Selenium, Watir, WebRat, Microsoft UI Automation, etc.
The example I provided is itself an automation tool, so the scenarios are demonstrating the behaviour of the automation tool through demonstrating the behaviour of a fake gui, just in case that gets confusing. Hope it helps anyway!
Since acceptance tests are a kind of functional tests, the general goal is to test your application with them end-to-end. On the other hand, you might need to consider efficiency (how much effort is to implement the test automation), maintainability, performance and reliability of the test automation. It is also important that the test automation can easily fit into the development process, so that it supports a kind of "test first" approach (to support outside-in development).
So this is a trade off, that can be different for each situation (that's why we provided the alternatives).
I'm pretty sure, that today the most widely fitting option is to test at the controller layer. (Maybe later as UI and UI automation frameworks will evolve, this will change.)