I might have totally misunderstood some fundamental things about SpecFlow, but I would like to have a data table and make use of NUnit attribute 'pairwise' to run my tests.
Using example on NUnit wiki on Pairwise attribute I would like to be able to write something like:
Scenario: Pairwise test...
Given I have the following values to calculate:
|value1|sign|value2|
|a |+ |x |
|b |- |y |
|c | | |
And this should result in 6 test runs as seen on NUnit page. Maybe this should be a scenario outline and put in examples instead.
But my questions is if its possible to use nunit attribute pairwise (or similar) and if so; how to do it?
First: There is no support of the Pairwise attribute in SpecFlow. You have to create each possible combination by your self and fill up the example table.
Second: You have in your example a Scenario. This will be one test. What you want is a Scenario Outline with an Example table. Look here for information about it: http://specflow.org/documentation/Using-Gherkin-Language-in-SpecFlow/
Full disclosure: I am one of the SpecFlow developers.
Related
I have a little over a month working with SpecFlow and got to a point where I configured a Background Scenario to setup/verify common data on a database, so the next step was trying to reuse the background for several feature files, to avoid cutting and pasting.
It has been asked before but I expected something else, more user-friendly, just as the Background scenario is easy to understand and update:
Background:
Given I have created the following currencies:
| Code | Name |
| USD | United States Dollar |
| EUR | Euro |
And I have created the following countries:
| Code | Currency | Name |
| US | USD | United States |
| ES | EUR | Spain |
| IT | EUR | Italy |
I found a quite naive solution that is working (or at least seems to, so far), but I'm concerned it may lead me the wrong way, because of my shallow knowledge of SpecFlow.
Taking a look at the generated code for a feature file I got to this:
Create a "feature" file that only has the background scenario, named something like "CommonDataSetup"
Create a step definition like:
[Given(#"common data configuration has been verified")]
public void GiveCommonDataConfigurationHasBeenVerified()
{
// this class is inside the generated feature file
var commonSetup = new CommonDataSetupFeature();
var scenarioInfo = new ScenarioInfo("Common data configuration", ((string[])(null)));
commonSetup.FeatureSetup();
commonSetup.ScenarioSetup(scenarioInfo);
commonSetup.FeatureBackground();
commonSetup.ScenarioCleanup();
commonSetup.FeatureTearDown();
}
In the Background of the other feature files write:
Background:
Given common data configuration has been verified
So now I can reuse the "common data configuration" step definition in as many feature files I need keeping DRY, and background scenarios can be much shorter.
I seems to work fine, But I wonder, is this a the right way to achieve background reuse?
Thanks in advance.
If you have a conversation with a business person who wants the feature, they probably don't say "Given common data configuration has been verified..."
They probably say something like, "Okay, you've got your standard currencies and country codes..."
Within that domain, as long the idea of standard countries and currencies is really well-known and understood, you don't need to include it. It has to be the case that every single person on the team is familiar with these, though. The whole business needs to be familiar with them. If they're that completely, totally familiar, then re-introducing a table full of them at the beginning of every scenario would be waste.
Anything you can do to eliminate that waste and get to the interesting bits of the scenario is good. Remember that the purpose of the conversations is to surface uncertainty and misunderstandings, and nobody's likely to get these wrong. The automation is a record of those conversations, and you really don't even need to have much of a conversation for this step.
Do have the conversations, though. Even if it's just one line and everyone knows what it is, using the business language for it is important. Without that, you'll end up discussing these really boring bits to try and work out what you each mean by "common data configuration" and "verify" before you can move on to the interesting parts of the scenarios.
Short version: I'd expect to see something like:
Given standard currencies and country codes
When...
You don't even need to use background for that, and however you implement it is fine. If you have a similar situation with standard data that's slightly less familiar, then include it in each feature file; it's important not to hide magic. Remember that readability trumps DRY in tests (which are really records of conversations).
I understand where the need comes, but reusing the same background in different feature files is against the idea behind Gherkin.
See https://github.com/cucumber/cucumber/wiki/Gherkin
Gherkin is the language that Cucumber understands. It is a Business Readable, Domain Specific Language that lets you describe software’s behaviour without detailing how that behaviour is implemented.
With the "Given common data configuration has been verified" step it is not more business readable.
Additional your current implementation messes with the internal state of SpecFlow. It is now somehow working, but when you will get in trouble with it.
If you need something setup in every test, did you had a look at the various Hooks?
http://www.specflow.org/documentation/Hooks/
With an [BeforeScenario]- hook you could setup your tests.
I have a set of preferences I want to test the exact same tests for, and make sure they all behave appropriately. So, the for loop would iterate over each preference configuration, and run the same set of tests is the previous and next iteration. *With IF statements of course, so tests that don't apply to the current configuration don't need to be tested.
Is there a way to do this / something similar?
I'm using rails 2.3.8
It sounds like you want cucumber example tables, where you can enumerate all your Preference values you want to test the same behavior of:
Given I have a preference defined
When I do some thing
Then I should see this behavior
Examples:
| preference | label_name |
| foo | It's Foo! |
| bar | So bar... |
Here's a link to a tutorial using table based examples for testing like this:
http://asciicasts.com/episodes/159-more-on-cucumber
Cucumber has a concept called Scenario Outlines that is used to describe a scenario that needs to be run repeatedly with differing data.
This sounds like something you use a scenario outline for:
https://github.com/cucumber/cucumber/wiki/Scenario-outlines
Scenario Outline:
blah blah
Scenarios:
| col | col |
| dat | dat |
Scenario Outline
blah blah
<use first scenarios table>
is ther a way to do this? I'd hate to have to copy a table... =\
The data you use shouldn't depend on the scenario, so it's no big deal if you copy/paste the data from one scenario to the next. I know this isn't DRY - so if you're copying the data more than a couple of times, consider using a Factory instead (see #2).
That being said, there's a couple of things you can do/questions to ask:
If blah blah is implemented as a step definition, then it's assumed that you'll reuse it when needed, that's partly what it's for
Your col/dat table might be able to be replaced with a Factory (via FactoryGirl). It's pretty common to use FactoryGirl as part of your testing, alongside Cucumber.
We've come to a point where we've realised that there are two options for specifying test data when defining a typical CRUD scenario:
Option 1: Describe the data to use, and let the implementation define the data
Scenario: Create a region
Given I have navigated to the "Create Region" page
And I have typed in a valid name
And I have typed in a valid code
When I click the "Save" button
Then I should be on the "Regions" page
And the page should show the created region details
Option 2: Explicitly state the test data to use
Scenario: Create a region
Given I have navigated to the "Create Region" page
And I have filled out the form as follows
| Label | Value |
| Name | Europe |
| Code | EUR |
When I click the "Save" button
Then I should be on the "Regions" page
And the page should show the following fields
| Name | Code |
| Europe | EUR |
In terms of benefits and drawbacks, what we've established is that:
Option 1 nicely covers the case when the definition of say a "valid name" changes. This could be more difficult to deal with if we went with Option 2 where the test data is in several places. Option 1 explicitly describes what's important about the data for this test, especially if it were a scenario where we were saying something like "has typed in an invalid credit card number". It also "feels" more abstract and BDD somehow, being more concerned with description than implementation.
However, Option 1 uses very specific steps which would be hard to re-use. For example "the page should show the created region details" will probably only ever be used by this scenario. Conversely we could implement Option 2's "the page should show the following fields" in a way that it could be re-used many times by other scenarios.
I also think Option 2 seems more client-friendly, as they can see by example what's happening rather than having to interpret more abstract terms such as "valid". Would Option 2 be more brittle though? Refactoring the model might mean breaking these tests, whereas if the test data is defined in code the compiler will help us with model changes.
I appreciate that there won't be a right or wrong answer here, but would like to hear people's opinions on how they would decide which to use.
Thanks!
I would say it depends. There are times when a Scenario might require a large amount of data to complete a successful run. Often the majority of that data is not important to the thing we are actually testing and therefore becomes noise distracting from the understanding we are trying to achieve with the Scenario. I started using something I call a Default Data pattern to provide default data that can be merged with data specific to the Scenario. I have written about it here:
http://www.cheezyworld.com/2010/11/21/ui-tests-default-dat/
I hope this helps.
I prefer option 2.
To the business user it is immediately clear what the inputs are and the outputs. With option 1 we don't know what valid data is, so your implementation may be wrong.
You can be even more expressive by adding invalid data too, when appropriate
Scenario: Filter for Awesome
Given I have navigated to the "Show People" page
And I have the following data
| Name | Value |
| John | Awesome|
| Bob | OK |
| Jane | Fail |
When I click the "Filter" button
Then the list should display
| Name | Value |
| John | Awesome |
You should however keep the data so its described in terms of the domain, rather that the specific implementation. This will allow you to test at different layers in your application. e.g. UI Service etc..
Every time I think about this I change my mind. But if you think about it - the test is to prove that you can create a region. A Criteria met by both options. But I agree that the visual cues with option 2 and developer friendliness are probably too good to turn down. In examples like this, at least.
I would suggest you take a step back and ask what stories and rules you are trying to illustrate with these scenarios. If there are rules about what makes a valid or invalid region code, and your stakeholders want to describe those using BDD, then you can use specific examples of valid and invalid region codes. If you want to describe what can happen after a region is created, then the exact data is not so interesting.
Your "Create a region" is not actually typical of the scenarios that we use in BDD. It can be characterised as "when I create a thing, then I can see the thing". It's not a useful scenario in that it doesn't by itself deliver anything valuable to the user. We look for scenarios in which something interesting or valuable is delivered to the end-user. Why is the user creating a region? What is the end goal? So that another user can assign other objects to that region, perhaps?
Example mapping, where stories are linked with rules and examples (where the examples become scenarios), is described in https://cucumber.io/blog/bdd/example-mapping-introduction/
I have around 30 scenarios that all bar one require this step to be at the top of the Background:
Given I have an account:
| name | path |
| ticketee | ticketee |
For the one that doesn't require this step it is not important that it exists or doesn't exist, because it's the feature for creating accounts. I can simply use a different account name and path for this.
Now, I was thinking rather than putting this in every single feature file 29 times that I could make use of the Before method in Cucumber which would mean placing a file in features/support/create_account.rb which would have this code:
Before do
steps(%Q{
Given I have an account:
| name | path |
| ticketee | ticketee |
})
end
The only downside from this is that it extracts what some would think belongs in the feature to a very difficult-to-track-down location and is probably not standard. But on the other hand, it saves quite a lot of repetition.
What should I do?
I'd use a tagged hook such as #with_ticketee_account. It still brings a bit of repetition, but it makes the background of the scenario more obvious than having it completely hidden.
If you wanted to make it so you only have to tag the one odd scenario, maybe create a tagged hook such as #without_ticketee_account which sets a variable which your generic before filter could check for before creating the ticketee account.
Tagged hook is IMHO a good idea.
Another Idea is to make that one line instead of three. perhaps:
Given I have an account called "ticketee"
which does all the same stuff, but is succinct as well. I have never used those tables with cucumber because I can usually make stuff on one line that reads better.
I'd go ahead and extract it into the Before block like you have proposed or use a factory (e. g. factory_girl).