I have developed a jBehave story to test some work flow implemented in our system.
Let’s say this story is called customer_registration.story
That story is a starting point of some other more complex work flows that our system supports.
Those more complex work flows are also covered by different stories.
Let’s say we have one of our more complex work flows covered by a customer_login.story
So the customer_login.story will look somehow like below:
Story: Customer Login
Narrative:
In order to access ABC application
As a registered customer
I want to login into the application
Scenario: Successfully login into the application
GivenStories: customer_registration.story
Given I am at the login page
When I type a valid password
Then I am able to see the application main menu
All works perfectly and I am happy with that.
3.The story at point 1 above (customer registration) is something I need to run on different sets of data.
Let’s say our system supports i18n and we need to check that customer registration story runs OK for all supported languages, say we want to test our customer registration works OK with both en-gb and zh-tw
So I need to implement a multi_language_customer_registration.story that will look something like that:
Story: Multi language customer registration
Narrative:
In order to access ABC application
As a potential customer
I want to register for using the application
Scenario: Successfully customer registration using different supported languages
GivenStories: customer_registration.story
Then some clean up step so the customer registration story can run again
Examples:
|language|
|en-gb |
|zh-tw |
Any idea about how I could achieve this?
Note that something like below is not an option as I do need to run the clean up step between runs.
GivenStories: customer_registration.story#{0},customer_registration.story#{1}
Moving the clean up step inside the customer registration story is not an option too as then the login story will stop working.
Thanks in advance.
P.S. As you could guess in reality the stories we created are more complex and it is not an easy task to refactor them, but I am happy to do this for a real gain.
First off, BDD is not the same as testing. I wouldn't use it for every single i18n scenario. Instead, isolate the bit which deals with i18n and unit test that, manually test for a couple and call it done. If you really need more thorough then use it with a couple of languages, but don't do it with all of them - just enough examples to give you some safety.
Now for the bit with the customers. First of all, is logging in and registration really that interesting? Are you likely to change them once you've got them working? Is there anything special about logging in or registration that's particular to your business? If not, try to keep that stuff out of the scenarios - it'll be more of a pain to maintain than it's worth, and if it's never going to change you can just test it once, manually.
Scenarios which show what the user is logging in for are usually more enticing and interesting to the business (you are having conversations with the business, right?).
Otherwise, here are the three ways in which you can set up a context (Given):
By hacking the data (so accessing the database directly)
Through the UI (or controller if you're automating from that level)
By using existing data.
You can also look to see if data exists, and if it doesn't, set it up. So for instance, if your customer is registered and you don't want him to be registered, you can delete his registration as part of setting up the context (running the Given step); or if you need him to be registered and he isn't, you can go through the UI to register him.
Lastly, JBehave has an #AfterScenario annotation which you can use to denote a clean-up step for that scenario. Steps are reusable - you can call the steps of the scenario from within another step in code, rather than using JBehave's mechanism (this is more maintainable anyway IMO) and this will allow you to avoid clearing registration when you log in.
Hope one of these options works for you!
From a tactical standpoint, I would do this:
In your .story file
Given I set my language to {language}
When I type a valid password {pass}
Then I am able to see the application main menu
Examples:
|language|pass|
|en-gb |password1|
|zh-tw |kpassword2|
Then in your Java file,
#Given ("I set my language to $lang")
#Alias ("I set my language to {language}")
// method goes here
#When ("I type a valid password $pwrd")
#Alias ("I type a valid password {pass}")
// method goes here
#Then ("I am able to see the application main menu")
most unit testing frameworks support this.
Look how mstest you can specify DataSource, nunit is similar
https://github.com/leblancmeneses/RobustHaven.IntegrationTests
unfortunately some of the bdd frameworks i've seen try to replace existng unit test frameworks when it should instead work together to reuse the infrastructure.
https://github.com/leblancmeneses/BddIsTddDoneRight
is fluent bdd syntax that can be used with mstest/nunit and works with existing test runners.
Related
I am on a development team where we have 2 separate mobile apps. One of the apps is for Users. The other app is for Admin of those users. My main objective is to execute a test case in the Admin app, and then run a test case in the Users app to verify its working properly. How can I approach this?
For example, I want to run a test case in the Admin app that revokes some privilege. I then want to run a test case from the Users app to check to confirm that privilege was revoked.
Maybe this is not a good strategy at all -- but it makes sense for my team because we have 2 apps that work together -- and if we do some function in the Admin app -- we want to see the expected result in the Users app
My plan was to mark each test with a Category, for example, "Privilege"
On Jenkins:
Run "Privilege" Category on the Admin app where I revoke some privilege
Run "Privilege" Category on the User app where I confirm revoked privilege
This seems like an ok test strategy right now. But if I have 20 UITests that means I'll have 20 different Jenkins projects in my dashboard, one for each UITest (per device, per platform). It seems that with 20 UITests I'll end up with over 100 Jenkins projects. Thats not really ideal to me.
Has anybody else come up with a testing strategy where they needed to test 2 separate projects back and forth. I understand that this does not really fall under unit testing - and I may get some vague answers around unit testing and general. But I do believe mobile is a different animal in the UITest world
There are couple of points in your question
do some function in the Admin app -- we want to see the expected result in the Users app
If you need to test such integrations between the two apps, you can go with proper labels for ones that are
mark each test with a Category
in any case, you will need some way to organise your suites. Good way to do so, are test annotations. I think Lazy setup is aplicable in your case. It will set the desired state for all marked tests, when needed.
needed to test 2 separate projects back and forth
End-to-end tests are mandatory, for the most business critical features. My suggestion is to employ Backdoor manipulation. Your other tests should already have covered the simpler cases (e.g. setting a privilege in Admin app), so if you already did exercise this feature, no point of redundancy.
It seems that with 20 UITests I'll end up with over 100 Jenkins projects. Thats not really ideal
You actually don't need a Jenkins project per suite, just configure the tests via CLI arguments and your harness will pick that up for you. What you need is a tag (or platform, or device) to be passed to the runner.
Generally, you do NOT want tests to depend on each other. Have a look at this example:
In the admin app, you set the privilege.
You open the user app.
The privilege should be set, but it's not.
You know that something went wrong, but you don't know whether it's the admin app that's not working or the user app.
Therefore, you should test them independently by mocking (=faking) the backend:
Open the admin app.
Set the privilege.
Ask the mocked backend: Did you receive a call from the admin app to set the privilege?
In an independent UI test for the user app, you do the following:
Set up a fake backend where the privilege is set
Open the user app
See whether the privilege is set.
By separating the tests for both apps, you will know which of the two does not work.
The software that I use at work to do such things is called WireMock but there are others out there, too.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
Suppose I'm designing some SaaS service. And I need to have a function that allow users create sites. User can makes special settings for each site in the admin panel (e.g. design of widget) and gets unique code for install service to his own site.
User story could be:
As a logged user I want to add the new site in the admin panel so that I could configure each instance of widgets separately and could get a unique code for install widget to my own site.
Form
But if I will try to describe this functionality with BDD or GWT (Given When Then) or Gherkin-style, I will face with some trouble. I start from next description:
GIVEN I'm logged into admin panel
AND I'm on "Sites" page
WHEN I click "Add site" button
THEN Pop-up window "Add site" come up
As you can see above realization suppose that site adding will be in pop-up window (e.g. it very important for UX). Pop-up window contains Site URL input field, drop-down control with languages and "Add" and "Cancel" buttons.
And we got a strange scenario which responsible for just pop-up opening. Is it correct? And how can I name this scenario ("Add site's form opening" ??)? Also this scenario has only a one case (when I click - pop-up open). Maybe this scenario not needed at all? I'm confused...
In this case we need to create another scenario when describes:
GIVEN "Add site" pop-up form is opened
WHEN I fill the "Site URL" field
AND click on "Add" button
THEN New site will be create in system
AND I will transfer to my own site's list
How do you think, where do I need to apply a business rules such as:
1) When new site is created a unique code must be generated and consist of minimum 8 characters including numbers and Alphabetical symbols.
2) checks doesn't apply for Site URL input field and user can input a Cyrillic symbols
3) etc?
I have a lot of additional question and hope on the community help!
The thing with BDD is to stay away from implementation details as much as possible. This scenario has multiple implementation details:
GIVEN I'm logged into admin panel AND I'm on "Sites" page
WHEN I click "Add site" button
THEN Pop-up window "Add site" come up
What happens if the "Sites" page becomes "Awesome Site" page or is simply deleted?
What happens if "Add site" is not a button anymore?
What happens if it's not a popup but a redirect occuring
What happens after? Is the value simply in showing the popup? I guess not...
For this specific example, a better approach would be:
GIVEN I'm an authorised administrator
WHEN I enter all the required information for a new site and save it
THEN I should see that site in my own sites list
With this scenario, if your implementation changes, you will only have to change your step definitions, you won't have to change your gherkin. Don't forget that those tests are supposed to explain the Behaviour of the system, not the way it is implemented.
The other questions you have are more related to unit testing in my opinion:
When new site is created a unique code must be generated and
consist of minimum 8 characters including numbers and Alphabetical
symbols => I'd do it at the class level, gherkin would not be
appropriate unless the customer specifically asked for this, then
the condition is "THEN a code having the required characteristics is
generated for that site" and you would have to define "required
characteristics" in a glossary the customer can read and understand.
Checks doesn't apply for Site URL input field and user can input a Cyrillic symbols => again, would put that at the class level en same as for 1. unless the customer wants to be able to read something about it, it should be at the unit level.
I hope that answers your question. I recommend this article by Dan North if you want to have a better idea of how to write better gherkin features.
EDIT 11/13/14
Based on your comments, I suggest we take a step back and describe a way to deal with requirements in your case. I have to tell you that I'm not a BDD expert and am only sharing my own personal experience, for more info on the subject I suggest you get in touch with the guys behind BDD Kickstart and Cucumber.pro where you will find online BDD courses. They will be able to give you lots of information and books to read.
That being said, let's dive into the subject.
The first thing you get is a list of features or stories that, if you follow Mike Cohn's story template would look like:
As a <type of user> I want <to do something> in order to <serve a business purpose>
I personally like to put the business purpose first to make sure we don't skip it in the discussions. You might also not follow that template and that's fine, but remember that it's a good idea to make sure the features you are listing with your customers have a business purpose. If there is no business value behind a feature then what's the point of doing it anyway...
So you do have a list of features/stories described as above. Now for each of these features, there are different cases or scenarios, that's what Dan describes in his article. This is where the Given-When-Then is introduced.
Scenario: Title
Given <some context>
When <there is an event>
Then <something happens>
Each of those scenarios are examples on how this specific feature behaves in different contexts. They are the different acceptance criteria for a specific feature, things the customer described as the expected behaviour of the system. They should be ignorant of any implementation details. So stuff like:
Given I am on page "First page"
When I click "Hello world"
Then I should see "You clicked hello world"
Is wrong for the reasons described prior to this edit.
Let's assume the following feature:
In order to save time when answering clients requests, as a webmaster,
I want to be able to manage the list of websites I am responsible for
Scenarios for this story would be:
Scenario 1: Show a list of websites
GIVEN I am an authorised administrator
AND I am managing several websites
THEN I should see a list of all the sites I manage
Scenario 2: Add website to list
GIVEN I am an authorised administrator
WHEN I enter all the required information for a new site and save it
THEN I should see that site in my own sites list
Scenario 3: Edit website from list
GIVEN I am an authorised administrator
WHEN I edit the site informations
THEN I the changes should be visible in my sites list
...
Now what if you want to go into data validation stuff like "site should have a title" for instance. To me there are two different ways to approach this. You can test that from the user's perspective with a full-stack test or test that there is some validation at the object level.
Let's assume the following scenario:
Scenario: New site has no title
GIVEN I'm an authorised administrator
WHEN I forget to fill in the title for a new site and save it
THEN I should be warned the site is not valid
You can use cucumber or specflow to run this scenario from the UX, therefore using some kind of browse-based proxy to test your app. That is usually slow as it hits the whole system and simulates a real user. It's an option, but I don't think it's the best though. IMO not all tests should be run against the UX and having too many Gherkin features can be a pain to maintain, that's why I prefer focusing on having the happy or critical path (usually I ask myself, where does the money comes from) tested full-stack and put the rest at lower levels.
You can still use Gherkin for these unit tests if you'd like. But that is not mandatory. You only need a way to show your customer you actually have a test for all those specific format controls and validation checks.
That doesn't mean you are not doing BDD anymore, you can still use the given-when-then-should pattern in rspec if you're a rubyist, or any other testing framework you use.
Hope that clarifies all this, let me know if there are any confusing parts...
I think Marc simply deserves the big green tick on this one, thanks to his amazingly thorough answer!
I just wanted to add a few comments.
You don't need to automate all your scenarios.
If you want to capture business requirements in a form that everyone (i.e. including non-tech savvy folks) can understand and Gherkin's Given/When/Then work for you, just go for it. There's nothing forcing you to automate all of your scenarios.
You don't need to automate all your scenarios through the UI.
Your software is made of layers that often respond to similar behaviours, via different interfaces (UI, HTTP, API, ...). Should you want to describe fine-grained business rules (i.e. site name constraints) with automated gherkin scenarios, you could write step definitions that talk directly to your domain layer instead of going through the user interface. That would probably still give you a decent level of confidence.
As a side note, I would recommend not to use Given/When/Then in classic testing frameworks (i.e. those that only devs can see!) if your purpose is to share your tests/requirements with non-tech people.
Have conversations!
Above all, BDD is about better communication: try to talk more, involve your developers (or some of them) earlier in the process so that they gain more knowledge, sooner. Formalising Gherkin scenarios comes in a second phase. Automating them should even be further down your priority list!
There is a certain flow within our application that makes separate calls with related data. These are in different controllers, but interact with the same user.
We're trying to build a test to confirm that the full flow works as expected. We've built individual tests for the constituent parts, but want a test for the full flow.
E.g., we have a user who checks in to work (checkin) and then builds a widget (widgetize). We have methods that will filter our users between who have checked in, and who have widgetized (and checked in). We can build little objects with FactoryGirl to ensure that the filter works, but we want a test that will have a user check in, another user check in, and the second one widgetize so that we can confirm that our filtering methods only capture the users we want it to capture.
My first thought was to build an rspec test that simply made a direct call to checkin from the widgetize spec, and then confirm the filter methods -- but I discovered that rspec does not allow cross controller calls (or at least I could not figure out how to make it work; posts and gets to that controller were not working). Also, people told me this was very bad practice.
How should I go about testing this?
This article goes over how you can use request-specs to go through integration tests fairly well.
http://everydayrails.com/2012/04/24/testing-series-rspec-requests.html
Basically you want to use a gem like capybara so you can simulate user input and get your tests to run through your app and check to see if everything is going as you expect it to.
I am implementing some smoke tests to our website.
I'm using a Given/When/Then format for existing automated acceptance tests/User stories. But now I want to do an initial smoke test of:
Given I'm on the homepage
Then I should see "Welcome To The Site"
Am I missing something? Is it "ok" to not have a When?
Tools Used: MVC3, SpecFlow, Nunit, Watin
It is completely valid syntax to omit any of Given, When or Then (and even to mix them in any order - specflow doesn't care.)
However, for the purpose of readability, rather than omit the When I often rephrase the Given, i.e.
When I view the homepage
Then I should see "Welcome To The Site"
I prefer to omit the Given section, because the When is supposed to indicate what the tested action is.
If the code for the step binding is the same and you want to re-use it, you can always bind your given and my when to the same method.
[Given(#"I'm on the homepage"]
[When(#"I view the homepage"]
public void NavigateToHomePage()
{
...
I think you are really missing the point here. You ALWAYS need a When. That is the thing you should be testing! What you can leave out are the Givens
What you should be saying is;
When I visit the homepage
Then I should see "Welcome To The Site"
Given When Then is really a nicer way of representing a state machine.
Given some initial state // in your case, non
When I perform some action // in your case, visiting the homepage
Then I have some final state // in your case, text displayed to a user
What I like to do is to think about all the things that must be present to allow the When to happen. In your case there doesn't seem to be any initial state. But consider if you had some web application. You would need to have an initial state before visiting the homepage (you'd need to make sure the user is logged in);
Given a user // user must be stored in the database
And the user is logged in // a logged in user must be in the session
When the user visits their homepage
Then the user should see "Welcome To Your Homepage"
An alternative scenario would be;
Given no logged in user // some people would leave this Given out, but I add it for completness
When a user visits their homepage
Then the user should be redirect to the login page
As someone correctly pointed out, most BDD tools don't actually differentiate between Given When Then but you should! The verbose nature of 'Given When Then' has been chosen as its easier for us humans to understand and helps our thought processes. A machine couldn't care less what you call the steps. With this being the case, you should make every effort to utilise the keywords correctly at all times.
Additional
BDD structure is no different to a well set-out test with arrange, act, assert.
The benefit of BDD though is that it gives a verbose structure to tests. This helps devs have domain appropriate conversations with product owners - Behaviour Driven Development.
If you aren't having these conversations, there's very little value in using BDD over normal tests practices.
I tend to see Given as the equivalent of traditional pre-conditions. When as the equivalent of the test action. And Then as the equivalent of the expected result.
Therefore, if there are no pre-conditions, I would leave out Given and simply focus on When and Then:
When I'm on the homepage
Then I should see "Welcome To The Site"
Specflow will allow you to use Given or When, but Visual Studio will also allow you to write a single class that is 1000's of lines long. Just because both are possible, does not mean either is 'right'.
Apologies for the thread resurrection...
I'd have probably gone with:
Given there is a homepage
When I view the homepage
Then I should see "Welcome To The Site"
I like to keep at least one Given, When and Then in each Scenario - don't forget you can also use And and But (not that they're particularly relevant to this scenario). You can even just make a bullet-point style list with *.
I would say:
Given I have requested the home page
When the home page loads
Then I should see 'Welcome To The Site'
You don't need a When. I like the think of the Given/When/Then keywords like
Given - This is a preparation step, do anything you need to be able to perform the test
When - This should be an action that your test will verify.
Then - This should be where you verify your test based on the action performed in the When steps.
As previously suggestion they only affect the execution order.
I am doing acceptance testing with Steak on a Ruby on Rails application. Imagine I want to test the functionality of a form.
It should create an user if all the fields are correct.
It should not create a user if any of the fields are incorrect.
In the first case, a message will inform that the user has been created: 'The user has been created'
In the second case error messages will indicate the errors.
I can base my tests on the information displayed. That is, the first test pass if the correct message is displayed.
Or I can base my tests on the changes on the database. That is, the first test pass if the database has a new user which contains the data entered.
Or I can assert for both conditions, database should change and the appropriate message should be displayed.
What is the conceptually adequate way of testing this kind of behaviour, according to Behaviour Driven Development?
What is the practical way of testing this kind of behaviour, according to being a Pragmatic Programmer?
Generally, keep to what the user sees.
However, sometimes there's more than one user of the information. Perhaps, once the data has reached the database, another application then uses that data (fairly typical for internal projects!)
If you can't automate with both applications together, then asserting the contents of the database, or perhaps verifying resources on a RESTful URL, can be a good way to go.
If you do this, it will probably change the nature of your scenarios. Rather than saying, Then the database should contain XYZ, try something like:
Given that Fred Brown lives at 25 Warrington Grove, Springfield
When Fred orders a deckchair
Then the warehouse should receive an order:
Deckchair
Quantity: 1
To: Fred Brown
25 Warrington Grove
Springfield
So you're always looking at the capabilities of the system, rather than just the data - even if the warehouse doesn't really receive it in your scenario, you're showing why the data has to be in that particular place and form.
This way, you also leave yourself the option of using a real UI, or a RESTful URL, etc. later - the step is independent of the implementation you choose.
My BDD practices involve using Cucumber. I try my hardest to keep it to what the user sees. That is, any When and Then statements should reflect user input or user visuals.
In your case I would make sure that a success message appears afterwards, and then I would (if possible) look for data on the screen indicating the application is aware of the object. For instance, if you added a comment to a blog post you should fill in the post, hit the 'accept' button, see a 'Message Posted' success message, and then also see your content on the page as a new post.
You should back up your user-driven BDD tests with unit and sometimes controller tests to make sure everything behind-the-scenes is functioning correctly as well.
I tend to use RSpec and verify that things get added to the database, methods are returning expected results etc. But I also use Cucumber to make sure the user sees what the user is expecting to see (But I don't really let cucumber care about what's in the DB).
I hope this helps. If you aren't using cucumber yet I highly recommend it for BDD testing in conjunction with RSpec. You should check out the book (comes in electronic PDF form) - I started with that and it helped a ton with my testing practices:
http://www.pragprog.com/titles/achbd/the-rspec-book
I would agree with keeping with what the user see's, up to a point.
If there is some kind of layer that could potentially persist a value in memory, such that it might show up on the site as if it was in the db, when it's not actually been written out TO the db.. then it might be a good idea to back up your 'what the user's see's' with some checks at the db level.