There is a certain flow within our application that makes separate calls with related data. These are in different controllers, but interact with the same user.
We're trying to build a test to confirm that the full flow works as expected. We've built individual tests for the constituent parts, but want a test for the full flow.
E.g., we have a user who checks in to work (checkin) and then builds a widget (widgetize). We have methods that will filter our users between who have checked in, and who have widgetized (and checked in). We can build little objects with FactoryGirl to ensure that the filter works, but we want a test that will have a user check in, another user check in, and the second one widgetize so that we can confirm that our filtering methods only capture the users we want it to capture.
My first thought was to build an rspec test that simply made a direct call to checkin from the widgetize spec, and then confirm the filter methods -- but I discovered that rspec does not allow cross controller calls (or at least I could not figure out how to make it work; posts and gets to that controller were not working). Also, people told me this was very bad practice.
How should I go about testing this?
This article goes over how you can use request-specs to go through integration tests fairly well.
http://everydayrails.com/2012/04/24/testing-series-rspec-requests.html
Basically you want to use a gem like capybara so you can simulate user input and get your tests to run through your app and check to see if everything is going as you expect it to.
Related
I have a rails-app and I'm trying making my cucumber/capybara/selenium-webdriver tests faster. Most of my scenarios look like:
#javascript
Feature: Writing reports
In order .....
Background:
Given I am logged in
Scenario: Scenario 1
...........
Scenario: Scenario 2
...........
Most of the time is spent in "Given I am logged in" where I go to the sign-up-page to create an user and log in. The authentication is implemented using Devise.
How can I make that I don't have to do the log-in in the test environment at all?
Are there other best practises to get around the tedious log-in-step? (And only keep the log-in for those scenarios that test the log-in/sign-up specifically)
Your cucumber scenarios are your living and breathing documentation. If you need to be logged in to use that part of the app then your cucumber steps should explicitly say so.
As for what you're calling under the hood in Capybara. You could look at the devise session controller and use the same method calls directly in your integration tests. However, if Devise changes it's implementation, your integration tests will break. Where possible your integration tests should not be concerned with any underlying implementation details.
The easiest code to maintain, and the most reliable test, will be to fill out the form as the user will.
You have a lot of UI deails in your scenarios. This makes it hard to understand which behaviour you really are verifying.
The scenario names gives some hints, but the concrete example in the Given/When/Then contains a lot of details that obfuscates the purpose.
Scenario: Login with invalid username/password
Given I am signed out
When I log in with invalid user credentials
Then should I get an error message
Scenario: Login with valid username/password and valid PIN
Given I am signed out
When I log in with valid credentials
then should I be greeted
These two examples does not contain any details about a link or pin form. I see those as implementation details that I would push down in the stack. I would push them to a helper class that the steps are using. You seem to be verifying a web application, I would let these details live in a page object that supports the steps.
I would also consider removing the usage of the personal pronome I and change that to a persona instead. A persona that is a typical for a user of your system. The persona has a name. Say that we create a persona called 'Ann' who is a 35 year old woman working at the administration department. This would give me the opportunity to rewrite the scenarios to
Scenario: Login with invalid username/password
Given Ann am signed out
When Ann log in with invalid user credentials
Then will she receive an error message
Scenario: Login with valid username/password and valid PIN
Given Ann am signed out
When Ann log in with valid credentials
then should Ann be greeted
This give us some context, we know who Ann is. We can keep her user name and password in our supporting classes and expect the implementation to use them properly.
If you are using Warden (which Devise runs on), might check out http://blog.crowdint.com/2012/11/02/using-warden-to-login-logout-within-your-cucumber-tests.html.
The author proposes a method to use Warden helpers instead of filling out the form. Basically logging in with pure Ruby instead of through the browser. For scenarios specifically testing login/logout functionality, you would still test filling in the individual fields, but this might speed up your other cases.
Don't sign the user up. Just User.create! as a part of the Given step. And only sign them in via web UI.
Avoid #javascript where possible. This way you can utilze devise test helper sign_in rather than going via web UI.
Use faster javascript driver. For example, as of this writing, poltergeist runs about twice as fast as Firefox on my machine.
I have developed a jBehave story to test some work flow implemented in our system.
Let’s say this story is called customer_registration.story
That story is a starting point of some other more complex work flows that our system supports.
Those more complex work flows are also covered by different stories.
Let’s say we have one of our more complex work flows covered by a customer_login.story
So the customer_login.story will look somehow like below:
Story: Customer Login
Narrative:
In order to access ABC application
As a registered customer
I want to login into the application
Scenario: Successfully login into the application
GivenStories: customer_registration.story
Given I am at the login page
When I type a valid password
Then I am able to see the application main menu
All works perfectly and I am happy with that.
3.The story at point 1 above (customer registration) is something I need to run on different sets of data.
Let’s say our system supports i18n and we need to check that customer registration story runs OK for all supported languages, say we want to test our customer registration works OK with both en-gb and zh-tw
So I need to implement a multi_language_customer_registration.story that will look something like that:
Story: Multi language customer registration
Narrative:
In order to access ABC application
As a potential customer
I want to register for using the application
Scenario: Successfully customer registration using different supported languages
GivenStories: customer_registration.story
Then some clean up step so the customer registration story can run again
Examples:
|language|
|en-gb |
|zh-tw |
Any idea about how I could achieve this?
Note that something like below is not an option as I do need to run the clean up step between runs.
GivenStories: customer_registration.story#{0},customer_registration.story#{1}
Moving the clean up step inside the customer registration story is not an option too as then the login story will stop working.
Thanks in advance.
P.S. As you could guess in reality the stories we created are more complex and it is not an easy task to refactor them, but I am happy to do this for a real gain.
First off, BDD is not the same as testing. I wouldn't use it for every single i18n scenario. Instead, isolate the bit which deals with i18n and unit test that, manually test for a couple and call it done. If you really need more thorough then use it with a couple of languages, but don't do it with all of them - just enough examples to give you some safety.
Now for the bit with the customers. First of all, is logging in and registration really that interesting? Are you likely to change them once you've got them working? Is there anything special about logging in or registration that's particular to your business? If not, try to keep that stuff out of the scenarios - it'll be more of a pain to maintain than it's worth, and if it's never going to change you can just test it once, manually.
Scenarios which show what the user is logging in for are usually more enticing and interesting to the business (you are having conversations with the business, right?).
Otherwise, here are the three ways in which you can set up a context (Given):
By hacking the data (so accessing the database directly)
Through the UI (or controller if you're automating from that level)
By using existing data.
You can also look to see if data exists, and if it doesn't, set it up. So for instance, if your customer is registered and you don't want him to be registered, you can delete his registration as part of setting up the context (running the Given step); or if you need him to be registered and he isn't, you can go through the UI to register him.
Lastly, JBehave has an #AfterScenario annotation which you can use to denote a clean-up step for that scenario. Steps are reusable - you can call the steps of the scenario from within another step in code, rather than using JBehave's mechanism (this is more maintainable anyway IMO) and this will allow you to avoid clearing registration when you log in.
Hope one of these options works for you!
From a tactical standpoint, I would do this:
In your .story file
Given I set my language to {language}
When I type a valid password {pass}
Then I am able to see the application main menu
Examples:
|language|pass|
|en-gb |password1|
|zh-tw |kpassword2|
Then in your Java file,
#Given ("I set my language to $lang")
#Alias ("I set my language to {language}")
// method goes here
#When ("I type a valid password $pwrd")
#Alias ("I type a valid password {pass}")
// method goes here
#Then ("I am able to see the application main menu")
most unit testing frameworks support this.
Look how mstest you can specify DataSource, nunit is similar
https://github.com/leblancmeneses/RobustHaven.IntegrationTests
unfortunately some of the bdd frameworks i've seen try to replace existng unit test frameworks when it should instead work together to reuse the infrastructure.
https://github.com/leblancmeneses/BddIsTddDoneRight
is fluent bdd syntax that can be used with mstest/nunit and works with existing test runners.
I am implementing some smoke tests to our website.
I'm using a Given/When/Then format for existing automated acceptance tests/User stories. But now I want to do an initial smoke test of:
Given I'm on the homepage
Then I should see "Welcome To The Site"
Am I missing something? Is it "ok" to not have a When?
Tools Used: MVC3, SpecFlow, Nunit, Watin
It is completely valid syntax to omit any of Given, When or Then (and even to mix them in any order - specflow doesn't care.)
However, for the purpose of readability, rather than omit the When I often rephrase the Given, i.e.
When I view the homepage
Then I should see "Welcome To The Site"
I prefer to omit the Given section, because the When is supposed to indicate what the tested action is.
If the code for the step binding is the same and you want to re-use it, you can always bind your given and my when to the same method.
[Given(#"I'm on the homepage"]
[When(#"I view the homepage"]
public void NavigateToHomePage()
{
...
I think you are really missing the point here. You ALWAYS need a When. That is the thing you should be testing! What you can leave out are the Givens
What you should be saying is;
When I visit the homepage
Then I should see "Welcome To The Site"
Given When Then is really a nicer way of representing a state machine.
Given some initial state // in your case, non
When I perform some action // in your case, visiting the homepage
Then I have some final state // in your case, text displayed to a user
What I like to do is to think about all the things that must be present to allow the When to happen. In your case there doesn't seem to be any initial state. But consider if you had some web application. You would need to have an initial state before visiting the homepage (you'd need to make sure the user is logged in);
Given a user // user must be stored in the database
And the user is logged in // a logged in user must be in the session
When the user visits their homepage
Then the user should see "Welcome To Your Homepage"
An alternative scenario would be;
Given no logged in user // some people would leave this Given out, but I add it for completness
When a user visits their homepage
Then the user should be redirect to the login page
As someone correctly pointed out, most BDD tools don't actually differentiate between Given When Then but you should! The verbose nature of 'Given When Then' has been chosen as its easier for us humans to understand and helps our thought processes. A machine couldn't care less what you call the steps. With this being the case, you should make every effort to utilise the keywords correctly at all times.
Additional
BDD structure is no different to a well set-out test with arrange, act, assert.
The benefit of BDD though is that it gives a verbose structure to tests. This helps devs have domain appropriate conversations with product owners - Behaviour Driven Development.
If you aren't having these conversations, there's very little value in using BDD over normal tests practices.
I tend to see Given as the equivalent of traditional pre-conditions. When as the equivalent of the test action. And Then as the equivalent of the expected result.
Therefore, if there are no pre-conditions, I would leave out Given and simply focus on When and Then:
When I'm on the homepage
Then I should see "Welcome To The Site"
Specflow will allow you to use Given or When, but Visual Studio will also allow you to write a single class that is 1000's of lines long. Just because both are possible, does not mean either is 'right'.
Apologies for the thread resurrection...
I'd have probably gone with:
Given there is a homepage
When I view the homepage
Then I should see "Welcome To The Site"
I like to keep at least one Given, When and Then in each Scenario - don't forget you can also use And and But (not that they're particularly relevant to this scenario). You can even just make a bullet-point style list with *.
I would say:
Given I have requested the home page
When the home page loads
Then I should see 'Welcome To The Site'
You don't need a When. I like the think of the Given/When/Then keywords like
Given - This is a preparation step, do anything you need to be able to perform the test
When - This should be an action that your test will verify.
Then - This should be where you verify your test based on the action performed in the When steps.
As previously suggestion they only affect the execution order.
I am doing acceptance testing with Steak on a Ruby on Rails application. Imagine I want to test the functionality of a form.
It should create an user if all the fields are correct.
It should not create a user if any of the fields are incorrect.
In the first case, a message will inform that the user has been created: 'The user has been created'
In the second case error messages will indicate the errors.
I can base my tests on the information displayed. That is, the first test pass if the correct message is displayed.
Or I can base my tests on the changes on the database. That is, the first test pass if the database has a new user which contains the data entered.
Or I can assert for both conditions, database should change and the appropriate message should be displayed.
What is the conceptually adequate way of testing this kind of behaviour, according to Behaviour Driven Development?
What is the practical way of testing this kind of behaviour, according to being a Pragmatic Programmer?
Generally, keep to what the user sees.
However, sometimes there's more than one user of the information. Perhaps, once the data has reached the database, another application then uses that data (fairly typical for internal projects!)
If you can't automate with both applications together, then asserting the contents of the database, or perhaps verifying resources on a RESTful URL, can be a good way to go.
If you do this, it will probably change the nature of your scenarios. Rather than saying, Then the database should contain XYZ, try something like:
Given that Fred Brown lives at 25 Warrington Grove, Springfield
When Fred orders a deckchair
Then the warehouse should receive an order:
Deckchair
Quantity: 1
To: Fred Brown
25 Warrington Grove
Springfield
So you're always looking at the capabilities of the system, rather than just the data - even if the warehouse doesn't really receive it in your scenario, you're showing why the data has to be in that particular place and form.
This way, you also leave yourself the option of using a real UI, or a RESTful URL, etc. later - the step is independent of the implementation you choose.
My BDD practices involve using Cucumber. I try my hardest to keep it to what the user sees. That is, any When and Then statements should reflect user input or user visuals.
In your case I would make sure that a success message appears afterwards, and then I would (if possible) look for data on the screen indicating the application is aware of the object. For instance, if you added a comment to a blog post you should fill in the post, hit the 'accept' button, see a 'Message Posted' success message, and then also see your content on the page as a new post.
You should back up your user-driven BDD tests with unit and sometimes controller tests to make sure everything behind-the-scenes is functioning correctly as well.
I tend to use RSpec and verify that things get added to the database, methods are returning expected results etc. But I also use Cucumber to make sure the user sees what the user is expecting to see (But I don't really let cucumber care about what's in the DB).
I hope this helps. If you aren't using cucumber yet I highly recommend it for BDD testing in conjunction with RSpec. You should check out the book (comes in electronic PDF form) - I started with that and it helped a ton with my testing practices:
http://www.pragprog.com/titles/achbd/the-rspec-book
I would agree with keeping with what the user see's, up to a point.
If there is some kind of layer that could potentially persist a value in memory, such that it might show up on the site as if it was in the db, when it's not actually been written out TO the db.. then it might be a good idea to back up your 'what the user's see's' with some checks at the db level.
So I just started using Steak, which in turn uses Capybara, which in turn uses Selenium.
So I've heard it's good RSpec practice to split testing into lots of little it-clauses, each testing a small piece of functionality. But then it makes a test run take a lot longer, because the same steps get repeated for each test.
Say I'm testing some ajax functionality. I want to test that:
a form appears when I click a certain link
error messages appear if I submit invalid input to that form
a confirmation message appears if I submit valid input, etc.
But if I have each of those things broken into its own test case, then Selenium has to start from scratch: it has to load the same page three times, click the link to make the form appear three times, etc. It strikes me as wasteful.
What I'd like to be able to do is specify a sequence of tests that preserve browser state, so each test continues where the previous one left off. Is that possible?
It is possible.
One way is to override Capybara's visit method and keep track of the last url and only revisit when it changes.
Something along the lines of:
module Capybara
alias original_visit visit
def visit(url)
original_visit url if url != $last_url
$last_url = url
end
end
but you should only do that if you really know what you are doing...