I am writing an MVC2 asp.net web app. I would like to have some performance tests within the test project, where I emulate hundreds of users all accessing the database layer at the same time. Is there a way that I can write a test that will make lots of concurrent calls to different controllers? Do I need to spawn lots of threads to make the multiuser aspect truly concurrent?
I would recommend you using a specific tool for performing load tests instead of writing this logic manually by spawning new threads. For example if you have the full version of Visual Studio you could create web tests which represent different user scenarios and then execute those scenarios in a load test. Another useful tool is Apache JMeter.
Yes you need multiple threads to do a full integration testing simulating concurrent user behavior.
You can either create spawn threads yourself or use something Parallel.Foreach() for a slightly more convenient way.
However these shouldn't be part of your unit test suite because they're a different kind of test than unit tests. Best to not mix them with your regular unit tests.
Related
I'm trying to sift through the myriad of test solutions out there, and I'm not even sure if I'm headed the right direction. The story is: we're running a RESTful Web service, implemented as a Rails app, which backs our mobile clients. We're unit testing the Web service (of course), but that involves mocking out many parts of the application, for instance, the search stack (Apache SOLR).
Moreover, our tests don't (i.e. cannot!) cover critical routes such as the mobile sign-in/sign-on process, since that involves communication between the API application and the mobile website, where the user can enter credentials, e.g. for SSO (Janrain Engage). Hence, a standard Rails integration test won't do.
I realize that in theory, if the test suite is very well designed, where mocking only happens strictly at the those join points where the tests for the next layer start, then by unit or functional testing the service API and the mobile website separately, one could get the same test coverage. I find that in practice though, this is an illusion if you have several developers working on the test suite independently; I just admit that our unit tests are simply not that well designed. Particularly when exercising TDD I found that while the tests can drive the application code, the test code design is only tailored to the unit under test, resulting in a rather wildly grown test suite.
Another thing I found is that sometimes we didn't detect regressions purely using unit tests, where e.g. bad queries were sent to the SOLR server due to knock-on effects. That's why I thought the only true way to ensure that the entire stack works along the critical routes is to automatically end-to-end test it on a staging server before every deployment, i.e. having actual HTTP requests sent to the app.
My questions would be:
do you think this is a reasonable thing to do at all? I found very little information about end-to-end testing a live API on the Web, leaving me wondering whether I'm making any sense at all
which tools/setup would you suggest? We use Watir to run acceptance tests for our website, but it seems to be overkill for a Web service (no browser environment needed, no JS or anything UI-ish). Something as simple as a Ruby script even?
any general best practices or advice you can give me w.r.t. to designing such tests?
This might be of interest to you: http://groups.google.com/group/ruby-capybara/browse_thread/thread/5c27bf866eb7fad3
What you might want to to try is combining Cucumber (or similar) with one of the tools mentioned in the link above, so you can do something like
Given I have 2 posts
When I send "DELETE" to post with id 1
Then I should have 1 post
This way you can test the API's full stack and check the result.
We've faced similar problems lately, like testing only on the unit level was fast but not enough to spot integration related anomalies.
I really like Cucumber's gherkin syntax, the value for me in it is that we can run the exercise once and then make several expectations after with separate titles, like if they would be separate RSpec it blocks.
There is a gem called Turnip, which makes possible to write RSpec tests with the gherkin syntax. We have been using it for a while and I'm quite happy with it, we even started having less integration level defects.
I collected my experience regarding in a blog post: https://blog.kalina.tech/2019/10/end-to-end-testing-for-api-only-rails-apps-with-cucumber-syntax.html
I am learning Behavior Driven Development with ASP.NET MVC and, based on a post from Steve Sanderson, understand that BDD can mean, at least, the following test types: individual units of code & UI interactions. Something similar is mentioned in this post. Do I need two different test frameworks if I want both unit and integration testing?
Unit testing repositories, controllers, & services using a context/specification framework, like MSpec. The results of testing with this will be useful to the development team.
Testing complete behaviors (integration) using a given/when/then framework, like SpecFlow with Watin. The results of this testing will be useful for my client.
The videos I have seen so far on using BDD have only been limited to testing the behaviour of entities without testing the behaviour of repositories, controllers, etc... Is there a sample project where I can see both automated Unit and Integration testing using a BDD approach?
Personally I use SpecFlow for building feature specific tests (i.e. "User creates new company record") where I'll sometimes (but not always) use Watin. For testing my respositories, or service classes, I'll use unit/integration tests with NUnit. Integration tests are for when I need to talk to the database during the test, unit is for when I simply run code in the target object under test without external interactions.
I would say that you don't need to use a BDD framework for your non UI tests. You can if you want, but there is no hard and fast rule on this. If you are going to do this, then I highly recommend creating more then one project for your tests. Keeping them split is a good idea, rather then mixing all the test into one project. You could name them:
MyProject.Tests.Features <-- For BDD
SpecFlow tests.
MyProject.Tests.Integration <-- For
tests that access an
external resource i.e. database.
MyProject.Tests.Unit
If you're not wanting to use two BDD frameworks, you can still use MSTest/NUnit in a BDD way. For example, this blog article describes a nice naming convention which is close to BDD, but aimed at MSTest/NUnit unit tests. You could use this for your non SpecFlow tests when your testing things like repositories.
In summary - you don't have to use SpecFlow and MSpec in your testing, but if you do, then I recommend separate test projects.
I generally agree with what Jason posted.
You might want to divide your specs into two categories, system/integration and unit-level tests. You can describe both categories with any framework, but keep in mind that code-only approaches (NUnit, MSpec, etc.) require a business analyst to be capable of writing C#. SpecFlow/Gherkin can be a better approach if you want to involve analysts and users in writing specifications. Since the syntax and rules (Given, When, Then) are easy to understand and writing specifications from a user's perspective are easy to jot down after little training. It's all about bridging the communication gap and having users helping your team form the ubiquitous language of your domain.
I recommend having specifications support both working "outside in" and "inside out". You may start with an "outside in" SpecFlow specification written by the user/analyst/product owner and work your way from "unimplemented" towards "green" writing the actual code. The code supporting the feature is developed using TDD with a more technically oriented framework like MSpec (the "inside out" part).
Here's a repository that use MSpec for both unit and integration tests: https://github.com/agross/duplicatefinder.
Seems like the internet doesn't have a definitive answer, or set of principles to help me answer the question. So I turn to the great folk on SO to help me find answers or guiding thoughts :)
SpecFlow is very useful for BDD in .NET. But when we talk about BDD are we just talking integration/acceptance tests, or are we also talking unit tests - a total replacement for TDD?
I've only used it on small projects, but I find that even for my unit tests, SpecFlow improves code documentation and thinking in terms of language. Converseley, I can't see the full code for a test in one place - as the steps are fragmented.
Now to you..........
EDIT: I forgot to mention that I see RSpec in the RoR community which uses BDD-style syntax for unit testing.
I've recently started to use SpecFlow for my BDD testing, but also, I still use unit and integration tests.
Basically, I split the tests into seperate projects:
Specs
Integration
Unit
My unit tests are for testing a single method and do not perform any database calls, or external references whatsoever. I use integration tests for single method calls (maybe sometime two) which do interact with an external resources, such as a database, or web service, etc.
I use BDD to describe tests which mimick the business/domain requirements of the project. For example, I would have specs for the invoice generation feature of a project; or for working with a shopping basket. These tests follow the
As a user, I want, In order to
type of semantics.
My advise is to split your tests based on your needs. Avoid trying to perform unit testing using SpecFlow.
We have started using Specflow even for our unit tests.
The main reason (and benefit) for this is that we find that it forces you to write the tests from a behavior point of view, which in turn forces you to write in a more implementation agnostic way and this ultimately results in tests which are less brittle and more refactoring friendly.
Sure this can also be done with standard unit testing frameworks, but you aren't guided that way as easily as we have found we are using specflow and the gherkin syntax.
There is some overhead setting things up for specflow, but we find this is quickly repaid when you have quite a few tests (due to the significant step reusability that you can get with specflow) or you need refactor your implementation.
Plus you get nice readable specs that are easy for newcomers to the team to understand.
Given:
Unit tests are test of (small) "units of code"
The customer of most “units of codes” are other programmers.
Part of the reason for having a unit test is to provide an example of how to call the code.
Therefore:
Clearly unit tests should normally be written in the programming language that the users of the “unit of code” will be calling it with.
However:
Sometimes data tables are needed to setup the conditions a unit test runs in.
Most unit test frameworks are not good at using tables of data.
Therefore:
Specflow may be the best option for some unit test, but should not be your default choose.
I see it as an integration testing which mean it doesn't replace your unit test cases written as part of your TDD process. Someone will have different opinion about this. IMHO unit test case only test the methods/functions and all the dependencies should be mocked and injected. When in it comes to integration testing, you will be injecting real dependencies instead of mocked one. You could do the same integration testing with any of the unit testing frameworks, but the BDD provides you cleaner way of explaining the integration test use case in a Domain Specific Language which is a plain English(or any localized language).
Ta,
Rajeesh
I used specflow for BDD testing on two different good sized applications. Once we worked through the kinks of the sentence naming conventions, it worked out pretty good. BA's and QA's, and even interns could write BDD tests for the application.
However, I ALSO used it for unit tests. Heresy! I can hear some of you scream. However, there were VERY good reasons for it. The system was responsible for making many calculations or determinations based off a lot of different data. With lots of unit tests that require all this data to be input for test purposes, it makes it a LOT easier to manage that data used for the unit tests via the table format provided by specflow. Effectively mocking the data repository in table format, allowing the different components to be vigorously tested.
I don't know if I would do it in every case, but in the ones I used it for, it made laying out the volumes of data necessary for for performing the unit tests so much easier and clearer.
In the end we are trying to deliver to the customer exactly what the customer wants and as such I really don't see the need to write unit tests in addition to SpecFlow. After all, it exercises the same code base. I am fairly new to BDD/ATDD/TDD but other than being "complete" and strictly adhering to TDD I'm finding it unnecessary to write more unit tests.
Now I suppose if the team was dispersed and the developer was not able to run the entire application then separate unit tests would be necessary but where the developer(s) has access to the entire code base and is able to run the application, then why bother write more tests.
I'm currently writing integration tests for a RoR REST API using cucumber.
I'd also like to stress-test the API and was wondering how I might be able to re-use my cucumber scenarios to do this.
I've used JMeter and httperf before for stress-tests, and of course with both tools had to write or record the tests that I wanted to include in the stress testing. Since I already have the scenarios in Cucumber that I want to test, I'm trying to avoid having to rewrite what would be largely the same scenarios in the stress-testing tool's syntax.
Can anybody recommend a way that I could use my cuke tests for this? Ideally, I'd like to simulate hundreds of concurrent users hitting the API.
You very well may have to create your own load harness. Essentially, you'll need to start up a number of threads, each calling the process to start your scenarios. You will want to have a delay to stagger when each thread starts because there will be some overhead involved with loading the Ruby interpreter. You may also have to spread the load across more than one machine for the same reasons. It would probably be best to do this in separate processes. Perhaps this might be good for a new project?
Hmm, what about creating some steps that perform a system call to httpref ?
I am looking at SpecFlow examples, and it's MVC sample contains several alternatives for testing:
Acceptance tests based on validating results generated by controllers;
Integration tests using MvcIntegrationTestFramework;
Automated acceptance tests using Selenium;
Manual acceptance tests when tester is prompted to manually validate results.
I must say I am quite impressed with how well SpecFlow examples are written (and I managed to run them within minutes after download, just had to configure a database and install Selenium Remote Control server). Looking at the test alternatives I can see that most of them complement each other rather than being an alternative. I can think of the following combinations of these tests:
Controllers are tested in TDD style rather than using SpecFlow (I believe Given/When/Then type of tests should be applied on higher, end-to-end level; they should provide good code coverage for respective components;
MvcIntegrationTestFramework is useful when running integration tests during development sessions, these tests are also part of daily builds;
Although Selenium-based tests are automated, they are slow and are mainly to be started during QA sessions, to quickly validate that there are no broken logic in pages and site workflow;
Manual acceptance tests when tester is prompted to confirm result validity are mainly to verify page look and feel.
If you use SpecFlow, Cucumber or other BDD acceptance test framework in you Web development, can you please share your practices regarding choosing between different test types.
Thanks in advance.
It's all behaviour.
Given a particular context, when an event occurs (within a particular scope), then some outcome should happen.
The scope can be a whole application, a part of a system or a single class. Even a function behaves this way, with inputs as context and the output as outcome (you can use BDD for functional language as well!)
I tend to use Unit frameworks (NUnit, JUnit, RSpec, etc.) at a class or integration level, because the audience is technical. Sometimes I document the Given / When / Then in comments.
At a scenario level, I try to find out who actually wants to help read or write the scenarios. Even business stakeholders can read text containing a few dots and brackets, so the main reason for having a natural language framework like MSpec or JBehave is if they want to write scenarios themselves, or show them to people who will really be put off by the dots and brackets.
After that, I look at how the framework will play with the build system, and how we'll give the ability to read or write as appropriate to the interested stakeholders.
Here's an example I wrote to show the kind of thing you can do with scenarios using simple DSLs. This is just written in NUnit.
Here's an example in the same codebase showing Given, When, Then in class-level example comments.
I abstract the steps behind, then I put screens or pages behind those, then in the screens and pages I call whatever automation framework I'm using - which could be Selenium, Watir, WebRat, Microsoft UI Automation, etc.
The example I provided is itself an automation tool, so the scenarios are demonstrating the behaviour of the automation tool through demonstrating the behaviour of a fake gui, just in case that gets confusing. Hope it helps anyway!
Since acceptance tests are a kind of functional tests, the general goal is to test your application with them end-to-end. On the other hand, you might need to consider efficiency (how much effort is to implement the test automation), maintainability, performance and reliability of the test automation. It is also important that the test automation can easily fit into the development process, so that it supports a kind of "test first" approach (to support outside-in development).
So this is a trade off, that can be different for each situation (that's why we provided the alternatives).
I'm pretty sure, that today the most widely fitting option is to test at the controller layer. (Maybe later as UI and UI automation frameworks will evolve, this will change.)