I have NUnit 2.5.10 and Specflow 1.8.1 installed with VS2010.
Right click on .feature file in VS, there is an item "Debug Specflow Scenarios" in context menu. But I always get "Could not find matching test runner". In Tools/Options/Specflow, there is no test runner option for NUnit.
Am I missing something?
Thanks in advance. :)
I don't know if NUnit has a seperate GUI for running tests as i use MSTests but this should be how u do it.
Setup a config file in your Test Project to tell Specflow what Test framework to use.
https://github.com/techtalk/SpecFlow/wiki/Configuration
The Tools > Options > Specflow > Test Runner tool is the tool that executes the Tests. The fact that nothing is showing here seems like the test runner is not installed. For example i have several, MSTest, Resharper, SpecRun, i normally leave it on Auto and just run the test from the test tool i want to use. I ran installers for these though which should add any VS extensions
You could try using the VS runner, Test > Windows > Test View
VS will just treat a Specflow test as a NUnit as the generated code behind is just an NUnit test (when configured that way). So i think the real question is how to debug an NUnit test in Visual Studio. A quick search suggested attaching the debugger to the NUnit test runner process.
Like Ryan I also have no test runner specifically for Nunit. However since I'm also running resharper I get additional resharper runners, and that is what Auto uses to run it.
Have you actually got nUnit installed on your machine? In which case you can test how successful the process is without VS and Specflow-addin getting in the way. If not, well installing may give you the runner you need.
Alternatively, I use nCrunch to automatically run all my tests. I've had it running nunit/specflow for the last few months on multiple solutions with no issues. No need to right click at all then. :-)
SpecFlow itself does not provide a test runner and if you want to right click on the .feature and run it, you'll have to re-configure SpecFlow to use MSTest, which should use built-in VS runner.
If you want to use NUnit, then additional tools are required. I would recommend Resharper, but it's not free. Check out this article for more options to run NUnit tests from VS.
Good day. To debug i do the following :
Put a break point on the scenario line that you are interested in
Make sure that you have NUnit running
Attach the NUnit process
Go to tools
Select attach process
In the available processes select NUnit agent.exe
Click attach
Run the test you want to debug in NUnit
Hope that helps
Related
I've just literally hit a brick wall trying to configure F# 4.5.0.0 under VS 15.8.1 to run xUnit tests. Here is the repo with the issue: https://github.com/kkkmail/fSharp-xUnit .
When I try to run all tests, VS comes back with this message: No test is available in C:\GitHub\fSharp-xUnit\DummyApp\DummyApp\bin\Debug\DummyApp.exe C:\GitHub\fSharp-xUnit\DummyApp\Tests\bin\Debug\Tests.dll C:\GitHub\fSharp-xUnit\DummyApp\DummyApp\DummyApp.fsproj C:\GitHub\fSharp-xUnit\DummyApp\Tests\Tests.fsproj. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Both the "DummyApp" and tests were created from scratch using VS and I cleared all temp files and all nuget caches prior to creating the solution as advised in various places on the web.
TL;DR Add xunit.runner.visualstudio to your test project.
For discovering tests Visual Studio relies on its test runner. This test runner can discover / run all tests for testing frameworks which implement the corresponding adaptor interfaces ITestDiscoverer and ITestExecutor. For xUnit, these interfaces are implemented in VsTestRunner which is published in xunit.runner.visualstudio.
See also the xUnit docs about running tests in VS
We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:
MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"
So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.
In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.
I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.
Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?
I found the solution, it was to switch from using "Test Plan" to "Test Assembly"
Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.
We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.
VSTest.Console.exe is optimized for performance and is used in place
of MSTest.exe in Visual Studio.
After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.
MSTest.exe You can use the MSTest.exe program to run automated tests
in a test assembly from a command line.
MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.
MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server.
More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe
Since you are using TFS2015, and if you don't have any VS2010 test projects,
we encourage you to use VStest instead of MStest.
My team is building out a regression suite of SpecFlow tests for one of our websites and have the tests working locally. We've been trying to get the build server (TFS 2013 build server) to be able to run these tests when we check in code as well. We haven't been able to get this to work.
Locally we have our tests running on the NUnit framework, which we thought would work the same on the build server. It seems the build server only wants to run the tests on VS Test Runner which is failing to run our SpecFlow tests every time.
If I log into the build server remotely and run the SpecFlow tests in VS against SpecRun the tests work and pass. We've been playing around with the build definition as well, but haven't had any luck. What needs to be set up to get this to run when code is checked in?
Here is our pattern matching for our SpecFlow test project:
And here is an image of our build output as VS Test Runner is trying to execute one of our SpecFlow tests:
UPDATE: We've got our tests to run on the build server. However, our tests that are actually trying to hit our web sites keep generating the following error:
Test method
AccessTheNDCHomePage threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote WebDriver server for URL http://localhost:7055/hub/session/ae10bda6-c46c-4c35-bd9f-f2f9400767ef/url timed out after 60 seconds. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OpenQA.Selenium.Remote.HttpCommandExecutor.CreateResponse(WebRequest request)
After looking around online it seems like this could potentially be because of Firefox update to version 38 and/or a recent Selenium/WebDriver update. Does anyone have any information or tips on fixing this issue?
If you are already able, with the same solution, to build and run other nUnit tests with the build then you likley only have a pattern match issue.
The build specified a pattern of assembly names to go find test in. If you change It to included "spec" then your tests will run.
In XAML build It is on the process tab of the build definition editor. In the new build system you select the "VS Test" task and edit It there.
Firstly you need to understand what SpecFlow is. It is basically a unit test generation tool. It generates the unit tests in the language defined in the config.
if you wrote plain NUnit tests in your solution what would you need to do to get them to run on TFS? you would have to have a NUnit runner on the the build server and have TFS configured to use this. It's no different for SpecFlow tests.
So the solution? Either generate your tests in MSTest format locally, then when you check these tests in TFS will run them. If you don't want to do that, then set TFS up to be able to run NUnit flavour tests.
EDIT
It sounds as though when you say 'fail to run' the specflow tests, these are not failing, but not being found. Probably you need to adjust either the name of the assemblies that contain your specflow tests so that the current patterns in tfs find the test assemblies, or add filters to allow TFS to find your new assemblies.
Here were some of the issues and solutions we found to set up SpecFlow on our build server:
The build server wouldn't always recognize our SpecFlow test project - Setting the Platform target of the SpecFlow test to x86 seemed to fix this issue
WebDriver time out after 60 seconds error - I am still getting this locally and it still happens when we try to run our tests via the browser on our build server. We were able to run our tests against our Selenium Grid as well as against SauceLabs via the build server. If I find a solution to this I will update my answer.
I have had the same issue in the past, the answer is to make sure that your build agent is configured to run interactive.
The build agent must be configured to run interactive in order to run the selenium/coded UI tests against a browser.
https://msdn.microsoft.com/en-us/library/ms181712.aspx#interactive
I am using TFS 2013 and need to know if I can setup a build to run unit tests without the need to use Lab Management. I don't have the ability to setup the Lab Management but want the build when completed to run the unit tests associated with it. Is that possible? If so, where are the tests run? Would the same be true with Coded-UI tests?
Running unit tests from the build server is straight forward. Under the process tab in the build definition, go to 2. Basic > Automated Tests. There you can filter by filename or select a test settings file.
If you want to run Coded UI Tests from the build there are two options.
The first is to configure the unit test run to run the tests interactively. This will allow CUIT tests to run. Basically they will run like you would run them from Visual Studio. I wrote this post for TFS 2010, but it hasn't changed for TFS 2013.
http://www.codesmartnothard.com/2010/10/04/ConfiguringATFS2010TeamBuildServerToRunCodedUITests.aspx
The other option requires you to modify the build template and add an ExecuteRemoteTestRun activity to the end of the definition. This allows you to choose the test suites that contain the test cases with associated automation.
http://msdn.microsoft.com/en-us/library/vstudio/ff934562.aspx#bkmk_executeremotetestrun
Let me know if you have any questions about these options.
Mike
I have some automated tests written in nunit and selenium. How can I call the dll to execute from Microsoft test Manager. as I couldnt figure out how to run automated tests from MTM. is it possible.
MTM does allow to automate the test.
We can browse a test case in Visual Studio and attach the executable to the test case (Created in MTM). Later we have to link the build to the test plan as well for the execution of the automated test. Test code should be developed as a test project (CodedUI or Unit test) if not done so the test manager will not detect the method to be tested.
For MTM to run automated tests with Selenium you need to associate the test with a Test Case in MTM. Currently MTM only supports MS Tests so you would need to build a wrapper for each of your selenium tests.
Solution #1: I would build the wrapper with a T4 template that generated the required MS Test stubs that call my Selenium tests.
You would then need to have a Test Case in TFS with the MS Test 'automation' associated with it.
Solution #2: You can use the "tcm import" command line tool to generate and keep your Test Cases in sync with the automation.
Now that you have all of the bits configured you can go ahead and call the generated tests from MTM and you hit the issue that #richard mentioned.
Solution #3: You can configure an Environment in MTM where you are going to collect the data and automate the trigger of the test run. This can be done as part of the Build, or better as part of your binary Pipeline in Release Management (http://nakedalm.com/execute-tests-release-management-visual-studio-2013/)
While there are a lot of hoops to get all setup I have found this to be a fairly robust way to execute the automation. It would be a lot easier if MTM supported other test frameworks, but that is not the case just now.
MTM doesn't run automated tests. It's a tool designed for manual testers. The only automation it offers is the record/replay of actions that a tester has recorded.
If you want to automatically run Selenium tests then look at running NUnit as part of the build process, just make sure the build agent is running as an interactive processes not a service so that Selenium can access the desktop and run a browser.
Alternatively you could look at using a headless browser such as XBrowser or HTMLUnit (though you'll need to use the Selenium2 remote driver)