Question:
How do I run NUnit tests on TFS?
Attempts:
I've searched the web and am still struggling to figure out the exact steps required to accomplish (to what I believe is) a trivial task.
I added the NUnit3TestAdapter package to my test project.
However, my tests are still not being discovered.
TFS displays the following message:
No test runs are available for this build. Enable automated tests in
your build pipeline by running your test framework of choice, such as
JUnit, Visual Studio Test, or xUnit. If you choose to run tests using
a custom task or runner, you can publish results using the Publish
Test Results task.
Expectation:
I'm surprised that NUnit is not an option that I can just select from.Hence, it's the standard unit test framework for .Net.
How do I run NUnit tests on TFS?
There is a blog here describes clearly how to run NUnit tests on TFS: Running NUnit Tests in a TFS 2015 Build vNext
Simply summarized as follows:
Add Nuget Package for NUnit Test Adapter
Specify path of custom Test Adapter inside build definition
Copying adapters inside Visual Studio TestWindows folder
Specify Path to Custom Test Adapter with nunit packages
Note：
You do need to edit the Visual Studio Test task’s advanced settings
to point at the NuGet packages folder for your solution (which will be
populated via NuGet restore) so the custom nUnit test adaptor can be
found i.e. usually setting it to $(Build.SourcesDirectory)\packages
Some other tutorial for you reference:
xUnit or NUnit with Visual Studio Online Build
Running nUnit and Jasmine.JS unit tests in TFS/VSO vNext build
Related
I have a visual studio loadTest project and LoadTest2010 result database, when I run the test from visual studio then test work fine and test data save in the LoadTest2010 DB but when run the test from VNext then test run fine (did all the configurations) but data does not save in the DB LoadTest2010.
In VNext VsoAgent agent is running the test.
Did any one face this issue?
vNext is TFS Test Agent and it is VS2015 Test Agent, we ever used Test Controller and Test Agent for VS2013 Update5. vNext is Test Agent for VS2015.
According to your description, whether you have added a web test project in load test, if no, please make sure you have added it.
and then please use MSTest to run load test locally to see the results.
What test steps do you use in your build definition? If you use the steps Visual Studio Test Agent deployment and Run functional test to run your load test, it may get the result like you posted above. Because these 2 steps are using VS Test agent 2015 and if you want to run load test, you need to install VS Test Agent 2013 according to this document.
So about using the VS 2013 test agent in Vnext build, you could follow the step below:
Add a test setting file for your test solution.
Next, specify test controller and Load test result store in the setting file.
Then add a Visual Studio Test step in your build definition and in this step, remember to specify the testsetting file.
this is the only one option right now in the VNext to run the loadtest and data will save to the loadtest database. Below is the snapshot from VNext
We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:
MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"
So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.
In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.
I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.
Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?
I found the solution, it was to switch from using "Test Plan" to "Test Assembly"
Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.
We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.
VSTest.Console.exe is optimized for performance and is used in place
of MSTest.exe in Visual Studio.
After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.
MSTest.exe You can use the MSTest.exe program to run automated tests
in a test assembly from a command line.
MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.
MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server.
More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe
Since you are using TFS2015, and if you don't have any VS2010 test projects,
we encourage you to use VStest instead of MStest.
We have a Visual Studio 2015 solution containing some C++ projects and some tests based on Google Test Framework.
Now I would like to run those tests with the quite new TFS 2015 build features. I know that there is the "Visual Studio Test" build step that is able to run custom test adapters (like the Google Test Adapter?).
Is this the easiest way to setup things? What exactly has to be installed on the (on premise) TFS2015 build server and how to configure the build steps?
Thanks for you help! Sebastian
Yes, the simplest way is just using google test adapter in ""Visual Studio Test" build " task. Just as the feature statement which will using VSTest.Console.exe
You need to install visual studio on your build server(agent). About how to conigure the build steps, there has been a detailed tutorial with Xunit test which also applies to google test. Plesae refer this blog: Running xUnit tests in TFS Build vNext
We have the following setup for unit and functional testing in TFS (under Build Definitions in TFS)
Install our program into a clean VM.
Deploy test agent (default vstf_testagent.exe)
Run tests
Some of the tests are written in NUnit3 and the C++ test in Google Tests. So, they are not seen by the default test runner.
In Visual Studio, the test adapters work great to run the tests.
But, I can't seem to figure out how to install these test adapters to vstf_testagent.exe on the virtual machine so that it sees and runs all the tests.
I found this page on how to install agents but it is with using Visual Studio.
You can install the test adapters to your project via Nuget and add a Nuget restore task in your build definition to restore the packages. "Run Functional Tests" task will search the test adapters in "packages" folder automatically.
Test Adapters: There is no explicit field to specify Test Adapter path in the task. The task automatically searches for "packages"
directory that exists in the same folder as the .sln file (nuget
restored directory structure). If your adapters are in a different
directory or you did not copy over the source files, use a runsettings
file with TestAdaptersPaths as described at
https://msdn.microsoft.com/en-us/library/jj635153.aspx
If the test adapters is in other folder, you can create a runsettings file and run the test with it.
Refer to this link for details: Visual Studio Test using Test Agent Task.
First, you need to install Visual Studio on your VM.
After that the directly way to make NUnit adapters available is downloading them from Visual Studio Gallery, unzip the .vsix extension file, then copy all the extension file in the folder just as the link mentioned in your question.
This operation should be done in every machine where you deployed Test Agents that will be used to run build with nunit tests. This can be annoying if you have lots of agents, if you prefer, Test Runner Task has an options to specify the path where the agent can find needed test runners.
You can also automate all the process by adding Nunit3 Test Adapter nuget package just as Eddie suggest to your test project, adding a standard Nuget Reference. Detail ways you can refer this blog: nUnit and Jasmine.JS unit tests in TFS/VSO vNext build
Nuget Package
NUnit 3 Test Adapter for Visual Studio 2012/13/15
It is also possible to copy the .dlls of your testadapter that you want to use directly from the nuget-package to the folder C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensionsof your computer where you want to run your tests.
Also see:
http://www.codewrecks.com/blog/index.php/2016/06/04/running-unit-tests-on-different-machine-during-tfs-2015-build/
How can I get my TFS build to run my Specflow tests? I have the tests running in the local VS build, but they aren't being picked up in the TFS build. Please could someone post the steps I would take to have the build run the specflow tests, and report on the results. Thanks.
I guess specflow uses nunit tests. you need to set up nunit in build server
http://www.mytechfinds.com/articles/software-testing/6-test-automation/72-running-nunit-tests-from-team-foundation-server-2012-continuous-integration-build
If you use SpecRun, it will automatically take care of TFS Build integration. Installing SpecRun is just referring the NuGet package from the SpecFlow project.