We have the following setup for unit and functional testing in TFS (under Build Definitions in TFS)
Install our program into a clean VM.
Deploy test agent (default vstf_testagent.exe)
Run tests
Some of the tests are written in NUnit3 and the C++ test in Google Tests. So, they are not seen by the default test runner.
In Visual Studio, the test adapters work great to run the tests.
But, I can't seem to figure out how to install these test adapters to vstf_testagent.exe on the virtual machine so that it sees and runs all the tests.
I found this page on how to install agents but it is with using Visual Studio.
You can install the test adapters to your project via Nuget and add a Nuget restore task in your build definition to restore the packages. "Run Functional Tests" task will search the test adapters in "packages" folder automatically.
Test Adapters: There is no explicit field to specify Test Adapter path in the task. The task automatically searches for "packages"
directory that exists in the same folder as the .sln file (nuget
restored directory structure). If your adapters are in a different
directory or you did not copy over the source files, use a runsettings
file with TestAdaptersPaths as described at
https://msdn.microsoft.com/en-us/library/jj635153.aspx
If the test adapters is in other folder, you can create a runsettings file and run the test with it.
Refer to this link for details: Visual Studio Test using Test Agent Task.
First, you need to install Visual Studio on your VM.
After that the directly way to make NUnit adapters available is downloading them from Visual Studio Gallery, unzip the .vsix extension file, then copy all the extension file in the folder just as the link mentioned in your question.
This operation should be done in every machine where you deployed Test Agents that will be used to run build with nunit tests. This can be annoying if you have lots of agents, if you prefer, Test Runner Task has an options to specify the path where the agent can find needed test runners.
You can also automate all the process by adding Nunit3 Test Adapter nuget package just as Eddie suggest to your test project, adding a standard Nuget Reference. Detail ways you can refer this blog: nUnit and Jasmine.JS unit tests in TFS/VSO vNext build
Nuget Package
NUnit 3 Test Adapter for Visual Studio 2012/13/15
It is also possible to copy the .dlls of your testadapter that you want to use directly from the nuget-package to the folder C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensionsof your computer where you want to run your tests.
Also see:
http://www.codewrecks.com/blog/index.php/2016/06/04/running-unit-tests-on-different-machine-during-tfs-2015-build/
Related
Question:
How do I run NUnit tests on TFS?
Attempts:
I've searched the web and am still struggling to figure out the exact steps required to accomplish (to what I believe is) a trivial task.
I added the NUnit3TestAdapter package to my test project.
However, my tests are still not being discovered.
TFS displays the following message:
No test runs are available for this build. Enable automated tests in
your build pipeline by running your test framework of choice, such as
JUnit, Visual Studio Test, or xUnit. If you choose to run tests using
a custom task or runner, you can publish results using the Publish
Test Results task.
Expectation:
I'm surprised that NUnit is not an option that I can just select from.Hence, it's the standard unit test framework for .Net.
How do I run NUnit tests on TFS?
There is a blog here describes clearly how to run NUnit tests on TFS: Running NUnit Tests in a TFS 2015 Build vNext
Simply summarized as follows:
Add Nuget Package for NUnit Test Adapter
Specify path of custom Test Adapter inside build definition
Copying adapters inside Visual Studio TestWindows folder
Specify Path to Custom Test Adapter with nunit packages
Note：
You do need to edit the Visual Studio Test task’s advanced settings
to point at the NuGet packages folder for your solution (which will be
populated via NuGet restore) so the custom nUnit test adaptor can be
found i.e. usually setting it to $(Build.SourcesDirectory)\packages
Some other tutorial for you reference:
xUnit or NUnit with Visual Studio Online Build
Running nUnit and Jasmine.JS unit tests in TFS/VSO vNext build
I am using the TFS 2013\VS 2013 professional editions for continuous build. Am looking to use an open source tool like OpenCover for code coverage. I have no prior experience in code coverage tools. I installed the OpenCover UI from Nuget but not sure how to include Codecoverage in integrated build in TFS. Getting "No Code Coverage Results" when the build runs.
I enabled code coverage under Automated tests node in build definition.
Any suggestion on how to implement code coverage in TFS\VS 2013 Profession edition would be greatly appreciated.
If you mean Enabled Code Coverage in the build definition just like below screeshot.
This is VS build-in code coverage, according to the Compare VS offerings site, only Visual Studio Enterprise has Code Coverage feature, so if you use TFS for your CI build and want to use the built-in code coverage feature, installing VS Enterprise on your build agent machine is required.
If you want to integrate OpenCover with TFS XAML build, you need to customize your build process template, add a custom activity in the build definition to trigger the opencover command for generating the report.
Created a RunCoverage.cmd at the root of my source folder
Installed / copied in TFS Servers the Opencover binaries and added into Path(must restart TFS Services to enable TFS to see it).
Created a new actitivity in the build definition, that is, editing the build template (before sonar execution activity) to run the cmd:
Running command line statements from TFS custom activity
http://msdn.microsoft.com/en-us/library/gg265783.aspx
There is the cmd contents:
#REM #echo suppresses command line. ^ is line continuation character
#ECHO Executing OpenCover...
OpenCover.Console.exe -register:user ^
-target:"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" ^
-targetargs:"/testcontainer:%~dp0test.dll /usestderr /resultsfile:%~dp0MSTestsResults.trx" ^
-output:%~dp0opencovertests.xml
More details please refer Naim Raja Díaz's answer in this similar question:Integration of OpenCover with TFS
I've been scouring various Nunit articles for the past several days trying to figure out the issue with why my Nunit tests are not running.
I am using TFS 2015 Build vNext engine and the Visual Studio Test step to execute my Nunit test cases. The VS 2015 solution used for generating the Nunit test dlls contains the several .net projects and 2 Nunit test projects. Nunit3.0 and Nunit3.0 Test Nuget packages are being used to create and execute the tests. Here is the nuget configuration and nunit.framework reference in my Visual Studio 2015 solution. The test cases run fine in the Visual Studio IDE and compile without error when the solution is build in vNext.
When the VS Test step executes in Build vNext, I get the following warning:
Warning: Dependent Assembly nunit.framework of E:\TFSBuild_Directory\14\s\EBS\B_WebEnable\ABC.Data\ABC.Data.Core.Nunit.Tests\obj\UnitTesting\ABC.Data.Core.Nunit.Tests.dll not found. Can be ignored if not a NUnit project.
The VS Test step is configured:
VS Test Step Configuration
I have confirmed on the build server that the nuget packages are getting restored to the build server and the Nunit.framework.dll appear to be in the expected file path as compared to my local path.
Any ideas as to what could be causing this issue? Any input is appreciated!
I have restored Nunit and Nunit3TestAdapter from Nuget to have a test on my side, but couldn't reproduce your issue. I'd like to share my steps here for your reference:
Restore Nunit and Nunit3TestAdapter from Nuget in VS for the test project (as your first screenshot), and check in the packages.config file.
In "Visual Studio Test" task, specify $(Build.SourcesDirectory)\packages in "Path to Custom Test Adaptors". As if there is a packages folder in the sources directory, it is automatically searched for testadapters (It seems you have a long path, you may need to check it).
I use the default VS template:
No error or warning occurred during this process. You can compare your steps with mine and correct yours. Hope this helps you.
Ok - after stepping away from the issue and coming back to it with fresh eyes, I found the issue causing the nunit.framework.dll issue.
After going step by step through the build definition, I forgot that I had an MSBuild parameter in the build step that was copying the compiled output to another location /p:OutDir=$(build.stagingDirectory)\ABC. Once I changed the test step to point to the new location, everything worked fine. Thanks everyone for the feedback!
Is there a way to run xUnit tests for UWP with TFS 2015 vNext build?
I try to do it with my local build Agent with it running in interactive mode.
When test needs to be done, the UWP test app runs but no tests discovered.
Is it supported to run this tests?
Yes, assuming you have installed xunit via Nuget and your packages are at the root level of your sources folder, you just need to set the Path to Custom Test Adapters to your packages folder.
In my case: $(Build.SourcesDirectory)\packages - adjust as you need to.
I am new to TFS and want to integrate OpenCover with TFS. If any has done this please help!
This question is rather old but maybe you are still interested. With current Version of TFS (2015 Update 2) this is now possible as a "vsts Extension".
See here for details: https://github.com/RobertK66/vsts-opencover
Since the answer doesn't specify the version of TFS, here is an answer for 2015/2017.
OpenCover can be run from TFS using the Powershell build step. You need to get the contents of the OpenCover NuGet package onto TFS and run OpenCover.console.exe from there.
Since TFS doesn't support the format produced by OpenCover, you need to take one additional step and convert the results to Cobertura format. It's possible using the OpenCoverToCoberturaConverter NuGet package.
I've described the whole process in much more detail on my blog:
http://codewithstyle.info/setting-up-coverage-reports-on-vsts-with-opencover/
OpenCover is just a console application so you can just modify your scripts to get OpenCover to run your unit tests.
I haven't used TFS for several years and it has changed since then however this blog post should help
To incorporate coverage measurement of OpenCover the build process of TFS (second half)
The original is in Japanese but if you are familiar with TFS then the screens will probably be obvious.
OpenCover also comes with an MSBuild task that may help you with your integration.
I've just integrated opencover with TFS Build, to generate a xml with the results that will be processed by sonar:
Created a RunCoverage.cmd at the root of my source folder
Installed / copied in TFS Servers the Opencover binaries and added into Path(must restart TFS Services to enable TFS to see it).
Created a new actitivity in the build definition, that is, editing the build template (before sonar execution activity) to run the cmd:
Running command line statements from TFS custom activity
http://msdn.microsoft.com/en-us/library/gg265783.aspx
There is the cmd contents:
#REM #echo suppresses command line. ^ is line continuation character
#ECHO Executing OpenCover...
OpenCover.Console.exe -register:user ^
-target:"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" ^
-targetargs:"/testcontainer:%~dp0test.dll /usestderr /resultsfile:%~dp0MSTestsResults.trx" ^
-output:%~dp0opencovertests.xml
But i'm facing three issues (that are related with my concrete implementation, but you may face them):
The Tests are runned twice (one for template itself, and other for OpenCover)
The MsTest.exe in the TFS Servers is not in the same path, so when the Controller asigns an agent (if the match is done by tags) then if the agent executing the build is in a TFS Server that does not have the MSTest in the right path it will fail.
How to inject in cmd the corresponding test runner (MsTest, XUnit, Nunit, etc...) depending on the test project
Hope that it helps! (and someone can help me ;-)