I have a TFS 2013 environment setup and I am trying to get TFSBuild to gather my code coverage results after a build.
The tests are running fine, but no matter what I try, I get a message saying "No Code Coverage Results".
I have read about a "Code Coverage Analysis Service", but that is not installed on my servers at all and I can't seem to find it anywhere.
I have gone through the process of the builds to make sure I have the "Automated Tests --> Test Source --> Run settings --> Type of run settings" set to CodeCoverageEnabled, but no matter what else I try, I can't get the results to get published to the Analysis services database so I can run the Code Coverage Reports.
I have tried a run settings file, but with no luck, so I have taken that out of the equation.
Any help would be greatly appreciated!
I just figured out what the problem was that my Code Coverage weren't been calculated on the buildserver. I figured this out with the helpdesk of Microsoft and it is a small UI bug.
Steps to reproduce: Create a new build definition.
Go to the process tab of the build definition and select the solution file to build.
Now expand the "Automated tests" section and expand the "Test Source".
Expand the "Run settings" and change the "Type of run settings" from "Default" to "CodeCoverageEnabled".
Save the builddefinition and run your build.
Examine build results and notice no code coverage is calculated.
To fix this issue use the other ui of the build definition.
Select the "Automated tests" node.
Now in that row on the right you will see a button with "...". Hit it and a dialog will be opened.
Hit "Edit" to edit your test source. You will see that Code coverage is enabled. If not select it.
Hit Ok and Ok again to close the two dialogs.
Now save the build definition and run your build again.
Now the code coverage does get calculated!!! The microsoft helpdesk has submitted a bug to the development team and it will be fixed I guess the next release. Small bug but it cost me more then two days to find the bug!
Do you have a Premium or ultimate visual studio installed on your build machine? Either of those is required to get code coverage results back to TFS
I think the "other window" is referring to right-clicking on the Solution's "Local.testsettings" file (or other [ ].testsettings files), selecting "Open" and navigating through the various setups. When you set the "Test Run location" radio button to "Run tests using local computer or a test controller", there is a setting under "Data and Diagnostics" for "Code Coverage (Visual Studio 2010)".
I am using local build server with Visual Studio online with a .runsettings file and I had exactly the same issue.
None of the trickery above helped, so I tested the build script on the hosted build controller and it worked fine, so I decided the problem must be the build server itself.
I changed the Build Service account from "Network Service" to a regular windows user account in the TFS Configuration Tool and now code coverage is collected. Note that this user will need access to the TFS build directories.
The above workarounds didn't work from me (VS2015 update5 & TFS2013 Update 3).
In my case, Visual Studio was not installed on the build server which is a requirement it seems for code coverage results to be returned.
From MSDN
Microsoft Visual Studio is required to be installed on the build server for the following scenarios:
To build any CPP test project, you must install Visual Studio Professional or later.
To run unit tests, or coded UI tests, you must install Visual Studio Professional or later.
To use data and diagnostic data adapters:
- Code coverage: Visual Studio Premium or later.
- Test impact: Visual Studio Ultimate.
- IntelliTrace: Visual Studio Ultimate.
To build any modern style apps on a build machine: Visual Studio Ultimate or Visual Studio Express for Windows 8 (Operating system on build server must be Windows 8).
To compile and run tests for a project with a faked assembly: Visual Studio Ultimate.
Related
We are trying to integrate the Coded UI Test automation into TFS 2015. The build definition is created and is working correctly. But Release Definition is having problem.
Below are the tasks added for Release Definition:
1. Windows Machine File Copy
2. Visual Studio Test Agent Deployment
3. Run Functional Tests
Please find the attached screen shots for each task:
WindowsMachineFileCopy
Visual Studio Test Agent Deployment
Run Functional Tests
Please find the attachment for the automated test case which is associated in Run Functional Test Task.
Automated Test Case
The Release runs successfully till Visual Studio Test Agent Deployment. Even the Test Agent pop up is opened. After this, the browser is not getting opened i.e., while running Run functional Tests task, it throws below error:
"Unit Test Adapter threw exception:
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.."
I have placed all the dll's including the references in bin folder. Please find below screen shot:
DLL's
I am using Visual Studio Premium 2012, Test Agent 2015 Update 3 and TFS 2015.
Installed Test Agent 2013 and now getting attached error in Deploy Test Agent task:
Test Agent Deploy error
Please suggest the solution so that my coded ui starts running through TFS.
You need to install Test Agent for Visual Studio 2013 on the test machine based on the Compatible versions of TFS,and test agent
Also there is a blog introduce the related issue :Test Agents support for Visual Studio 2015
So, just try to use Test Agent for VS 2013 instead of Test Agent 2015 Update 3, then try it again.
I am using the TFS 2013\VS 2013 professional editions for continuous build. Am looking to use an open source tool like OpenCover for code coverage. I have no prior experience in code coverage tools. I installed the OpenCover UI from Nuget but not sure how to include Codecoverage in integrated build in TFS. Getting "No Code Coverage Results" when the build runs.
I enabled code coverage under Automated tests node in build definition.
Any suggestion on how to implement code coverage in TFS\VS 2013 Profession edition would be greatly appreciated.
If you mean Enabled Code Coverage in the build definition just like below screeshot.
This is VS build-in code coverage, according to the Compare VS offerings site, only Visual Studio Enterprise has Code Coverage feature, so if you use TFS for your CI build and want to use the built-in code coverage feature, installing VS Enterprise on your build agent machine is required.
If you want to integrate OpenCover with TFS XAML build, you need to customize your build process template, add a custom activity in the build definition to trigger the opencover command for generating the report.
Created a RunCoverage.cmd at the root of my source folder
Installed / copied in TFS Servers the Opencover binaries and added into Path(must restart TFS Services to enable TFS to see it).
Created a new actitivity in the build definition, that is, editing the build template (before sonar execution activity) to run the cmd:
Running command line statements from TFS custom activity
http://msdn.microsoft.com/en-us/library/gg265783.aspx
There is the cmd contents:
#REM #echo suppresses command line. ^ is line continuation character
#ECHO Executing OpenCover...
OpenCover.Console.exe -register:user ^
-target:"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" ^
-targetargs:"/testcontainer:%~dp0test.dll /usestderr /resultsfile:%~dp0MSTestsResults.trx" ^
-output:%~dp0opencovertests.xml
More details please refer Naim Raja Díaz's answer in this similar question:Integration of OpenCover with TFS
We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:
MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"
So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.
In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.
I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.
Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?
I found the solution, it was to switch from using "Test Plan" to "Test Assembly"
Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.
We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.
VSTest.Console.exe is optimized for performance and is used in place
of MSTest.exe in Visual Studio.
After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.
MSTest.exe You can use the MSTest.exe program to run automated tests
in a test assembly from a command line.
MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.
MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server.
More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe
Since you are using TFS2015, and if you don't have any VS2010 test projects,
we encourage you to use VStest instead of MStest.
We were using TFS and VS 2013, everything was working fine. We upgraded to TFS and VS 2015. We added a new build server (Windows Server 2012 R2, Visual Studio 2015 Enterprise, DevExpress and Wix) to have a clean install.
The builds were failing Task could not find "LC.exe" using the SdkToolsPath "" ... so we changed the build template from DefaultTemplate.11.1.xaml to TfvcTemplate.12.xaml
Now it's building, but when we open the builds from the build explorer under code coverage we have No build code coverage data available. Code coverage is enabled in the build definition.
Assuming you enabled Code coverage by going to the process tab of the build definition and then expanding the Tests--Automated tests--Test Source--Run settings, and change the Type of run settings from Default to CodeCoverageEnabled.
I have tried the same steps in TFS 2015, and I can't get Code coverage either in this way.
Instead, I tried the steps below, then I can get Code coverage now:
Go to Process--Test--Automated tests--Test source in build definition.
Now in Test source row, click the button with "...".
Under "Options", select Enabled Code Coverage and save the build definition. No matter the Enabled Code Coverage option has been selected or not, you still need to save this edit. Now queue the build, you'll see the code coverage.
Similarly to How to use MsTest in Continuous Integration without VS?, I want to run mstest.exe on a TeamBuild server.
My context is Trapping Error Status in MSBuild - i.e., I'm only trying to use mstest.exe /publish to upload the results in to the TFS repository. Thus the full rigmarole in http://www.shunra.com/shunrablog/index.php/2009/04/23/running-mstest-without-visual-studio/ is (you'd hope) likely to be overkill, esp as MSTEST.exe, as covered in http://social.msdn.microsoft.com/Forums/en-US/tfsgeneral/thread/e4575890-8f88-413c-a7f3-0d09d3b9cb01 suggests all I need to do is install Team Explorer.
I've installed VS2008 Team Explorer SP1, and mstest.exe /? still isnt telling me about the publish options.
Recall that it says only Team Explorer is required of mstest.exe /? for publishing:
The following options are also available if Team Explorer is installed:
/publish:[server name] Publish results to the Team Foundation
Server.
/publishbuild:[build name] The build identifier to be used to publish
test results.
/publishresultsfile:[file name] The name of the test results file to
publish. If none is specified, use the
file produced by the current test run.
/teamproject:[team project name] The name of the team project to which the
build belongs. Specify this when publishing
test results.
/platform:[platform] The platform of the build against which
to publish test results.
/flavor:[flavor] The flavor of the build against which
to publish test results.
I'm going down the procmon.exe path to figure out what's missing, but ideally someone in the know would step in and answer:-
Should mstest.exe /publish only require Team Explorer as stated, or does it require VSTT and Team Explorer?
The /publish option of MSTest.exe does a license check to ensure that a "Team" version of Visual Studio is installed (i.e. the Developer or Test edition) before it does the publish part of the code.
This restriction appears to have been introduced to allow the basic unit testing capabilities in standard versions of Visual Studio but requires you to have one of the more expensive Team editions of Visual Studio to enable the "Team" related features in testing - such as to publish your test results to TFS.
In total there are 3 missing features if you do not have a Team edition of Visual Studio with Team Epxlorer installed.
Publishing to TFS
Code coverage
Connect to remote agent.
If you have the Development Edition installed then you get publish and codecoverage but not the remote agent agent capabilities (i.e. for doing Load lesting).
The Test Edition and Visual Studio Team Suite have everything.
If you would like to see an example of a custom MSBuild task that uses MSTest.exe to publish unit test data from a build server (in this example JUnit test data), then take a look at the Teamprise Build Extensions that I wrote. The source code for these is available under the permissive MS-PL open source license.
I've read on the MSDN forums that the /publish functionality of mstest requires purchasing either VS Team Developer or VS Team Test edition. Any lesser editions such as standard/professional or team explorer don't have it.
I'd really like to see some more evidence though, as it seems fairly unreasonable that with "standard" VS2008 you can write and run unit tests, but you arbitrarily can't run them on your build machine :-(
To answer the question:
Should mstest.exe /publish only require Team Explorer, or does it require VSTT and Team Explorer?
It appears that it requires VSTT (which I assume means visual studio team-tester edition)
Well, the message you get when running mstest.exe /? is actually very misleading. It tells you that Team Explorer is required to publish your test results, but unfortunately installing Team Explorer doesn't change anything. As Martin stated, MsTest does a license check to ensure that appropriate edition of Visual Studio is installed and thus '/publish' and other switches are not available.
And by 'appropriate' edition MsTest means Premium or Ultimate, not sure about other editions.
Microsoft says you should install Visual Studio on your server to have a fully featured CI and this is the only official and supported solution, but if you really don't like this concept (and I don't), you can make MsTest think that Visual Studio is installed on your server.
It is enough to export some registry keys from your development machine to your CI server and voila! - MsTest does publish your test results to TFS.
The registry keys are at
HKLM\SOFTWARE\Microsoft\VisualStudio\12.0\Licenses
or at
HKLM\Software\Wow6432Node\Microsoft\VisualStudio\12.0\Licenses
depending on your system.
You can find more details of this solution here:
http://blog.foxxtrot.net/2010/02/hacking-mstest-out-of-visual-studio.html