Good day.
I have a TFS build and a Visual Studio Test step there. All test are run, but I can't get detailed info about them, only quantity, passed\failed\others, run duration.
Also, I see "Outcome: Failed" and "This build doesn't have test results for the outcome you've selected".
How to configure this outcome?
You can't directly use the build-in outcome to analysis the detail test result.
You need to run the Publish Test Result task. First generate an XML file which contains the results. And then use the publish results step and pointing to that file so that the test results will show up in the build output.
More details about how to use this, please refer this similar tutorial with xUnit test: Execute and publish xUnit Tests results with .NET Core and VSTS
Note: For now VSTS/TFS does not support NUnit3 format. Source Link: Support NUnit2 format
Related
We have a release pipeline which deploys our product, then runs some cucumber tests which produce junit output. We then use PublishTestResults to publish this junit results.
This will then show the results in the Test Tab of the pipeline.
These tests are checked in to TFS source control but do not exist in the Test Tab of TFS.
We would like to create an empty TFS Test for each cucumber tests, and have the result of the test, updated after each pipeline is deployed, from the junit test report. Is this possible with PublishTestResults or another method.
It's not able to do this through publish test result test task.
This task publishes test results to Azure Pipelines or TFS when tests
are executed to provide a comprehensive test reporting and analytics
experience. You can use the test runner of your choice that supports
the results format you require. Supported results formats include
CTest, JUnit (including PHPUnit), NUnit 2, NUnit 3, Visual Studio
Test (TRX), and xUnit 2.
The published test results are displayed in the Tests tab in the pipeline summary and help you to measure pipeline quality, review
traceability, troubleshoot failures, and drive failure ownership.
You could also take a look at the concept of TFS test case. Sorry not similar with cucumber tests, but after google, it looks like also not suit for build-in tfs test case.
So it's easy to publish the generated test result to tests tab in pipeline summary. But it's not possible to publish cucumber tests to TFS Test Manager with PublishTestResults or another method.
I have a TFS build definition setup to run the task "Visual Studio Test" using the Nunit3 Test Adapter. This runs successfully and publishes test results. However, neither the TestFixture name or the Test's Fully Qualified Name are available in the test reuslts. This makes it difficult to determine what exactly what test failed without using lengthy test names.
Is there a way to include the Test Fixture names in the published test results from the build?
Please refer to the Tests summary tab of your build. It shows the detail information of test results of your Nunit3 tests.
You could use the Filter to find out those Failed or Succeeded tests.
You could also click the link in the picture below to check detail data for this test run.
Using the default build template on TFS 2013, we use the Post-test script to collect trx files from protractor tests, in order to include those in the build report summary.
We use vs2015 MsTest:
MSTest.exe
/publishresultsfile:...\TRXTestResult\CHROME-54.0.2840.71-results.trx
/publish:https://.../tfs/defaultcollection
/teamproject:...
/publishbuild:...
/platform:AnyCpu
/flavor:Release
Result:
Waiting to publish...
Publishing results of test run (test name) to
https://.../tfs/defaultcollection ...
Publish completed successfully.
However, the published test result wont show up in the build report (only the default from unit test).
This can't be achieved by a simple test script. You need to create a custom activity to handle this.
Inside this activity you can handle everything that is necessary to run the protractor tests and publish the test results. With the custom activity you are also able to integrate the automated protractor tests very easy into different build templates. This improves the reusability in a great way.
Then you need to create the custom build template and integration into a TFS Build. More details please refer the tutorial from this blog: Automated UI testing of AngularJS applications using TFS
I have generated my xml coverage file as part of CI build on the TFS server using vNext 2015 build definition. How would one display the results in the TFS 2015 summary ? Either using the xml report or the html generated using the ReportGenerator.
You need to convert the results produced by OpenCover to a format which can be parsed by TFS.
One way to do this would be to use the OpenCoverToCoberturaConverter NuGet package.
Once you have that, you can use the Publish Code Coverage Results build step.
I have described the whole process on my blog.
In “TFS 2015 - Update 2” this is possible by writing your own vsts extension (see here: https://www.visualstudio.com/en-us/docs/integrate/extensions/overview ).
I set up my own 'learning project' for building this as .vsix here: https://github.com/RobertK66/vsts-opencover.
My custom build step uses nunit3 console runner to execute tests under opencover.
I managed to upload the OpenCover xml result file as 'testrun-attachment' via REST-interface and got the pieces in place to show the summary values on both the build summary tab and on its own extended “build-results-view”.
Base for this first version were a lot of examples provided by MS on github: https://github.com/Microsoft/vsts-tasks and https://github.com/Microsoft/vsts-extension-samples
To get a first feeling what places on your TFS Web Portal can be extended/customized you can download and install this extension https://aha.gallery.vsassets.io/items?itemName=ms-samples.samples-contributions-guide from the Visual Studio Marketplace.
Hope this helps to get you started.
Currently, these customizations are not supported. You can't edit the displays in new build summary. However, you can customize Code Coverage Analysis.
You can submit it to User Voice site at:
http://visualstudio.uservoice.com/forums/121579-visual-studio
Instead of using OpenCover extension, it is very convenience for you to generate code coverage result and include it in the build summary page this way:
Select the Visual Studio Test step, check the Code Coverage Enabled option.
Then, the code coverage result shows on the build summary page:
Has anybody had any success getting Team Build to show xUnit.net test results and code coverage in the build report? The MSBuild runner is running the tests just fine and the results are in the log file, but the test results and code coverage results areas of the build report say "No test result" and "No coverage result" respectively. Is Team Build looking for a certain location / format for the test results to be exported by the xUnit.net runner?
TFS/TeamBuild definitely requires the test results in a particular format, they also need to be specifically published to TFS as well.
I'm currently looking at this problem for the Gallio test runner, and can offer two potential options:
Try wrapping the xUnit tests inside the 'Generic Test' project type
that VSTS offers - this is where MSTest executes the xUnit test
runner.
Come up with an XSLT transform (or similar) that can map the
xUnit results to the MSTest schema (see
http://www.codeplex.com/nunit4teambuild for an example of this
approach for NUnit tests).
See also http://jonnekats.wordpress.com/2009/05/07/integrate-xunit-tests-into-your-daily-team-build/