Publish Test Results task mentions that "You can also use this task in a build pipeline to publish code coverage results produced when running tests to Azure Pipelines or TFS in order to obtain coverage reporting."
However, I have tried this for my test report of type .cobertura format. And it doesn't create the coverage tab. Though it works for .coverage and .covx/.covb files (tested)
<DataCollector friendlyName="Code Coverage">
<Configuration>
<Format>Cobertura</Format>
</Configuration>
</DataCollector>
I'm using VSTPI 17.1 to get cobertura support from VS 2022 however my ADO is using VS 2019.
So is it actually supported? If yes then what's required to make it work?
Alternatively, I can use PublishCodeCoverageResults task, however
I miss the power of VsTest task followed by publish-test-results task's auto aggregating results from multiple jobs within the same build or will not? https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results doesn't specify it however with publish-test-results, I always see aggregated results (desired)
And the doc mentions
"
Tasks such as Visual Studio Test, .NET Core also provide the option to publish code coverage data to the pipeline. If you are using these tasks, you do not need a separate Publish Code Coverage Results task in the pipeline."
And also there is a limit of 7 KB for this task.
Hence "Publish Test Results" should publish cobertura type as well because it's supported by the VsTest
Else I would run into needing to self merge multiple of them https://github.com/microsoft/vstest/issues/3497#issuecomment-1084741316
Expected: Publish Test Results Task to also publish the coverage to Coverage tab from .cobertura files.
Related
I am using TFS 2017 Update3 and I have a Build definition which runs Grunt build task at two different HTML projects and generates two cobertura.xml files at their respective paths.
To display the code coverage result I have used Publish Code Coverage Results task in my build pipeline but it displays the Code coverage for only 1 project and doesn't display the code coverage result of other Html Project.
I tried using two Publish Code Coverage Results task each pointing to different cobertura.xml files but still it displays the code coverage result for only one project on TFS build summary.
Is there any other ways by which I can publish multiple code coverage result?
I am running my test in TFS (Nunit plus Visual Studio with Adapter) and I have set the build definition as below
Build succeeds but no test result file was generated
Does TFS writes this Xml file ?
Log
2017-02-08T08:08:40.8151428Z Executing the powershell script: D:\A1\agent\tasks\PublishTestResults\1.0.20\PublishTestResults.ps1
2017-02-08T08:08:41.0963795Z ##[warning]No test result files were found using search pattern 'D:\A1_work\1\s**\TEST-*.xml'.
If the Nunit plus Visual Studio with Adapter means you have two test steps: one for unit tests and the other one for vs tests.
Please also add two "Publish Test Results" step one for Nunit format.
Also run your test manually on the build server to see if test result file .trx generated on the machine.
In our continuous integration process we are using Jenkins, NUnit and OpenCover.
Every Jenkins job runs NUnit and OpenCover, but OpenCover calls the NUnit batch file in order to determine code coverage; therefore NUnit is executed twice.
For example we have a first (simplified) batch:
nunit-console-x86 [PathToTestAssemblies] /xml=NunitResult.xml /noshadow /nodots /process=Separate
And we have a second batch for OpenCover:
OpenCover.Console.exe -target:"NUnit.bat" -output:"./OpenCoverResults.xml" -register -targetdir:".\bin" -coverbytest:*.dll
The problem is that OpenCover does not provide NUnit result (The NunitResult.xml file in my previous command). So in order to have less test time for every Jenkins job we want to get back the NUnit result or find a way to have the following features with OpenCover in the Jenkins job web page:
Latest Tests result from every Jenkins job, so it is easy for a developer to see the latest result.
Test result trend
Is there a way to have both NunitResult and OpenCover results from an single run of NUnit?
I finally found were was the NUnitResult.xml file. In fact it was in the folder were I put all test assemblies (ex: bin) while the Nunit batch put it at the root location.
I think this is because I use the -targetdir args with "bin" to indicate OpenCover were are my assemblies.
I'm trying to use JaCoCo agent for collecting FT code coverage of a web-site by adding "-javaagent..." to Jetty and running external Selenium tests (or, for example, manual testing).
At the end of testing I have results only for web-module, that was actually run in Jetty, but not for "core" modules (in the same project) being used in web-module. So, is there any ability to collect calls for all modules of the project?
Indeed, I now tried to start all project webservices (it has 5) with the same JaCoCo report destination (i.e. /tmp/jacoco-res.exec), append=true and ran:
mvn sonar:sonar -Dsonar.jacoco.itReportPath=/tmp/jacoco-res.exec
So, sonar now used the same JaCoCo report file for every module in project and sonar shows tests coverage both for jetty-run and core modules.
Is this approach applicable and coverage results correct?
I had a simular question that I posed on the Sonar User forum a couple of days ago.
In short, your approach is correct and was also proposed by a Sonar consultant (as can be seen on the post).
Perform a Sonar analyze on each Maven project.
For each analysis, reuse the report and pass your JaCoCo report (using -Dsonar.jacoco.itReportPath)
Sonar will retrieve from this JaCoCo report
the coverage data related to the Maven project being analyzed. (one of your core projects)
Thus, for each Maven project, you'll get the coverage by your integration tests.
Finally, to get the coverage of your webservice by your integration
tests, you will have to use the Views plugin:
http://www.sonarsource.com/products/plugins/governance/portfolio-management/
Create a view that is an aggregation of all the Maven
projects composing the webservice.
Run a Sonar analysis of one of its
Maven projects in order for Sonar to compute the view.
On the view dashboard, you'll be able to get the code coverage
of the webservice by your integration tests.
We use TFS 2010 and Automated builds.
We also make use of MSTests.
I would like some concrete information about the build server's test execution method.
Will the test engine (on build server) run the unit tests sequentially or in parallel?
By default it will run them sequentially. You can customize the build workflow by adding a Parallel activity and running different sets of tests in each. Or if you want to parallelize the test run across multiple build machines you can have the build use multiple RunOnAgent activities to do so (http://blogs.msdn.com/b/jimlamb/archive/2010/09/14/parallelized-builds-with-tfs2010.aspx).
Note: If you execute the tests across multiple test runs you will end up with multiple test reports (.trx files) that will not be merged together without further customization of the build.
#Dylan Smiths answer is correct, but does not cover option # 3.
Executing Unit Tests in parallel on a multi-CPU/core machine
DANGER WILL ROBINSON: This is only applicable to VS2010 and mstest.exe. VS2012 has a new test runner that does not support parallel test execution Visual Studio UserVoice Run unit tests in parallel The VS2012 test system can use the legacy testrunner, and you can make it work if you specify a .testsettings file using the MSTest/SettingsFile element. Configuring Unit Tests by using a .runsettings File
How to: Enable parallel test execution
Ensure you have a multi-core/CPU machine
Ensure you are running only unit tests
Ensure your tests are thread-safe
Ensure you do not have any data adapters on
Ensure you are running locally (cannot use TestController/TestAgent)
Modify your test settings file.
Right-click the test setting file and select "Open With" -> Open as Xml
Set the parallelTestCount attribute on the Execution element
Options are:
blank = 1 CPU/Core - this is the default
0 = Auto configure: We will use as many tests as we can based on your CPU and core count
n = The number of tests to run in parallel
Save your settings file