In our continuous integration process we are using Jenkins, NUnit and OpenCover.
Every Jenkins job runs NUnit and OpenCover, but OpenCover calls the NUnit batch file in order to determine code coverage; therefore NUnit is executed twice.
For example we have a first (simplified) batch:
nunit-console-x86 [PathToTestAssemblies] /xml=NunitResult.xml /noshadow /nodots /process=Separate
And we have a second batch for OpenCover:
OpenCover.Console.exe -target:"NUnit.bat" -output:"./OpenCoverResults.xml" -register -targetdir:".\bin" -coverbytest:*.dll
The problem is that OpenCover does not provide NUnit result (The NunitResult.xml file in my previous command). So in order to have less test time for every Jenkins job we want to get back the NUnit result or find a way to have the following features with OpenCover in the Jenkins job web page:
Latest Tests result from every Jenkins job, so it is easy for a developer to see the latest result.
Test result trend
Is there a way to have both NunitResult and OpenCover results from an single run of NUnit?
I finally found were was the NUnitResult.xml file. In fact it was in the folder were I put all test assemblies (ex: bin) while the Nunit batch put it at the root location.
I think this is because I use the -targetdir args with "bin" to indicate OpenCover were are my assemblies.
Related
Publish Test Results task mentions that "You can also use this task in a build pipeline to publish code coverage results produced when running tests to Azure Pipelines or TFS in order to obtain coverage reporting."
However, I have tried this for my test report of type .cobertura format. And it doesn't create the coverage tab. Though it works for .coverage and .covx/.covb files (tested)
<DataCollector friendlyName="Code Coverage">
<Configuration>
<Format>Cobertura</Format>
</Configuration>
</DataCollector>
I'm using VSTPI 17.1 to get cobertura support from VS 2022 however my ADO is using VS 2019.
So is it actually supported? If yes then what's required to make it work?
Alternatively, I can use PublishCodeCoverageResults task, however
I miss the power of VsTest task followed by publish-test-results task's auto aggregating results from multiple jobs within the same build or will not? https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results doesn't specify it however with publish-test-results, I always see aggregated results (desired)
And the doc mentions
"
Tasks such as Visual Studio Test, .NET Core also provide the option to publish code coverage data to the pipeline. If you are using these tasks, you do not need a separate Publish Code Coverage Results task in the pipeline."
And also there is a limit of 7 KB for this task.
Hence "Publish Test Results" should publish cobertura type as well because it's supported by the VsTest
Else I would run into needing to self merge multiple of them https://github.com/microsoft/vstest/issues/3497#issuecomment-1084741316
Expected: Publish Test Results Task to also publish the coverage to Coverage tab from .cobertura files.
i have 2 test reports. One for backend i.e. junit and one for front end i.e. karma.
i have to publish test result to jenkins. For junit there is plugin. but for karma there is not.
What can be used.
As of now i tried using junit and xunit to publish the report. But what jenkins is doing here is -- "It is merging both the result reports into a single trend graph."
As noted, you can get Karma to publish in JUnit format.
Alternatively, if it can output HTML, you can publish those: https://jenkins.io/doc/pipeline/steps/htmlpublisher/
Karma is a test runner for JavaScript. (for angular that runs jasmine test cases)
For Karma, you can use "karma-junit-reporter" which will generate an XML file of test cases and then use publish JUnit reporter from post-build action and give path of that XML file
I am running my test in TFS (Nunit plus Visual Studio with Adapter) and I have set the build definition as below
Build succeeds but no test result file was generated
Does TFS writes this Xml file ?
Log
2017-02-08T08:08:40.8151428Z Executing the powershell script: D:\A1\agent\tasks\PublishTestResults\1.0.20\PublishTestResults.ps1
2017-02-08T08:08:41.0963795Z ##[warning]No test result files were found using search pattern 'D:\A1_work\1\s**\TEST-*.xml'.
If the Nunit plus Visual Studio with Adapter means you have two test steps: one for unit tests and the other one for vs tests.
Please also add two "Publish Test Results" step one for Nunit format.
Also run your test manually on the build server to see if test result file .trx generated on the machine.
I need to include the output of my failing tests (NUnit) on the final build report of TFS.
I've looked around and found solutions for VSTest, but not NUnit unfortunately.
you can edit the build process template xaml you use and add a custom task. https://tfsbuildextensions.codeplex.com/wikipage?title=How%20to%20integrate%20the%20nUnit%20build%20activity&referringTitle=Documentation
Alternatively (and arguably worse, but it's what I do) is to not modify the template xaml but to target a custom msbuild file (.proj) into which I'd run the nunit tasks. You then get your build def to build this .proj rather that a sln and it's this that runs and outputs your nunit results.
I have started using Qunit to test my JS code. I am looking into JSCoverage to generate the coverage reports later. We have a CI server (Jenkins) which already do a few things with our PHP code and I was wondering if anyone can comment on how I can integrate the report from my Qunit and JSCoverage into Jenkins
Thanks
Sparsh
QUnit: use QUnit API to generate junit XML files. Here's a sample.
In Post-build Actions for your job you then check Publish JUnit test result report and specify your junit XML files (or their file pattern). Jenkins will then mark builds that have failed tests as unstable and produce a nice trend graph of successful/failing tests.
A few more details, for those actually attempting this:
Putting together QUnit and Jenkins
If you want to run QUnit and publish the results in Jenkins, you'll need to do the following:
Step 1: Getting QUnit to generate an XML file compatible with JUnit.
If you're using Apache Ant, this question explains how to get
QUnit to generate XML.
If not, you can use Grunt and
grunt-qunit-junit, together with grunt-contrib-qunit, to
run your .html tests.
And if you're not into either Ant or Grunt, here is
a script for PhantomJS to run your tests directly and produce
JUnit-style XML.
Step 2: Processing that XML file
This is the easy step - look in "Post-build Actions" for your job in Jenkins, and add the path to the XML file.