I am new to Jenkins and SonarQube configuration. Wanted to display the code coverage of PCs layer of the project in SonarQube Dashboard. (Test cases are written using MS test framework)
Facing a problem as SonarQube Dashboard is showing 0% coverage inspite of having 40 + test cases for PCs layer.
Following are the ‘Build’ configuration:
Part 2:
SonarQube Dashboard
Assuming the probable cause – the sonarqube dashboard showing results of WebProject(as highlighted in screenshot) and not for the business layer. Need help in showing coverage of Business Layer of the Project(PCs as highlighted in screenshot)
You have 24k lines of code and 40 unit tests? The coverage percentage likely comes in at <1%... less than .5% even. So that's being rounded to 0%.
If you want to narrow it down, you can set exclusions for the stuff you don't want included in the coverage calculation (you'll have to re-analyze after making the updates for them to take effect) but you'll still likely weigh in around 0% until you add more tests.
This after months I was able to resolve. As the requirement was to achieve code coverage for Business layers like PC's as highlighted in above question. This was fixed by following the below steps:
Sonar plugin order should be:
Execute Windows batch command - to delete the .trx or .coveragexml files
SonarQube Scanner for MSBUILD - Begin Analysis
Build a VS project or solution using MSBUILD
Run Unit test with VSTest.console
Execute windows batch command- to convert the test report from trx to coveragexml
SonarQube Scanner for MSBUILD -End Analysis
Note: Make sure to Run Unit test with VSTest.console and not with MS Test
Change the path in windows batch command plugin to delete the test result file
d:\jenkins\tools\nuget\nuget.exe restore "%WORKSPACE%\<Project>.sln"
if exist TestResults\ del TestResults\*.trx
if exist TestResults\ del TestResults\*.coveragexml
Changed the path of the unit test dll to - "Bin\.UnitTest.dll".
Related
Publish Test Results task mentions that "You can also use this task in a build pipeline to publish code coverage results produced when running tests to Azure Pipelines or TFS in order to obtain coverage reporting."
However, I have tried this for my test report of type .cobertura format. And it doesn't create the coverage tab. Though it works for .coverage and .covx/.covb files (tested)
<DataCollector friendlyName="Code Coverage">
<Configuration>
<Format>Cobertura</Format>
</Configuration>
</DataCollector>
I'm using VSTPI 17.1 to get cobertura support from VS 2022 however my ADO is using VS 2019.
So is it actually supported? If yes then what's required to make it work?
Alternatively, I can use PublishCodeCoverageResults task, however
I miss the power of VsTest task followed by publish-test-results task's auto aggregating results from multiple jobs within the same build or will not? https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results doesn't specify it however with publish-test-results, I always see aggregated results (desired)
And the doc mentions
"
Tasks such as Visual Studio Test, .NET Core also provide the option to publish code coverage data to the pipeline. If you are using these tasks, you do not need a separate Publish Code Coverage Results task in the pipeline."
And also there is a limit of 7 KB for this task.
Hence "Publish Test Results" should publish cobertura type as well because it's supported by the VsTest
Else I would run into needing to self merge multiple of them https://github.com/microsoft/vstest/issues/3497#issuecomment-1084741316
Expected: Publish Test Results Task to also publish the coverage to Coverage tab from .cobertura files.
Using Jenkins I am trying to pass the result of the Nunit task and the OpenCover task to SonarQube.
SonarQube displays properly the test coverage but there is no trace of unit test results.
Here is what I do :
Build Visual Studio project with MSBuild
Batch windows for opencover via nunit
Launch SonarQube scanner
Batch windows for deployement
Here is my actual properties for the Sonar Scanner task:
sonar.projectKey=FC
sonar.projectName=FC
sonar.projectVersion=%SVN_REVISION%
sonar.sources=src
sonar.binaries=build
sonar.exclusions=src/FC.Test/**/*.*
sonar.tests=src/FC.Test
sonar.cs.nunit.reportsPaths=TestResult.xml
sonar.cs.opencover.reportsPaths=reports/opencovertests.xml
I'm using SonarQube 5.6, Jenkins 2.10, SonarQube plugin 2.4.3, OpenCover 4.5 and Nunit 2.6.4
Is there any known problems (compatibility issues...) ?
Note that I tryed to wrap my process with the Prepare and Terminate Sonar analysis tasks but it systematicaly deletes the coverage panel. So i removed it.
EDIT 2017
I stopped doing this way. I understood that the good way to do this was using the SonarQube Scanner for MSBuild which is integrated at the Jenkins's plugin.
The coverage panel was not shown because i didn't configure my quality profiles properly.
But...
I still don't have my tests' result in Sonar.
I checked the absolute path of my report and its parsing by sonarQube : "INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=25ms". I also tryed with a .nunit file. I saw that apply a filter to opencover could help, so I did.
Do you have a clue?
Scanning C# projects requires to use the Sonar Scanner for MSBuild and follow a specific 3 steps process which I guess you don't follow for now. See doc about the 3 steps process at: http://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+MSBuild and http://docs.sonarqube.org/display/SCAN/From+Jenkins
Your properties look correct. Yet they shall not be provided in a sonar properties file but instead in the command line arguments of the MSBuild.SonarQube.Runner.exe begin step, for example:
MSBuild.SonarQube.Runner.exe begin <allMandatoryParameters> /d:sonar.cs.nunit.reportsPaths=TestResult.xml /d:sonar.cs.opencover.reportsPaths=reports\opencovertests.xml
Olivier
In our continuous integration process we are using Jenkins, NUnit and OpenCover.
Every Jenkins job runs NUnit and OpenCover, but OpenCover calls the NUnit batch file in order to determine code coverage; therefore NUnit is executed twice.
For example we have a first (simplified) batch:
nunit-console-x86 [PathToTestAssemblies] /xml=NunitResult.xml /noshadow /nodots /process=Separate
And we have a second batch for OpenCover:
OpenCover.Console.exe -target:"NUnit.bat" -output:"./OpenCoverResults.xml" -register -targetdir:".\bin" -coverbytest:*.dll
The problem is that OpenCover does not provide NUnit result (The NunitResult.xml file in my previous command). So in order to have less test time for every Jenkins job we want to get back the NUnit result or find a way to have the following features with OpenCover in the Jenkins job web page:
Latest Tests result from every Jenkins job, so it is easy for a developer to see the latest result.
Test result trend
Is there a way to have both NunitResult and OpenCover results from an single run of NUnit?
I finally found were was the NUnitResult.xml file. In fact it was in the folder were I put all test assemblies (ex: bin) while the Nunit batch put it at the root location.
I think this is because I use the -targetdir args with "bin" to indicate OpenCover were are my assemblies.
I am trying to get code coverage with Sonar and Jenkins. I see Jenkins' Sonar plugin successfully executes JUnit test cases and completes build successfully. But Sonar does not show Code Coverage results (always shows 0.0% as the code coverage) on the project. But Sonar does show "Unit test success".
I am using Maven with Jenkins and Sonar.
I get the below message in Jenkins logs while executing the Sonar plugin:
Project coverage is set to 0% as no JaCoCo execution data has been dumped: .../sonar/target/jacoco.exec
Can any one help me how to get correct code coverage on any Sonar project.
Just because Sonar invoked Surefire correctly (and you received the "Unit test success" message) doesn't mean that JaCoCo instrumented your code.
Try executing JaCoCo directly. You might find out why JaCoCo is failing directly:
mvn jacoco:prepare-agent test jacoco:report
JaCoCo will place jacoco.exec as well as its XML/HTML reports within target/jacoco. Or it will fail, and hopefully you'll have a better idea as to why.
One very common problem is that the JaCoCo javaagent will not run if you've changed the Surefire argLine at all, because jacoco:prepare-agent just sets the argLine property which in this scenario, is conveniently ignored. You can set prepare-agent's propertyName property to something else (like jacocoArgLine) and include that in your argLine config:
<argLine>-Xmx1024m ${jacocoArgLine}</argLine>
I read https://docs.sonarqube.org/display/PLUG/Code+Coverage+by+Unit+Tests+for+Java+Project and used cobertura as my code coverage plugin then I see code coverage displays for small projects. When I check for a big project in sonar I just see code coverage as - that means its empty. In logs I could find that Cobertura report was not found at /.../coverage.xml path.
coverage.xml was not generated due to OutOfMemeryError:heapspace. Since my project is such a big project when I set heap memory to 2GB and cobertura plugin memory to 1.5GB sonar gets code coverage displayed.
According to this blog post, you probably forget to set the sonar.binaries property in your project properties
Don’t forget sonar.binaries, otherwise, you might get something like « Project coverage is set to 0% since there is no directories with classes. » in your logs.
[...]
sonar.surefire.reportsPath=target/surefire-reports
sonar.jacoco.reportPath=target/jacoco.exec
sonar.binaries=target/classes
[...]
Your Sonar plug-in cannot find the report file generated by Jacoco.
In your pom(parent or child) look for the <destFile> tag under jacoco/prepare-agent execution. Change the name and the location to target/jacoco.exec and you are good to go!
Instead of using the Jenkins sonar plugin for running sonar, try to do it with maven, by executing the sonar goal. Usually this is easier then setting up an additional build step and tweaking all the paths till sonar finds all the relevant files.
In your case I guess it can read the junit/testng report, but can't find the jacoco results (jacoco.exec). Is the path sonar outputs(../sonar/target/jacoco.exec) correct?
I'm trying to use JaCoCo agent for collecting FT code coverage of a web-site by adding "-javaagent..." to Jetty and running external Selenium tests (or, for example, manual testing).
At the end of testing I have results only for web-module, that was actually run in Jetty, but not for "core" modules (in the same project) being used in web-module. So, is there any ability to collect calls for all modules of the project?
Indeed, I now tried to start all project webservices (it has 5) with the same JaCoCo report destination (i.e. /tmp/jacoco-res.exec), append=true and ran:
mvn sonar:sonar -Dsonar.jacoco.itReportPath=/tmp/jacoco-res.exec
So, sonar now used the same JaCoCo report file for every module in project and sonar shows tests coverage both for jetty-run and core modules.
Is this approach applicable and coverage results correct?
I had a simular question that I posed on the Sonar User forum a couple of days ago.
In short, your approach is correct and was also proposed by a Sonar consultant (as can be seen on the post).
Perform a Sonar analyze on each Maven project.
For each analysis, reuse the report and pass your JaCoCo report (using -Dsonar.jacoco.itReportPath)
Sonar will retrieve from this JaCoCo report
the coverage data related to the Maven project being analyzed. (one of your core projects)
Thus, for each Maven project, you'll get the coverage by your integration tests.
Finally, to get the coverage of your webservice by your integration
tests, you will have to use the Views plugin:
http://www.sonarsource.com/products/plugins/governance/portfolio-management/
Create a view that is an aggregation of all the Maven
projects composing the webservice.
Run a Sonar analysis of one of its
Maven projects in order for Sonar to compute the view.
On the view dashboard, you'll be able to get the code coverage
of the webservice by your integration tests.