I have a Jenkins job to copy some test reports(XML) to local and convert them to JUnit reports via xUnit.
The issue is that there are no failed or skipped tests, but I always get
[xUnit] [INFO] - Check 'Failed Tests' threshold.
[xUnit] [INFO] - Check 'Skipped Tests' threshold.
[xUnit] [INFO] - Setting the build status to FAILURE
[xUnit] [INFO] - Stopping recording.
Finished: FAILURE
What causes this?
On my previous project I was running 3 development teams with 4 solutions that together consisted an integrated platform all on one Jenkins Master.
I had 2 DOS batch files that were held in a JenkinsScripts/ subdirectory at the root of each solution. One was called CodeCoverage.bat (which ensured OpenCover and ReportGenerator watched the unit tests) which in turn called RunUnitTests.bat passing parameters.
In RunUnitTests.bat the key line with many parameters resolved to:
packages\xunit.runner.console.2.1.0\tools\xunit.console.exe <MyUnitTestProject.dll> -nologo -noshadow -nunit TestReports\TestResults.xml
Hence, I was converting XUnit result XML to NUnit result XML and using the NUnit Plugin to publish the results.
Since I used this approach, the options for mapping of XUnit and NUnit 3 XML to Jenkins friendly JUnit (at my behest mind you) has improved.
Related
Using Jenkins I am trying to pass the result of the Nunit task and the OpenCover task to SonarQube.
SonarQube displays properly the test coverage but there is no trace of unit test results.
Here is what I do :
Build Visual Studio project with MSBuild
Batch windows for opencover via nunit
Launch SonarQube scanner
Batch windows for deployement
Here is my actual properties for the Sonar Scanner task:
sonar.projectKey=FC
sonar.projectName=FC
sonar.projectVersion=%SVN_REVISION%
sonar.sources=src
sonar.binaries=build
sonar.exclusions=src/FC.Test/**/*.*
sonar.tests=src/FC.Test
sonar.cs.nunit.reportsPaths=TestResult.xml
sonar.cs.opencover.reportsPaths=reports/opencovertests.xml
I'm using SonarQube 5.6, Jenkins 2.10, SonarQube plugin 2.4.3, OpenCover 4.5 and Nunit 2.6.4
Is there any known problems (compatibility issues...) ?
Note that I tryed to wrap my process with the Prepare and Terminate Sonar analysis tasks but it systematicaly deletes the coverage panel. So i removed it.
EDIT 2017
I stopped doing this way. I understood that the good way to do this was using the SonarQube Scanner for MSBuild which is integrated at the Jenkins's plugin.
The coverage panel was not shown because i didn't configure my quality profiles properly.
But...
I still don't have my tests' result in Sonar.
I checked the absolute path of my report and its parsing by sonarQube : "INFO: Sensor C# Unit Test Results Import [csharp] (done) | time=25ms". I also tryed with a .nunit file. I saw that apply a filter to opencover could help, so I did.
Do you have a clue?
Scanning C# projects requires to use the Sonar Scanner for MSBuild and follow a specific 3 steps process which I guess you don't follow for now. See doc about the 3 steps process at: http://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+MSBuild and http://docs.sonarqube.org/display/SCAN/From+Jenkins
Your properties look correct. Yet they shall not be provided in a sonar properties file but instead in the command line arguments of the MSBuild.SonarQube.Runner.exe begin step, for example:
MSBuild.SonarQube.Runner.exe begin <allMandatoryParameters> /d:sonar.cs.nunit.reportsPaths=TestResult.xml /d:sonar.cs.opencover.reportsPaths=reports\opencovertests.xml
Olivier
I am trying to get code coverage with Sonar and Jenkins. I see Jenkins' Sonar plugin successfully executes JUnit test cases and completes build successfully. But Sonar does not show Code Coverage results (always shows 0.0% as the code coverage) on the project. But Sonar does show "Unit test success".
I am using Maven with Jenkins and Sonar.
I get the below message in Jenkins logs while executing the Sonar plugin:
Project coverage is set to 0% as no JaCoCo execution data has been dumped: .../sonar/target/jacoco.exec
Can any one help me how to get correct code coverage on any Sonar project.
Just because Sonar invoked Surefire correctly (and you received the "Unit test success" message) doesn't mean that JaCoCo instrumented your code.
Try executing JaCoCo directly. You might find out why JaCoCo is failing directly:
mvn jacoco:prepare-agent test jacoco:report
JaCoCo will place jacoco.exec as well as its XML/HTML reports within target/jacoco. Or it will fail, and hopefully you'll have a better idea as to why.
One very common problem is that the JaCoCo javaagent will not run if you've changed the Surefire argLine at all, because jacoco:prepare-agent just sets the argLine property which in this scenario, is conveniently ignored. You can set prepare-agent's propertyName property to something else (like jacocoArgLine) and include that in your argLine config:
<argLine>-Xmx1024m ${jacocoArgLine}</argLine>
I read https://docs.sonarqube.org/display/PLUG/Code+Coverage+by+Unit+Tests+for+Java+Project and used cobertura as my code coverage plugin then I see code coverage displays for small projects. When I check for a big project in sonar I just see code coverage as - that means its empty. In logs I could find that Cobertura report was not found at /.../coverage.xml path.
coverage.xml was not generated due to OutOfMemeryError:heapspace. Since my project is such a big project when I set heap memory to 2GB and cobertura plugin memory to 1.5GB sonar gets code coverage displayed.
According to this blog post, you probably forget to set the sonar.binaries property in your project properties
Don’t forget sonar.binaries, otherwise, you might get something like « Project coverage is set to 0% since there is no directories with classes. » in your logs.
[...]
sonar.surefire.reportsPath=target/surefire-reports
sonar.jacoco.reportPath=target/jacoco.exec
sonar.binaries=target/classes
[...]
Your Sonar plug-in cannot find the report file generated by Jacoco.
In your pom(parent or child) look for the <destFile> tag under jacoco/prepare-agent execution. Change the name and the location to target/jacoco.exec and you are good to go!
Instead of using the Jenkins sonar plugin for running sonar, try to do it with maven, by executing the sonar goal. Usually this is easier then setting up an additional build step and tweaking all the paths till sonar finds all the relevant files.
In your case I guess it can read the junit/testng report, but can't find the jacoco results (jacoco.exec). Is the path sonar outputs(../sonar/target/jacoco.exec) correct?
I agree that this question is asked many a times by many people and many solutions are already been provided. However, after referring to umpteen links, not being able to display the Emma report in Sonar is driving me crazy. Below is a description of what I am trying to do.
My java project is a multi module maven project. We are using EMMA as the code coverage tool and Jenkins to build the project, Sonar to analyze the code and configured Sonar for our build job using the Jenkins sonar plugin. Up to this point, the story is great. The problem comes when I try to reuse the reports generated by Emma during Sonar analysis. There is no apparent error in the Sonar analysis logs during the build, however, in Sonar project dashboard the coverage report is 0%. Below is configuration data that I provide in the Sonar configuration section of my Jenkins build job.
-Dsonar.core.codeCoveragePlugin=emma
-Dsonar.emma.reportPath=/target
-Dsonar.dynamicAnalysis=reuseReports
-Dsonar.surefire.reportsPath=/target/surefire-reports
Sonar Version : 3.0
Emma plugin versions:
emma-maven-plugin : 1.0-alpha-2
Sonar Emma Plugin :1.0.1
emma : 2.0.5312
The Maven build generates the coverage.em files in the individual modules' target folder. No coverage.ec file is generated. Is it mandatory to have both coverage.ec and coverage.em files for Sonar?
The coverage.es,html,txt,xml files are generated in the project build directory/emma folder
The Sonar analysis logs for a module looks like below.
[INFO] [15:06:02.829] Sensor org.sonar.plugins.emma.EmmaSensor#bfc33...
[WARN] [15:06:02.844] Resource will be ignored in next Sonar versions, index is locked: com.mycompany.mypackage.tests.util.Activator
[INFO] [15:06:02.876] Sensor org.sonar.plugins.emma.EmmaSensor#bfc33 done: 47 ms
The above logs make me think that Sonar is indeed doing something with the generated Emma reports, but not displaying anything on the Sonar project dashboard.
Please let me know where I may be going wrong or what additional thing needs to be done to get this working.
I think this thread might me useful. http://sonar.15.n6.nabble.com/Receiving-quot-index-is-locked-quot-warning-when-Emma-sensor-runs-td4676490.html
It describes the same problem and gives you some workarounds
I followed the instructions here
I am NOT using Maven.
My Jenkins job output says:
12:32:33.951 INFO Sensor JaCoCoSensor...
12:32:33.961 INFO Project coverage is set to 0% as no JaCoCo execution data has been dumped: /var/lib/jenkins/workspace/SeqGen/SeqGen/jacoco.exec
12:32:35.152 INFO Sensor JaCoCoSensor done: 1201 ms
Here are my properties:
project.home=SeqGen
sonar.projectKey=com.skyboximaging:seqgen
sonar.projectName="SeqGen"
sonar.projectVersion=1.0
sonar.dynamicAnalysis=true
sources=src/java
tests=test/java
binaries=classes
sonar.jacoco.reportPath=jacoco.exec
sonar.jacoco.antTargets=test-with-coverage
On the Sonar server, I set General Settings > Code Coverage > Code coverage plugin to jacoco
I am very confused by the documentation at http://docs.codehaus.org/display/SONAR/Code+coverage+plugins I am particularly puzzled by this sentence:
"During Sonar analysis, the Sonar Jacoco plugin will take care to attach the Jacoco agent to the JVM and to launch the unit tests."
As far as I can tell, my Ant target is not getting invoked. How does the Sonar Jacoco plugin know where to find my build.xml?
What am I doing wrong?
They just released what you need : https://wiki.jenkins-ci.org/display/JENKINS/JaCoCo+Plugin
Providing an answer for the next poor soul who spends hours trying to find out why this doesn't work. Apparently this feature has been removed:
Remove "sonar.jacoco.antTargets" property and related classes in the plugin
One possibility is that the name of your ant task for junit does not match the name specified in the properties.
You have specified
sonar.jacoco.antTargets=test-with-coverage
As per the documentation, there should be an ant target with exactly the same name. Also in this target, junit should be set to fork as well.
my setup is:
Jenkins 1.448
Sonar 2.13
Jenkins Sonar Plugin 1.72
a MySql database for Sonar
I have the following project:
Eclipse Plugin Project
using Maven
multiple modules (parent maven project, source plugin project, test fragment project)
a simple JUnit test testing a class of the source plugin project
What the process is at the moment:
Jenkins checks out the parent project from SVN
Jenkins builds the project and its modules using Maven and Tycho
Jenkins uses Surefire to execute the tests and produce a report
Jenkins uses the Jenkins Sonar plugin to start mvn sonar:sonar to execute code analysis
Static code analysis is done on each module
JaCoCo fails
Sonar has static code analysis statistics available and test success percentage is shown.
The only thing that's missing is JaCoCo's code coverage...or maybe even Cobertura's code coverage.
I don't really care at this point, I just want one of these things to produce code coverage results.
Cobertura seems to be unable to produce results possibly because either
- it can't instrument the code
- it looks for the instrumented code at the wrong place
- it can't cope with Tycho or the separate source/test code setup
JaCoCo should be able to "watch" the code coverage on the fly, but its output is
[INFO] [17:44:04.708] Sensor JaCoCoSensor...
[INFO] [17:44:04.717] Project coverage is set to 0% as no JaCoCo execution data has been dumped: /var/lib/jenkins/jobs/testPlugin/workspace/testSource/target/jacoco.exec
[INFO] [17:44:04.854] Sensor JaCoCoSensor done: 146 ms
[INFO] [17:44:10.587] Sensor JaCoCoSensor...
[INFO] [17:44:10.587] Project coverage is set to 0% as no JaCoCo execution data has been dumped: /var/lib/jenkins/jobs/testPlugin/workspace/testTest/target/jacoco.exec
[INFO] [17:44:10.631] Sensor JaCoCoSensor done: 44 ms
[INFO] [17:44:12.402] Sensor JaCoCoSensor...
[INFO] [17:44:12.402] Project coverage is set to 0% as build output directory doesn't exists: /var/lib/jenkins/jobs/testPlugin/workspace/testParent/target/classes
[INFO] [17:44:12.402] Sensor JaCoCoSensor done: 0 ms
What am I doing wrong? Do I need to enable JaCoCo in my pom.xml somewhere? Do I need surefire? What do I need to do to use Cobertura?
I've found a page with an example tycho multi-module project (separate source and test projects). After copying some of the properties and configurations...
it seems that I now use a maven jacoco plugin to generate the jacoco code coverage report and then tell Sonar where to find the report and reuse it.
Why on earth is there next to no documentation for this? Or why is it so hard to find?
https://github.com/Godin/sonar-experiments/tree/master/jacoco-examples/tycho-example
You can have a look on the blog post I wrote few months ago : http://mdwhatever.free.fr/index.php/2011/09/quality-analysis-on-eclipse-plugins-with-tycho-sonar-jacoco-and-swtbot/
When the code is done being instrumented, surefire has to be ran on that specific code that the instrumentation generates. I've not used your setup, but your error seems very similar to what I got when I was getting Clover code coverage to work; I had to point surefire to run tests out of the directory that the compiled instrumented code was being stored in.
Also, I found specifying the -X option on the Maven command line to be invaluable as reading the logs of each plugin during a build shows you exactly where it's looking for whatever it's trying to find.
Edit:
Instrumented code will have to be compiled before surefire can run against it, so you'll want to point to the post-compilation directory and not the instrumented source itself.