I have a maven Jenkins job (Jenkins version 2.105) with Jacoco and Sonar (Version 6.0) configured. The project has multiple jacoco.exec created and I need to put the path for the same under sonar.jacoco.reportpath. The code coverage comes up in sonar if I add for only one exec. While adding the others are comma separted values, code coverage in not displayed in Sonar.
As the version of SonarQube is prior to 6.2 I understand we are required to use sonar.jacoco.reportPath property and not sonar.jacoco.reportPaths. How do we configure multiple path here?
You need to merge your JaCoCo .exec files into a single binary file.
To achieve this use JaCoCo's merge mojo.
Cristian (from cristian.io) has an excellent walkthrough of how to achieve this here. The following is a slightly modified version of the code from that blog post.
def allTestCoverageFile = "$buildDir/jacoco/allTestCoverage.exec"
task jacocoMergeTest(type: JacocoMerge) {
destinationFile = file(allTestCoverageFile)
executionData = project.fileTree(dir: '.', include:'**/build/jacoco/test.exec')
}
sonarqube {
properties {
property "sonar.projectKey", "your.org:YourProject"
property "sonar.projectName", "YourProject"
property "sonar.jacoco.reportPath", allTestCoverageFile
}
}
Related
Is it possible to define/specify a runner when starting tests from cucumber's command line(cucumber.api.cli.Main)?
My reason for this is so i can generate xml reports in Jenkins and push the results to ALM Octane.
I kind of inherited this project and its using gradle to do a javaexect and call cucumber.api.cli.Main
I know its possible to do this with #RunWith(OctaneCucumber.class) when using JUnit runner + maven (or only JUnit runner), otherwise that tag is ignored. I have the custom runner with that tag but when i run from cucumber.api.cli.Main i can't find a way to run with it and my tag just gets ignored.
What #Grasshopper suggested didn't exactly work but it made me look in the right direction.
Instead of adding the code as a plugin, i managed to "hack/load" the octane reporter by creating a copy of the cucumber.api.cli.Main, using it as a base to run the cli commands and change a bit the run method and add the plugin at runtime. Needed to do this because the plugin required quite a few parameters in its constructor. Might not be the perfect solution, but it allowed me to keep the gradle build process i initially had.
public static byte run(String[] argv, ClassLoader classLoader) throws IOException {
RuntimeOptions runtimeOptions = new RuntimeOptions(new ArrayList<String>(asList(argv)));
ResourceLoader resourceLoader = new MultiLoader(classLoader);
ClassFinder classFinder = new ResourceLoaderClassFinder(resourceLoader, classLoader);
Runtime runtime = new Runtime(resourceLoader, classFinder, classLoader, runtimeOptions);
//====================Added the following lines ================
//Hardcoded runner(?) class. If its changed, it will need to be changed here also
OutputFile outputFile = new OutputFile(Main.class);
runtimeOptions.addPlugin(new HPEAlmOctaneGherkinFormatter(resourceLoader, runtimeOptions.getFeaturePaths(), outputFile));
//==============================================================
runtime.run();
return runtime.exitStatus();
}
I have two runners in my automation project as follows:
Main runner - Executes all the #ui-test tagged test cases and if a scenario is failed target/rerun.txt will be populated with the scenario location (e.g. features/Dummy.feature:22):
#RunWith(Cucumber.class)
#CucumberOptions(
features = "classpath:features",
plugin = {"pretty", "html:target/cucumber-html-report", "json:target/cucumber.json", "rerun:target/rerun.txt"},
tags = {"#ui-test", "~#ignore"}
)
public class RunCukesTest {
}
Secondary runner - Re-executes the scenarios from target/rerun.txt:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "#target/rerun.txt",
plugin = {"pretty", "html:target/cucumber-html-report-rerun", "json:target/cucumber_rerun.json"}
)
public class ReRunFailedCukesTest {
}
When the execution is performed two result json files are created:
cucumber.json
cucumber_rerun.json
Jenkins will collect the results via Cucumber-JVM Reports plugin and will create a combined report.
The problem is, even if all the target/rerun.txt tests are passed in the second run, the report status will remain failed because of the cucumber.json.
Is there a way (to set up Cucumber-JVM Reports plugin or modify the upper presented runners) to overwrite cucumber.json with the results from cucumber_rerun.json and to publish only the modified cucumber.json?
Another sub-keywords: maven, java, cucumber-java8, cucumber-junit, junit
I had problem similar to yours, though, I've used single runner, handled re-runs from testNG(re-runs was one of the reasons I've switched from JUnit to TestNG) directly and as a results I had increased amount of tests in my json report.
My solution was to clean json files afterwards, despite the fact that Jenkins knows about failed tests it won't mark build as failed or as unstable.
In your particular case you may try to somehow match tests from rerun.json and exclude them from regular json report.
For parsing jsons I may recommend using Jackson FasterXML
I use Jenkins cucumber reporting latest release with below config in Jenkins.
Image Of Config In Jenkins
1st Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="FolderFeature",
glue={"Gluefolder"},
plugin={"html:target/cucumberpf-html-report",
"json:target/cucumberpf.json"}
)
public class RunPF {
}
2nd Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="Blah/Test.feature",
glue={"mygluefolder"},
plugin={"html:target/cucumber-html-report",
"json:target/cucumber.json"}
)
public class RunRA {
}
I had failed in both .json files and when it passed both were merged and updated correctly in one cucumber report.
Here is the error:
[CucumberReport] Preparing Cucumber Reports
[CucumberReport] JSON report directory is "C:\Users\ajacobs\workspace\com.mytest.framework\target\"
[CucumberReport] Copied 2 json files from workspace "C:\Users\admin\workspace\yourtest\target" to
reports directory "C:\Users\admin\.jenkins\jobs\Regression\builds\21\cucumber-html-reports\.cache"
[CucumberReport] Processing 2 json files:
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumber.json
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumberpf.json
Finished: SUCCESS
My Code Coverage in Sonar is showing 0% which isn't true as I do have Unit Tests.
Gradle
sonarqube {
properties {
property "sonar.binaries", "build/intermediates/classes/release"
property "sonar.java.binaries", "build/intermediates/classes/release"
property "sonar.java.test.binaries", "build/intermediates/classes/test/release"
property "sonar.sources", "src"
property "sonar.junit.reportsPath", "build/reports/tests/release"
property "sonar.java.junit.reportsPath", "build/reports/tests/release"
property "sonar.android.lint.report", "build/outputs/lint-results.xml"
property "sonar.jacoco.reportPath", "${project.buildDir}/jacoco/testReleaseUnitTest.exec"
}
}
When I open up the index.html inside build/reports/tests/release then I can see successful unit tests.
I run sonarqube as a gradle task within my Jenkins environment. My SonarQube instance shows Code Smells and everything but for code coverage, it shows 0%.
Update
I do get an index.html created for the Code Coverage but that's all showing 0% too:
app/build/reports/jacoco/jacocoTestDebugUnitTestReport/html/index.html
Update
Still getting 0% but this is what I have so far:
android {
...
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
testCoverageEnabled true
}
debug {
testCoverageEnabled true
}
}
jacoco {
version "0.7.8.201607051106"
}
}
Excerpt from SonarQube Documentation:
The Java Plugin reuses reports; it does not generate them. So before trying to configure your analysis to import these reports, make sure they are correctly generated and non-empty.
Since you don't seem to be using the Gradle Jacoco plugin, SonarQube is probably reporting that 0% because you have not generated a report. You will need to add Jacoco to your build and ensure that you have fed SonarQube the path of the generated report (sonar.jacoco.reportPath) so it can read it.
To add Jacoco to your project, you will need to add the following to build.gradle:
//...
apply plugin: "jacoco"
//...
jacoco {
toolVersion = "0.7.6.201602180812"
//Note: unless "reportsDir" is set here, default is “$buildDir/reports/jacoco”
}
You will also need to ensure the following: First of all, you need to ensure the jacocoTestReport task is being executed (either by adding it to tasks yourself; or by adding the task to your gradle invocation). Secondly, you need to make sure that SonarQube is looking in the correct location for the test report by setting sonar.jacoco.reportPath to point to your /reports/jacoco directory (it's defaulting to target/jacoco.exec, so it won't find a report on default settings).
I fixed the issue by using this plugin. The problem as I see it was that Jacoco was trying to look for Instrumentation Tests androidTests and not Unit Tests tests. The plugin which I used made sure that it ran the tests before hand, created a report based on the tests AND make Jacoco point to those tests.
When you set up a Jenkins job various test result plugins will show regressions if the latest build is worse than the previous one.
We have many jobs for many projects on our Jenkins and we wanted to avoid having a 'job per branch' set up. So currently we are using a parameterized build to build eg different development branches using a single job.
But that means when I build a new branch any regressions are measured against the previous build, which may be for a different branch. What I really want is to measure regressions in a feature branch against the latest build of the master branch.
I thought we should probably set up a separate 'master' build alongside the parameterized 'branches' build. But I still can't see how I would compare results between jobs. Is there any plugin that can help?
UPDATE
I have started experimenting in the Script Console to see if I could write a post-build script... I have managed to get the latest build of master branch in my parameterized job... I can't work out how to get to the test results from the build object though.
The data I need is available in JSON at
http://<jenkins server>/job/<job name>/<build number>/testReport/api/json?pretty=true
...if I could just get at this data structure it would be great!
I tried using JsonSlurper to load the json via HTTP but I get 403, I guess because my script has no auth session.
I guess I could load the xml test results from disk and parse them in my script, it just seems a bit stupid when Jenkins has already done this.
I eventually managed to achieve everything I wanted, using a Groovy script in the Groovy Postbuild Plugin
I did a lot of exploring using the script console http://<jenkins>/script and also the Jenkins API class docs are handy.
Everyone's use is going to be a bit different as you have to dig down into the build plugins to get the info you need, but here's some bits of my code which may help.
First get the build you want:
def getProject(projectName) {
// in a postbuild action use `manager.hudson`
// in the script web console use `Jenkins.instance`
def project = manager.hudson.getItemByFullName(projectName)
if (!project) {
throw new RuntimeException("Project not found: $projectName")
}
project
}
// CloudBees folder plugin is supported, you can use natural paths:
project = getProject('MyFolder/TestJob')
build = project.getLastCompletedBuild()
The main test results (jUnit etc) seem to be available directly on the build as:
result = build.getTestResultAction()
// eg
failedTestNames = result.getFailedTests().collect{ test ->
test.getFullName()
}
To get the more specialised results from eg Violations plugin or Cobertura code coverage you have to look for a specific build action.
// have a look what's available:
build.getActions()
You'll see a list of stuff like:
[hudson.plugins.git.GitTagAction#2b4b8a1c,
hudson.scm.SCMRevisionState$None#40d6dce2,
hudson.tasks.junit.TestResultAction#39c99826,
jenkins.plugins.show_build_parameters.ShowParametersBuildAction#4291d1a5]
These are instances, the part in front of the # sign is the class name so I used that to make this method for getting a specific action:
def final VIOLATIONS_ACTION = hudson.plugins.violations.ViolationsBuildAction
def final COVERAGE_ACTION = hudson.plugins.cobertura.CoberturaBuildAction
def getAction(build, actionCls) {
def action = build.getActions().findResult { act ->
actionCls.isInstance(act) ? act : null
}
if (!action) {
throw new RuntimeException("Action not found in ${build.getFullDisplayName()}: ${actionCls.getSimpleName()}")
}
action
}
violations = getAction(build, VIOLATIONS_ACTION)
// you have to explore a bit more to find what you're interested in:
pylint_count = violations?.getReport()?.getViolations()?."pylint"
coverage = getAction(build, COVERAGE_ACTION)?.getResults()
// if you println it looks like a map but it's really an Enum of Ratio objects
// convert to something nicer to work with:
coverage_map = coverage.collectEntries { key, val -> [key.name(), val.getPercentageFloat()] }
With these building blocks I was able to put together a post-build script which compared the results for two 'unrelated' build jobs, then using the Groovy Postbuild plugin's helper methods to set the build status.
Hope this helps someone else.
We use clover for code coverage testing but it interferes with stack traces and error information. I want to be able to use cloverGenerateReport when doing automated builds via jenkins but to skip this step entirely when doing local builds.
I've tried the various suggestions from searches for 'gradle optional dependencies' but I can't seem to get clover completely out of the way.
Suggestions?
You can use the method onlyIf.
cloverGenerateReport.onlyIf {
project.hasProperty('enableClover') ? Boolean.valueOf(project.getProperty('enableClover')) : false
}
On the command line you can enable it by providing the project property:
gradle cloverGenerateReport -PenableClover=true
One solution would be to check if the environment variable "JENKINS_HOME" exists. If it does, then set cloverGenerateReport as a dependency to another task.
In your build.gradle:
def env = System.getenv()
if(env.containsKey('JENKINS_HOME')){
reportTask.dependsOn cloverGenerateReport
}