I am working on a grails application. We use cobertura to generate the code coverage reports. Now I want to modify the grails project such that the build should fail if the code coverage is less than say, 90%.
How can I achieve this in grails?
I don't think the code-coverage plugin supports this directly, but it's easy enough to do by hooking into the powerful grails build events infrastructure. By placing this in your scripts/_Events.groovy, the build will fail if coverage is below a certain threshold:
eventStatusFinal = { message ->
if (message ==~ /.*Cobertura Code Coverage Complete.*/) {
def report = new XmlSlurper().parse(new File("target/test-reports/cobertura/coverage.xml"))
if (Float.parseFloat(report.'#line-rate'.text()) < 0.90) {
throw new RuntimeException("coverage too low!")
}
}
}
This requires you to turn on the XML report generation with this in grails-app/conf/BuildConfig.groovy:
coverage {
xml = true
}
Adjust the attribute (line-rate, branch-rate) and value as needed.
Related
The multi branch pipeline plugin, awesome as it is, doesn't build tags out of the box. The usage of the basic-branch-build-strategies-plugin is required to enable tag discovery and building.
My question is directly related to: Is there a way to automatically build tags using the Multibranch Pipeline Jenkins plugin?
This plugin works great in the UI but doesn't appear to be easily configurable using the Jenkins job dsl. Does anyone have any examples of how to set the branch strategies using the dsl (or dsl configure->) so that tags will be discovered and built?
Having examined the delta between the config.xml files when the settings are changed via ui, it looks like I need to be able to add this trait:
<org.jenkinsci.plugins.github__branch__source.TagDiscoveryTrait />
and this section under build strategies:
<buildStrategies
<jenkins.branch.buildstrategies.basic.TagBuildStrategyImpl
plugin="basic-branch-build-strategies#1.1.1">
<atLeastMillis>-1</atLeastMillis>
<atMostMillis>172800000</atMostMillis>
</jenkins.branch.buildstrategies.basic.TagBuildStrategyImpl>
</buildStrategies>
Something like
multibranchPipelineJob('pipline') {
...
branchSources {
branchSource {
source {
github {
...
traits {
...
gitTagDiscovery()
}
}
buildStrategies {
buildTags {
atLeastDays '-1'
atMostDays '20'
}
}
}
}
}
}
is what I've been working with. It's not documented in the plugin, but that doesn't stop the job-dsl plugin from dynamically generating the API calls for it.
You can see what the API for your specific Jenkins installation is by going to {your_jenkins_url}/plugin/job-dsl/api-viewer/index.html.
Sometimes things won't appear there because a plugins lacks support for job-dsl.
In that case you can still generate the xml with the Configure Block.
However, this is pretty clumsy to use.
Edit: At least if I use gitHubTagDiscovery() as suggested by the dynamically generated API, Jenkins will crash. Instead, the configure block has to be used to get all the discovery methods for github.
configure {
def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits
traits << 'org.jenkinsci.plugins.github__branch__source.BranchDiscoveryTrait' {
strategyId(1)
}
traits << 'org.jenkinsci.plugins.github__branch__source.OriginPullRequestDiscoveryTrait' {
strategyId(1)
}
traits << 'org.jenkinsci.plugins.github__branch__source.TagDiscoveryTrait'()
}
I have a maven Jenkins job (Jenkins version 2.105) with Jacoco and Sonar (Version 6.0) configured. The project has multiple jacoco.exec created and I need to put the path for the same under sonar.jacoco.reportpath. The code coverage comes up in sonar if I add for only one exec. While adding the others are comma separted values, code coverage in not displayed in Sonar.
As the version of SonarQube is prior to 6.2 I understand we are required to use sonar.jacoco.reportPath property and not sonar.jacoco.reportPaths. How do we configure multiple path here?
You need to merge your JaCoCo .exec files into a single binary file.
To achieve this use JaCoCo's merge mojo.
Cristian (from cristian.io) has an excellent walkthrough of how to achieve this here. The following is a slightly modified version of the code from that blog post.
def allTestCoverageFile = "$buildDir/jacoco/allTestCoverage.exec"
task jacocoMergeTest(type: JacocoMerge) {
destinationFile = file(allTestCoverageFile)
executionData = project.fileTree(dir: '.', include:'**/build/jacoco/test.exec')
}
sonarqube {
properties {
property "sonar.projectKey", "your.org:YourProject"
property "sonar.projectName", "YourProject"
property "sonar.jacoco.reportPath", allTestCoverageFile
}
}
I have two runners in my automation project as follows:
Main runner - Executes all the #ui-test tagged test cases and if a scenario is failed target/rerun.txt will be populated with the scenario location (e.g. features/Dummy.feature:22):
#RunWith(Cucumber.class)
#CucumberOptions(
features = "classpath:features",
plugin = {"pretty", "html:target/cucumber-html-report", "json:target/cucumber.json", "rerun:target/rerun.txt"},
tags = {"#ui-test", "~#ignore"}
)
public class RunCukesTest {
}
Secondary runner - Re-executes the scenarios from target/rerun.txt:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "#target/rerun.txt",
plugin = {"pretty", "html:target/cucumber-html-report-rerun", "json:target/cucumber_rerun.json"}
)
public class ReRunFailedCukesTest {
}
When the execution is performed two result json files are created:
cucumber.json
cucumber_rerun.json
Jenkins will collect the results via Cucumber-JVM Reports plugin and will create a combined report.
The problem is, even if all the target/rerun.txt tests are passed in the second run, the report status will remain failed because of the cucumber.json.
Is there a way (to set up Cucumber-JVM Reports plugin or modify the upper presented runners) to overwrite cucumber.json with the results from cucumber_rerun.json and to publish only the modified cucumber.json?
Another sub-keywords: maven, java, cucumber-java8, cucumber-junit, junit
I had problem similar to yours, though, I've used single runner, handled re-runs from testNG(re-runs was one of the reasons I've switched from JUnit to TestNG) directly and as a results I had increased amount of tests in my json report.
My solution was to clean json files afterwards, despite the fact that Jenkins knows about failed tests it won't mark build as failed or as unstable.
In your particular case you may try to somehow match tests from rerun.json and exclude them from regular json report.
For parsing jsons I may recommend using Jackson FasterXML
I use Jenkins cucumber reporting latest release with below config in Jenkins.
Image Of Config In Jenkins
1st Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="FolderFeature",
glue={"Gluefolder"},
plugin={"html:target/cucumberpf-html-report",
"json:target/cucumberpf.json"}
)
public class RunPF {
}
2nd Runner
#RunWith(Cucumber.class)
#CucumberOptions(
features="Blah/Test.feature",
glue={"mygluefolder"},
plugin={"html:target/cucumber-html-report",
"json:target/cucumber.json"}
)
public class RunRA {
}
I had failed in both .json files and when it passed both were merged and updated correctly in one cucumber report.
Here is the error:
[CucumberReport] Preparing Cucumber Reports
[CucumberReport] JSON report directory is "C:\Users\ajacobs\workspace\com.mytest.framework\target\"
[CucumberReport] Copied 2 json files from workspace "C:\Users\admin\workspace\yourtest\target" to
reports directory "C:\Users\admin\.jenkins\jobs\Regression\builds\21\cucumber-html-reports\.cache"
[CucumberReport] Processing 2 json files:
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumber.json
[CucumberReport] C:\Users\admin\yourtest\builds\21\cucumber-html-reports\.cache\cucumberpf.json
Finished: SUCCESS
My Code Coverage in Sonar is showing 0% which isn't true as I do have Unit Tests.
Gradle
sonarqube {
properties {
property "sonar.binaries", "build/intermediates/classes/release"
property "sonar.java.binaries", "build/intermediates/classes/release"
property "sonar.java.test.binaries", "build/intermediates/classes/test/release"
property "sonar.sources", "src"
property "sonar.junit.reportsPath", "build/reports/tests/release"
property "sonar.java.junit.reportsPath", "build/reports/tests/release"
property "sonar.android.lint.report", "build/outputs/lint-results.xml"
property "sonar.jacoco.reportPath", "${project.buildDir}/jacoco/testReleaseUnitTest.exec"
}
}
When I open up the index.html inside build/reports/tests/release then I can see successful unit tests.
I run sonarqube as a gradle task within my Jenkins environment. My SonarQube instance shows Code Smells and everything but for code coverage, it shows 0%.
Update
I do get an index.html created for the Code Coverage but that's all showing 0% too:
app/build/reports/jacoco/jacocoTestDebugUnitTestReport/html/index.html
Update
Still getting 0% but this is what I have so far:
android {
...
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
testCoverageEnabled true
}
debug {
testCoverageEnabled true
}
}
jacoco {
version "0.7.8.201607051106"
}
}
Excerpt from SonarQube Documentation:
The Java Plugin reuses reports; it does not generate them. So before trying to configure your analysis to import these reports, make sure they are correctly generated and non-empty.
Since you don't seem to be using the Gradle Jacoco plugin, SonarQube is probably reporting that 0% because you have not generated a report. You will need to add Jacoco to your build and ensure that you have fed SonarQube the path of the generated report (sonar.jacoco.reportPath) so it can read it.
To add Jacoco to your project, you will need to add the following to build.gradle:
//...
apply plugin: "jacoco"
//...
jacoco {
toolVersion = "0.7.6.201602180812"
//Note: unless "reportsDir" is set here, default is “$buildDir/reports/jacoco”
}
You will also need to ensure the following: First of all, you need to ensure the jacocoTestReport task is being executed (either by adding it to tasks yourself; or by adding the task to your gradle invocation). Secondly, you need to make sure that SonarQube is looking in the correct location for the test report by setting sonar.jacoco.reportPath to point to your /reports/jacoco directory (it's defaulting to target/jacoco.exec, so it won't find a report on default settings).
I fixed the issue by using this plugin. The problem as I see it was that Jacoco was trying to look for Instrumentation Tests androidTests and not Unit Tests tests. The plugin which I used made sure that it ran the tests before hand, created a report based on the tests AND make Jacoco point to those tests.
When you set up a Jenkins job various test result plugins will show regressions if the latest build is worse than the previous one.
We have many jobs for many projects on our Jenkins and we wanted to avoid having a 'job per branch' set up. So currently we are using a parameterized build to build eg different development branches using a single job.
But that means when I build a new branch any regressions are measured against the previous build, which may be for a different branch. What I really want is to measure regressions in a feature branch against the latest build of the master branch.
I thought we should probably set up a separate 'master' build alongside the parameterized 'branches' build. But I still can't see how I would compare results between jobs. Is there any plugin that can help?
UPDATE
I have started experimenting in the Script Console to see if I could write a post-build script... I have managed to get the latest build of master branch in my parameterized job... I can't work out how to get to the test results from the build object though.
The data I need is available in JSON at
http://<jenkins server>/job/<job name>/<build number>/testReport/api/json?pretty=true
...if I could just get at this data structure it would be great!
I tried using JsonSlurper to load the json via HTTP but I get 403, I guess because my script has no auth session.
I guess I could load the xml test results from disk and parse them in my script, it just seems a bit stupid when Jenkins has already done this.
I eventually managed to achieve everything I wanted, using a Groovy script in the Groovy Postbuild Plugin
I did a lot of exploring using the script console http://<jenkins>/script and also the Jenkins API class docs are handy.
Everyone's use is going to be a bit different as you have to dig down into the build plugins to get the info you need, but here's some bits of my code which may help.
First get the build you want:
def getProject(projectName) {
// in a postbuild action use `manager.hudson`
// in the script web console use `Jenkins.instance`
def project = manager.hudson.getItemByFullName(projectName)
if (!project) {
throw new RuntimeException("Project not found: $projectName")
}
project
}
// CloudBees folder plugin is supported, you can use natural paths:
project = getProject('MyFolder/TestJob')
build = project.getLastCompletedBuild()
The main test results (jUnit etc) seem to be available directly on the build as:
result = build.getTestResultAction()
// eg
failedTestNames = result.getFailedTests().collect{ test ->
test.getFullName()
}
To get the more specialised results from eg Violations plugin or Cobertura code coverage you have to look for a specific build action.
// have a look what's available:
build.getActions()
You'll see a list of stuff like:
[hudson.plugins.git.GitTagAction#2b4b8a1c,
hudson.scm.SCMRevisionState$None#40d6dce2,
hudson.tasks.junit.TestResultAction#39c99826,
jenkins.plugins.show_build_parameters.ShowParametersBuildAction#4291d1a5]
These are instances, the part in front of the # sign is the class name so I used that to make this method for getting a specific action:
def final VIOLATIONS_ACTION = hudson.plugins.violations.ViolationsBuildAction
def final COVERAGE_ACTION = hudson.plugins.cobertura.CoberturaBuildAction
def getAction(build, actionCls) {
def action = build.getActions().findResult { act ->
actionCls.isInstance(act) ? act : null
}
if (!action) {
throw new RuntimeException("Action not found in ${build.getFullDisplayName()}: ${actionCls.getSimpleName()}")
}
action
}
violations = getAction(build, VIOLATIONS_ACTION)
// you have to explore a bit more to find what you're interested in:
pylint_count = violations?.getReport()?.getViolations()?."pylint"
coverage = getAction(build, COVERAGE_ACTION)?.getResults()
// if you println it looks like a map but it's really an Enum of Ratio objects
// convert to something nicer to work with:
coverage_map = coverage.collectEntries { key, val -> [key.name(), val.getPercentageFloat()] }
With these building blocks I was able to put together a post-build script which compared the results for two 'unrelated' build jobs, then using the Groovy Postbuild plugin's helper methods to set the build status.
Hope this helps someone else.