How do I fail a Jenkins pipeline with below a certain PHPUnit code coverage percentage? - jenkins

I am using a Jenkins declarative pipeline to implement continuous integration with my code. The code is a PHP project that uses PHPUnit for unit testing.
Currently, the pipeline
builds the project
runs a code sniffer to find coding violations
runs PHPUnit tests, generating coverage reports and publishing them
If any PHPUnit test fails, the Jenkins job fails (which is good).
Now, I also want to fail the job if the code coverage percentage is below a certain amount. I can't find how to do so.
The current Jenkinsfile looks like this:
pipeline {
agent any
stages {
stage ('Build') {
steps {
sh '''
cd symfony && composer install --prefer-dist
./vendor/bin/pcov clobber
'''
}
}
stage('PHP CodeSniffer') {
steps {
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh './symfony/vendor/bin/phpcs-meq.sh ./symfony/src/AppBundle'
}
}
}
stage('PHPUnit Tests') {
steps {
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh '''
cd symfony
cp phpunit.xml.dist phpunit.xml
./bin/console cache:warmup --env=test
./vendor/bin/phpunit\
--coverage-clover '../reports/coverage/coverage.xml'\
--coverage-html '../reports/coverage'\
--log-junit '../reports/unitreport.xml'
'''
}
junit 'reports/unitreport.xml'
publishHTML([
allowMissing: true,
alwaysLinkToLastBuild: false,
keepAll: true,
reportDir: 'reports/coverage',
reportFiles: 'index.html',
reportName: 'PHPUnit Test Coverage Report'
])
}
}
}
}
I have found this snippet, which publishes the clover report. I thought that perhaps making the "failingTarget" high enough would exit 1 (fail the pipeline) if the coverage didn't make the requirement. It doesn't -- it outputs a warning, but the step still succeeds.
step([
$class:'CloverPublisher',
cloverReportDir: '../reports/coverage',
cloverReportFileName: 'coverage.xml',
healthyTarget: [methodCoverage: 90, conditionalCoverage: 90, statementCoverage: 90],
unhealthyTarget: [methodCoverage: 89, conditionalCoverage: 89, statementCoverage: 89],
failingTarget: [methodCoverage: 88, conditionalCoverage: 88, statementCoverage: 88]
])
How can I fail a Jenkins pipeline if my PHP code coverage is below a certain threshold?

Is xml the only file format available to output the coverage report in?
I would focus on reading the file and then evaluating the necessary elements to determine whether you want to throw an error.
for XML, that would involve importing groovy.util.XmlSlurper, reading the file, parsing it and then navigating the structure.
JSON would be easier, could just use readJSON jenkins basic step and then access the map.

Related

Fail build at 80% pass rate NUnit-Console + Jenkins

Could you tell me how to run tests in NUnit-Console so that they don't fail the build in Jenkins.
I want my build to fail at 80% pass rate. But under the current conditions, it fails if at least 1 test fails. Because the build returns an error code equal to the number of failed tests.
stage('Run tests') {
steps {
bat """nunit3-console.exe "Tests.dll" --skipnontestassemblies --result="TestResult.xml" """
}
}
I planned to fail the build using the Nunit plugin for Jenkins
post {
always {
step([$class: 'NUnitPublisher', testResultsPattern: """TestResult.xml""", healthScaleFactor: 1.0, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true, failedTestsFailBuild: false ])
}
}

Cannot access recordIssues in Jenkins Pipeline using Warnings Next Generation

I have a simple Jenkinsfile with a recordIssues step. The relevant code looks like this:
step([
$class: 'recordIssues',
aggregatingResults: true,
enabledForFailure: true,
tools: [pyLint()]
])
I have installed the latest version of the Warnings Next Generation plugin (https://plugins.jenkins.io/warnings-ng) but I run into the following issue:
[Pipeline] step
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.UnsupportedOperationException: no known implementation of interface jenkins.tasks.SimpleBuildStep is named recordIssues
at org.jenkinsci.plugins.structs.describable.DescribableModel.resolveClass(DescribableModel.java:478)
Is it somehow possible to check that the extension is installed correctly?
This is my working version for a Python project CI configuration in Jenkins to use JUnit, PEP8, Pylint and Coverage reports:
...
stage('Test: Run') {
steps {
// Run my project tests.
sh 'coverage run manage.py tests'
// Dump coverage metrics to XML.
sh 'coverage xml'
// Run Pylint.
sh 'pylint --rcfile=.pylintrc my_project > reports/pylint.report'
// Run Pycodestyle (PEP8 checks).
sh 'pycodestyle my_project > reports/pep8.report'
}
post {
always{
// Generate JUnit, PEP8, Pylint and Coverage reports.
junit 'reports/*junit.xml'
recordIssues(
tool: pep8(pattern: 'reports/pep8.report'),
unstableTotalAll: 200,
failedTotalAll: 220
)
recordIssues(
tool: pyLint(pattern: 'reports/pylint.report'),
unstableTotalAll: 20,
failedTotalAll: 30
)
cobertura coberturaReportFile: 'reports/coverage.xml'
}
}
}
...
It works with Cobertura plugin, JUnit plugin and Warnings Next Generation. Python packages I used are the traditional coverage and pylint and for PEP8 I used pycodestyle.
Hope this helps somebody else, as finding good examples of this Jenkinsfile stuff is not easy.
Just for the record:
Jenkins v2.204.2
Jenkins Warnings Next Generation Plugin v8.0.0
stage('Static Analysis') {
steps {
recordIssues(
tool: pyLint(pattern: '**/pylint.out'),
unstableTotalAll: 100,
)
}
This works for me (Jenkins ver. 2.164.1):
stage('Static Analysis') {
recordIssues(
tool: pyLint(pattern: '**/pylint.out'),
unstableTotalAll: '100',
)
recordIssues(
tool: pep8(pattern: '**/pep8.out'),
unstableTotalAll: '100',
)
}

No such DSL method 'publishHTML' found among steps

I have a jenkins DSL step that runs my python nosetests and creates a unittest coverage report.
Here is my jenkins stage.
stage ('Unit Tests') {
steps {
sh """
#. venv/bin/activate
export PATH=${VIRTUAL_ENV}/bin:${PATH}
make unittest || true
"""
}
post {
always {
junit keepLongStdio: true, testResults: 'report/nosetests.xml'
publishHTML target: [
reportDir: 'report/coverage',
reportFiles: 'index.html',
reportName: 'Coverage Report - Unit Test'
]
}
}
}
I get this error -
java.lang.NoSuchMethodError: No such DSL method 'publishHTML' found among steps.
How can I fix this error? I got this piece of code from a different repository.
The publishHTML method is provided by the HTLMPublisher Jenkins plugin. After installing the plugin on the primary Jenkins server, the publishHTML method will be available in the Jenkins Pipeline for both scripted and declarative syntax.

Jenkins Pipeline publish html report

Maven clean install generate new html file in following location
/var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/DocsJmeterTests_20170601_151330/index.html
Here "DocsJmeterTests_20170601_151330" will change for every run. So i am trying to publish html report using publish html report plugin. Following is my Pipeline script
node {
build job: 'Docs_LoadTest'
stage('Results') {
publishHTML([allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir:
'/var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/*/',
reportFiles: 'index.html',
reportName: 'Docs Loadtest Dashboard'
])
}
}
Getting following error while running the job
[htmlpublisher] Archiving HTML reports...
[htmlpublisher] Archiving at BUILD level
/var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/* to
/var/lib/jenkins/jobs/Docs_Pipeline/builds/10/htmlreports/Docs_Loadtest_Dashboard
ERROR: Specified HTML directory '/var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/*' does not exist.
Even we tried following options didnt worked
/var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/**/
/var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/DocsJmeterTests_*
/var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/DocsJmeterTests_*
_*
The HTML Publisher plugin does not seem to understand wildcards. What you could do in your Pipeline is using Linux's copy command, since that can work with wildcards.
This copies over the contents of all directories in the [Docs_LoadTest]/jmeter/reports folder to a jmeter_results folder in the local workspace:
sh 'cp -r /var/lib/jenkins/workspace/Docs_LoadTest/target/jmeter/reports/*/. target/jmeter_results/'
Note that you must clean both your target folder in the Docs_LoadTest folder and your Pipeline in between runs, else multiple reports will be copied over with this solution.
A better solution:
Would be to apply this trick in the Docs_LoadTest and use the Publish Artifact and Copy Artifact features. This works around having to hardcode the path to the other job and will work even if the Pipeline executes on another slave than the Docs_LoadTest. This does require the Copy Artifacts plugin.
Assuming Docs_LoadTest is a Freestyle job:
Add an Execute Shell Build step that copies the results to a fixed folder, e.g. jmeter_results:
mkdir -p target/jmeter_results/
cp -r target/jmeter/reports/*/. target/jmeter_results/
Then add an Archive Artifacts Post Build Archive Step with the following files to archive:
target/jmeter_results/*
In your Pipeline:
Use the Copy Artifact step to copy the files to target/jmeter_results folder in the local workspace:
step ([$class: 'CopyArtifact',
projectName: 'Docs_LoadTest',
filter: 'target/jmeter_results/*']);
Change the call to the HTML publisher to use this folder:
publishHTML([allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: 'target/jmeter_results',
reportFiles: 'index.html',
reportName: 'Docs Loadtest Dashboard'
])
I was having similar problem, only that I wanted to publish multiple reports.
What I ended up doing was I added simple groovy script to iterate through files in reports directory. You can use same/similar approach to get file name.
stage('publish reports') {
steps {
unstash 'source'
script {
sh 'ls target/jmeter/reports > listFiles.txt'
def files = readFile("listFiles.txt").split("\\r?\\n");
sh 'rm -f listFiles.txt'
for (i = 0; i < files.size(); i++) {
publishHTML target: [
allowMissing:false,
alwaysLinkToLastBuild: false,
keepAll:true,
reportDir: 'target/jmeter/reports/' + files[i],
reportFiles: 'index.html',
reportName: files[i]
]
}
}
}
}
Note: this example is used in declarative pipeline. Docs about readFile function.
I have tried simply the followings.
stage('Test-Junit') {
steps {
sh 'gradle test'
}
post {
always {
script {
def moduleNames = ["app", "core", ...]
for(i=0; i<moduleNames.size(); i++ ) {
publishHTML target: [
allowMissing:false,
alwaysLinkToLastBuild: false,
keepAll:true,
reportDir: moduleNames[i] + '/build/reports/tests/test',
reportFiles: 'index.html',
reportName: 'Test Report:' + moduleNames[i]
]
}
}
}
}
}
It will make all modules report and thus you can find them on left nav-bar of project dash-board.
It is not exactly the same scenario, but decided to publish my code because was really hard to understand, clarify and get documentation and accurate examples on how to publish different reports in just one final consolidated report, using the publishHTML plug-in for Jenkins.
A bit of background, we are executing different packages of testing, but some test cases can't run together because they could kill each other, so we need to execute, from the same code, in two different time frames due that we run test cases in parallel.
The solution was to execute by tags, so once the different execution using a Jenkins DSL - pipeline happens the builds produce just one report with different tabs on it.
So this is the final code that works for me:
pipeline {
agent any
stages {
stage('Git') {
steps {
git .....
}
}
stage('Exec-1') {
steps {
bat 'mvn -B clean verify -Dcucumber.filter.tags=#exec1 -Dserenity.outputDirectory=reports/site/serenity/exec1'
}
stage('Exec-2') {
steps {
bat 'mvn -B clean verify -Dcucumber.filter.tags=#exec2 -Dserenity.outputDirectory=reports/site/serenity/exec2'
}
}
stage('Exec-3') {
steps {
bat 'mvn -B clean verify -Dcucumber.filter.tags=#exec3 -Dserenity.outputDirectory=reports/site/serenity/exec3'
}
}
}
post {
always {
publishHTML target: [
reportName: 'Test',
reportDir: 'reports/site/serenity',
reportFiles: 'exec1/index.html, exec2/index.html, exec3/index.html',
reportTitles: 'Exec-1, Exec-2, Exec-3',
keepAll: true,
alwaysLinkToLastBuild: true,
allowMissing: false
]
}
}
}

Jenkins Pipeline always fails

My Jenkins pipeline stages are all successful yet the build always says it failed. To be clear, the build was a success and I would like it to register with Jenkins as successful but for some reason Jenkins thinks it has failed. All the stages say success in the dashboard yet the build is marked with a red ball and the console output ends with Finished: FAILURE.
Here is my pipeline file
node {
try{
stage 'Clone repo'
sh 'gcloud source repos clone <repo-name> --project=<project-name>'
dir('<repo-name>') {
try{
stage 'Run tests'
sh './gradlew test'
stage 'Run integration tests'
sh './gradlew integrationTest'
publishHTML(target: [
allowMissing: false,
alwaysLinkToLastBuild: false,
keepAll: false,
reportDir: '<repo-name>/build/reports/integrationTest',
reportFiles: 'index.html',
reportName: 'Integration Test Reports'])
} finally {
stage 'Stop and remove docker containers'
sh 'docker-compose down'
sh 'docker-compose rm -f'
}
}
} finally {
deleteDir()
}
}
I realised I had included the full path to the html reports when I was actually inside a dir block. There was no indication in the logs of this.
is your problem solved ?
Anyway the deleteDir function can be a problem sometimes. Because deleteDir recursively deletes the current directory and its contents, you can raise an error if you try to delete the complete job workspace.
Take care to use it in a dir function as below:
dir('directoryToDelete') {
deleteDir()
}

Resources