Jenkins build gets failed, when unit test cases are failed,
Here am using pipeline script in jenkins, Need to generate HTML report using mocha-awesome, I can get the HTML report only when all test cases are passed, Build fails if any functions failed in my testcases.Here you can see the screenshot
The call to the unit test run returns with exit status 1.
What you can do is use the returnStatus option for the sh or bat step which will then not fail the build itself but leaves that up to you:
def exitStatus = sh returnStatus: true, script: 'unittests'
or:
def exitStatus = bat returnStatus: true, script: 'unittests.exe'
Having this you can selectively fail the build, i.e. if exitStatus == 1 then ignore as it signals a test fail, if it is anything but 0 or 1 then fail the build using the error step.
Related
I have a stage in my pipeline that is running some UI tests currently this is the behaviour I get:
If the tests pass the stage goes green the next stage runs, and at
the end the build goes green.
If a test fails the stage goes green, the next stage runs, and at the
end the build is yellow (unstable)
How can I make it so that instead of moving on to the next stage if a test fails the pipeline is failed?
This is the stage of my pipeline, I've tried adding a post section but even when a test fails it reports success.
stage('UITests') {
steps {
withCredentials([file(credentialsId: 'env_file', variable: 'envFile')]) {
sh '''
cat $envFile > .env.dev
make run_tests
'''
}
}
// Fail build if test fail
post{
success {
echo "UI Tests passed moving to Build stage"
}
failure {
error "UI Tests Failed, stopping the build"
}}
}
In the Jenkins log for the stage I can see when a test fails I get
error Command failed with exit code 1.
This doesn't happen when a test passes so is there a reason the post block is always going to success?
The sh step returns the same status code that your actual sh command: https://www.jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script
You can get this status code, assign its value to a variable and then fail with a custom message:
def statusCode = sh script:"cat $envFile > .env.dev && make run_tests", returnStatus:true
if (statusCode != 0) {
error 'UI Tests Failed, stopping the build'
}
I'm using Cucumber reports plugin in my declarative pipeline like that:
cucumber '**/cucumber.json'
I'm able to check if some tests fail through link on the sidebar, but do I need to do something to mark the stage containing cucumber.json check as failed if some cucumber reports are failed? Because the problem is the build and stage are both green and successful despite there are some failed cucumber reports.
Jenkins version is 2.176.3
Cucumber reports version is 4.10.0
Cucumber command you are using just generates the report regardless the test result.
So yes, you have to make your pipeline fail somehow as the problem you are facing is that your test command is not returning making your pipeline fail.
The way to go is to make that the command that runs the tests returns non-zero exit code (exit 1) if something went wrong on your tests. That would make your pipeline stage to go red.
In case you run your tests using Maven this would be automatically managed on 'mvn test' (or whatever).
Otherwise, if you cannot do that, you will have to manage to make something like for example an sh script
that returns the exit code (0 pass / 1 fail) or a groovy function inside 'script' tag that sets the pipeline currentBuild.result value:
def checkTestResult() {
// Check some file to see if tests went fine or not
return 'SUCCESS' // or 'FAILURE'
}
...
stage {
script {
currentBuild.result = checkTestResult()
if (currentBuild.result == 'FAILURE') {
sh "exit 1" // Force pipeline exit with build result failed
}
}
}
...
I recommend you to use cucumber command on a 'always' post build action of your declarative pipeline
as it is a step that you will likely execute every time at the end of the pipeline either if it passes or fails. See the following example:
pipeline {
stages {
stage('Get code') {
// Whatever
}
stage('Run tests') {
steps {
sh "mvn test" // run_tests.sh or groovy code
}
}
}
post {
always {
cucumber '**/cucumber.json'
}
}
}
It is possible to set BuildStatus : 'FAILURE' to mark build as failed if a report marked as failed.
cucumber fileIncludePattern: '**/cucumber.json', buildStatus: 'FAILURE'
I have a Jenkins setup with a pipeline that uses pytest to run some test suites. Sometimes a test fails and sometimes the test environment crashes (random HTTP timeout, external lib error, etc.). The job parses the XML test result but the build is marked as FAILURE as long as pytest returns non-zero.
I want Jenkins to get exit code zero from pytest even if there are failed tests but I also want other errors to be marked as failures. Are there any pytest options that can fix this? I found pytest-custom_exit_code but it can only suppress the empty test suite error. Maybe some Jenkins option or bash snippet?
A simplified version of my groovy pipeline:
pipeline {
stages {
stage ('Building application') {
steps {
sh "./build.sh"
}
}
stage ('Testing application') {
steps {
print('Running pytest')
sh "cd tests && python -m pytest"
}
post {
always {
archiveArtifacts artifacts: 'tests/output/'
junit 'tests/output/report.xml'
}
}
}
}
}
I have tried to catch exit code 1 (meaning some tests failed) but Jenkins still received exit code 1 and marks the build as FAILURE:
sh "cd tests && (python -m pytest; rc=\$?; if [ \$rc -eq 1 ]; then exit 0; else exit \$rc; fi)"
Your attempt does not work because Jenkins runs the shell with the errexit (-e) option enabled and that causes the shell to exit right after the pytest command before it reaches your if statement. There is a one-liner however that will work because it is executed as one statement: https://stackoverflow.com/a/31114992/1070890
So your build step would look like this:
sh 'cd tests && python -m pytest || [[ $? -eq 1 ]]'
My solution was to implement the support myself in pytest-custom-exit-code and create a pull request.
From version 0.3.0 of the plugin I can use pytest --suppress-tests-failed-exit-code to get the desired behavior.
I am using scrippted pipeline in Jenkins and I want to compile a solution using MsBuild.
Problem is when I run it using bat command: bat ' MsBuild.exe solution.sln /p:Configuration=Debug' (which runs it as a batch file) and when the build FAILS the job doesn't fail.
It's like it doesn't recognize that the MsBuild failed to compile the solution and continues to the next steps.
How can I run MsBuild and analize the output so that if the build fails, then the job will fail too?
Thank you
Try following and see how it goes:
def msbuild = "path/to/msbuild/MsBuild.exe"
def exitStatus = bat(returnStatus: true, script: "${msbuild} solution.sln /p:Configuration=Debug")
if (exitStatus != 0){
currentBuild.result = 'FAILURE'
}
And if you don't want to execute it any further, you can throw an error if exit status is not 0:
if (exitStatus != 0){
currentBuild.result = 'FAILURE'
error 'build failed'
}
I am running
sh "terraform fmt -list=true -write=false -diff=true -check=true"
within a Jenkins pipeline, and got the following weird error message:
ERROR: Terraform Lint failed due to: hudson.AbortException: script returned exit code 3
Finished: FAILURE
If I just run the terraform fmt ... without the pipeline, then it is ok. Any suggestion on how I can fix this weird error?
As mentioned in the Release Notes for Terraform 0.10.5 (but not in the documentation), the -check parameter
makes it return a non-zero exit status if any formatting changes are required
When you run it locally, it will not output any error, but if you would afterwards check the exit code echo $?, it would be again 3.
In Jenkins, however, any sh step returning non-zero status is treated as failure. So you either remove the -check option, or you deal with it correctly, i.e.,
def fmtStatus = sh "terraform fmt -list=true -write=false -diff=true -check=true"
if (fmtStatus == 0) {
echo "All correct"
} else {
echo "Changes necessary"
}