Retry a Jenkins pipeline job on a specific error message - jenkins

So without Jenkins Pipeline the Naginator Plugin allows to restart a specific build on failure using regular expressions.
I like the retry option in Jenkins pipeline but I am not sure if I can catch an error from the build in the catch block and do a retry.
Is there a way to do so?
Eg: I have jenkins build which runs make. now make fails with an error: "pg_config.h missing". I want to catch this error and retry the build again a couple of times.
How can I do the above? Also, is it possible to catch multiple errors similar to regular expressions in Naginator somehow using pipelines?

retry("3"){
try {
sh "${cmd} 2>&1 > cmdOutput.txt"
sh "cat cmdOutput.txt"
} catch(FlowInterruptedException interruptEx) {
throw interruptEx
} catch(err) {
def cmdOutput = readFile('cmdOutput.txt').trim()
if (cmdOutput.contains("pg_config.h missing")) {
error "Command failed with error : ${err}. Retrying ...."
} else {
echo "Command failed with error other than `pg_config.h missing`"
}
}
}

I use the 'waitUntil' step and a counter to retry a shell command. I capture the output of the shell command so that I can run regex checks against the output and then continue or exit the loop.
// example pipeline
pipeline {
agent {
label ""
}
stages {
// stage('clone repo') {
// steps {
// git url: 'https://github.com/your-account/project.git'
// }
// }
// stage ('install') {
// steps {
// sh 'npm install'
// }
// }
stage('build') {
steps {
script {
// wrap with timeout so the job aborts if no activity
timeout(activity: true, time: 5, unit: 'MINUTES') {
// loop until the inner function returns true
waitUntil {
// setup or increment "count" counter and max value
count = (binding.hasVariable('count')) ? count + 1 : 1
countMax = 3
println "try: $count"
// Note: you must include the "|| true" after your command,
// so that the exit code always returns as 0. The "sh" command is
// actually running '/bin/sh -xe'. The '-e' option forces the script
// to exit on non-zero exit code. Prevent this by forcing a 0 exit code
// by adding "|| true"
// execute command and capture stdout
// Uncomment one of these 3 lines to test different conditions.
output = sh returnStdout: true, script: 'echo "Finished: SUCCESS" || true'
// output = sh returnStdout: true, script: 'echo "BUILD FAILED" || true'
// output = sh returnStdout: true, script: 'echo "something else happened" || true'
// show the output in the log
println output
// run different regex tests against the output to check the state of your build
buildOK = output ==~ /(?s).*Finished: SUCCESS.*/
buildERR = output ==~ /(?s).*BUILD FAILED.*/
// then check your conditions
if (buildOK) {
return true // success, so exit loop
} else if (buildERR) {
if (count >= countMax) {
// count exceeds threshold, so throw an error (exits pipeline)
error "Retried $count times. Giving up..."
}
// wait a bit before retrying
sleep time: 5, unit: 'SECONDS'
return false // repeat loop
} else {
// throw an error (exits pipeline)
error 'Unknown error - aborting build'
}
}
}
}
}
}
}
// post {
// always {
// cleanWs notFailBuild: true
// }
// }
}

Related

Continuous Integration pipeline

I am looking to trigger the on_failure step in my pipeline. I have a very simple script.
2 resources and 1 job. The job has a run step in which I would like to trigger failure manually. I have tried many things and they all leaded to an error.
Is there an shell script exit code that could make the task to fail and not being errored
Triggering post → failure and not failing the build is not possible:
failure
Only run the steps in post if the current Pipeline’s or stage’s run has a "failed" status, typically denoted by red in the web UI.
However, you can do the following:
def status
pipeline {
agent any
stages {
stage('Failing stage') {
steps {
script {
status = sh script: 'exit 99', returnStatus: true
}
}
}
}
post {
always {
script {
if ( status == 99 )
echo 'Script failed...'
else
echo 'Script succeeded...'
}
}
}
}
Example post -> failure
post {
always {
cleanWs()
}
success {
sendEmail('SUCCESSFUL')
}
unstable {
sendEmail('UNSTABLE')
}
failure {
sendEmail('FAILED')
}
}

How to create a script loop in Jenkins which continues or return success / error to pipeline based on json received by shell command?

I am using latest Jenkins in my Linux Box. I am trying to create a pipline with script block like below;
pipeline {
agent any
stages {
stage('TestStage') {
steps {
script {
sh "testfile.sh"
}
}
}
}
}
testfile.sh will return a json text like below;
{
"Worker": [
{
"Status": "running"
}
]
}
The Status can be either running or success or failure. If it is running, the code has to call testfile.sh again and check the status. If its success, pipeline has to continue to next step and if it is failure, pipeline has to terminate. Is it possible to achieve this?
Thanks.
You can achieve by creating a while loop that executes your script, reads the output and checks its value until a given timeout, when the status is no longer running you can check the result and fail the build according to the final status.
Something like:
pipeline {
agent any
stages {
stage('TestStage') {
steps {
script {
timeout(time: 1, unit: 'HOURS') { // timeout for the 'running' period
def status = sh script: 'testfile.sh', returnStdout: true
while (status == 'running') {
sleep time: 10, unit: 'SECONDS' // sleep between iterations
def output = sh script: 'testfile.sh', returnStdout: true
def dict = readJSON text: output
status = dict.Worker.Status
}
}
if(status == 'failure'){
error "Operation has ended with status 'failure'"
}
}
}
}
}
}
You can find more info on the relevant steps: sleep, timeout, error and readJSON.

Jenkins pipeline: Run all steps in stage, even if the first one fails

I have a series of steps in a stage that I want to run even if the first one fails. I want the stage result to fail and the build to get aborted, but only after all steps have run. For example,
pipeline {
agent any
stages {
stage('Run Test') {
steps {
sh "echo running unit-tests"
sh "echo running linting && false" // failure
sh "echo generating report" // This should still run (It currently doesn't)
publishCoverage adapters: [coberturaAdapter("coverage.xml")] // This should still run (It currently doesn't)
junit 'unit-test.xml' // This should still run (It currently doesn't)
}
}
stage('Deploy') {
steps {
echo "deploying" // This should NOT run
}
}
}
}
The result should be a failed build where the "Run Test" stage failed and the "Deploy" stage did not run. Is this possible?
P.S.
I am NOT asking for the same behavior as in Continue Jenkins pipeline past failed stage. I want to run the steps following the failure, but not any of the stages afterwards. I tried to enclose each of the test steps with catchError (buildResult: 'FAILURE', stageResult: 'FAILURE'), but the "Deploy" stage still runs.
EDIT:
I cannot combine all the steps into one big sh step and capture its return code because some of the steps are not shell commands, but instead jenkins steps like junit and publishCoverage.
A script witha non-zero exit code will always cause a jenkins step to fail. You can use returnStatus as true so that jenkins does not fails the step.
Additionally considering your use case, you could use a post always execution, so that the steps are always carried out.
Please see below reference example:
stage('Run Test') {
steps {
def unit_test_result= sh returnStatus: true, script: 'echo "running unit-tests"'
def lint_result= sh returnStatus: true, script: 'echo "running linting"'
if (unit_test_result!=0 || lint_result!=0 ) {
// If the unit_test_result or lint_result status is not 0 then mark this stage as unstable to continue ahead
// and all later stages will be executed
unstable ('Testing failed')
// You can also mark as failed as below and it will not conintue other stages:
// error ('Testing failed')
}
}
post {
always {
// This block would always be executed inspite of failure
sh "echo generating report"
publishCoverage adapters: [coberturaAdapter("coverage.xml")]
junit 'unit-test.xml'
}
}
}
I found a slightly hacky way to get the behavior I want. The other answers didn't work for me, either because they need all the steps to be sh steps, or they don't stop the deploy stage from running. I used catchError to set the build and stage result. But to prevent the next stage from running, I needed to an explicit call to error if the stage failed.
pipeline {
agent any
stages {
stage('Run Test') {
steps {
script {
// catchError sets the stageResult to FAILED, but does not stop next stages from running
catchError (buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "echo running unit-tests"
}
catchError (buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "echo running linting && false" // failure
}
catchError (buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "echo generating report" // This still runs
}
publishCoverage adapters: [coberturaAdapter("coverage.xml")] // This still runs
junit 'unit-test.xml' // This still runs
if (currentBuild.result == "FAILURE") { // This is needed to stop the next stage from running
error("Stage Failed")
}
}
}
}
stage('Deploy') {
steps {
echo "deploying" // This should NOT run
}
}
}
}
Theoretically you should be able to use sh "<command>||true" It would ignore the error on command and continue. However, Jenkins will not fail as it would ignore the error.
If you don't want Jenkins to ignore the error and want it to stop at the end of the stage, you can do something like: sh "<command>||$error=true" then fail the build based on the $error variable. (sh "$error" might be enough but I am not sure, may require an if statement at the end.) It will be only set to true iff command fails.
Another option is to wrap your build steps in a try-catch block! if there's an exception, i.e. return code of build is not 0 you can catch it, mark the build as unstable and then the rest of the pipeline continues on.
here's an example `
pipeline {
agent {
node {
label 'linux'
}
}
options {
timestamps()
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '3'))
}
tools {
maven 'Maven 3.6.3'
jdk 'jdk11'
}
stages {
stage('CleanWS') {
steps {
cleanWs()
}
}
stage('Build') {
steps {
withMaven(options: [artifactsPublisher(disabled: true)]) {
sh "export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn -f pom.xml clean install -DskipTests -Pregression-test -Dmaven.javadoc.skip=true"
}
}
}
stage('Test') {
steps {
script {
try {
withMaven(options: [artifactsPublisher(disabled: true)]) {
sh "export MAVEN_OPTS=\"-Xmx2048m\" && export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn -B verify -Dmaven.source.skip=true -Dmaven.javadoc.skip=true"
}
} catch (exc) {
currentBuild.result = 'UNSTABLE'
}
}
}
post {
always {
script {
junit "**/surefire-reports/*.xml"
}
}
}
}
stage('Sonar Analyse') {
steps {
script {
withMaven(options: [artifactsPublisher(disabled: true)]) {
withSonarQubeEnv("SonarQube") {
sh "export MAVEN_OPTS=\"-Xmx2048m\" && export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn sonar:sonar"
}
}
}
}
}
stage('Deploy to Nexus') {
steps {
sh "export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn -f pom.xml -B clean deploy -DdeployAtEnd=true -DskipTests"
}
}
}
post {
failure {
script {
emailext(
body: "Please go to ${env.BUILD_URL}/console for more details.",
to: emailextrecipients([developers(), requestor()]),
subject: "Nightly-Build-Pipeline Status is ${currentBuild.result}. ${env.BUILD_URL}"
)
}
}
unstable {
script {
emailext(
body: "Please go to ${env.BUILD_URL}/console for more details.",
to: emailextrecipients([developers(), requestor()]),
subject: "Nightly-Build-Pipeline Build Status is ${currentBuild.result}. ${env.BUILD_URL}"
)
}
}
}
}`

Jenkins - set timeout to Stage and continue with next stage

I have a stage which runs one shell script at remote node. If the script takes very long time to execute, the stage should wait for sometime and should move to next stage without aborting the subsequent stages.
Could you please let me know the required syntax to achieve this.
in a declarative pipeline, add this stage:
stage ("do-and-skip-for-timeout") {
steps {
script {
try {
timeout (time: 10, unit: 'MINUTES') {
echo "do something here, that can take some time" // replace this line with your code
}
}
catch (error) {
def user = error.getCauses()[0].getUser()
if('SYSTEM' == user.toString()) { // SYSTEM means timeout.
echo "Timeout reached, continue to next stage"
}
else {
throw new Exception("[ERROR] stage failed!")
}
}
}
}

Docker build unexpected EOF in Jenkins pipeline

I am sure I am not the only one who is interested in how to handle something like this: docker build stage in Jenkins pipeline fails with Unexpected EOF (there can be a lot of reasons, in my case the docker daemon was restarted on the slave)
appImage = docker.build ("${projectName}:${env.BRANCH_NAME}-${gitCommit}", "--build-arg APP_ENV=${appEnv} --build-arg SKIP_LINT=true .")
The deploy phase kicks in, because the Unexpected EOF does not actually throw any error, there is no exception to catch so the build status is null.
I know that it's not a regular situation but still how can we handle smth like this so that the following stages do not run in case the build is interrupted.
Additional details:
#JRichardsz , thanks for the answer! Usually currentBuild.result . defaults to null e.g. https://issues.jenkins-ci.org/browse/JENKINS-46325 so unless you set it to success explicitly upon successful stage's execution , it will be null. But all in all the same can be achieved with try catch like :
if (deployableBranches.contains(env.BRANCH_NAME)) {
try {
stage('Build image') {
ansiColor('xterm') {
appImage = docker.build
("${projectName}:${env.BRANCH_NAME}-${gitCommit}", "--build-arg
SKIP_LINT=true .")
}
}
stage('Push image') {
docker.withRegistry("${registryUrl}", "${dockerCredsId}") {
appImage.push()
appImage.push "${env.BRANCH_NAME}-latest"
}
}
stage('Deploy') {
build job: 'kubernetes-deploy', parameters: [
/////
]
}
} catch (e) {
// A shell step returns with a nonzero exit code
// When pipeline is in a shell step, and a user presses abort
if (e.getMessage().contains('script returned exit code 143')) {
currentBuild.result = "ABORTED"
} else {
currentBuild.result = "FAILED"
}
throw e
} finally {
// Success or failure or abort, always send notifications
stage('Send deployment status') {
helpers.sendDeploymentStatus(projectName, currentBuild.result,
helpers.getCommitHashShort())
}
}
}
But the issue is that stage('Build image') may exit without any error code like it was in my case.
I had a similar requirement : "If some rule is executed in stage A, following stages must not run"
This worked for me :
def flag;
node {
stage('A') {
flag = 1;
}
stage('B') {
// exit this stage if flag == 1
if(flag == 1){
return;
}
//start of stage B tasks
...
}
}
Also you could use some jenkins variable like currentBuild.result instead of flag like this :
node {
stage('A') {
//stage A tasks
//this stage could modify currentBuild.result variable
}
stage('B') {
// exit this stage if currentBuild.result is null , empty, "FAILURE", etc
if(currentBuild.result == null ||
currentBuild.result == "" ||
currentBuild.result=="FAILURE" ){
return;
}
//stage B tasks
}
}

Resources