I have a Jenkins pipeline where I have some stages implemented.
Jenkins outputs some messages while the job is running.
Code:
stage('My stage') {
options {
timeout(time: 2, unit: "MINUTES")
}
steps {
script {
./script.sh
}
}
}
}
Output:
...
20:12:40 Timeout set to expire in 2 min 0 sec
[Pipeline] {
[Pipeline] script
[Pipeline] {
20:12:40 ./script.sh '
20:14:40 Cancelling nested steps due to timeout
20:14:40 Sending interrupt signal to process
20:14:48 ...
20:14:48 + JOB_NAME=...
20:14:48 + BUILD_URL=...
20:14:48 + BUILD_ID=567
20:14:48 + BUILD_RESULT=SUCCESS
In this example I am in the middle of a job and I have lots of messages been displayed. I need to capture this message "Cancelling nested steps due to timeout".
When jenkins outputs this message I want to be able to send notifications to a user.
I am not sure how to do this.
After executing the script.sh you can check the console log like below.
stage('My stage') {
options {
timeout(time: 2, unit: "MINUTES")
}
steps {
script {
sh "./script.sh"
// Checking the console log
def consoleLog = Jenkins.getInstance().getItemByFullName(env.JOB_NAME).getBuildByNumber(Integer.parseInt(env.BUILD_NUMBER)).logFile.text
if(consoleLog.contains("Cancelling nested steps due to timeout")) {
echo "Send the Notification"
}
}
}
}
}
Also if you don't want to do this sequentially, you can create a parallel stage and keep on checking the logs.
Related
I have a stage in my pipeline job that requires user input to proceed to the next stage. The problem is sometime I forget to click the proceed button after N minutes of waiting. I want to send a Slack message to notify me that the stage has been paused for N minutes. Is there a way to achieve it?
Below is the sample of my pipeline script:
pipeline {
agent any
stages {
stage('A') {
steps {
echo 'Starting Stage A'
input message: 'Continue to the next stage?'
// send Slack message after 15 minutes user didn't click Proceed/Abort button
// but still wait for user input (don't mark it as failed even after 15 minutes)
}
}
stage('B') {
steps {
echo 'Starting Stage B'
}
}
}
}
I have tried using Jenkins timeout feature using the pipeline script below
But, if the timeout is reached, it will continue to the next stage automatically. What I want is even after the timeout is reached, still wait for user input on Stage A (don't continue to the Stage B immediately)
pipeline {
agent any
stages {
stage('A') {
steps {
script {
try {
echo 'Starting Stage A'
timeout(time: 15, unit: 'MINUTES') {
input message: 'Continue to the next stage?'
}
} catch (err) {
def user = err.getCauses()[0].getUser()
if ('SYSTEM' == user.toString()) { // failed because of timeout
// send Slack message
// how to still wait for user input on this stage even after timeout is reached?
}
}
}
}
}
stage('B') {
steps {
echo 'Starting Stage B'
}
}
}
}
Thanks
I figured it out using the script below:
pipeline {
agent any
stages {
stage('A') {
steps {
script {
echo 'Starting Stage A'
def showInputAgain = false
def proceed = true
try {
timeout(time: 15, unit: 'SECONDS') {
input message: 'Continue?'
}
} catch (err) {
def user = err.getCauses()[0].getUser()
if ('SYSTEM' == user.toString()) {
showInputAgain = true
echo 'Stage A failed. Reach timeout. Sending message to Slack'
} else {
proceed = false
echo 'Stage A failed. User abort'
}
}
if (proceed && showInputAgain) {
input message: 'Continue?'
}
}
}
}
stage('B') {
steps {
echo 'Starting Stage B'
}
}
}
}
But any other inputs/answers will be appreciated
In my Jenkins pipeline, I have 15 stages. Now I have a post function at the end of the Jenkins file to send me an email about whether the whole process is failed or success. I would like to include all the stages that are failed in the email too. Using post in each stage is not a good idea, because I would receive 15 emails each time the job runs.
I am thinking of creating a list and save all failed env.STAGE_NAME in the list and print it at the end? But it would not allow me to do such a thing in the post.
I want to achieve something like:
pipeline {
agent { label 'master'}
stages {
stage('1') {
steps {
echo 'make fail'
}
}
stage('2') {
steps {
sh 'make fail'
}
}
...
stage('15') {
steps {
sh 'make fail'
}
}
}
post {
always {
echo 'ok'
}
failure {
"There are 3 stages have failed the test, which are: '1', '2' '15'"
}
}
}
How would I do it?
I have a pipeline with stages where one of the stages, intermittently takes longer than expected and hence using timeout to abort it. But if the stage is aborted, build also marked as aborted. Following is the code for the pipeline:
pipeline {
agent any
stages {
stage('First') {
options {
timeout(time: 10, unit: 'SECONDS')
}
steps {
script {
catchError(buildResult: 'SUCCESS') {
echo "Executing stage I"
sleep 12
}
}
}
}
stage('Second') {
steps {
script {
echo "Executing stage II"
}
}
}
}
}
Even though the stage is marked as Aborted, I want to mark build as Success. Can you please help how I can achieve this?
I would suggest one improvement to Michael's answer (which is correct btw). You can use catchError to mark stage ABORTED (or UNSTABLE) and mark the build SUCCESS, but you need to wrap the code that may timeout with try-catch block to control the error. Consider the following example:
pipeline {
agent any
stages {
stage('First') {
options {
timeout(time: 3, unit: 'SECONDS')
}
steps {
script {
catchError(buildResult: 'SUCCESS', stageResult: 'ABORTED') {
try {
echo "Executing stage I"
sleep 4
} catch(org.jenkinsci.plugins.workflow.steps.FlowInterruptedException e) {
error "Stage interrupted with ${e.toString()}"
}
}
}
}
}
stage('Second') {
steps {
script {
echo "Executing stage II"
}
}
}
}
}
When you run this pipeline, the stage that timed out is marked as ABORTED, but the pipeline continues and if there is no failure in the remaining stages, it is marked as SUCCESS.
And here is what the UNSTABLE stage status looks like.
Michael's solution works as well, but it produces a slightly different result - the stage that times out is marked as SUCCESS, and this might be less intuitive. You need to click on the stage to check if it timed out or not.
pipeline {
agent any
stages {
stage('First') {
options {
timeout(time: 3, unit: 'SECONDS')
}
steps {
script {
try {
echo "Executing stage I"
sleep 4
} catch(Exception e) {
currentBuild.result = "SUCCESS"
}
}
}
}
stage('Second') {
steps {
script {
echo "Executing stage II"
}
}
}
}
}
Your catchError() won't work in your case. The documantation (Source) tells the following:
buildResult (optional)
If an error is caught, the overall build result
will be set to this value. Note that the build result can only get
worse, so you cannot change the result to SUCCESS if the current
result is UNSTABLE or worse. Use SUCCESS or null to keep the build
result from being set when an error is caught.
The build status is set with currentBuild.currentResult which can have three values: SUCCESS, UNSTABLE, or FAILURE.
If you want to mark the build as SUCCESS on abortion the post-option (Source) aborted can be used:
pipeline {
agent any
stages {
stage('Example') {
steps {
echo 'Hello World'
}
}
}
post {
aborted {
// Executed only if stage is aborted
currentBuild.result = 'SUCCESS'
}
}
}
I have a stage which runs one shell script at remote node. If the script takes very long time to execute, the stage should wait for sometime and should move to next stage without aborting the subsequent stages.
Could you please let me know the required syntax to achieve this.
in a declarative pipeline, add this stage:
stage ("do-and-skip-for-timeout") {
steps {
script {
try {
timeout (time: 10, unit: 'MINUTES') {
echo "do something here, that can take some time" // replace this line with your code
}
}
catch (error) {
def user = error.getCauses()[0].getUser()
if('SYSTEM' == user.toString()) { // SYSTEM means timeout.
echo "Timeout reached, continue to next stage"
}
else {
throw new Exception("[ERROR] stage failed!")
}
}
}
}
My Jenkins pipeline runs several tasks in parallel. It appears that if one stage fails, all subsequent stages will run their failure post block (whether they actually failed or not). I don't know if this is by design or if I'm doing something wrong.
Note: This pipeline runs on a Windows node, hence the bat('exit /b 1')
pipeline {
agent any
stages {
stage('Parallel steps') {
parallel {
stage('Successful stage') {
steps {
script { sleep 10 }
}
post {
failure {
echo('detected failure: Successful stage')
}
}
}
stage('Failure stage') {
steps {
script { bat('exit /b 1') }
}
post {
failure {
echo('detected failure: Failure stage')
}
}
}
}
}
}
}
In the above pipeline, only 'Failure stage' fails, yet in the output I see this, indicating the failure conditional executed for both steps!
Started by user Doe, John
Running on WINDOWS_NODE in D:\workspace
[Successful stage] Sleeping for 10 sec
[Failure stage] [test] Running batch script
[Failure stage] D:\workspace>exit /b 1
Post stage
[Pipeline] [Failure stage] echo
[Failure stage] detected failure: Failure stage
[Failure stage] Failed in branch Failure stage
Post stage
[Pipeline] [Successful stage] echo
[Successful stage] detected failure: Successful stage
ERROR: script returned exit code 1
Finished: FAILURE
What's the best way for me to detect which parallel stage failed and report it to the overall pipeline?
It looks like this is a known bug with Declarative Pipelines. I had to give up using the built-in post->failure block and use try/catch instead, which has its own problems:
You have to catch and then re-throw the error in order to make the stage fail appropriately.
The UI can get a little confusing, as the step that failed is no longer highlighted in red (but the error message is still in the log).
The code is slightly less readable.
This code works correctly. Only the failing stage echoes "detected failure" instead of both.
pipeline {
agent any
stages {
stage('Parallel steps') {
parallel {
stage('Successful stage') {
steps {
script {
try {
sleep 10
} catch (e) {
echo('detected failure: Successful stage')
throw(e)
}
}
}
}
stage('Failure stage') {
steps {
script {
try {
bat('exit /b 1')
} catch (e) {
echo('detected failure: Failure stage')
throw(e)
}
}
}
}
}
}
}
}