I'd like to leverage the existing Mailer plugin from Jenkins within a Jenkinsfile that defines a pipeline build job. Given the following simple failure script I would expect an email on every build.
stage 'Test'
node {
try {
sh 'exit 1'
} finally {
step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: 'me#me.com', sendToIndividuals: true])
}
}
The output from the build is:
Started by user xxxxx
[Pipeline] stage (Test)
Entering stage Test
Proceeding
[Pipeline] node
Running on master in /var/lib/jenkins/jobs/rpk-test/workspace
[Pipeline] {
[Pipeline] sh
[workspace] Running shell script
+ exit 1
[Pipeline] step
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE
As you can see, it does record that it performs the pipeline step immediately after the failure, but no emails get generated.
Emails in other free-style jobs that leverage the mailer work fine, its just invoking via pipeline jobs.
This is running with Jenkins 2.2 and mailer 1.17.
Is there a different mechanism by which I should be invoking failed build emails? I don't need all the overhead of the mail step, just need notifications on failures and recoveries.
In Pipeline failed sh doesn't immediately set the currentBuild.result to FAILURE whereas its initial value is null. Hence, build steps that rely on the build status like Mailer might work seemingly incorrect.
You can check it by adding a debug print:
stage 'Test'
node {
try {
sh 'exit 1'
} finally {
println currentBuild.result // this prints null
step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: 'me#me.com', sendToIndividuals: true])
}
}
This whole pipeline is wrapped with exception handler provided by Jenkins that's why Jenkins marks the build as failed in the the end.
So if you want to utilize Mailer you need to maintain the build status properly. For instance:
stage 'Test'
node {
try {
sh 'exit 1'
currentBuild.result = 'SUCCESS'
} catch (any) {
currentBuild.result = 'FAILURE'
throw any //rethrow exception to prevent the build from proceeding
} finally {
step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: 'me#me.com', sendToIndividuals: true])
}
}
If you don't need to re-throw the exception, you can use catchError. It is a Pipeline built-in which catches any exception within its scope, prints it into console and sets the build status. For instance:
stage 'Test'
node {
catchError {
sh 'exit 1'
}
step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: 'me#me.com', sendToIndividuals: true])
}
In addition to izzekil's excellent answer, you may wish to choose email recipients based on the commit authors. You can use email-ext to do this (based on their pipeline examples):
step([$class: 'Mailer',
notifyEveryUnstableBuild: true,
recipients: emailextrecipients([[$class: 'CulpritsRecipientProvider'],
[$class: 'RequesterRecipientProvider']])])
If you're using a recent email-ext (2.50+), you can use that in your pipeline:
emailext(body: '${DEFAULT_CONTENT}', mimeType: 'text/html',
replyTo: '$DEFAULT_REPLYTO', subject: '${DEFAULT_SUBJECT}',
to: emailextrecipients([[$class: 'CulpritsRecipientProvider'],
[$class: 'RequesterRecipientProvider']]))
If you're not using a declarative Jenkinsfile, you will need to put checkout scm so Jenkins can find the committers. See JENKINS-46431.
If you're still on an older version of email-ext, you'll hit JENKINS-25267. You could roll your own HTML email:
def emailNotification() {
def to = emailextrecipients([[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider']])
String currentResult = currentBuild.result
String previousResult = currentBuild.getPreviousBuild().result
def causes = currentBuild.rawBuild.getCauses()
// E.g. 'started by user', 'triggered by scm change'
def cause = null
if (!causes.isEmpty()) {
cause = causes[0].getShortDescription()
}
// Ensure we don't keep a list of causes, or we get
// "java.io.NotSerializableException: hudson.model.Cause$UserIdCause"
// see http://stackoverflow.com/a/37897833/509706
causes = null
String subject = "$env.JOB_NAME $env.BUILD_NUMBER: $currentResult"
String body = """
<p>Build $env.BUILD_NUMBER ran on $env.NODE_NAME and terminated with $currentResult.
</p>
<p>Build trigger: $cause</p>
<p>See: $env.BUILD_URL</p>
"""
String log = currentBuild.rawBuild.getLog(40).join('\n')
if (currentBuild != 'SUCCESS') {
body = body + """
<h2>Last lines of output</h2>
<pre>$log</pre>
"""
}
if (to != null && !to.isEmpty()) {
// Email on any failures, and on first success.
if (currentResult != 'SUCCESS' || currentResult != previousResult) {
mail to: to, subject: subject, body: body, mimeType: "text/html"
}
echo 'Sent email notification'
}
}
I think a better way to send mail notifications in jenkins pipelines is to use the post section of a pipeline as described in the jenkins docs instead of using try catch:
pipeline {
agent any
stages {
stage('whatever') {
steps {
...
}
}
}
post {
always {
step([$class: 'Mailer',
notifyEveryUnstableBuild: true,
recipients: "example#example.com",
sendToIndividuals: true])
}
}
}
}
}
Related
I have a pipeline with some information detailed behind
pipeline {
parameters {
booleanParam(name: 'RERUN', defaultValue: false, description: 'Run Failed Tests')
}
stage('Run tests ') {
steps {
runTest()
}
}
post {
always {
reRun()
}
}
}
def reRun() {
if ("SUCCESS".equals(currentBuild.result)) {
echo "LAST BUILD WAS SUCCESS"
} else if ("UNSTABLE".equals(currentBuild.result)) {
echo "LAST BUILD WAS UNSTABLE"
}
}
but I want that after the stage "Run tests" execute, if some tests fail I want to re-run the pipeline with parameters RERUN true instead of false. How can I replay via script instead of using plugins ?
I wasn't able to find how to re-run using parameters on my search, if someone could help me I will be grateful.
First of you can use the post step to determine if the job was unstable:
post{
unstable{
echo "..."
}
}
Then you could just trigger the same job with the new parameter like this:
build job: 'your-project-name', parameters: [[$class: 'BooleanParameterValue', name: 'RERUN', value: Boolean.valueOf("true")]]
Within Jenkins, I would like to parse the ansible playbook "Play Recap" output section for the failing hostname(s). I want to put the information into an email or other notification. This could also be used to fire off another Jenkins job.
I'm currently submitting an ansible-playbook as a jenkins job to deploy software across a number of systems. I'm using a Jenkins Pipeline script, which was necessary to implement for sshagent to be applied correctly.
pipeline {
agent any
options {
ansiColor('xterm')
}
stages {
stage("setup environment") {
steps {
deleteDir()
} //steps
} //stage - setup environment
stage("clone the repo") {
environment {
GIT_SSH_COMMAND = "ssh -o StrictHostKeyChecking=no"
} //environment
steps {
sshagent(['my_git']) {
sh "git clone ssh://git#github.com/~usr/ansible.git"
} //sshagent
} //steps
} //stage - clone the repo
stage("run ansible playbook") {
steps {
sshagent (credentials: ['apps']) {
withEnv(['ANSIBLE_CONFIG=ansible.cfg']) {
dir('ansible') {
ansiblePlaybook(
becomeUser: null,
colorized: true,
credentialsId: 'apps',
disableHostKeyChecking: true,
forks: 50,
hostKeyChecking: false,
inventory: 'hosts',
limit: 'production:&*generic',
playbook: 'demo_play.yml',
sudoUser: null,
extras: '-vvvvv'
) //ansiblePlaybook
} //dir
} //withEnv
} //sshagent
} //steps
} //stage - run ansible playbook
} //stages
post {
failure {
emailext body: "Please go to ${env.BUILD_URL}/consoleText for more details.",
recipientProviders: [[$class: 'DevelopersRecipientProvider'], [$class: 'RequesterRecipientProvider']],
subject: "${env.JOB_NAME}",
to: 'our.dev.team#gmail.com',
attachLog: true
office365ConnectorSend message:"A production system appears to be unreachable.",
status:"Failed",
color:"f00000",
factDefinitions: [[name: "Credentials ID", template: "apps"],
[name: "Build Duration", template: "${currentBuild.durationString}"],
[name: "Full Name", template: "${currentBuild.fullDisplayName}"]],
webhookUrl:'https://outlook.office.com/webhook/[really long alphanumeric key]/IncomingWebhook/[another super-long alphanumeric key]'
} //failure
} //post
} //pipeline
There are several Jenkins plug-ins for parsing the console output, but none will let me capture and utilize text. I have looked at log-parser and text finder.
The only lead I have is using groovy to script this.
https://devops.stackexchange.com/questions/5363/jenkins-groovy-to-parse-console-output-and-mark-build-failure
An example of "Play Recap" within the console output is:
PLAY RECAP **************************************************************************************************************************************************
some.host.name : ok=25 changed=2 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0
some.ip.address : ok=22 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
I am trying to get either a list or a delimited string of each host that is failing. Although, in the case of a list, I need to figure out how to send multiple notifications.
If anyone could help me with the full solution, I would very much appreciate your help.
Q: "Parse the ansible playbook 'Play Recap' output section."
A: Use json callback and parse the output with jq. For example
shell> ANSIBLE_STDOUT_CALLBACK=json ansible-playbook pb.yml | jq .stats
There are a few 'gotchas' that I came across as I solved this problem.
The only successful way I could access the output of the ansible plugin was through pulling the raw log file. def log = currentBuild.rawBuild.getLog(100) In this case I only pulled the last 100 lines, as I'm only looking for the Play Recap box. This method requires special permissions. The console log will display the error and provide a link where the functions can be allowed.
The ansible output should not be colorized. colorized: false Colorized output is quite difficult to parse. The 'console log' doesn't show you the colorized markup, however if you look at the 'consoleText' you will see it.
When using regex, you will most likely have a matcher object which is non-serializable. To use this in Jenkins, it may need to be placed in a function tagged #NonCPS which stops Jenkins from trying to serialize the object. I had mixed results with needing this, so I don't exhaustively understand where it's required.
The regex statement was one of the harder parts for me. I came up with a generic statement that could be easily modified for different scenarios e.g. failed or unreachable. I also had more luck using the 'slashy-style' regex in groovy which places a forward slash on either end of the statement with no need for quotes of any kind. You'll note the 'failed' portion is different failed=([1-9]|[1-9][0-9]), so that it only matches a statement where the failure is non-zero.
/([0-9a-zA-Z\.\-]+)(?=[ ]*:[ ]*ok=([0-9]|[1-9][0-9])[ ]*changed=([0-9]|[1-9][0-9])[ ]*unreachable=([0-9]|[1-9][0-9])[ ]*failed=([1-9]|[1-9][0-9]))/
Here's the full pipeline code that I came up with.
pipeline {
agent any
options {
ansiColor('xterm')
}
stages {
stage("setup environment") {
steps {
deleteDir()
} //steps
} //stage - setup environment
stage("clone the repo") {
environment {
GIT_SSH_COMMAND = "ssh -o StrictHostKeyChecking=no"
} //environment
steps {
sshagent(['my_git']) {
sh "git clone ssh://git#github.com/~usr/ansible.git"
} //sshagent
} //steps
} //stage - clone the repo
stage("run ansible playbook") {
steps {
sshagent (credentials: ['apps']) {
withEnv(['ANSIBLE_CONFIG=ansible.cfg']) {
dir('ansible') {
ansiblePlaybook(
becomeUser: null,
colorized: false,
credentialsId: 'apps',
disableHostKeyChecking: true,
forks: 50,
hostKeyChecking: false,
inventory: 'hosts',
limit: 'production:&*generic',
playbook: 'demo_play.yml',
sudoUser: null,
extras: '-vvvvv'
) //ansiblePlaybook
} //dir
} //withEnv
} //sshagent
} //steps
} //stage - run ansible playbook
} //stages
post {
failure {
script {
problem_hosts = get_the_hostnames()
}
emailext body: "${problem_hosts} has failed. Please go to ${env.BUILD_URL}/consoleText for more details.",
recipientProviders: [[$class: 'DevelopersRecipientProvider'], [$class: 'RequesterRecipientProvider']],
subject: "${env.JOB_NAME}",
to: 'our.dev.team#gmail.com',
attachLog: true
office365ConnectorSend message:"${problem_hosts} has failed.",
status:"Failed",
color:"f00000",
factDefinitions: [[name: "Credentials ID", template: "apps"],
[name: "Build Duration", template: "${currentBuild.durationString}"],
[name: "Full Name", template: "${currentBuild.fullDisplayName}"]],
webhookUrl:'https://outlook.office.com/webhook/[really long alphanumeric key]/IncomingWebhook/[another super-long alphanumeric key]'
} //failure
} //post
} //pipeline
//#NonCPS
def get_the_hostnames() {
// Get the last 100 lines of the log
def log = currentBuild.rawBuild.getLog(100)
print log
// GREP the log for the failed hostnames
def matches = log =~ /([0-9a-zA-Z\.\-]+)(?=[ ]*:[ ]*ok=([0-9]|[1-9][0-9])[ ]*changed=([0-9]|[1-9][0-9])[ ]*unreachable=([0-9]|[1-9][0-9])[ ]*failed=([1-9]|[1-9][0-9]))/
def hostnames = null
// if any matches occurred
if (matches) {
// iterate over the matches
for (int i = 0; i < matches.size(); i++) {
// if there is a name, concatenate it
// else populate it
if (hostnames?.trim()) {
hostnames = hostnames + " " + matches[i]
} else {
hostnames = matches[i][0]
} // if/else
} // for
} // if
if (!hostnames?.trim()) {
hostnames = "No hostnames identified."
}
return hostnames
}
I'm trying to send an email when a Pipeline build on Jenkins fails. An example can be found here: https://github.com/jenkinsci/pipeline-examples/blob/master/jenkinsfile-examples/nodejs-build-test-deploy-docker-notify/Jenkinsfile
My concrete groovy script looks as follows:
#!groovy
node('') {
def env = ["JAVA_HOME=${tool 'jdk1.8.0_131'}", "PATH+MAVEN=${tool 'maven_3.1.1'}/bin:${env.JAVA_HOME}/bin", "PATH+GRADLE=${tool 'gradle_4.1'}/bin:${env.JAVA_HOME}/bin" ]
def err = null
currentBuild.result = "SUCCESS"
try {
stage('errorStage') {
dir('error') {
git url: "unknown", branch: "master"
withEnv(env) {
sh "mvn -Pjenkins-build clean deploy"
}
}
}
} catch (caughtError) {
println "caught error :" + caughtError
err = caughtError
currentBuild.result = "FAILURE"
mail (body:
"Pipeline error: ${err}\nFix me.",
from: 'jenkins#x.com',
subject: 'Pipeline build failed',
to: 'recipient#x.com')
} finally {
/* Must re-throw exception to propagate error */
if (err) {
throw err
}
}
}
Actually, nothing ever happens, although the exception is being caught correctly and the build fails. Is there anything required to be able to use mail?! Maybe another Jenkins plugin or something?
I was able to fix it myself by using emailext plugin on Jenkins instead of mail. The code looks as follows now:
emailext body: "Pipeline error: ${err}\nPlease go to ${BUILD_URL} and verify the build",
recipientProviders: [[$class: 'DevelopersRecipientProvider'], [$class: 'RequesterRecipientProvider']],
subject: "'${JOB_NAME}' (${BUILD_NUMBER}) failed",
to: '...'
I am looking to send out an email on failure and/or list of all of the stages and their status. I have wrapped my code in a try{} and am catching any failures. On my Console output it says that the email was sent to the appropriate email but I am not receiving it... Under my Configuration on Jenkins I have set up my Jenkins Location, E-mail Notification and Extended E-mail Notifications set up. I have confirmed that the E-mail Notifications work properly and added the same credentials to the Extended E-mail Notifications. Here is my console out put:
Running on Dobby in D:Pipelinefolder
[Pipeline] {
[Pipeline] emailext
Sending email to: myemail#school.edu
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] checkout
and
Results (nunit3) saved as TestResult.xml
[Pipeline] }
[Pipeline] // stage
[Pipeline] emailext
Sending email to: myemail#school.edu
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Please view my code below:
node('Dobby') {
try {
notifyBuild('STARTED')
stage('Checkout') {
checkout([$class: 'SubversionSCM',
additionalCredentials: [],
excludedCommitMessages: '',
excludedRegions: '',
excludedRevprop: '',
excludedUsers: 'buildbot',
filterChangelog: false,
ignoreDirPropChanges: false,
includedRegions: '',
locations: [[credentialsId: 'aPpr0pR1at3.CrDEnt1Al5.83',
depthOption: 'infinity',
ignoreExternalsOption: true,
local: '.',
remote: "http://jenkins.svn.link.edu/svn/my/repo"]],
workspaceUpdater: [$class: 'UpdateUpdater']])
}
stage('Build webApp') {
bat 'C:/"Program Files (x86)/Microsoft Visual Studio"/2017/Community/MSBuild/15.0/Bin/MSBuild.exe webApp/webApp.sln /m /p:VisualStudioVersion=15.0' //msbuild
}
stage('Test: Check if IIS webApp ON') {
bat 'C:/"Program Files (x86)"/NUnit.org/nunit-console/nunit3-console.exe SeleniumNunit/SeleniumNunit/bin/Debug/SeleniumNunit.dll'
}
} catch (e) {
// If there was an exception thrown, the build failed
currentBuild.result = "FAILED"
throw e
} finally {
// Success or failure, always send notifications
notifyBuild(currentBuild.result)
}
}
def notifyBuild(String buildStatus = 'STARTED') {
// build status of null means successful
buildStatus = buildStatus ?: 'SUCCESSFUL'
emailext(
to: 'myemail#school.edu',
subject: "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'",
body: "details",
recipientProviders: [[$class: 'DevelopersRecipientProvider']]
)
}
Can you identify any reasons why I am not receiving emails?
So... I unchecked Use SMTP Authentication under Extended E-mail Notifications and it worked! I need to wrap my head around this now...
When using the Jenkins pipeline where each stage runs on a different agent, it is good practice to use agent none at the beginning:
pipeline {
agent none
stages {
stage('Checkout') {
agent { label 'master' }
steps { script { currentBuild.result = 'SUCCESS' } }
}
stage('Build') {
agent { label 'someagent' }
steps { bat "exit 1" }
}
}
post {
always {
step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: "test#test.com", sendToIndividuals: true])
}
}
}
But doing this leads to Required context class hudson.FilePath is missing error message when the email should go out:
[Pipeline] { (Declarative: Post Actions)
[Pipeline] step
Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node
[Pipeline] error
[Pipeline] }
When I change from agent none to agent any, it works fine.
How can I get the post step to work without using agent any?
wrap the step that does the mailing in a node step:
post {
always {
node('awesome_node_label') {
step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: "test#test.com", sendToIndividuals: true])
}
}
}
I know this is old but I stumbled on this looking for something related. If you want to run the post step on any node, you can use
post {
always {
node(null) {
step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: "test#test.com", sendToIndividuals: true])
}
}
}
https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#node-allocate-node Says that the label may be left blank. Many times in a declarative pipeline if something is left blank this results in an error. To work around this, setting it to null will often work.