Docker build unexpected EOF in Jenkins pipeline - docker

I am sure I am not the only one who is interested in how to handle something like this: docker build stage in Jenkins pipeline fails with Unexpected EOF (there can be a lot of reasons, in my case the docker daemon was restarted on the slave)
appImage = docker.build ("${projectName}:${env.BRANCH_NAME}-${gitCommit}", "--build-arg APP_ENV=${appEnv} --build-arg SKIP_LINT=true .")
The deploy phase kicks in, because the Unexpected EOF does not actually throw any error, there is no exception to catch so the build status is null.
I know that it's not a regular situation but still how can we handle smth like this so that the following stages do not run in case the build is interrupted.
Additional details:
#JRichardsz , thanks for the answer! Usually currentBuild.result . defaults to null e.g. https://issues.jenkins-ci.org/browse/JENKINS-46325 so unless you set it to success explicitly upon successful stage's execution , it will be null. But all in all the same can be achieved with try catch like :
if (deployableBranches.contains(env.BRANCH_NAME)) {
try {
stage('Build image') {
ansiColor('xterm') {
appImage = docker.build
("${projectName}:${env.BRANCH_NAME}-${gitCommit}", "--build-arg
SKIP_LINT=true .")
}
}
stage('Push image') {
docker.withRegistry("${registryUrl}", "${dockerCredsId}") {
appImage.push()
appImage.push "${env.BRANCH_NAME}-latest"
}
}
stage('Deploy') {
build job: 'kubernetes-deploy', parameters: [
/////
]
}
} catch (e) {
// A shell step returns with a nonzero exit code
// When pipeline is in a shell step, and a user presses abort
if (e.getMessage().contains('script returned exit code 143')) {
currentBuild.result = "ABORTED"
} else {
currentBuild.result = "FAILED"
}
throw e
} finally {
// Success or failure or abort, always send notifications
stage('Send deployment status') {
helpers.sendDeploymentStatus(projectName, currentBuild.result,
helpers.getCommitHashShort())
}
}
}
But the issue is that stage('Build image') may exit without any error code like it was in my case.

I had a similar requirement : "If some rule is executed in stage A, following stages must not run"
This worked for me :
def flag;
node {
stage('A') {
flag = 1;
}
stage('B') {
// exit this stage if flag == 1
if(flag == 1){
return;
}
//start of stage B tasks
...
}
}
Also you could use some jenkins variable like currentBuild.result instead of flag like this :
node {
stage('A') {
//stage A tasks
//this stage could modify currentBuild.result variable
}
stage('B') {
// exit this stage if currentBuild.result is null , empty, "FAILURE", etc
if(currentBuild.result == null ||
currentBuild.result == "" ||
currentBuild.result=="FAILURE" ){
return;
}
//stage B tasks
}
}

Related

conditional execution in Jenkins nested stages pipelines

I've seen Conditional execution based on result of previous stage in declarative pipeline and I got a flavour of the question: how the things would look if we got a simplified workflow like
A-BC-{B1-B2-B3}-D in sequential mode?
What I'm trying to achieve is to have stage B2 executed only if B1 succeeds and B3 should be executed on condition B1 fails and finally proceed with D.
Thanks for the input.
The following did work for me - define an extra map stageResultMap.ObjectdircreationFailed = true in the exception phase and call it in the stage which should be invoked upon failure. Example below.
stage ('Build'){
stages {
stage ('Objectdircreation'){
steps {
script {
try {
sh'''#!/usr/bin/env bash
PATH=/sdev/user/bin:/opt/rational/clearcase/bin:$PATH
export PATH
mkdir -p $OBJECTDIR
'''
stageResultMap.didObjectdircreationSucceed = true
}
catch (Exception e) {
unstable("${STAGE_NAME} failed!")
currentBuild.result = 'FAILURE'
stageResultMap.didObjectdircreationSucceed = false
//set an extra map to true and test it
stageResultMap.ObjectdircreationFailed = true
}
}
}
} // end of Objectdircreation stage
stage ('Inject objects under an objdir'){
//
// execute only if Objectdircreation succeeded
//
when {
expression {
return stageResultMap.find{ it.key == "didObjectdircreationcreationSucceed" }?.value
}
}
steps {
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE')
{
sh """#!/usr/bin/env bash
make ${ARGS} $OBJDIRPREFIX=$OBJDIR build
"""
}
}
} // end of build under an object tree
stage ('Inject build under current objdir') {
//
// execute only if object dir creation failed
//
when {
expression {
return stageResultMap.find{ it.key == "didProjecttreecreationFailed" }?.value
}
}
steps {
catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh """#!/usr/bin/env bash
make ${ARGS} build
"""
}
}
} // end of build under root
} // end of internal build stages
//
} // end of build nested stages
} // end of main stages section
//
//
} //pipeline end

Setting the build status to failed when failingFast

In a declarative jenkins pipeline, when failFast is used within a set of parallel stages. How do you set the build status to 'FAILED' instead of 'ABORTED'?
Assuming something fails, this will exit with 'ABORTED'
pipeline {
agent any
stages {
stage('Parallel') {
failFast true
parallel {
stage('Branch A') {
steps {
sh "foo"
}
}
stage('Branch B') {
steps {
sh "bar"
}
}
stage('Branch C') {
steps {
sh "baz"
}
}
}
}
}
}
If FailFast is removed the build will eventually fail with 'FAILED'. But I need the FailFast behaviour as well.
OK, after a bit of investigation this appears to an issue with Jenkins: https://issues.jenkins-ci.org/browse/JENKINS-55459
I was able to get by this by adding a script with a try/catch to those steps within the parallel stages.
steps {
script {
try {
MyCode()
}
catch (Exception err) {
if (currentBuild.result == null) {
error "The stage '${env.STAGE_NAME}' has failed."
} else {
echo "Exiting stage early."
}
}
}
}

How to exit from the Jenkins pipeline if a stage sets build fail/unstable status?

I have a declarative Jenkins pipeline with stage1, stage2, stage3 and so on. I want to stop stage2 from running if stage1 sets the build unstable/fail.
I know I can stop the steps in stage1 from running using return when the build is not success but couldn't find a way where I can just exit the pipeline without running the stages below stage1
Here is what I have:
stage('stage1') {
steps {
script{
//somesteps
if ("${stdout}" == "1"){
currentBuild.result = 'UNSTABLE'
return
} //if
//somesteps
} //script
} //steps
} //stage
// run only when stage1 is success
stage('stage2'){
when {
expression {
params.name ==~ /x|y/
}
}
steps {
script{
//stage2 steps
}
}
}
If params.name ==~ /z/ stage 3 will be executed skippping stage2
Note: I cannot include the steps in stage2/3/.. in stage1. It should be that way. Based on the build paramters stage2/3/4... will be called after stage1
The easiest way to skip remaining pipeline stages is to set up a variable which will control if following stages should be skipped or not. Something like this:
def skipRemainingStages = false
pipeline {
agent any
stages {
stage("Stage 1") {
steps {
script {
skipRemainingStages = true
println "skipRemainingStages = ${skipRemainingStages}"
}
}
}
stage("Stage 2") {
when {
expression {
!skipRemainingStages
}
}
steps {
script {
println "This text wont show up...."
}
}
}
stage("Stage 3") {
when {
expression {
!skipRemainingStages
}
}
steps {
script {
println "This text wont show up...."
}
}
}
}
}
This is very simple example that sets skipRemainingStages to true at Stage 1 and Stage 2 and Stage 3 get skipped because expression in the when block does not evaluates to true.
Alternatively you can call error(String message) step to stop the pipeline and set its status to FAILED. For example, if your stage 1 calls error(msg) step like:
stage("Stage 1") {
steps {
script {
error "This pipeline stops here!"
}
}
}
In this case pipeline stops whenever error(msg) step is found and all remaining stages are ignored (when blocks are not even checked).
Of course you can call error(msg) depending on some condition to make it FAILED only if specific conditions are met.
You can use post in a stage to exit as follows:
pipeline {
stages {
stage('stage 1') {
steps {
//step 1
}
}
stage('stage 2') {
steps {
script{
//step 2
}
}
post{
success {
}
failure {
script{
sh "exit 1"
//or
error "Failed, exiting now..."
}
}
aborted {
}
unstable {
script{
sh "exit 1"
//or
error "Unstable, exiting now..."
}
}
}
}
}
}
This will abort the build and job wouldn't run further.
You can also simply throw an Exception. That will abort the build.
In fact simply setting the build status in a catch clause works pretty well.
You can also then add custom logic in the finally block for sending notifications for build status changes (email, Slack message etc)
So perhaps something like the following. NOTE: I have copied some of this from an existing Jenkinsfile. So not 100% sure this is the same syntax as you were using:
pipeline {
try {
stages {
stage("stage1") {
if (something) {
throw new RuntimeException("Something went wrong")
}
}
stage("stage2") {
}
}
} catch (e) {
currentBuild.result = "FAILED"
throw e
}
}
You can try:
stage('Set skipRemainingStages variable which decides, whether to run next stages or not') {
steps {
script {
skipRemainingStages = true
try {
println("In if block")
skipRemainingStages = true
}
catch (Exception exc) {
println("Exception block: ${exc}")
skipRemainingStages = false
}
if (skipRemainingStages) {
currentBuild.result = 'FAILURE'
error("Stopping early!")
}
}
}
}
stage('This will not execute if skipRemainingStages=true')
{.
.
.
}
You can use mark the build as failed and then use sh "exit 1" to interrupt its execution in your Jenkins pipelines like below:
pipeline {
stages {
stage('stage 1') {
steps {
}
}
stage('stage 2') {
steps {
script{
if (something) {
currentBuild.result = "FAILURE"
sh "exit 1"
}
}
}
}
}
}

Abort current build from pipeline in Jenkins

I have a Jenkins pipeline which has multiple stages, for example:
node("nodename") {
stage("Checkout") {
git ....
}
stage("Check Preconditions") {
...
if(!continueBuild) {
// What do I put here? currentBuild.xxx ?
}
}
stage("Do a lot of work") {
....
}
}
I want to be able to cancel (not fail) the build if certain preconditions are not met and there is no actual work to be done. How can I do this? I know the currentBuild variable is available, but I can't find the documentation for it.
You can mark the build as ABORTED, and then use the error step to cause the build to stop:
if (!continueBuild) {
currentBuild.result = 'ABORTED'
error('Stopping early…')
}
In the Stage View, this will show that the build stopped at this stage, but the build overall will be marked as aborted, rather than failed (see the grey icon for build #9):
After some testing I came up with the following solution:
def autoCancelled = false
try {
stage('checkout') {
...
if (your condition) {
autoCancelled = true
error('Aborting the build to prevent a loop.')
}
}
} catch (e) {
if (autoCancelled) {
currentBuild.result = 'ABORTED'
echo('Skipping mail notification')
// return here instead of throwing error to keep the build "green"
return
}
// normal error handling
throw e
}
This will result into following stage view:
failed stage
If you don't like the failed stage, you have to use return. But be aware you have to skip each stage or wrapper.
def autoCancelled = false
try {
stage('checkout') {
...
if (your condition) {
autoCancelled = true
return
}
}
if (autoCancelled) {
error('Aborting the build to prevent a loop.')
// return would be also possible but you have to be sure to quit all stages and wrapper properly
// return
}
} catch (e) {
if (autoCancelled) {
currentBuild.result = 'ABORTED'
echo('Skipping mail notification')
// return here instead of throwing error to keep the build "green"
return
}
// normal error handling
throw e
}
The result:
custom error as indicator
You can also use a custom message instead of a local variable:
final autoCancelledError = 'autoCancelled'
try {
stage('checkout') {
...
if (your condition) {
echo('Aborting the build to prevent a loop.')
error(autoCancelledError)
}
}
} catch (e) {
if (e.message == autoCancelledError) {
currentBuild.result = 'ABORTED'
echo('Skipping mail notification')
// return here instead of throwing error to keep the build "green"
return
}
// normal error handling
throw e
}
Following this documentation from Jenkins, you should be able to generate an error to stop the build and set the build result like this:
currentBuild.result = 'ABORTED'
Hope that helps.
The thing that we use is:
try {
input 'Do you want to abort?'
} catch (Exception err) {
currentBuild.result = 'ABORTED';
return;
}
The "return" at the end makes sure that no further code is executed.
I handled in a declarative way as shown below:
Based on catchError block it will execute post block.
If post result falls under failure category, the error block will be executed to stop upcoming stages like Production, PreProd etc.
pipeline {
agent any
stages {
stage('Build') {
steps {
catchError {
sh '/bin/bash path/To/Filename.sh'
}
}
post {
success {
echo 'Build stage successful'
}
failure {
echo 'Compile stage failed'
error('Build is aborted due to failure of build stage')
}
}
}
stage('Production') {
steps {
sh '/bin/bash path/To/Filename.sh'
}
}
}
}
Inspired by all the answers I have put all the stuff together into one Scripted Pipeline. Keep in mind this is not a Declarative Pipeline.
To get this example working you will need:
QuickFIX form this answer Jenkins CI Pipeline Scripts not permitted to use method groovy.lang.GroovyObject
discord notifier plugin - https://plugins.jenkins.io/discord-notifier/
Discord channel webhook url filled in the code
The idea I had was to abort the pipeline if it is "replayed" vs started by "run button"(in branches tab of Jenskins BlueOcean):
def isBuildAReplay() {
// https://stackoverflow.com/questions/51555910/how-to-know-inside-jenkinsfile-script-that-current-build-is-an-replay/52302879#52302879
def replyClassName = "org.jenkinsci.plugins.workflow.cps.replay.ReplayCause"
currentBuild.rawBuild.getCauses().any{ cause -> cause.toString().contains(replyClassName) }
}
node {
try {
stage('check replay') {
if (isBuildAReplay()) {
currentBuild.result = 'ABORTED'
error 'Biuld REPLAYED going to EXIT (please use RUN button)'
} else {
echo 'NOT replay'
}
}
stage('simple stage') {
echo 'hello from simple stage'
}
stage('error stage') {
//error 'hello from simple error'
}
stage('unstable stage') {
unstable 'hello from simple unstable'
}
stage('Notify sucess') {
//Handle SUCCESS|UNSTABLE
discordSend(description: "${currentBuild.currentResult}: Job ${env.JOB_NAME} \nBuild: ${env.BUILD_NUMBER} \nMore info at: \n${env.BUILD_URL}", footer: 'No-Code', unstable: true, link: env.BUILD_URL, result: "${currentBuild.currentResult}", title: "${JOB_NAME} << CLICK", webhookURL: 'https://discordapp.com/api/webhooks/')
}
} catch (e) {
echo 'This will run only if failed'
if(currentBuild.result == 'ABORTED'){
//Handle ABORTED
discordSend(description: "${currentBuild.currentResult}: Job ${env.JOB_NAME} \nBuild: ${env.BUILD_NUMBER} \nMore info at: \n${env.BUILD_URL}\n\nERROR.toString():\n"+e.toString()+"\nERROR.printStackTrace():\n"+e.printStackTrace()+" ", footer: 'No-Code', unstable: true, link: env.BUILD_URL, result: "ABORTED", title: "${JOB_NAME} << CLICK", webhookURL: 'https://discordapp.com/api/webhooks/')
throw e
}else{
//Handle FAILURE
discordSend(description: "${currentBuild.currentResult}: Job ${env.JOB_NAME} \nBuild: ${env.BUILD_NUMBER} \nMore info at: \n${env.BUILD_URL}\n\nERROR.toString():\n"+e.toString()+"\nERROR.printStackTrace():\n"+e.printStackTrace()+" ", footer: 'No-Code', link: env.BUILD_URL, result: "FAILURE", title: "${JOB_NAME} << CLICK", webhookURL: 'https://discordapp.com/api/webhooks/')
throw e
}
} finally {
echo 'I will always say Hello again!'
}
}
Main trick was the order of lines to achive abort state:
currentBuild.result = 'ABORTED'
error 'Biuld REPLAYED going to EXIT (please use RUN button)'
First set the state then throw an exception.
In the catch block both work:
currentBuild.result
currentBuild.currentResult
If you're able to approve the constructor for FlowInterruptedException, then you can do the following:
throw new FlowInterruptedException(Result.ABORTED, new UserInterruption(getCurrentUserId()))
You can add to your shared library repo a file var/abortError.groovy:
import org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
import jenkins.model.CauseOfInterruption.UserInterruption
def call(message)
{
currentBuild.displayName = "#${env.BUILD_NUMBER} $message"
echo message
currentBuild.result = 'ABORTED'
throw new FlowInterruptedException(Result.ABORTED, new UserInterruption(env.BUILD_USER_ID))
}
Then you can use it this way (after importing library):
abortError("some message")
Note that if you se following error in console logs:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new org.jenkinsci.plugins.workflow.steps.FlowInterruptedException hudson.model.Result jenkins.model.CauseOfInterruption[]
You need follow the link form log and approve security exception.
You can go to the script console of Jenkins and run the following to abort a hung / any Jenkins job build/run:
Jenkins .instance.getItemByFullName("JobName")
.getBuildByNumber(JobNumber)
.finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"));
This is the way to abort the currently running build pipeline in Jenkins UI(in Build History there is a cancel button), for capture:
The Executor.interrupt(Result) method is the cleanest, most direct way I could find to stop a build prematurely and choose the result.
script {
currentBuild.getRawBuild().getExecutor().interrupt(Result.ABORTED)
sleep(1) // Interrupt is not blocking and does not take effect immediately.
}
Pros:
Works in a declarative pipeline just as well as a scripted one.
No try/catch or exceptions to handle.
Marks the calling stage and any successive stages as green/passing in the UI.
Cons:
Requires a number of in-process script approvals, including one that is considered insecure. Approve and use with caution.
Taken from my answer on devops.stackexchange.com.
As for currentBuild, have a look at the docs for the RunWrapper class.

Show a Jenkins pipeline stage as failed without failing the whole job

Here's the code I'm playing with
node {
stage 'build'
echo 'build'
stage 'tests'
echo 'tests'
stage 'end-to-end-tests'
def e2e = build job:'end-to-end-tests', propagate: false
result = e2e.result
if (result.equals("SUCCESS")) {
stage 'deploy'
build 'deploy'
} else {
?????? I want to just fail this stage
}
}
Is there any way for me to mark the 'end-to-end-tests' stage as failed without failing the whole job? Propagate false just always marks the stage as true, which is not what I want, but Propagate true marks the job as failed which I also don't want.
This is now possible, even with declarative pipelines:
pipeline {
agent any
stages {
stage('1') {
steps {
sh 'exit 0'
}
}
stage('2') {
steps {
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
sh "exit 1"
}
}
}
stage('3') {
steps {
sh 'exit 0'
}
}
}
}
In the example above, all stages will execute, the pipeline will be successful, but stage 2 will show as failed:
As you might have guessed, you can freely choose the buildResult and stageResult, in case you want it to be unstable or anything else. You can even fail the build and continue the execution of the pipeline.
Just make sure your Jenkins is up to date, since this is a fairly new feature.
Stage takes a block now, so wrap the stage in try-catch. Try-catch inside the stage makes it succeed.
The new feature mentioned earlier will be more powerful. In the meantime:
try {
stage('end-to-end-tests') {
node {
def e2e = build job:'end-to-end-tests', propagate: false
result = e2e.result
if (result.equals("SUCCESS")) {
} else {
sh "exit 1" // this fails the stage
}
}
}
} catch (e) {
result = "FAIL" // make sure other exceptions are recorded as failure too
}
stage('deploy') {
if (result.equals("SUCCESS")) {
build 'deploy'
} else {
echo "Cannot deploy without successful build" // it is important to have a deploy stage even here for the current visualization
}
}
Sounds like JENKINS-26522. Currently the best you can do is set an overall result:
if (result.equals("SUCCESS")) {
stage 'deploy'
build 'deploy'
} else {
currentBuild.result = e2e.result
// but continue
}
I recently tried to use vaza's answer
Show a Jenkins pipeline stage as failed without failing the whole job as template for writing a function that excutes a job in an own stage named like the job name. Surprisingly it worked, but maybe some groovy experts have a look at it :)
Here is how it looks like if one of the jobs is aborted:
def BuildJob(projectName) {
try {
stage(projectName) {
node {
def e2e = build job:projectName, propagate: false
result = e2e.result
if (result.equals("SUCCESS")) {
} else {
error 'FAIL' //sh "exit 1" // this fails the stage
}
}
}
} catch (e) {
currentBuild.result = 'UNSTABLE'
result = "FAIL" // make sure other exceptions are recorded as failure too
}
}
node {
BuildJob('job1')
BuildJob('job2')
}
In order to show a successful build with a failed stage when a downstream job fails AND support a user being able to cancel a build (including all subsequent stages), I had to use a combination of various solutions, specifically when, try/catch, throw and catchError().
env.GLOBAL_BUILD_ABORTED = false // Set if the user aborts the build
pipeline {
agent any
stages {
stage('First Stage') {
when { expression { env.GLOBAL_BUILD_ABORTED.toBoolean() == false } }
steps {
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
myLocalBuildMethod('Stage #1, build #1')
myLocalBuildMethod('Stage #1, build #2')
}
}
}
stage('Second Stage') {
when { expression { env.GLOBAL_BUILD_ABORTED.toBoolean() == false } }
steps {
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
myLocalBuildMethod('Stage #2, build #1')
myLocalBuildMethod('Stage #2, build #2')
myLocalBuildMethod('Stage #2, build #3')
}
}
}
}
}
def myLocalBuildMethod(myString) {
/* Dummy method to show User Aborts vs Build Failures */
echo "My Local Build Method: " + myString
try {
build (
job: "Dummy_Downstream_Job"
)
} catch (e) {
/* Build Aborted by user - Stop All Test Executions */
if (e.getMessage().contains("was cancelled") || e.getMessage().contains("ABORTED")) {
env.GLOBAL_BUILD_ABORTED = true
}
/* Throw the execiption to be caught by catchError() to mark the stage failed. */
throw (e)
}
// Do other stuff...
}
You could add a explicit fail task, such as 'sh "not exist command"' in the stage.
if (result.equals("SUCCESS")) {
stage 'deploy'
build 'deploy'
} else {
try {
sh "not exist command"
}catch(e) {
}
}
Solution steps
You must emit an error in a stage to mark it as an error
Outside the scope of the stage, handle the exception and choose the build status
This makes the effect desired by a couple of users here, including myself, #user3768904, #Sviatlana
Success with failed Step Example
node("node-name") {
try {
stage("Process") {
error("This will fail")
}
} catch(Exception error) {
currentBuild.result = 'SUCCESS'
return
}
stage("Skipped") {
// This stage will never run
}
}
Aborted with failure Step Example
node("node-name") {
try {
stage("Process") {
error("This will fail")
}
} catch(Exception error) {
currentBuild.result = 'ABORTED'
return
}
stage("Skipped") {
// This stage will never run
}
}
You can use the following code in your else statement:
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
error "some err msg"
}
This could be a general pattern showing how to customize the stage result with nice messages using the built-in functions and propagate the sub-job's result to the stage result. That the overall build is marked unstable if a sub-job is not successful is just a implementation choice for this example.
def run_sub_job() {
def jobBuild = build(job: 'foo', wait: true, propagate: false)
def result = jobBuild.getResult()
def msg = 'sub-job: ' + result
if ('SUCCESS' == result) {
println(msg)
} else if ('UNSTABLE' == result) {
unstable(msg) // will also set buildResult to UNSTABLE
} else { // anything else (FAILURE, ABORTED ...) is considered an error
catchError(
buildResult: 'UNSTABLE',
stageResult: result // propagate sub-job result
) {
error(msg)
}
}
}

Resources