Can a Jenkins pipeline have an optional input step? - jenkins

Is it possible to create a Jenkins pipeline with an optional input stage?
The below snippet doesn't achieve this goal.
Expected behaviour
The stage (and therefore the input prompt) should only run for specific branches.
Actual behaviour
This stage runs for all branches. The when filter is ignored when an input step is used.
stage('Approve') {
when {
expression { BRANCH_NAME ==~ /^qa[\w-_]*$/ }
}
input {
message "Approve release?"
ok "y"
submitter "admin"
parameters {
string(name: 'IS_APPROVED', defaultValue: 'y', description: 'Deploy to master?')
}
}
steps {
script {
if (IS_APPROVED != 'y') {
currentBuild.result = "ABORTED"
error "User cancelled"
}
}
}
}

The filter is not ignored, it is just evaluated after the input step. In your example, you would always be asked whether to deploy, and in the case that you are not on a QA branch, nothing would happen.
Now you could ask why Jenkins isn't evaluating the 'when' directive first. In that case, you could not use the input parameter in your when condition.
And having multiple when directives would be like scripting within the declarative pipeline.
However, there is an expression that allows you controlling when the 'when' directive is evaluated. This is beforeAgent. It allows you to evaluate the when statement before the agent is allocated. Similar to that, you would need something like beforeInput. You could create a feature request for that.
I stepped away from using the input directive and I use input within a scripting block now, because that provides much more flexibility, e.g. I am sending Slack notifications when somebody has to approve something, which is impossible with the declarative approach. You would need a notify directive for that. And if there was one, is that going to be evaluated before or after the input step?
You see, doing everything declarative is not always the best way. So my recommended approach is the following (disclaimer: this is untested!):
pipeline {
// We want to use agents per stage to avoid blocking our build agents
// while we are waiting for user input.
agent none
...
// The question mark naming convention is helpful to show you which
// approval stage belongs to which work stage.
stage('Release?') {
// Don't allocate an agent because we don't want to block our
// slaves while waiting for user input.
agent none
when {
// You forgot the 'env.' in your example above ;)
expression { env.BRANCH_NAME ==~ /^qa[\w-_]*$/ }
}
options {
// Optionally, let's add a timeout that we don't allow ancient
// builds to be released.
timeout time: 14, unit: 'DAYS'
}
steps {
// Optionally, send some notifications to the approver before
// asking for input. You can't do that with the input directive
// without using an extra stage.
slackSend ...
// The input statement has to go to a script block because we
// want to assign the result to an environment variable. As we
// want to stay as declarative as possible, we put noting but
// this into the script block.
script {
// Assign the 'DO_RELEASE' environment variable that is going
// to be used in the next stage.
env.DO_RELEASE = input ...
}
// In case you approved multiple pipeline runs in parallel, this
// milestone would kill the older runs and prevent deploying
// older releases over newer ones.
milestone 1
}
}
stage('Release') {
// We need a real agent, because we want to do some real work.
agent any
when {
// Evaluate the 'when' directive before allocating the agent.
beforeAgent true
// Only execute the step when the release has been approved.
environment name: 'DO_RELEASE', value: 'yes'
}
steps {
// Make sure that only one release can happen at a time.
lock('release') {
// As using the first milestone only would introduce a race
// condition (assume that the older build would enter the
// milestone first, but the lock second) and Jenkins does
// not support inter-stage locks yet, we need a second
// milestone to make sure that older builds don't overwrite
// newer ones.
milestone 2
// Now do the actual work here.
...
}
}
}

The correct syntax would be more like (completely untested):
stage('Approve') {
when {
expression { BRANCH_NAME ==~ /^qa[\w-_]*$/ }
}
steps {
script {
def IS_APPROVED = input(
message: "Approve release?"
ok: "y"
submitter: "admin"
parameters: [
string(name: 'IS_APPROVED', defaultValue: 'y', description: 'Deploy to master?')
]
)
if (IS_APPROVED != 'y') {
currentBuild.result = "ABORTED"
error "User cancelled"
}
}
}
}
So essentially, you're hitting the limits of declarative pipelines have to fall back to groovy scripting / scripted pipelines.

Related

Parallel matrix and global variables without race condition?

I have the following declarative pipeline where I write a global build variable
during a parallel matrix, the write in stage Build Detection is probably (wasn't clear to me) a race condition but I am not sure. I have 3 questions regarding the below simple pipeline:
Is it correct that since Build-Detection uses the same agent (note only Build uses a different agent), it is definitely a race condition ?
If I would have one agent for each parallel line, it would not be a
race condition as the global build is different in each agent?
Is there a way to make a variable copy of build inside the stage such that its not global anymore?
How should we deal with global variable communicating stuff (for when steps etc)
and parallel matrix feature?
Map<String,Boolean> build
pipeline {
stages {
stage('Test') {
failFast false
matrix {
axes {
axis {
name 'CONTAINER'
values 'A', 'B'
}
}
stages {
stage('Build Detection') {
steps {
script {
build[CONTAINER] = CONATAINER == 'A'
echo "Should Build: ${build[CONTAINER]}"
}
}
}
stage('Build') {
agent {
kubernetes {
yamlFile '.jenkins/pods/build-kaniko.yaml'
}
}
when {
beforeAgent true
expression { return build[CONTAINER] }
}
steps {
echo "BUILDING....."
}
}
}
}
}
}
}
No, it has nothing to do with build agents. The JVM that's executing the compiled groovy code is running on the Jenkins master, not a build agent. Therefore, using a global variable is shared by each thread running in the Jenkins master JVM. Whether there's a possible race condition is not related to stages using the same or different build agents.
Same answer as 1.
Yes, simply define a variable using "def" or a specific type in the stage's script block. Just be sure to not reference a new variable without a type because in Groovy that causes it to be declared globally.
Using a map with a key that is specific to each thread like you're doing seems like a good way to me. If you really want to make sure there is no possibility of two unsafe thread operations modifying the map at the same time, then make sure that a threadsafe map is used. You could print out the class of the map to find out what implementation is getting instantiated. I would hope it's something threadsafe like ConcurrentHashMap.

Jenkins Declarative Pipeline: how to wait before and after input on when directive

In my Jenkinsfile, I have 2 stages: Pre Live and Live. I ask the user for input on stage Pre Live to know whether a deploy should be done to a pre live environment, and then on stage Live, I ask the user again for input to know whether to do a deploy to a live environment or not.
I managed to implement this. This is how the code looks:
stage("Pre Live") {
input {
message 'Deploy to Pre Live?'
parameters {
booleanParam(name: 'RELEASE_PRE_LIVE', defaultValue: false)
}
}
when {
beforeInput false
expression {
return RELEASE_PRE_LIVE.toBoolean()
}
}
steps {
// ...
}
}
stage("Live") {
input {
message 'Deploy to Live?'
parameters {
booleanParam(name: 'RELEASE_LIVE', defaultValue: false)
}
}
when {
beforeInput false
expression {
return RELEASE_LIVE.toBoolean()
}
}
steps {
// ...
}
}
What I am not able to do, however, is too keep all of this logic, but also only ask for input on the stage Live if the the previous stage (Pre Live) was executed. Normally, this could be done through the when directive in the Live stage, but the problem is that I need my when directive in that stage to evaluate after the input, because I need the input value to know if the user wants to deploy to live or not, but I also don't want to unnecessarily wait for input on this stage if Pre Live was never ran, because it doesn't make sense.
Have you considered using build step that would trigger another pipeline and splitting your pipeline into two pipelines instead of one?
Here is an example article for declarative pipeline syntax:
https://support.cloudbees.com/hc/en-us/articles/360019828412-Pipeline-How-to-write-a-declarative-pipeline-to-invoke-another-job
I believe you can implement an additional check by setting then passing a variable from the Pre-Live stage to the Live stage. Then acting on that variable in the Live stage.
You already have this next part down, but I thought it would be good here for context.
Evaluating when before the input directive
By default, the when condition for a stage will not be evaluated before the input, if one is defined. However, this can be changed by specifying the beforeInput option within the when block. If beforeInput is set to true, the when condition will be evaluated first, and the input will only be entered if the when condition evaluates to true.
beforeInput true takes precedence over beforeAgent true.
From: https://jenkins.io/doc/book/pipeline/syntax/#when

How can I skip a stage if the agent is offline?

In my pipeline, I have a stage that checks to see if a specific computer (node) is offline. If it is, I want to skip the next stage. However, the next stage is set to use the offline agent, so it doesn't seem to be able to check the When clause.
Here's a simplified version of my pipeline:
pipeline {
agent none
environment {
CONTINUERUN = true
}
stages {
stage('Check Should Run') {
agent any
steps {
script {
CONTINUERUN = false
}
}
}
stage('Skip this stage') {
agent {
label 'offlineAgent'
}
when {
expression {
CONTINUERUN
}
}
steps {
//Do stuff here
}
}
}
}
When I run this, the build just hangs at the 'Skip this stage' stage. I'm assuming, because the agent is offline. How can I skip this stage, when the agent is known to be offline?
In order to evaluate expression before allocating agent, you need to add beforeAgent directive to the when block.
Relevant part of documentation:
Evaluating when before entering agent in a stage
By default, the when condition for a stage will be evaluated after entering the agent for that stage, if one is defined. However, this can be changed by specifying the beforeAgent option within the when block. If beforeAgent is set to true, the when condition will be evaluated first, and the agent will only be entered if the when condition evaluates to true.

Jenkins pipeline milestone not cancelling previous ongoing build

I am experimenting with Jenkins pipeline and milestones and cannot figure out why Jenkins is not cancelling the previous build when a new build crosses the milestone.
Example Jenkinsfile
pipeline {
agent any
parameters {
booleanParam(defaultValue: true, description: '', name: 'userFlag')
}
stages {
stage("foo") {
steps {
milestone(ordinal: 1, label: "BUILD_START_MILESTONE")
sh 'sleep 1000'
}
}
}
}
Triggering this pipeline twice does not cancel the 1st job
Try this:
/* This method should be added to your Jenkinsfile and called at the very beginning of the build*/
#NonCPS
def cancelPreviousBuilds() {
def jobName = env.JOB_NAME
def buildNumber = env.BUILD_NUMBER.toInteger()
/* Get job name */
def currentJob = Jenkins.instance.getItemByFullName(jobName)
/* Iterating over the builds for specific job */
for (def build : currentJob.builds) {
/* If there is a build that is currently running and it's not current build */
if (build.isBuilding() && build.number.toInteger() != buildNumber) {
/* Than stopping it */
build.doStop()
}
}
}
I don't think the behavior is "If I'm a newer build that crosses this milestone, then all older build that crossed this milestone will be cancelled"
The actual behavior of the milestone step is that when a more recent pipeline crosses it first, then it prevents older pipeline from crossing that milestone.
I have a simple work around with milestone plugin, according to the document:
Builds pass milestones in order (taking the build number as sorter field).
Older builds will not proceed (they are aborted) if a newer one already passed the milestone.
When a build passes a milestone, any older build that passed the previous milestone but not this one is aborted.
Once a build passes the milestone, it will never be aborted by a newer build that didn't pass the milestone yet.
you can try something like this:
pipeline {
agent any
stages {
stage('Stop Old Build') {
steps {
milestone label: '', ordinal: Integer.parseInt(env.BUILD_ID) - 1
milestone label: '', ordinal: Integer.parseInt(env.BUILD_ID)
}
}
}
}
you can put this at the start of any pipeline.
Assume you just start a new build, #5. The first milestone, will be used to passes #4's second milestone, and the second milestone(of #5) will be used to kill #4's process, if it's currently running.
The disableConcurrentBuilds property has been added to Pipeline. The Pipeline syntax snippet generator offers the following syntax hint:
properties([disableConcurrentBuilds(abortPrevious: true)])
That property is used on ci.jenkins.io to cancel older plugin build jobs when newer plugin build jobs start.
Declarative Pipeline also includes the disableConcurrentBuilds option that is documented in the Pipeline syntax page.
The declarative directive generator suggests the following:
options {
disableConcurrentBuilds abortPrevious: true
}
as per https://jenkins.io/blog/2016/10/16/stage-lock-milestone/, a pair of 'milestone()' works for me to kill the previous jobs while the pipeline kicked off for times,
stage('Build') {
// The first milestone step starts tracking concurrent build order
milestone()
node {
echo "Building"
}}
// The Deploy stage does not limit concurrency but requires manual input
// from a user. Several builds might reach this step waiting for input.
// When a user promotes a specific build all preceding builds are aborted,
// ensuring that the latest code is always deployed.
stage('Deploy') {
timeout(time: 60, unit: 'SECONDS') {input "Deploy?"}
milestone()
node {
echo "Deploying"
}
}
The last milestone helps kill previous builds if reached, say deploy button clicked for the above case. Or the locked resource released for the below case,
// This locked resource contains both Test stages as a single concurrency Unit.
// Only 1 concurrent build is allowed to utilize the test resources at a time.
// Newer builds are pulled off the queue first. When a build reaches the
// milestone at the end of the lock, all jobs started prior to the current
// build that are still waiting for the lock will be aborted
lock(resource: 'myResource', inversePrecedence: true){
node('test') {
stage('Unit Tests') {
echo "Unit Tests"
}
stage('System Tests') {
echo "System Tests"
}
}
milestone()
}
Building on #D.W.'s answer, i found a simple pattern that works. It seems to fit into D.W.'s bullet #3 (which is the official doc): When a build passes a milestone, Jenkins aborts older builds that passed the previous milestone but not this one.
Adding an earlier milestone that everything will pass, and then one after the thing that is going to wait, makes it all work like you think it should. In my case:
steps {
milestone 1
input 'ok'
milestone 2
}
Create two active builds with this, and only approve the second one. You'll see the first one get automatically canceled, because build 2 passed milestone 2 first.
Try taking out milestone 1, and you'll see that build 1 does not get canceled when build 2 passes milestone 2.
Adding the early milestone satisfies the requirement. It seems that a build has to pass any milestone before a future milestone passed by a newer build will cause it to cancel.

How to differentiate build triggers in Jenkins Pipeline

I'm hoping to add a conditional stage to my Jenkinsfile that runs depending on how the build was triggered. Currently we are set up such that builds are either triggered by:
changes to our git repo that are picked up on branch indexing
a user manually triggering the build using the 'build now' button in the UI.
Is there any way to run different pipeline steps depending on which of these actions triggered the build?
The following code should works to determine if a user has started the pipeline or a timer/other trigger:
def isStartedByUser = currentBuild.rawBuild.getCause(hudson.model.Cause$UserIdCause) != null
In Jenkins Pipeline without currentBuild.rawBuild access the build causes could be retrieved in the following way:
// started by commit
currentBuild.getBuildCauses('jenkins.branch.BranchEventCause')
// started by timer
currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause')
// started by user
currentBuild.getBuildCauses('hudson.model.Cause$UserIdCause')
You can get a boolean value with:
isTriggeredByTimer = !currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause').isEmpty()
Or, as getBuildCauses() returns an array, the array's size will work correctly with Groovy truthy semantics:
if (currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause')) {
The ability to get causes for a workflow run was released in version 2.22 (2018 Nov 02) to the Pipeline Supporting APIs Plugin. The feature was requested in JENKINS-41272.
A couple methods were added to the currentBuild global variable with that release:
getBuildCauses
Returns a JSON array of build causes for the current build
EXPERIMENTAL - MAY CHANGE getBuildCauses(String causeClass)
Takes a string representing the fully qualified Cause class and returns a JSON array of build causes filtered by that type for the current build, or an empty JSON array if no causes of the specified type apply to the current build
And an example from me submitting:
echo "${currentBuild.buildCauses}" // same as currentBuild.getBuildCauses()
echo "${currentBuild.getBuildCauses('hudson.model.Cause$UserCause')}"
echo "${currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause')}"
And the output:
[Pipeline] echo
[[_class:hudson.model.Cause$UserIdCause, shortDescription:Started by user anonymous, userId:null, userName:anonymous], [_class:org.jenkinsci.plugins.workflow.cps.replay.ReplayCause, shortDescription:Replayed #12]]
[Pipeline] echo
[]
[Pipeline] echo
[]
[Pipeline] End of Pipeline
Finished: SUCCESS
NOTE
There appears to be an issue with the currentBuild.getBuildCauses(type) when the type is a type of Cause contributed by a plugin. For example, currentBuild.getBuildCauses('org.jenkinsci.plugins.workflow.cps.replay.ReplayCause') fails with a java.lang.ClassNotFoundException. This was reported in JENKINS-54673 for the 2.22 version of the Pipeline: Supporting APIs (workflow-support) plugin. It is reportedly fixed in the 2.24 version.
I might be missing something, but you can achieve what you want easily by making use of the when directive:
pipeline {
agent any
stages {
stage('Always') {
steps {
echo "I am always executed"
}
}
stage('ManualTimed') {
steps {
echo "I am only executed when triggered manually or timed"
}
when {
beforeAgent true
anyOf {
triggeredBy 'TimerTrigger'
triggeredBy cause: 'UserIdCause'
}
}
}
stage('GitLabWebHookCause') {
steps {
echo "I am only executed when triggered by SCM push"
}
when {
beforeAgent true
triggeredBy 'GitLabWebHookCause'
}
}
}
}
You will find many similar useful examples for various use cases in the documentation of the when directive.
Edit:
thanks to Jean-Francois Larvoire's answer, I was able to figure out 'my trigger' GitLabWebHookCause I required for my use case.
#vitalii-blagodir:
Your answer works for detecting builds triggered by users and timers, but not by commits.
Instead, I found this to work in my case:
def isTriggeredByIndexing = currentBuild.getBuildCauses('jenkins.branch.BranchIndexingCause').size()
def isTriggeredByCommit = currentBuild.getBuildCauses('com.cloudbees.jenkins.GitHubPushCause').size()
def isTriggeredByUser = currentBuild.getBuildCauses('hudson.model.Cause$UserIdCause').size()
def isTriggeredByTimer = currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause').size()
The .size() suffix returns 0 if the object is missing, or 1 if it's present. This makes the result usable as a boolean.
For finding the object name to use, I found it convenient to display this in the log:
echo "# Build causes"
def buildCauses = currentBuild.buildCauses
def numCause = 0
for (cause in buildCauses) {
echo "${numCause++}: ${cause.shortDescription}" // Display a human-readable index and description
echo "${cause}" // Display the object class name. This allows knowing what names to use in getBuildCauses(name) calls below.
}
Finally, if the goal is to abort a pipeline build in specific cases, then the test must be done before the beginning of the pipeline.
For example, we had a problem with the branch indexing triggering extra useless builds. This was fixed by adding this before the pipeline:
// Avoid useless buils: The branch indexing should only trigger the initial build of a new branch.
def isTriggeredByBranchIndexing = currentBuild.getBuildCauses('jenkins.branch.BranchIndexingCause').size()
if (isTriggeredByBranchIndexing && currentBuild.previousBuild) { // Then it's not the initial build.
echo "# Reindexing a branch already built. It is useless to rebuild it now. Aborting."
currentBuild.result = 'SUCCESS' // Make sure the build is not displayed in red in the Jenkins UI.
return // Abort before the pipeline even starts. (Inside the pipeline, this would only abort one stage.)
}
I think that the answers here are incomplete and do not provide an actual ready to use answer. Here's my code to get it working:
import com.cloudbees.groovy.cps.NonCPS
#NonCPS
def isStartedByTimer() {
def buildCauses = currentBuild.rawBuild.getCauses()
echo buildCauses
boolean isStartedByTimer = false
for (buildCause in buildCauses) {
if ("${buildCause}".contains("hudson.triggers.TimerTrigger\$TimerTriggerCause")) {
isStartedByTimer = true
}
}
echo isStartedByTimer
return isStartedByTimer
}
// [...]
// Other pipeline stuff
script {
isStartedByTimer()
}
When started by user:
00:00:01.353 [hudson.model.Cause$UserIdCause#fa5cb22a]
[Pipeline] echo
00:00:01.358 false
When started by timer:
00:00:01.585 [hudson.triggers.TimerTrigger$TimerTriggerCause#5]
[Pipeline] echo
00:00:01.590 true
Note: the NonCPS decorator is needed because otherwise the next non-script step will throw.
Assuming the two different build causes are "timer" and "push" (to a git repo), you can add the following stage to your Jenkinsfile (in a declarative Jenkins pipeline) to make use of getBuildCauses():
pipeline {
stages {
stage('preparation') {
steps {
script {
// get build cause (time triggered vs. SCM change)
def buildCause = currentBuild.getBuildCauses()[0].shortDescription
echo "Current build was caused by: ${buildCause}\n"
// e.g. "Current build was caused by: Started by GitHub push by mirekphd"
// vs. "Started by timer"
}
}
}
}
}
Then I can decide whether to perform certain stages conditionally (depending on the build cause). For example, pulling a docker base image and inspecting for changes in system libraries (likely security updates) should be done periodically, regardless of whether there was a source code change or not.
We can use "BUILD_CAUSE" variable for getting the information about who initiated the run
for [jenkins-pipeline] you may use
currentBuild.rawBuild.getCauses()
(see github.com/jenkinsci/pipeline-examples/blob/master/… for more details)
There was a similar requirement, where user detail who triggered the build should be there in success / failure notification. The job was already had time based triggered, hence could not use wrap([$class: 'BuildUser']) directly.
I used below step, which print username if the job is triggered manually or timer triggered. So, I used this:
pipeline {
agent any
stages {
stage('Test') {
steps {
script{
env.buildCauses = currentBuild.rawBuild.getCauses()
if (buildCauses.contains("hudson.triggers.TimerTrigger")){
env.builduser = "TimerTrigger"
} else {
wrap([$class: 'BuildUser']) {
env.builduser = "${BUILD_USER}"
}
}
}
echo "Initiated by: ${env.builduser}"
}
}
}
}

Resources