Branch Indexing in jenkins multibranch pipeline triggers extra build, which is already built by poll SCM - jenkins

I have a multibranch pipeline.
I have configured Trigger in jenkinsFile properties:
pipelineTriggers([pollSCM('H/2 * * * *')])])
The multibranch pipeline is configured to "scan Multibranch Pipeline Triggers" periodically.
Expected behavior: Trigger builds only on build triggers configured through jenkinsFile
Actual: It triggers build on Poll SCM and also on "re-indexing" for the same commit.
We are using
Jenkins: 2.107.1
git plugin: 3.8.0
Pipeline multibranch: 2.17

I guess we can not stop the build trigger from scanning. But our build should be able to identify and stop the additional build.
// execute this before anything else, including requesting any time on an agent
if (currentBuild.getBuildCauses().toString().contains('BranchIndexingCause')) {
print "INFO: Build skipped due to trigger being Branch Indexing"
currentBuild.result = 'ABORTED' // optional, gives a better hint to the user that it's been skipped, rather than the default which shows it's successful
return
}
Source: https://www.jvt.me/posts/2020/02/23/jenkins-multibranch-skip-branch-index/

Related

Jenkins add/remove a branch from a multi-branch pipeline job

I have a multi-branch pipeline job which does a build/test on a specific branch.
I would like to create a new Jenkins job, which 'loads' a new branch into the multi-branch pipeline, runs the pipeline 3 times with 3 different options then cleans up and removes the branch.
Here is my use case example:
Set up - add branch release-19.0.4 branch to multi-branch pipeline
Run multi-branch pipeline on branch release-19.0.4 with parameter support-version=19.0.3
Run multi-branch pipeline on branch release-19.0.4 with parameter support-version=19.0.0
Run multi-branch pipeline on branch release-19.0.4 with parameter support-version=18.0.0
Write report: support-version=19.0.3 - ok, support-version=19.0.0 - ok, support-version=18.0.0 - failed
Tear down - remove release-19.0.4 branch from mule-branch pipeline job.
I can use build pipeline task to build another job with different options and read the success or failure of that job in order to create my report. However I'm stuck on how to do steps 1) and 6) from within a Jenkinsfile.
I've tried to read through the Jenkins pipeline API's but nothing seems to fit. Any ideas how to do this via automation?

What does the pollSCM trigger refer to in this Jenkinsfile?

Consider the following setup using Jenkins 2.176.1:
A new pipeline project named Foobar
Poll SCM as (only) build trigger, with: H/5 * * * * ... under the assumption that this refers to the SCM configured in the next step
Pipeline script from SCM with SCM Git and a working Git repository URL
Uncheck Lightweight checkout because of JENKINS-42971 and JENKINS-48431 (I am using build variables in the real project and Jenkinsfile; also this may affect how pollSCM works, so I include this step here)
Said repository contains a simple Jenkinsfile
The Jenkinsfile looks approximately like this:
#!groovy
pipeline {
agent any
triggers { pollSCM 'H/5 * * * *' }
stages {
stage('Source checkout') {
steps {
checkout(
[
$class: 'GitSCM',
branches: [],
browser: [],
doGenerateSubmoduleConfigurations: false,
extensions: [],
submoduleCfg: [],
userRemoteConfigs: [
[
url: 'git://server/project.git'
]
]
]
)
stash 'source'
}
}
stage('OS-specific binaries') {
parallel {
stage('Linux') {
agent { label 'gcc && linux' }
steps {
unstash 'source'
echo 'Pretending to do a build here'
}
}
stage('Windows') {
agent { label 'windows' }
steps {
unstash 'source'
echo 'Pretending to do a build here'
}
}
}
}
}
}
My understanding so far was that:
a change to the Jenkinsfile (not the whole repo) triggers the pipeline on any registered agent (or as configured in the pipeline project).
said agent (which is random) uses the pollSCM trigger in the Jenkinsfile to trigger the pipeline stages.
But where does the pollSCM trigger poll (what SCM repo)? And if it's a random agent then how can it reasonably detect changes across poll runs?
then the stages are being executed on the agents as allocated ...
Now I am confused what refers to what. So here my questions (all interrelated which is why I keep it together in one question):
The pipeline project polls the SCM just for the Jenkinsfile or for any changes? The repository in my case is the same (for Jenkinsfile and source files to build binaries from).
If the (project-level) polling triggers at any change rather than changes to the Jenkinsfile
Does the pollSCM trigger in the Jenkinsfile somehow automagically refer to the checkout step?
Then ... what would happen, would I have multiple checkout steps with differing settings?
What determines what repository (and what contents inside of that) gets polled?
... or is this akin to the checkout scm shorthand and pollSCM actually refers to the SCM configured in the pipeline project and so I can shorten the checkout() to checkout scm in the steps?
Unfortunately the user handbook didn't answer any of those questions and pollSCM has a total of four occurrences on a single page within the entire handbook.
I'll take a crack at this one:
The pipeline project polls the SCM just for the Jenkinsfile or for any
changes? The repository in my case is the same (for Jenkinsfile and
source files to build binaries from).
The pipeline project will poll the repo for ANY file changes, not just the Jenkinsfile. A Jenkinsfile in the source repo is common practice.
If the (project-level) polling triggers at any change rather than
changes to the Jenkinsfile Does the pollSCM trigger in the Jenkinsfile
somehow automagically refer to the checkout step?
Your pipeline will be executed when a change to the repo is seen, and the steps are run in the order that they appear in your Jenkinsfile.
Then ... what would happen, would I have multiple checkout steps with
differing settings?
If you defined multiple repos with the checkout step (using multiple checkout SCM calls) then the main pipeline project repo would be polled for any changes and the repos you define in the pipeline would be checked out regardless of whether they changed or not.
What determines what repository (and what contents inside of that)
gets polled? ... or is this akin to the checkout scm shorthand and
pollSCM actually refers to the SCM configured in the pipeline project
and so I can shorten the checkout() to checkout scm in the steps?
pollSCM refers to the pipeline project's repo. The entire repo is cloned unless the project is otherwise configured (shallow clone, lightweight checkout, etc.).
The trigger defined as pollSCM polls the source-control-management (SCM), at the repository and branch in which this jenkinsfile itself (and other code) is located.
For Pipelines which are integrated with a source such as GitHub or BitBucket, triggers may not be necessary as webhooks-based integration will likely already be present. The triggers currently available are cron, pollSCM and upstream.
It works for a multibranch-pipeline as trigger to execute the pipeline.
When Jenkins polls the SCM, exactly this repository and branch, and detects a change (i.e. new commit), then this Pipeline (defined in jenkinsfile) is executed.
Usually then the following SCM Step checkout will be executed, so that the specified project(s) can be built, tested and deployed.
See also:
SCM Poll in jenkins multibranch pipeline
SehllHacks(2020): Jenkins: Scan Multibranch Pipeline Without Build

How to "Scan Repository Now" from a Jenkinsfile

I can call another jenkins job using the build command. Is there a way I can tell another job to do a branch scan?
A multibranch pipeline job has a UI button "Scan Repository Now". When you press this button, it will do a checkout of the configured SCM repository and detect all the branches and create subjobs for each branch.
I have a multibranch pipeline job for which I have selected the "Suppress automatic SCM triggering" option because I only want it to run when I call it from another job. Because this option is selected, the multibranch pipeline doesn't automatically detect when new branches are added to the repository. (If I click "Scan Repository Now" in the UI it will detect them.)
Essentially I have a multibranch pipeline job and I want to call it from another multibranch pipeline job that uses the same git repository.
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
//scanScm "../my-other-multibranch-job" <--- scanScm is a fake command I made up
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
I get an error on that build line, because the target multibranch pipeline job does not yet know that BRANCH_NAME exists. I need a way to trigger an SCM re-scan in the target job from this current job.
Similar to what you figured out yourself, I can contribute my optimization that actually waits until the scan has finished (but is subject to Script Security):
// Helper functions to trigger branch indexing for a certain multibranch project.
// The permissions that this needs are pretty evil.. but there's currently no other choice
//
// Required permissions:
// - method jenkins.model.Jenkins getItemByFullName java.lang.String
// - staticMethod jenkins.model.Jenkins getInstance
//
// See:
// https://github.com/jenkinsci/pipeline-build-step-plugin/blob/3ff14391fe27c8ee9ccea9ba1977131fe3b26dbe/src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerStepExecution.java#L66
// https://stackoverflow.com/questions/41579229/triggering-branch-indexing-on-multibranch-pipelines-jenkins-git
void scanMultiBranchAndWaitForJob(String multibranchProject, String branch) {
String job = "${multibranchProject}/${branch}"
// the `build` step does not support waiting for branch indexing (ComputedFolder job type),
// so we need some black magic to poll and wait until the expected job appears
build job: multibranchProject, wait: false
echo "Waiting for job '${job}' to appear..."
while (Jenkins.instance.getItemByFullName(job) == null || Jenkins.instance.getItemByFullName(job).isDisabled()) {
sleep 3
}
}
Ended up figuring this out shortly after posting the question. Calling build against the base multibranch pipeline job as opposed to a branch causes it to re-scan. The solution to my above snippet would have ended up looking something like...
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
build job: "../my-other-multibranch-job", wait: false, propagate: false // scan for branches
sleep 2 // scanning takes time
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
The wait: false is important because otherwise you get "ERROR: Waiting for non-job items is not supported". The multibranch "parent" job is closer to a folder than a job, but it's a folder that supports the build command, and it does so by scanning the SCM.
But solving this just led to another problem, which is that with wait: false we have no way of knowing when the SCM Scan finished. If you have a large repository (or you're short on jenkins agents), the branch won't get discovered until after the second build command has already failed due to the branch not existing. You could bump the sleep time even higher, but that doesn't scale.
Fortunately, it turns out manually initiating the SCM scan isn't even needed if you have github webhooks set up for your jenkins. The branch will be discovered more-or-less instantly, so for my purposes this is solved another way. The reason I was running into it is we don't have webhooks set up in our dev jenkins, but once I move this code to prod it will work fine.
If you're trying to use JobDSL to set up multibranches calling multibranches and you don't have webhooks or something equivalent, the better path is probably to abandon multibranch for your second tier of jobs and use JobDSL to create folders and manage the branch jobs yourself.

How do I configure "Scan Multibranch Pipeline Triggers" in my jenkinsfile?

Right now I manually configure my all my multibranch pipeline jobs and set "Scan Multibranch Pipeline Triggers" to 3 minutes.
How do I put this in my Jenkinsfile? I can't find examples of this. Is "Scan Multibranch Pipeline Triggers" available in the triggers{} block?
The settings on the multibranch configuration page only configure the multibranch scan job itself, not the individual jobs created inside the multibranch "folder".
The option under "Scan Multibranch Pipeline Triggers" that says "Periodically if not otherwise run" is only a trigger for when the multibranch job will scan for new branches. If changes are found to existing branches, or if new branches are discovered with a Jenkinsfile that match your branch specifications, a new build will be triggered, but this is not intended to be the way a job is triggered.
Actually, you can disable the automatic build when changes are found by adding a property to the SCM configuration to "Disable automatic SCM Triggering". Then then you will see the multibranch scan trigger, but the jobs themselves won't build, even if there are changes found.
To trigger jobs, ideally you should use a webhook if you can. If you use a git hook using the git plugin (not the github plugin), then you need to enable the PollSCM trigger (though you can set it to only poll rarely, or not at all).
If you just want normal triggering options, as of 2.22, you can configure the either cron or pollSCM triggers.
pipeline {
triggers {
cron('H/4 * * * 1-5')
pollSCM('0 0 * * 0')
}
Then I believe you can configure webhooks to inform your multibranch job when to do a scan. I haven't tried that. I just tell it to scan every hour or a couple times per day using the "Periodically if not otherwise run".
Note, the same thing applies for the build discarder and other things you configure in your multibranch job. In the web UI, you can only configure the multibranch job itself, not the individual jobs created from it. You have to use Pipeline to configure the jobs.
If your're using the JobDSL Jenkins plugin for creating jobs, then you can add following lines to configure "Scan Multibranch Pipeline Triggers":
configure {
it / 'triggers' << 'com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger'{
spec '* * * * *'
interval "60000"
}
}
Using the JobDSL Jenkins Plugin for multibranch pipeline job, the periodic folder trigger can be configured as given below. In this example, the maximum amount of time since the last indexing that is allowed to elapse before an indexing is triggered will be seven days.
multibranchPipelineJob('my-awesome-job') {
triggers {
periodicFolderTrigger {
interval("7d")
}
}
}

Jenkins multi-branch pipeline and specifying upstream projects

We currently generate a lot of Jenkins jobs on a per Git branch basis using Jenkins job DSL; the multi-branch pipeline plugin looks like an interesting way to potentially get first-class job generation support using Jenkinsfiles and reduce the amount of Job DSL we maintain.
For example we have libwidget-server and widget-server develop branch projects. When the libwidget-server build finishes then the widget-server job is triggered (for the develop branch). This applies to other branches too.
This makes use of the Build after other projects are built to trigger upon completion of an upstream build (e.g. libwidget-server causes widget-server to be built).
It seems that the multi-branch pipeline plugin lacks the Build after other projects are built setting - how would we accomplish the above in the multi-branch pipeline build?
You should add the branch name to your upstream job (assuming you are using a multi-branch pipeline for the upstream job too).
Suppose you have a folder with two jobs, both multi-branch pipeline jobs: jobA and jobB; jobB should trigger after jobA's master.
You can add this code snippet to jobB's Jenkinsfile:
properties([
pipelineTriggers([
upstream(
threshold: 'SUCCESS',
upstreamProjects: '../jobA/master'
)
])
])
(Mind that any branch of jobB here will trigger after jobA's master!)
I'm currently trying to get this to work for our deployment.
The closest I've got is adding the following to the downstream Jenkinsfile;
properties([
pipelineTriggers([
triggers: [
[
$class: 'jenkins.triggers.ReverseBuildTrigger',
upstreamProjects: "some_project", result: hudson.model.Result.SUCCESS
]
]
]),
])
That at least gets Jenkins to acknowledge that it should be triggering when
'some_project' get's built i.e it appears in the "View Configuration" page.
However so far builds of 'some_project' still don't trigger the downstream
project as expected.
That being said maybe you'll have more luck.
Let me know if it works for you.
(Someone else has asked a similar question here -> Jenkins: Trigger Multi-branch pipeline on upstream change )

Resources