Multibranch Pipeline vs Pipeline job - jenkins

Now that the Multibranch Pipeline job type has matured, is there any reason to use the simple Pipeline job type any longer? Even if you only have one branch today, it's probably wise to account for the possibility of multiple branches in the future, so what would the motivation be to use the Pipeline job type for your Jenkins Pipeline vs. always using the Multibranch Pipeline job type, assuming you are storing your Jenkinsfile in SCM? Is there feature parity between the two job types now?

In my experience with multibranch pipelines, the ONLY downside is that you can't see the last success/failure/duration columns on the Jenkins main page. They just show "NA" on the Jenkins front page since it's technically a 'folder' of sub-jobs.
Other than that I can't think of any other "cons" to using multibranch.
I disagree with the other answer.... that case was that multibranch sends changes for "any" branch. That's not necessarily true. If a Jenkinsfile exists on a random feature branch, but that branch is not defined in the pipeline then you can just not do anything with it using typical if/else conditionals.
For example:
node {
checkout scm
def workspace = pwd()
if (env.BRANCH_NAME == 'master') {
stage ('Some Stage 1 for master') {
sh 'do something'
}
stage ('Another Stage for Master') {
sh 'do something else here'
}
}
else if (env.BRANCH_NAME == 'stage') {
stage ('Some stage branch step') {
sh 'do something'
}
stage ('Deploy to stage target') {
sh 'do something else'
}
}
else {
sh 'echo "Branch not applicable to Jenkins... do nothing"'
}
}

Multibranch Pipeline works well if your Jenkins job deals with a single git repository. On the other hand, the pipeline job can be repository-neutral and branch-neutral and very flexible when working with multiple git repositories with a single Jenkins job.
For example, assume you have artifact-1 from repo-1, artifact-2 from repo-2, and integration tests from repo-3. And artifact-2 depends on artifact-1. A Jenkins job has to build artifact-1, then build artifact-2, and finally run integration tests from repo-3. And assume your code change goes to a feature-1 branch of repo-1 and feature-1 branch for new tests in repo-3. In this case, the Jenkins job builds feature-1 for artifact-1, then uses 'dev' branch as default from repo-2 (if feature-1 is not detected in repo-2), and runs 'feature-1' from repo-3 for new integration tests. As you can see, the job works well with three git repositories. A repo-neutral/branch-neutral pipeline job is ideal in this setting.

In a CI/CD situation, it may not be desirable to send every branch to the target environment. Using pipeline and specifying a single branch would allow you to filter, and send only /master to Staging or Production environments. Multibranch would be useful for sending any change on any branch specifically to a test environment.
On the other hand, if the QA/AutomatedTesting process is thorough enough, the risk with sending any branch to Production could be acceptable.

If you are still developing your flow, the simple pipeline has the added advantage of supporting parameterized projects. This feature is useful for developing the declarative pipelines in the jenkins gui, using the parameter to control what branch/repository you are targeting.

Related

Jenkins with Jenkinsfile Pipeline - Trigger Builds Remotely

I have a Jenkinsfile setup for our CI/CD pipeline, and it runs through the pipeline on git actions like Pull Requests, Branch Creation, Tag Pushes, etc..
Prior to this setup, I was used to setting up Jenkins build jobs in the Jenkins UI. The advantage of this, was that I could setup dedicated build jobs that I could trigger remotely, and independently of git webhook actions. I could do a POST to the job endpoint with parameters to trigger various actions.
Documentation for this process would be referenced here - see "Trigger Builds Remotely"
I could also hit the big button that says "Build", or "Build with Parameters" in the UI, which was super nice.
How would one do this with a Jenkinsfile? Is this even possible to define build jobs in a pipeline definition within a Jenkinsfile? I.E. define functions / build jobs that have dedicated URLs that could be called on the Jenkins URL independent of webhook callbacks?
What's the best practice here?
Thanks for any tips, references, suggestions!
I would recommend starting with Multibranch pipelines. In general you get all the things you mentioned, but a little better. Because thhe paramteres can be defined within your Jenkinsfile. In short just do it like this:
Create a Jenkinsfile an check this into a Git Repository.
To create a Multibranch Pipeline: Click New Item on Jenkins home page.
Enter a name for your Pipeline, select Multibranch Pipeline and click OK.
Add a Branch Source (for example, Git) and enter the location of the repository.
Save the Multibranch Pipeline project.
A declarative Jenkinsfile can look like this:
pipeline {
agent any
parameters {
string(name: 'Greeting', defaultValue: 'Hello', description: 'How should I greet the world?')
}
stages {
stage('Example') {
steps {
echo "${params.Greeting} World!"
}
}
}
}
A good tutorial with screenshhots can be found here: https://www.jenkins.io/doc/book/pipeline/multibranch/

How to "Scan Repository Now" from a Jenkinsfile

I can call another jenkins job using the build command. Is there a way I can tell another job to do a branch scan?
A multibranch pipeline job has a UI button "Scan Repository Now". When you press this button, it will do a checkout of the configured SCM repository and detect all the branches and create subjobs for each branch.
I have a multibranch pipeline job for which I have selected the "Suppress automatic SCM triggering" option because I only want it to run when I call it from another job. Because this option is selected, the multibranch pipeline doesn't automatically detect when new branches are added to the repository. (If I click "Scan Repository Now" in the UI it will detect them.)
Essentially I have a multibranch pipeline job and I want to call it from another multibranch pipeline job that uses the same git repository.
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
//scanScm "../my-other-multibranch-job" <--- scanScm is a fake command I made up
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
I get an error on that build line, because the target multibranch pipeline job does not yet know that BRANCH_NAME exists. I need a way to trigger an SCM re-scan in the target job from this current job.
Similar to what you figured out yourself, I can contribute my optimization that actually waits until the scan has finished (but is subject to Script Security):
// Helper functions to trigger branch indexing for a certain multibranch project.
// The permissions that this needs are pretty evil.. but there's currently no other choice
//
// Required permissions:
// - method jenkins.model.Jenkins getItemByFullName java.lang.String
// - staticMethod jenkins.model.Jenkins getInstance
//
// See:
// https://github.com/jenkinsci/pipeline-build-step-plugin/blob/3ff14391fe27c8ee9ccea9ba1977131fe3b26dbe/src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerStepExecution.java#L66
// https://stackoverflow.com/questions/41579229/triggering-branch-indexing-on-multibranch-pipelines-jenkins-git
void scanMultiBranchAndWaitForJob(String multibranchProject, String branch) {
String job = "${multibranchProject}/${branch}"
// the `build` step does not support waiting for branch indexing (ComputedFolder job type),
// so we need some black magic to poll and wait until the expected job appears
build job: multibranchProject, wait: false
echo "Waiting for job '${job}' to appear..."
while (Jenkins.instance.getItemByFullName(job) == null || Jenkins.instance.getItemByFullName(job).isDisabled()) {
sleep 3
}
}
Ended up figuring this out shortly after posting the question. Calling build against the base multibranch pipeline job as opposed to a branch causes it to re-scan. The solution to my above snippet would have ended up looking something like...
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
build job: "../my-other-multibranch-job", wait: false, propagate: false // scan for branches
sleep 2 // scanning takes time
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
The wait: false is important because otherwise you get "ERROR: Waiting for non-job items is not supported". The multibranch "parent" job is closer to a folder than a job, but it's a folder that supports the build command, and it does so by scanning the SCM.
But solving this just led to another problem, which is that with wait: false we have no way of knowing when the SCM Scan finished. If you have a large repository (or you're short on jenkins agents), the branch won't get discovered until after the second build command has already failed due to the branch not existing. You could bump the sleep time even higher, but that doesn't scale.
Fortunately, it turns out manually initiating the SCM scan isn't even needed if you have github webhooks set up for your jenkins. The branch will be discovered more-or-less instantly, so for my purposes this is solved another way. The reason I was running into it is we don't have webhooks set up in our dev jenkins, but once I move this code to prod it will work fine.
If you're trying to use JobDSL to set up multibranches calling multibranches and you don't have webhooks or something equivalent, the better path is probably to abandon multibranch for your second tier of jobs and use JobDSL to create folders and manage the branch jobs yourself.

How git merge works for multibranch pipeline?

I am setting up a multi-branch pipeline in Jenkins for my application. I am unsure about how different jenkinsfile checked into the repository handles merging.
For example, I have 3 branches, master, develop and feature-1, all those branches require a different pipeline configuration (see below).
master:
1. build
2. test
3. deploy to PROD
4. tell everyone using email
develop:
1. build
2. test
3. deploy to DEV
feature-1:
1. build
2. test
From day 1, master has Jenkinsfile for production release, develop was branched with the same Jenkinsfile content, changes were made to deploy to DEV and removed last step, committed change.
Day 2: feature-1 branched off develop, changes were made to remove deploy step and committed.
When feature-1 is completed, we are merging feature-1 into develop branch, since the commit that removed last step is dated after the develop commit, this is going to result in Auto-Merge and will make "develop" branch not deploy. Same will happen when develop is merged into master
Based on above potential issues, I think I must be doing something wrong here, what is the best practice for managing Jenkinsfile in Multi-branch pipeline?
Implement some flow control (https://jenkins.io/doc/book/pipeline/syntax/#flow-control) and maintain the same Jenkinsfile in all your branches.
if (env.BRANCH_NAME == 'master'){
echo 'do all the things'
} else if (env.BRANCH_NAME == 'develop'){
echo 'do a few things'
} else if (env.BRANCH_NAME == 'feature-1'){
echo 'do a couple things'
}
The BRANCH_NAME information is exposed by Jenkins when using Multibranch Pipelines (https://jenkins.io/doc/book/pipeline/multibranch/#additional-environment-variables).
You could use conditional statements like
// Note: this is a declarative pipeline
stage('deploy PROD'){
when { branch 'master' }
steps {
...do deploy...
}
}
stage('deploy DEV'){
when { branch 'develop' }
steps {
...do deploy...
}
}
...
For more info see:
https://github.com/jenkinsci/pipeline-examples/blob/master/declarative-examples/simple-examples/whenBranchMaster.groovy#L13
https://jenkins.io/doc/book/pipeline/syntax/#when

Skipping stages in Jenkins pipeline on a Commit but not a PR

We are currently using the Jenkins/Groovy pipeline method for CI.
I'm trying to create a single pipeline for building packages and running unit tests on a branch (and let me know if this is bad practice).
The problem is on a commit I don't want to execute my test steps (due to the large number of commits and time to execute the full pipeline), but I still want the packaging stage to run for our manual testers to be able to pull and install on instances.
Is there any way to distinguish between a run for a PR vs a commit in the pipeline steps or in the job configuration?
Using the (assuming github, but theres bitbucket equiv as well)
https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin to discover Repositories to build branches and PR's will allow you to fall back on Jenkins ENV variables.
This allows simple if statement to determine if the build is for a branch, or for a PR, as PR's are built on a 'branch' of PR-n
Once a PR is open however, all commits would be built.
https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/index.html#github-branch-source
you can use the changeRequest built-in condition from Jenkins.
Executes the stage if the current build is for a "change request" (a.k.a. Pull Request on GitHub and Bitbucket, Merge Request on GitLab, Change in Gerrit, etc.). When no parameters are passed the stage runs on every change request, for example: when { changeRequest() }.
stage('Run only for pull requests to master branch or at the master branch') {
when {
anyOf {
branch 'master'
changeRequest target: 'master'
}
}
steps {
// this is a very long step for integration test that we don't want to execute often, but we need to execute before to merge to master
sh "${mvn} " +
"clean " +
"test-compile " +
"failsafe:integration-test failsafe:verify"
}
}

Visualizing build steps in Jenkins pipeline

In my Jenkins pipeline, I trigger several other jobs using the build step and pass some parameters to it. I'm having issues visualizing the different jobs I've triggered in addition to my pipeline. I have set up the Jenkins Delivery Pipeline plugin but the documentation for it is extremely vague and I have only been able to visualize the steps within my pipeline, despite tagging the jobs with both a stage and task name.
Example:
I have two jobs in Jenkins as pipelines/workflow jobs with the following pipeline script:
Job Foo:
stage('Building') {
println 'Triggering job'
build 'Bar'
}
Job Bar:
node('master') {
stage('Child job stage') {
println 'Doing stuff in child job'
}
}
When visualizing this with the Jenkins Pipeline Delivery plugin, I only get this:
How do I make it also show the stage in job Bar in a separate box?
Unfortunately, this use case is currently not supported in the Delivery Pipeline plugin version 1.0.0. The delivery pipeline plugin views for Jenkins pipelines are only rendering what is contained within one pipeline definition at this point. This feature request is tracked in JENKINS-43679.

Resources