We're seeing multiple builds of our Jenkins multi-branch pipeline being triggered by the same merge commit in Bitbucket.
One build is marked triggered by a: "Branch event at "
The other by a: "commit notification "
We have our Jenkins urls setup in a plugin
Bitbucket Server Webhook to Jenkins
and have a our trigger in the Jenkinsfile setup as follows:
triggers {
pollSCM ""
cron "H 0 * * *"
}
There seem to be a couple of old questions that unfortunately don't have concrete answers.
We were using Bitbucket with Jenkins integration and had this issue. Our problem was that on the Bitbucket webhook we had selected notifications from push and from PR. This configuration created a Job with the branch name and a second job called PR-XXX when the PR was created.
Maybe this is the reason?
It looks like both the: Branch API Plugin and the Git client plugin were sending events to Jenkins that triggered a build.
We solved the issue by suppressing automatic triggering.
This can be done either in the UI simply by going to your job, selecting configure from the left then adding the property
Suppress automatic SCM triggering.
Alternatively for a code solution (which I ended up using) add it to the seedjob.groovy as follows:
multibranchPipelineJob("${service.name}-build") {
// ... unrelated code omitted
configure { project ->
project.remove(project / 'sources' / 'data' / 'jenkins.branch.BranchSource' / 'strategy' / 'properties')
def s = project / 'sources' / 'data' / 'jenkins.branch.BranchSource' / 'strategy' {
properties(class: 'java.util.Arrays$ArrayList') {
a(class: 'jenkins.branch.NoTriggerBranchProperty') {
'jenkins.branch.NoTriggerBranchProperty' ''
}
}
}
}
}
UPDATE:
It seems like Julian's answer worked but had a bug where it didn't automatically trigger pushes to feature branches. In our Jenkinsfile we added:
properties([overrideIndexTriggers(true)])
Which ensures git commit still triggers a build despite NoTriggerBranchProperty in our seedjob.
Related
I am new Jenkins. I was able to configure commit-based job trigger using Freestyle job. This way, any new commit to GitHub was triggering the given job.
But when it comes to pipeline job, I am not able to achieve the same. Please help regarding the same.
In Build Triggers section of pipeline, I have enabled GitHub hook trigger for GITScm polling.
pipeline{
agent {
node 'npm-linux'
}
options {
timeout(time: 15, unit: 'MINUTES')
disableConcurrentBuilds()
}
stages {
stage('build') {
steps {
sh 'git clone link'
sh 'mvn clean install'
}
}
}
}
As you're able to successfully see the freestyle job getting triggered on commit, we know for sure that GitHub has been configured correctly. Now, to fix the declarative pipeline issue, you have to make use of triggers in your pipeline code.
For example,
pipeline {
agent any
triggers {
// Instead of '* * * * *', you may use 'H/2 * * * *' which will check for source code changes every two minutes
pollSCM '* * * * *'
}
stages {
stage ('Echo') {
steps {
echo 'Hello, World!'
}
}
}
}
Note:
In Build Triggers section of your job configuration, you still have to keep the check box enabled for GitHub hook trigger for GITScm polling
Even after the above setting, the declarative pipeline job did not trigger automatically when i pushed my commit. When i ran the job manually once, thereafter things went fine. All subsequent commits triggered the pipeline job. So, just manually trigger the job once and things should be fine.
Also remember that polling is resource intensive so it is generally not a good idea to use it. However, in this case, where you are using GitHub, i'm not sure whether post-commit functionality can be configured using any way other than GitHub Webhooks. And sadly, just with GitHub Webhooks enabled, pipeline job wasn't getting triggered without the help of triggers directive.
I have created a jenkins pipeline job called "pipelinejob" with the below script:
pipeline {
agent any
stages {
stage ('Setup'){
steps{
//echo "${BRANCH_NAME}"
echo "${env.BRANCH_NAME}"
//echo "${GIT_BRANCH}"
echo "${env.GIT_BRANCH}"
}
}
}
}
Under General, I have selected "GitHub project" and inserted my company's github in the form:
https://github.mycompany.com/MYPROJECTNAME/MY_REPOSITORY_NAME/
Under Build Triggers, i have checked "GitHub hook trigger for GITScm polling
I have created a simple job called "simplejob" with same configuration as 1) and 2)
In my company's Github, i have created a webhook like "jenkins_url/jenkins/github-webhook/"
I commit a change in "mybranch" in "MY_REPOSITORY_NAME"
My simple job "simplejob" is triggered and built successfully
My pipeline job "pipelinejob" is not triggered
In Jenkins log i see the below:
Sep 12, 2019 2:42:45 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run
Poked simplejob
Nothing regarding my "pipelinejob".
Could you please point me to the right directions as to what to check next?
P.S. I have manually executed my "pipelinejob" successfully
I wasted two days of work on this, as none of the previous solutions worked for me. :-(
Eventually I found the solution on another forum:
The problem is that if you use a Jenkinsfile that is stored in GitHub, along with your project sources, then this trigger must be configured in the Jenkinsfile itself, not in the Jenkins or project configuration.
So add a triggers {} block like this to your Jenkinsfile:
pipeline {
agent any
triggers {
githubPush()
}
stages {
...
}
}
Then...
Push your Jenkinsfile into GitHub
Run one build manually, to let Jenkins know about your will to use this trigger.
You'll notice that the "GitHub hook trigger for GITScm polling" checkbox will be checked at last!
Restart Jenkins.
The next push should trigger an automated build at last!
On the left side-pane of your pipeline job, click GitHub Hook log. If it says 'Polling has not run yet', you will need to manually trigger the pipeline job once before Jenkins registers it to poke on receiving hooks.
Henceforth, the job should automatically trigger on GitHub push events.
I found an answer to this question with scripted pipeline file. We need to declare the Github push event trigger in Jenkins file as follows.
properties([pipelineTriggers([githubPush()])])
node {
git url: 'https://github.com/sebin-vincent/Treasure_Hunt.git',branch: 'master'
stage ('Compile Stage') {
echo "compiling"
echo "compilation completed"
}
stage ('Testing Stage') {
echo "testing completed"
echo "testing completed"
}
stage("Deploy") {
echo "deployment completed"
}
}
}
The declaration should be in the first line.
git url: The URL on which pipeline should be triggered.
branch: The branch on which pipeline should be triggered. When you specify the branch as master and make changes to other branches like develop or QA, that won't trigger the pipeline.
Hope this could help someone who comes here for an answer for the same problem with Jenkins scripted pipeline :-(.
The thing is whenever you create a pipeline job for git push which is to be triggered by github-webhook, first you need to build the pipeline job manually for one time. If it builds successfully, then Jenkins registers it to poke on receiving hooks. And from the next git push, your pipeline job will trigger automatically.
Note: Also make sure that the pipeline job built manually for the first time should be built successfully, otherwise Jenkins will not poke it. If it fails to build, you can never trigger the job again.
I can call another jenkins job using the build command. Is there a way I can tell another job to do a branch scan?
A multibranch pipeline job has a UI button "Scan Repository Now". When you press this button, it will do a checkout of the configured SCM repository and detect all the branches and create subjobs for each branch.
I have a multibranch pipeline job for which I have selected the "Suppress automatic SCM triggering" option because I only want it to run when I call it from another job. Because this option is selected, the multibranch pipeline doesn't automatically detect when new branches are added to the repository. (If I click "Scan Repository Now" in the UI it will detect them.)
Essentially I have a multibranch pipeline job and I want to call it from another multibranch pipeline job that uses the same git repository.
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
//scanScm "../my-other-multibranch-job" <--- scanScm is a fake command I made up
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
I get an error on that build line, because the target multibranch pipeline job does not yet know that BRANCH_NAME exists. I need a way to trigger an SCM re-scan in the target job from this current job.
Similar to what you figured out yourself, I can contribute my optimization that actually waits until the scan has finished (but is subject to Script Security):
// Helper functions to trigger branch indexing for a certain multibranch project.
// The permissions that this needs are pretty evil.. but there's currently no other choice
//
// Required permissions:
// - method jenkins.model.Jenkins getItemByFullName java.lang.String
// - staticMethod jenkins.model.Jenkins getInstance
//
// See:
// https://github.com/jenkinsci/pipeline-build-step-plugin/blob/3ff14391fe27c8ee9ccea9ba1977131fe3b26dbe/src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerStepExecution.java#L66
// https://stackoverflow.com/questions/41579229/triggering-branch-indexing-on-multibranch-pipelines-jenkins-git
void scanMultiBranchAndWaitForJob(String multibranchProject, String branch) {
String job = "${multibranchProject}/${branch}"
// the `build` step does not support waiting for branch indexing (ComputedFolder job type),
// so we need some black magic to poll and wait until the expected job appears
build job: multibranchProject, wait: false
echo "Waiting for job '${job}' to appear..."
while (Jenkins.instance.getItemByFullName(job) == null || Jenkins.instance.getItemByFullName(job).isDisabled()) {
sleep 3
}
}
Ended up figuring this out shortly after posting the question. Calling build against the base multibranch pipeline job as opposed to a branch causes it to re-scan. The solution to my above snippet would have ended up looking something like...
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
build job: "../my-other-multibranch-job", wait: false, propagate: false // scan for branches
sleep 2 // scanning takes time
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
The wait: false is important because otherwise you get "ERROR: Waiting for non-job items is not supported". The multibranch "parent" job is closer to a folder than a job, but it's a folder that supports the build command, and it does so by scanning the SCM.
But solving this just led to another problem, which is that with wait: false we have no way of knowing when the SCM Scan finished. If you have a large repository (or you're short on jenkins agents), the branch won't get discovered until after the second build command has already failed due to the branch not existing. You could bump the sleep time even higher, but that doesn't scale.
Fortunately, it turns out manually initiating the SCM scan isn't even needed if you have github webhooks set up for your jenkins. The branch will be discovered more-or-less instantly, so for my purposes this is solved another way. The reason I was running into it is we don't have webhooks set up in our dev jenkins, but once I move this code to prod it will work fine.
If you're trying to use JobDSL to set up multibranches calling multibranches and you don't have webhooks or something equivalent, the better path is probably to abandon multibranch for your second tier of jobs and use JobDSL to create folders and manage the branch jobs yourself.
Right now I manually configure my all my multibranch pipeline jobs and set "Scan Multibranch Pipeline Triggers" to 3 minutes.
How do I put this in my Jenkinsfile? I can't find examples of this. Is "Scan Multibranch Pipeline Triggers" available in the triggers{} block?
The settings on the multibranch configuration page only configure the multibranch scan job itself, not the individual jobs created inside the multibranch "folder".
The option under "Scan Multibranch Pipeline Triggers" that says "Periodically if not otherwise run" is only a trigger for when the multibranch job will scan for new branches. If changes are found to existing branches, or if new branches are discovered with a Jenkinsfile that match your branch specifications, a new build will be triggered, but this is not intended to be the way a job is triggered.
Actually, you can disable the automatic build when changes are found by adding a property to the SCM configuration to "Disable automatic SCM Triggering". Then then you will see the multibranch scan trigger, but the jobs themselves won't build, even if there are changes found.
To trigger jobs, ideally you should use a webhook if you can. If you use a git hook using the git plugin (not the github plugin), then you need to enable the PollSCM trigger (though you can set it to only poll rarely, or not at all).
If you just want normal triggering options, as of 2.22, you can configure the either cron or pollSCM triggers.
pipeline {
triggers {
cron('H/4 * * * 1-5')
pollSCM('0 0 * * 0')
}
Then I believe you can configure webhooks to inform your multibranch job when to do a scan. I haven't tried that. I just tell it to scan every hour or a couple times per day using the "Periodically if not otherwise run".
Note, the same thing applies for the build discarder and other things you configure in your multibranch job. In the web UI, you can only configure the multibranch job itself, not the individual jobs created from it. You have to use Pipeline to configure the jobs.
If your're using the JobDSL Jenkins plugin for creating jobs, then you can add following lines to configure "Scan Multibranch Pipeline Triggers":
configure {
it / 'triggers' << 'com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger'{
spec '* * * * *'
interval "60000"
}
}
Using the JobDSL Jenkins Plugin for multibranch pipeline job, the periodic folder trigger can be configured as given below. In this example, the maximum amount of time since the last indexing that is allowed to elapse before an indexing is triggered will be seven days.
multibranchPipelineJob('my-awesome-job') {
triggers {
periodicFolderTrigger {
interval("7d")
}
}
}
We are using the pipeline plugin with multibranch configuration for our CD.
We have checked in the Jenkinsfile which works off git.
git url: "$url",credentialsId:'$credentials'
The job works fine, but does not auto trigger when a change is pushed to github.
I have set up the GIT web hooks correctly.
Interestingly, when I go into a branch of the multibranch job and I click "View Configuration", I see that the "Build when a change is pushed to Github" is unchecked. There is no way to check it since I can not modify the configuration of the job (since it takes from parent) and the same option is not there in parent.
Any ideas how to fix this?
For declarative pipelines try:
pipeline {
...
triggers {
githubPush()
}
...
}
For me this enables the checkbox "GitHub hook trigger for GITScm polling", but polling is not actually required. This requires the GitHub plugin.
I found a way to check the checkbox "Build when a change is pushed to Github".
This line did the trick:
properties([pipelineTriggers([[$class: 'GitHubPushTrigger'], pollSCM('H/15 * * * *')])])
I think the polling is needed to make it work. Would be nice if no polling is needed.
Here's a Jenkinsfile example with this implemented:
#!/usr/bin/env groovy
node ('master'){
stage('Build and Test') {
properties([pipelineTriggers([[$class: 'GitHubPushTrigger'], pollSCM('H/15 * * * *')])])
checkout scm
env.PATH = "${tool 'Maven 3'}/bin:${env.PATH}"
sh 'mvn clean package'
}
}
For declarative pipelines, try this:
pipeline {
agent any
triggers {
pollSCM('') //Empty quotes tells it to build on a push
}
}
If you use Stash for example you can register a Post-Receive WebHook where you have to insert your URL form Jenkins like : http://jenkinsHost:9090/git/notifyCommit?url=ssh://git#gitHost:1234/test.git
In your jenkins Job you have to set at least the Build trigger "Poll SCM".
And set a polling time of e.g 5 mins.
This enables also the automatic branch indexing for your multibranch project configuration.
Resorting to polling adds latency - time that it takes for a build to start and hence finish giving back a result.
It seemed to me that the basic plugins have a low level of abstraction, so I switched to the Github Organization Folder plugin, which depends on all of them and sets up an organization hook for triggering builds branches and/or pull requests.
Before I start, I would like to emphasize that I had no previous experience with Jenkins so far, so there might be a bunch of better solutions out there.
What I wanted to achieve in a nutshell:
After every push made to a Bitbucket repo(test2), on every branch,
pull and build another Bitbucket repo(test1), from an identical
branch name and right after that, build test2 using test1 as a
dependency.
How I managed to achieve that?
I started a new job with type 'Multibranch Pipeline'
I added the following Jenkinsfile to test2:
pipeline {
agent any
stages {
stage('build') {
steps {
dir('test1') {
git branch: BRANCH_NAME, url: 'git#bitbucket.org:user/test1.git', credentialsId: 'credentials_id'
}
sh('build_process')
}
}
}
}
I come across the issue that you can't set up a Bitbucket hook for pipelines
I added Bitbucket Branch Source Plugin to Jenkins
I selected Bitbucket at 'Branch Sources' when setting up the job
I added credentials and put a checkmark to Auto-register webhook
Under 'Scan Multibranch Pipeline Triggers' I put a checkmark to Periodically if not otherwise run, with an interval of 1 min
I added a webhook to my Bitbucket repo
I updated all my plugins, restarted Jenkins and it's ready to go
Other plugins I have installed: Bitbucket Plugin, Pipeline plugin. Hope this helps for somebody, I did manage to solve it this way after hours of struggling without the Bitbucket Branch Source Plugin.
node{
stage('Build and Test') {
properties([pipelineTriggers([[$class: 'GitHubPushTrigger'], pollSCM('* * * * *')])])
checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'xxx-xxx-xxxx[your credentails Id]', url: 'https://github.com/git']]])
echo 'Build and Test has been done'
}
}