Jenkins version 2.121.2
I have a multi-branch pipeline set up. I am using a declarative Jenkinsfile.
I have a set of tests which take a long time to run. I want these to run over night for any branches which have changes.
I have tried a few things but my current failing attempt is:
Under the job > configure, I have enabled 'Suppress automatic SCM triggering'
Have 'Scan Multibranch Pipeline Triggers' > 'Periodically if not otherwise run' set to 1 minute (just for testing, I will increase this when it is working)
In my Jenkinsfile (example for a 4am run), I have also tried with pollSCM():
triggers {
cron('0 4 * * *')
}
In the 'Scan multibranch pipeline log' I see the following but no job runs at 4am (time in the trigger() in my Jenkinsfile):
Changes detected: my-feature-branch (1234567890abcdefgh → abcdefgh123456789)
Did not schedule build for branch: my-feature-branch
What am I doing wrong please?
Edit:
So I've tried this set up instead:
Set the cron to every 15 minutes
triggers {
cron('5,20,35,50 * * * *')
}
Removed the setting under configure in the UI 'Suppress automatic SCM triggering'
But it just starts running the minute polling has happened (16 minutes past the hour in this test).
What ever I do nothing seems to pay attention to my cron settings?
If I got to 'View configuration' under the branch job in the UI it shows the UI settings from my Jenkinsfile ok.
Edit (again!):
So with the last edit, it did actually run immediately and then again at the cron time.
Now enabled again in the UI the setting 'Suppress automatic SCM triggering'.
And I have it working! The main issue I realised (a) changes are not applied I do not think until the run after the first run with a change in the Jenkinsfile? (b) Also I installed the next execution plugin so I can see what it is planning on better.
The issue here was that trigger declared in multibranchPipelineJob is for scanning multibranch. To run job periodically declare trigger in pipeline like this:
pipeline {
triggers {
cron('45 6 * * 1-5')
}
agent {
...
Related
I can call another jenkins job using the build command. Is there a way I can tell another job to do a branch scan?
A multibranch pipeline job has a UI button "Scan Repository Now". When you press this button, it will do a checkout of the configured SCM repository and detect all the branches and create subjobs for each branch.
I have a multibranch pipeline job for which I have selected the "Suppress automatic SCM triggering" option because I only want it to run when I call it from another job. Because this option is selected, the multibranch pipeline doesn't automatically detect when new branches are added to the repository. (If I click "Scan Repository Now" in the UI it will detect them.)
Essentially I have a multibranch pipeline job and I want to call it from another multibranch pipeline job that uses the same git repository.
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
//scanScm "../my-other-multibranch-job" <--- scanScm is a fake command I made up
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
I get an error on that build line, because the target multibranch pipeline job does not yet know that BRANCH_NAME exists. I need a way to trigger an SCM re-scan in the target job from this current job.
Similar to what you figured out yourself, I can contribute my optimization that actually waits until the scan has finished (but is subject to Script Security):
// Helper functions to trigger branch indexing for a certain multibranch project.
// The permissions that this needs are pretty evil.. but there's currently no other choice
//
// Required permissions:
// - method jenkins.model.Jenkins getItemByFullName java.lang.String
// - staticMethod jenkins.model.Jenkins getInstance
//
// See:
// https://github.com/jenkinsci/pipeline-build-step-plugin/blob/3ff14391fe27c8ee9ccea9ba1977131fe3b26dbe/src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerStepExecution.java#L66
// https://stackoverflow.com/questions/41579229/triggering-branch-indexing-on-multibranch-pipelines-jenkins-git
void scanMultiBranchAndWaitForJob(String multibranchProject, String branch) {
String job = "${multibranchProject}/${branch}"
// the `build` step does not support waiting for branch indexing (ComputedFolder job type),
// so we need some black magic to poll and wait until the expected job appears
build job: multibranchProject, wait: false
echo "Waiting for job '${job}' to appear..."
while (Jenkins.instance.getItemByFullName(job) == null || Jenkins.instance.getItemByFullName(job).isDisabled()) {
sleep 3
}
}
Ended up figuring this out shortly after posting the question. Calling build against the base multibranch pipeline job as opposed to a branch causes it to re-scan. The solution to my above snippet would have ended up looking something like...
node {
if(env.BRANCH_NAME == "the-branch-I-want" && other_criteria) {
build job: "../my-other-multibranch-job", wait: false, propagate: false // scan for branches
sleep 2 // scanning takes time
build "../my-other-multibranch-job/${env.BRANCH_NAME}"
The wait: false is important because otherwise you get "ERROR: Waiting for non-job items is not supported". The multibranch "parent" job is closer to a folder than a job, but it's a folder that supports the build command, and it does so by scanning the SCM.
But solving this just led to another problem, which is that with wait: false we have no way of knowing when the SCM Scan finished. If you have a large repository (or you're short on jenkins agents), the branch won't get discovered until after the second build command has already failed due to the branch not existing. You could bump the sleep time even higher, but that doesn't scale.
Fortunately, it turns out manually initiating the SCM scan isn't even needed if you have github webhooks set up for your jenkins. The branch will be discovered more-or-less instantly, so for my purposes this is solved another way. The reason I was running into it is we don't have webhooks set up in our dev jenkins, but once I move this code to prod it will work fine.
If you're trying to use JobDSL to set up multibranches calling multibranches and you don't have webhooks or something equivalent, the better path is probably to abandon multibranch for your second tier of jobs and use JobDSL to create folders and manage the branch jobs yourself.
We use CloudBees-2.138.2.2 Jenkins and have had lot of challenges to trigger jenkins jobs only based on commits in our Github repo. I am still looking for the exact working script for scripted pipeline and not declarative one.
so For example :- pipelineTriggers([cron('0-59/2 * * * *')]), works
but pipelineTriggers([pollSCM('0-59/2 * * * *')]), never works despite new commits
was able to make it work using :-
pipelineTriggers([pollSCM('* * * * *')]) under properties
This way the job polls the repo every minute and will trigger build only when it detects a new commit.
You can try adding the following in your scripted pipeline
properties([pipelineTriggers([githubPush()])])
Right now I manually configure my all my multibranch pipeline jobs and set "Scan Multibranch Pipeline Triggers" to 3 minutes.
How do I put this in my Jenkinsfile? I can't find examples of this. Is "Scan Multibranch Pipeline Triggers" available in the triggers{} block?
The settings on the multibranch configuration page only configure the multibranch scan job itself, not the individual jobs created inside the multibranch "folder".
The option under "Scan Multibranch Pipeline Triggers" that says "Periodically if not otherwise run" is only a trigger for when the multibranch job will scan for new branches. If changes are found to existing branches, or if new branches are discovered with a Jenkinsfile that match your branch specifications, a new build will be triggered, but this is not intended to be the way a job is triggered.
Actually, you can disable the automatic build when changes are found by adding a property to the SCM configuration to "Disable automatic SCM Triggering". Then then you will see the multibranch scan trigger, but the jobs themselves won't build, even if there are changes found.
To trigger jobs, ideally you should use a webhook if you can. If you use a git hook using the git plugin (not the github plugin), then you need to enable the PollSCM trigger (though you can set it to only poll rarely, or not at all).
If you just want normal triggering options, as of 2.22, you can configure the either cron or pollSCM triggers.
pipeline {
triggers {
cron('H/4 * * * 1-5')
pollSCM('0 0 * * 0')
}
Then I believe you can configure webhooks to inform your multibranch job when to do a scan. I haven't tried that. I just tell it to scan every hour or a couple times per day using the "Periodically if not otherwise run".
Note, the same thing applies for the build discarder and other things you configure in your multibranch job. In the web UI, you can only configure the multibranch job itself, not the individual jobs created from it. You have to use Pipeline to configure the jobs.
If your're using the JobDSL Jenkins plugin for creating jobs, then you can add following lines to configure "Scan Multibranch Pipeline Triggers":
configure {
it / 'triggers' << 'com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger'{
spec '* * * * *'
interval "60000"
}
}
Using the JobDSL Jenkins Plugin for multibranch pipeline job, the periodic folder trigger can be configured as given below. In this example, the maximum amount of time since the last indexing that is allowed to elapse before an indexing is triggered will be seven days.
multibranchPipelineJob('my-awesome-job') {
triggers {
periodicFolderTrigger {
interval("7d")
}
}
}
Given a Jenkins multibranch pipeline job that uses a property strategy to "suppress automatic SCM triggering" for all branches but 'default', how do you allow Jenkins to wait until night (say 7pm-6am) to build every other branch?
We used to be able to set the Poll SCM strategy for each job individually, which worked well.
Pipeline scripts allow you to set a pollSCM pipeline trigger property. However it won't get set unless the job has run at least once and there seems to be a defect where jobs are continuously triggered by scm changes, making it less useful.
Jenkinsfile properties can (now) configure polling triggers and override the default triggering behaviour. This example enables daily builds for everything except 'default' and release branches (which are always built)
def alwaysBuild = (env.BRANCH_NAME == "default" || env.BRANCH_NAME ==~ /release-.*/);
properties([
overrideIndexTriggers(alwaysBuild),
pipelineTriggers([pollSCM('#daily')])
]);
NOTE: As of 2016-Sept, there seems to be a bug where pollSCM triggers multiple builds per change. Possibly this bug: https://issues.jenkins-ci.org/browse/JENKINS-38443
I have one pipeline build called, for example UAT. This build is scheduled every 3 minutes. And another build called DEV. DEV is scheduled every minute. The task is: to run UAT only if the last DEV execution was SUCCESS. If not - skip the execution. And run it after other 3 minutes with the same condition.
How can I achieve that ?
Don't schedule your UAT job as a separate job but instead trigger the launch once your first DEV pipeline finishes with success.
As you are using pipelines you actually have 2 solutions :
1)
Don't call another job but just call a Groovy function to integrate the DEV part, such as :
node() {
stage "UAT"
// Your existing UAT pipeline content here
stage "DEV"
git 'http://urlToYourGit/projectContainingYourDevScript'
pipeline = load 'functions.groovy'
pipeline.dev()
}
2) Just call a second Jenkins job with this kind of line :
node() {
stage "UAT"
// Your existing UAT pipeline content here
build job: "dev-job"
}
With these 2 solutions you can configure your first job to run every minute and it will trigger the second part/job only if the first one finishes with success (otherwise Jenkins will just fail the build as it would normally do).