I have a Jenkins parallel build issue. I have a large configuration options blob, and I want to call the same build job repeatedly with changes to the config blob. I also want to delay a little bit between each run. Also, the number of calls is based off selections so could be anywhere from 1 job to say 7 jobs at once hence building it grammatically. Given a much paired down version of the code below assuming 'opts' are what is coming in from selection, could someone help me accomplish this. The long job has a timestamp inside it and so we don't want them all kicking off at the exact same moment.
Without the 'sleep' I will see "Starting building: myLongJob#100", then 101, then 102 etc.. However there is no delay between the jobs. With the sleep I see "Starting building: myLongJob #100" then 100, then 100 again.
Other than the given code, I tried adding quietPeriod: diffnum+1 but that either did nothing or waited until the long job finished. I also tried adding wait: true (and wait: false). Still I see no delay or the same job build number repeatedly.
cfgOtps = """
server: MYOPTION
prefix: Testing
"""
def opts = ["Windows", "Linux", "Mac"]
// function will create a list of cfg blobs with server replaced by 'opt' so in this example
// cfgs will be length of 3 with server beign Windows, Linux, and Mac respectively
def cfgs = generate_cfg_list(cfgOtps, opts)
//update so each entry in branches has a different key
def branchnum = 0
def branches = [:]
//variable to have different inside the closure
def diffnum = -1
def runco = ""
cfgs.each {
branchnum += 1
branches["branch${branchnum}"] = {
diffnum += 1
//sleep(diffnum+5)
def deployResult = build job: 'myLongJob', parameters: [[$class: 'TextParameterValue', name: 'ConfigObj', value: cfgs[diffnum]],
string(name:'dummy', value: "${diffnum}")]
}
}
parallel branches
I expected the output to be something like the following with a short delay between them.
Starting building: myLongJob #100
Starting building: myLongJob #101
Starting building: myLongJob #102
Which I do get if I do not have the sleep. The issue there is the long job needs to not be run at the same time so sometimes overwrites things.
Adding the sleep results in the following
Starting building: myLongJob #100
Starting building: myLongJob #100
Starting building: myLongJob #100
or maybe 2 with the same number and one a different build number. Not sure why a sleep would induce that behavior?
Related
I am developing software for an embedded device. The steps involved in building and verifying it all are complicated: creating the build environment (via containers), building the actual SD card image, running unit tests, automated tests on target hardware, license compliance checks and so on - details aren't important here.
Currently I have this in one long declarative Jenkinsfile as a multibranch-pipeline (for all intents and purpose here, we're doing gitflow). In doing this I've hit a limit on the size of a Jenkinsfile (https://issues.jenkins.io/browse/JENKINS-37984) and can't actually get all the stages in that I want to.
It's too big so i need to cut this massive pipeline up. I broke this all up in little pipeline jobs with parameters to pass data/context between each part of the pipeline and came up with something like this:
I've colour-coded the A and B artifacts as they're used a lot and the lines would make things messy. What this tries to show is an order of running things, where things in a column depend on artifacts created in column to the left.
I'm struggling to discover how to do the "waiting" for multiple upstream jobs (for instance in Job Foxtrot in the diagram) before starting another downstream job that depends on them.
I specifically do not want to turn each column in the diagram into a parallel group of things, because for instance Job Delta might take 2 minutes but Job Charlie take 20 minutes. The exact duration of each job is variable and unpredictable as for some parameter combinations will mean building from scratch and others will cause an existing artifact to be output.
I think I need something like the join plugin (https://plugins.jenkins.io/join/), but for pipeline jobs (join only works on freestyle jobs and is quite aged).
The one approach I've explored is to have a "controller" job (maybe job Alpha in the diagram?) that uses the build step (https://www.jenkins.io/doc/pipeline/steps/pipeline-build-step/_) with the wait parameter set to false to trigger the downstream jobs in correct order, with the correct parameters. It would involve searching Jenkins.instance.getItems() to locate the Runs for the downstream projects, which have an upstream cause that matches the currently executing "controller" job. This involves polling waiting for the job to appear and then polling for the job to complete. This feels like I'm "doing it wrong". Below is the source for this polling approach - be gentle, i'm new to groovy!
Is this polling approach a good way? What problems could I encounter with this approach? Should I be using the ItemListener Jenkins ExtensionPoint and writing a plugin to do this sort of thing in a generic way? Is there another way I've not found?
I feel like I'm not "holding it right" when it comes to the overall pipeline design/architecture here.
Finally after writing this I notice that Jobs India, Juliet and Kilo could be collapsed into a single Job, but I don't think that solve much.
#NonCPS
Integer getTriggeredBuildNumber(String project, String causeJobName, Integer causeBuildNumber) {
//find the job/project first
def job = Jenkins.instance.getAllItems(org.jenkinsci.plugins.workflow.job.WorkflowJob.class).find { job -> job.getFullName() == project }
//find a build for this job that was caused by the current build
def build = job.getBuilds().find { build ->
build.getCauses().findAll{ it.class == hudson.model.Cause.UpstreamCause.class }.find { cause ->
cause.getUpstreamProject() == causeJobName && cause.getUpstreamBuild() == causeBuildNumber
} != null
}
if(build != null) {
return build.getNumber()
} else {
return -1
}
}
#NonCPS
Boolean isBuildComplete(String jobName, Integer buildNumber) {
def job = Jenkins.instance.getAllItems(org.jenkinsci.plugins.workflow.job.WorkflowJob.class).find { job -> job.getFullName() == jobName }
if(job) {
def build = job.getBuildByNumber(buildNumber)
return build.isBuilding() == false && build.getResult() != null
} else {
println "WARNING: job '" + jobName + "' not found."
return false
}
}
We've hit the "Code too large" too many times, but the way to cope with it is to refactor your pipeline to remain deep under the limit. The following may be used:
You can run a combination of scripted and declarative pipeline. So some stages in the beginning and/or in the end may be refactored out.
You can build some of the parallel stages dynamically. This code would not be counted towards the limited code size.
Lastly, the issue mentions transformation variables, and that can help too.
We used the combination of the above and have expanded our pipeline well beyond what it was when we first encountered the issue you're facing.
have a pipeline script that executes child jobs in parallel.
Say I have 5 data (a,b,c,d,e) that has to be executed on 3 jobs (J1, J2, J3)
My pipeline script is in the below format
for (int i = 0; i < size; i++) { def index = i branches["branch${i}"] = { build job: 'SampleJob', parameters: [ string(name: 'param1', value:'${data}'), string(name:'dummy', value: "${index}")] } } parallel branches
My problem is, say the execution is happening on Job 1 with the data 1,2,3,4,5 and if the data 3 execution is failed on Job 1 then the data 3 execution should be stopped there itself and should not happen on the subsequent parallel execution on Jobs 2 and 3.
Is there any way that I can read the execution status of parallelly execution job status on the Pipeline script so that I can restrict data 3 execution to block in Jobs 2 and 3.
I am quite blocked here for a long time. Hoping for a solution from my community. Thanks a lot in advance.
In summary, it sounds like you want to
run multiple jobs in parallel against different pieces of data. I will call the set of related jobs the "batch".
avoid starting a queued job if any of the jobs in the batch have failed
automatically abort a running job if any of the jobs in the batch have failed
The jobs need some way to communicate their failure to the others. Use a shared storage location to store the "failure flag". If the file exists, then one or more of the jobs have failed.
For example, a shared NFS path: /shared/jenkins/jobstate/<BATCH_ID>/failed
At the start of the job, check for the existence of this path. Exit if it does. The file doesn't necessarily need to contain any data - its presence is enough.
Since you need running jobs to abort early if the failure flag exists, you will need to poll that location periodically. For example, after each unit of work. Again, if the file exists then exit early.
If you don't use NFS, that's ok. You could also use an object storage bucket. The important thing is that the state is accessible to all the relevant build jobs.
I'm new to Jenkins pipeline and trying to understand how can I throttle an "entire" pipeline, which basically means that the following will take place:
1) I will be able to run the same pipeline maximum number of concurrent runs, say MAX_CONCURRENT_RUNS = 2
2) Each run (essentially build) can have its own parameters, with the following "extra requirement", that two (or more) different builds CAN have (if required) the same parameters sent to it.
3) In the case where at a particular point in time there are already MAX_CONCURRENT_RUNS builds (runs) of the pipeline, then the MAX_CONCURRENT_RUNS + 1 run will "hold" itself until the first currently running build will terminate and only then will start to execute.
I have looked in this SO question and also this SO question, but they both not "exactly" applicable to my situation (requirements).
I'm using Jenkins server version 2.176.1
After some research I did mainly in these two links:
The throttle plugin official GitHub page and JENKINS-45140 issue where some of the comments were very useful, I have composed this solution:
1) First thing is install the required plugin, that can be found in the Manage Jenkins --> Manage Plugins "search tab" by typing throttle-concurrents (the official plugin page can be found here).
2) A "simple" throttle category needs to be added to the global configuration of the "throttle builder plugin" within Jenkins' global configuration. This can be done by going to Manage Jenkins --> Configure system. There under the "Throttle Concurrent Builds" section the "new" category needs to be added. In the below example, I have set the name of the category to: simpleThrottleCatagory and the following parameters:
This way, the pipeline that would be able to run several builds at the same time, with some "upper limit" on how many builds, which is essentially the MAX_CONCURRENT_RUNS (in this case 2).
3) In this example I will keep the pipeline "itself" implementation "as simple as possible" in order to focus on the "throttling" considerations and not the "common pipeline stuff".
3.1) The "simple concurrent pipeline" will simply receive two parameters from the user:
Number of seconds to sleep:NumSecToSleep.
Some sample choice parameter named BocaOrRiver with two possible values: boca or river.
3.2) The entire pipeline implementation in this case is as follows (note that some extra "approvals" needs to take place so that Calendar.getInstance().getTime().format('YYYY/MM/dd-hh:mm:ss',TimeZone.getTimeZone('CST')) function will work. In case you are unable to perform these changes, replace the two lines with this function call with any other implementation that will get the current time stamp):
// Do NOT place within the pipeline block
properties([ [ $class: 'ThrottleJobProperty',
categories: ['simpleThrottleCatagory'],
limitOneJobWithMatchingParams: false,
maxConcurrentPerNode: 2,
maxConcurrentTotal: 2,
paramsToUseForLimit: '',
throttleEnabled: true,
throttleOption: 'category' ] ])
pipeline
{
agent any parameters
{
string(name: "NumSecToSleep", description: "Number of second to sleep in the Sleep stage")
choice(name: "BocaOrRiver", choices: "boca\nriver", description: "Which Team in Buenos Aires do you prefer?")
}
stages
{
stage("First stage")
{
steps
{
echo "WORKSPACE is:${WORKSPACE}"
echo "Build number is:${env.BUILD_NUMBER}"
}
}
stage("Sleep stage")
{
steps
{
script
{
def time = params.NumSecToSleep echo "Sleeping for ${params.NumSecToSleep} seconds"
def timeStamp = Calendar.getInstance().getTime().format('YYYY/MM/dd-hh:mm:ss',TimeZone.getTimeZone('CST'))
println("Before sleeping current time is:" + timeStamp)
sleep time.toInteger() // seconds
timeStamp = Calendar.getInstance().getTime().format('YYYY/MM/dd-hh:mm:ss',TimeZone.getTimeZone('CST'))
println("After sleeping current time is:" + timeStamp)
echo "Done sleeping for ${params.NumSecToSleep} seconds"
}
}
}
}
}
3.3) NOTES:
3.3.1) The code within the actual pipeline block is essentially straight forward: Simply display some "build specific" parameters just to be sure that each build of the job gets its specific user defined parameters and it sleeps for some number of seconds also so that two (in this case) or more builds can be run indeed concurrently and it would be able to see "for our own eyes" (at run time) that the two jobs run together (in parallel).
3.3.2) The more interesting part of the pipeline is the properties block (at the top):
3.3.2.1) Note that it needs to be defined OUTSIDE of the pipeline block section.
3.3.2.2) I think that most of the settings defined within this properties block are very "self explanatory" YET the two that should be mentioned are:
$class: 'ThrottleJobProperty': This is a "predefined" value of Jenkins to indicate that this "job" (can be also pipeline) can be throttled.
categories: ['simpleThrottleCatagory']: This is the "global throttle category" defined in the previous step.
4) Basic illustration:
In the figure below there is a screen shot of a situation where three builds where started one after the other, with "enough" time to sleep in each one of them so that the first two (build 17 & 18 pointed in points 2 & 3 respectively) won't "finish too soon", meaning, so that indeed the "third" build (build 19) will "have to wait" for an available executor (pointed in point 4):
5) Here I have described a very simple and minimal yet (IMMO) representative implementation along with "global configuration" of an "entire" concurrent pipeline. Off course this topic can be discussed MUCH further, for example, it is also possible to throttle only single step within a pipeline.
Right now, I have two sets of benchmarks, a short one and a long one. The short one runs on checkin for every branch. Which set to run is a parameter - SHORT or LONG. The long one always runs nightly on the dev branch. How can I trigger other branches to build and run the long benchmark if the branch was built successfully today?
If you want to run those long tests only over the night - I find it easiest to just duplicate the job and modify it so its triggered in the night and has additional checks added after the normal job, I.e. your post-commit jobs just do the short test, the nightly triggered do the short first and then (if no errors) the long one.
I find that much easier to handle then the added complexity of chaining jobs on some condition, like evaluating time of day to skip some tests.
Example 1st job that runs after every commit
node() {
stage('Build') {
// Build
}
stage('Short Test') {
// Short Test
}
}
2nd job that triggers nightly
node() {
stage('Build') {
// Build
}
stage('Short Test') {
// Short Test, fail the build here when not successful
}
stage('Long Tests')
// Long Test, runs only when short test successful
}
}
Edit
A solution that got it all in a single job, however it adds alot of complexity and makes some followup use cases harder to integrate, i.e. different notification for the integration test branch, tracking of build durations etc. I still find it more manageable to have it split in 2 jobs.
Following job must be configured to be triggered by post commit hook and a nightly timer. It runs the long test when
the last build is younger then set (you dont want it to trigger from the last nightly),
last run was successful (dont want to run long test for a broken build), and
was triggered by said timer (dont want to trigger on a check in).
def runLongTestMaxDiffMillis = 20000
def lastRunDiff = (currentBuild.getStartTimeInMillis().toInteger() - currentBuild.getPreviousBuild().getStartTimeInMillis().toInteger())
def lastBuildTooOld = (lastRunDiff > runLongTestMaxDiffMillis)
def isTriggeredByTimer = currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause')
def lastBuildSuccessful = (currentBuild.getPreviousBuild().getResult() == 'SUCCESS')
def runLongTest = (!lastBuildTooOld && isTriggeredByTimer && lastBuildSuccessful)
node() {
if (runLongTest) {
println 'Running long test'
} else {
println 'Skipping long test'
}
}
You can create another pipeline that calls the parameterized pipeline with the LONG parameter, for example:
stage('long benchmark') {
build job: 'your-benchmark-pipeline', parameters: [string(name: 'type', value: 'LONG')]
}
When you configure this new pipeline you can tick the Build after other projects are built checkbox in the Build Triggers section and choose which short benchmarks should trigger it once they complete successfully (the default behavior).
You can use the Schedule Build Plugin to schedule a build of the long job when the short job succeed.
The short job runs on every branch, when a build succeed for a certain branch, it schedules a build of the long job (in the night) with the branch in parameter, so the long job will run on this particular branch.
I'm trying to trigger a downstream job from my current job like so
pipeline {
stages {
stage('foo') {
steps{
build job: 'my-job', propagate: true, wait: true
}
}
}
}
The purpose is to wait on the job result and fail or succeed according to that result. Jenkins is always failing with the message Waiting for non-job items is not supported . The job mentioned above does not have any parameters and is defined like the rest of my jobs, using multibranch pipeline plugin.
All i can think of is that this type of jenkins item is not supported as a build step input, but that seems counterintuitive and would prove to be a blocker to me. Can anyone confirm if this is indeed the case?
If so, can anyone suggest any workarounds?
Thank you
I actually managed to fix this by paying more attention to the definition of the build step. Since all my downstream jobs are defined as multibranch pipeline jobs, their structure is folder-like, with each item in the folder representing a separate job. Thus the correct way to call the downstream jobs was not build job: 'my-job', propagate: true, wait: true, but rather build job: "my-job/my-branch-name", propagate: true, wait: true.
Also, unrelated to the question but related to the issue at hand, make sure you always have at least one more executor free on the jenkins machine, since the wait on syntax will consume one thread for the waiting job and one for the job being waited on, and you can easily find yourself in a resource-starvation type situation.
Hope this helps
This looks like JENKINS-45443 which includes the comment
Pipeline has no support for the upstream/downstream job system, in part due to technical limitations, in part due to the fact that there is no static job configuration that would make this possible except by inspecting recent build metadata.
But it also offer the workaround:
as long as the solution is still ongoing, I include here our workaround. It is based in the rtp (Rich Text Publisher) plugin, that you should have installed to make it work:
At the end of our Jenkinsfile and after triggering the job, we wait it to finish. In that case, build() returns the object used to run the downstream job. We get the info from it.
Warning: getAbsoluteUrl() function is a critical one. Use it at your own risk!
def startedBld = build(
job: YOUR_DOWNSTREAM_JOB,
wait: true, // VERY IMPORTANT, otherwise build () does not return expected object
propagate: true
)
// Publish the started build information in the Build result
def text = '<h2>Downstream jobs</h2>Started job ' + startedBld.rawBuild.toString () + ''
rtp (nullAction: '1',parserName: 'HTML', stableText: text)
This issue is part of JENKINS-29913, opened for the past two years:
Currently DependencyGraph is limited to AbstractProject, making it impossible for Workflows to participate in upstream/downstream relationships (in cases where job chaining is required, for example due to security constraints).
It refers the RFE (Request for Enhancement) JENKINS-37718, based on another (unanswered) Stack Overflow question.