Run Jenkins Pipelines jobs in parallel from closure - jenkins

I have a Jenkins server using the Pipeline plugin. In this, I want to launch several builds in parallel, and wait for the builds to complete before moving on to the next stage of my pipeline.
I am able to do this successfully, if I write out the build jobs explicitly, like so:
parallel 'one': {
build job: 'job1',
parameters: [
[$class: 'StringParameterValue', name: 'CONFIG', value: "foo"]
],
propagate: false,
wait: true
}, 'two': {
build job: 'job2',
parameters: [
[$class: 'StringParameterValue', name: 'CONFIG', value: "foo"]
],
propagate: false,
wait: true
}, 'three': {
build job: 'job3',
parameters: [
[$class: 'StringParameterValue', name: 'CONFIG', value: "foo"]
],
propagate: false,
wait: true
}
However, in reality, there will be a variable number of jobs that need to be built, so explicitly writing these out is not feasible. I have tried to wrap the builds in a closure like so:
def createParallel = { String parallelName ->
parallelName: {
build job: 'jobX',
parameters: [
[$class: 'StringParameterValue', name: 'CONFIG', value: "foo"]
],
propagate: false,
wait: true
}
}
parallel (
createParallel('one'),
createParallel('two'),
createParallel('three')
)
The problem with this method is the builds do not actually kick off in parallel - They build one at a time, waiting for the previous build to complete. What am I doing wrong?

You are not specifying any build parameters, so Jenkins coalesces the queue items, by design.

Set wait: false and build never wait for another build.But if you need to trigger another stage ,you have to get the result

The short answer is that presently this cannot be done in Jenkins, due to bugs JENKINS-33051 and JENKINS-25979.
What I was doing with the above was essentially creating three parallels, each with a single parameter. What needs to be done is place the closures in a list, and then spread the list as parameters, like so:
def list = [ createParallel('one'), createParallel('two'), createParallel('three') ]
parallel (*list)
Unfortunately, this is not implemented in Jenkins right now.

Related

Missing parameters from downstram jobs after Jenkins upgrade

I upgraded my jenkins recently from 2.164.3 to 2.249.3 version and also upgraded the build pipeline plugin in addition to many other plugins.
and since than my E2E pipe broke on parameters that are passed to downstran job from upstream job.
whay my pipeline does is something like that:
UpstreamJob
stage('Job_1') {
stage('Create') {
steps {
//whatever the job_1 does
}
}
}
stage('Job_2'){
steps {
script {
try {
create_cluster_build_id = Job_1.getId()
} catch (err) {
println "No id was found"
}
down_stream_job_2 = build job: 'Job_2', wait: true, propagate: true, parameters: [
[$class: 'StringParameterValue', name: 'Branch', value: branch_Name],
[$class: 'StringParameterValue', name: 'Docker_Tag', value: docker_tag],
[$class: 'StringParameterValue', name: 'upstream_name', value: env.JOB_NAME],
[$class: 'StringParameterValue', name: 'upstream_build_id', value: env.BUILD_NUMBER],
[$class: 'StringParameterValue', name: 'upstream_build_url', value: env.BUILD_URL],
[$class: 'StringParameterValue', name: 'Job_1_id', value: Job_1_id],
]
}
}
and the error I get inside job2 is
groovy.lang.MissingPropertyException: No such property: Job1_id for class: groovy.lang.Binding
Now...
1)Iv'e found a documentation for similar issue - JENKINS-34871
2)as WA I tried to add the parameter manually in the job2 configuration (in old jenkins configuration it was not part of the job parameters) , it worked but... after several runs the parameter was gone again although I saved it in the job2 configuration.
So my question to the jenkins experts outthere are:
How can it be that I add parameter to a job, save and apply, and after few runs it (the parameter) dissappears?
Found a solution in this thread - Pre-Defined parameters no longer passed to child job
Copy this line to Script console
System.setProperty("hudson.model.ParametersAction.keepUndefinedParameters", "true")

Is there an option to call Jenkinsfile from Jenkins job (like how we specify script path in pipeline)?

If the above is not possible, is there a way to write groovy script to call Jenkinsfiles within Jenkins job
You can call Job B from Job A as a downstream job in pipeline and even pass parameters to it like so:
build job: 'CloudBees/Folder/To/JobB', propagate: false, wait: false, parameters: [[$class: 'StringParameterValue', name: 'PARAM_NAME', value: "${env.SOMEVALUE}"]]
I use this to only call Job B from Job A at the end of Job A in a post build success clause. The success clause only executes if Job A still has a job status of SUCCESS at that point.
post {
success {
script {
build job: 'CloudBees/Folder/To/JobB', propagate: false, wait: false, parameters: [[$class: 'StringParameterValue', name: 'PARAM_NAME', value: "${env.SOMEVALUE}"]]
}
}
}

Call parameterized Jenkins pipeline from another pipeline

There is any way to trigger a pipeline job from another pipeline with parameters, i already tried
build job: '/myjob', parameters: [string(name: 'param1', value:'val1')], wait: false
also tried
build job: 'myjob', parameters: [string(name: 'param1', value:'val1')], wait: false
and
build job: 'myjob', parameters: [[$class: 'StringParameterValue', name: 'param1', value: 'val1']], wait: false
with no luck, it says:
Item type does not support parameters
Since the subjob was another multibranch pipeline project i needed to specify the branch i wanted to run so with
build job: 'myjob/master', parameters: [string(name: 'param1', value:'val1')], wait: false
it now works
Depending on your Jenkins job / pipeline structure, you should prefix the job with "../" e.g.:
build job: '../myjob/master', parameters: [string(name: 'param1', value:'val1')], wait: false
The below worked for me in order to pass parameters "test_1" and "test_2" from pipeline "master" to pipeline "sub-1"
In the master pipeline
build job: 'sub-1', parameters: [[$class: 'StringParameterValue', name: 'test_1', value: 'nameValue'], [$class: 'StringParameterValue', name: 'test_2', value: 'valueValue']], wait: true
In the sub pipeline "sub-1" use by referencing the "params" variable
node {
echo params.test_1
echo params.test_2
}
Reference:
https://support.cloudbees.com/hc/en-us/articles/221400287-How-to-pass-parameter-to-downstream-job-in-Pipeline-job-

Jenkins: MatrixCombinationsParameterValue from a pipeline

I'm want to start a matrix-build from a pipeline job but I want to build only one Axis.
I tried with this:
build job: "Build_Android_Matrix", propagate: false, wait: true,
parameters: [[$class: 'StringParameterValue', name: 'branch', value: "$branch"],
[$class: 'BooleanParameterValue', name: 'production', value: true],
[$class: 'BooleanParameterValue', name: 'beta', value: false],
[$class: 'MatrixCombinationsParameterValue', name: 'paramFilter', description: null, combinations: ['buildType=Release']]]
I have 2 Axes, flavor and buildType, and paramFilter is the matrix combinations parameter.
The matrix-build starts with all the job parameters but it builds nothing because the matrix combinations selection is empty.
I've also tried with ['buildType==Release'] and ['buildType=="Release"'] but I always get the same result.
I've also tried with:
build job: "Build_Android_Matrix", propagate: false, wait: true, parameters: [
new hudson.plugins.matrix_configuration_parameter.MatrixCombinationsParameterValue
("paramFilter",
null,
['buildType=Release'])
]
but it fails because RejectedAccessException: Scripts not permitted to use new.
I'm almost sure that I'm not providing the combinations in the right way but I don't know what else I can try.
Update
After Christopher Orr answer I tried to set the parameters like this:
[$class: 'MatrixCombinationsParameterValue', name: 'paramFilter', description: null, combinations: ['buildType=Release,flavor=Italy']]]
with this as my Axes:
flavor: Germany Italy Mexico UnitedStates
buildType: Debug Release
And was not working because I forgot that I have also a Slaves Axis and that must be specified as well.
So this is what worked for me:
[$class: 'MatrixCombinationsParameterValue', combinations: ["buildType=Release,flavor=Italy,label=android"], description: '', name: 'paramFilter']
When you use the Matrix Combinations plugin from the web UI, you need to explicitly specify all of the combinations that you want to run. So in Pipeline you need to do the same, for example:
combinations: ['buildType=Release,flavor=beta',
'buildType=Release,flavor=production']
Order of parameters matters.

How to migrate from build flow plugin to pipeline

I have a build flow similar to below code
parallel (
{
build("job1A")
build("job1B")
build("job1C")
},
{
build("job2A")
build("job2B")
build("job2C")
}
)
How can I make the same flow run in a pipeline job
It is really simple, you can use the parallel step:
def jobs = [
"part1": {
build("job1A")
build("job1B")
build("job1C")
},
"part2": {
build("job2A")
build("job2B")
build("job2C")
}
]
parallel jobs
If you don't know how to write the pipeline script refer to the Snippet Generator.
Parameterized job calls:
build job: 'job1A', parameters: [[$class: 'StringParameterValue', name: 'Test', value: 'testvalue']]
More details about the classes can be found in the reference.

Resources