Jenkins 2 dimensional array parallel stages - jenkins

Question

I am trying to implement Jenkins 2D parallel stages.
This is the example that I am testing.
I expect 12 different stages which are A1/B1/C1/A2/B3/C2/A3/B3/C3/A4/B4/C4 and they are executing 3 parallel builds and each build has 4 serial build stages.
build_list = ['B1', 'B2', 'B3']
l1 = ['A1', 'A2', 'A3', 'A4']
l2 = ['B1', 'B2', 'B3', 'B4']
l3 = ['C1', 'C2', 'C3', 'C4']
array_build_list = [ l1, l2, l3 ]
index = 1
index = -index
def parallelStagesMap = build_list.collectEntries {
index++
["${it}" : generateStage(it, index)]
}
def generateStage(job, index) {
return {
stage("${job}") {
echo "This is ${job}"
script {
build_project_list = array_build_list[index]
echo "build_project_list = ${build_project_list}"
for(int i=0; i<build_project_list.size(); i++) {
stage(build_project_list[i]) {
echo "${build_project_list[i]}"
}
}
}
}
}
}
pipeline {
agent any
stages {
stage("start") {
steps {
echo "start"
}
}
stage('parallel stage') {
steps {
script {
echo "Inside script"
parallel parallelStagesMap
}
}
}
}
}
But this is the build result:
Result
The result stages are C1/C1/C1/C2/C2/C2/C3/C3/C3/C4/C4/C4 which means only the last parallel run is repeated 3 times.
Blue ocean Result
Do you have any suggestions or comments?
This is the console output:
[Pipeline] Start of Pipeline
[Pipeline] echo
After value B1 at index 0
[Pipeline] echo
After value B2 at index 1
[Pipeline] echo
After value B3 at index 2
[Pipeline] node
Running on Jenkins in /Users/.jenkins/workspace/2D_array_test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (start)
[Pipeline] echo
start
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (parallel stage)
[Pipeline] script
[Pipeline] {
[Pipeline] echo
Inside script
[Pipeline] parallel
[Pipeline] { (Branch: B1)
[Pipeline] { (Branch: B2)
[Pipeline] { (Branch: B3)
[Pipeline] stage
[Pipeline] { (B1)
[Pipeline] stage
[Pipeline] { (B2)
[Pipeline] stage
[Pipeline] { (B3)
[Pipeline] echo
This is B1
[Pipeline] sh
[Pipeline] echo
This is B2
[Pipeline] sh
[Pipeline] echo
This is B3
[Pipeline] sh
[Pipeline] script
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] echo
build_project_list = [A1, A2, A3, A4]
[Pipeline] echo
build_project_list = [B1, B2, B3, B4]
[Pipeline] echo
build_project_list = [C1, C2, C3, C4]
[Pipeline] stage
[Pipeline] { (C1)
[Pipeline] stage
[Pipeline] { (C1)
[Pipeline] stage
[Pipeline] { (C1)
[Pipeline] echo
C1
[Pipeline] }
[Pipeline] echo
C1
[Pipeline] }
[Pipeline] echo
C1
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (C2)
[Pipeline] stage
[Pipeline] { (C2)
[Pipeline] stage
[Pipeline] { (C2)
[Pipeline] echo
C2
[Pipeline] }
[Pipeline] echo
C2
[Pipeline] }
[Pipeline] echo
C2
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (C3)
[Pipeline] stage
[Pipeline] { (C3)
[Pipeline] stage
[Pipeline] { (C3)
[Pipeline] echo
C3
[Pipeline] }
[Pipeline] echo
C3
[Pipeline] }
[Pipeline] echo
C3
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (C4)
[Pipeline] stage
[Pipeline] { (C4)
[Pipeline] stage
[Pipeline] { (C4)
[Pipeline] echo
C4
[Pipeline] }
[Pipeline] echo
C4
[Pipeline] }
[Pipeline] echo
C4
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // script
[Pipeline] // script
[Pipeline] // script
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Answer 1

Try refactoring your code like shown below.
build_list = ['B1X', 'B2X', 'B3X']
l1 = ['A1', 'A2', 'A3', 'A4']
l2 = ['B1', 'B2', 'B3', 'B4']
l3 = ['C1', 'C2', 'C3', 'C4']
array_build_list = [ l1, l2, l3 ]
index = 1
index = -index
def parallelStagesMap = build_list.collectEntries {
index++
["${it}" : generateStage(it, index)]
}
def generateStage(job, index) {
return {
stage("${job}") {
echo "This is ${job}"
script {
build_project_list = array_build_list[index]
echo "build_project_list = ${build_project_list}"
runStages(build_project_list)
}
}
}
}
def runStages(def build){
for(int i=0; i<build.size(); i++) {
stage(build[i]) {
echo "Build ~~~~~ ${build[i]}"
}
}
}
pipeline {
agent any
stages {
stage("start") {
steps {
echo "start"
}
}
stage('parallel stage') {
steps {
script {
echo "Inside script"
parallel parallelStagesMap
}
}
}
}
}
Explanation
Basically, this has to do with the variable scopes. In your pipeline, the culprit is this line build_project_list = array_build_list[index] Since you don't have def in front of the variable declaration the build_project_list becomes a global variable that binds to the scope of the script. So when things execute in parallel all the stages will update this variable in parallel.
So there are two ways to fix this, method one is the one I added above, by scoping the variables by moving them to a new function. Method two is simply adding def key work to the variable build_project_list. Something like below.
def build_project_list = array_build_list[index]