I want to execute all the stages in parallel with the loop based on user input.
This gives error because script is not allowed under stages.
How should I achieve the same?
pipeline {
agent {
node {
label 'ec2'
}
}
stages{
script{
int[] array = params.elements;
for(int i in array) {
parallel{
stage('Preparation') {
echo 'Preparation'
println(i);
}
stage('Build') {
echo 'Build'
println(i);
}
}
}
}
}
}
If you are using declarative pipelines you have two options, first is to use static parallel stages which is an integral part of the declarative syntax but does not allow dynamic or runtime modifications.
The second option (which is probably what you attempted) is to use the scripted parallel function:
parallel firstBranch: {
// do something
}, secondBranch: {
// do something else
},
failFast: true|false```
When using it inside a declarative pipeline it should be used inside a script block like you did but the declarative basic directive must still be kept: pipeline -> stages -> stage -> steps -> script. In addition the scripted parallel function receives a specifically formatted map alike the example above.
In your case it can look somethong like:
pipeline {
agent {
node {
label 'ec2'
}
}
stages {
stage('Parallel Execution') {
steps {
script {
parallel params.elements.collectEntries {
// the key of each entry is the parallel execution branch name
// and the value of each entry is the code to execute
["Iteration for ${it}" : {
stage('Preparation') {
echo 'Preparation'
println(it);
}
stage('Build') {
echo 'Build'
println(it);
}
}]
}
}
}
}
}
}
Or if you want to use the for loop:
pipeline {
agent {
node {
label 'ec2'
}
}
stages {
stage('Parallel Execution') {
steps {
script {
map executions = [:]
for(int i in params.elements) {
executions["Iteration for ${it}" ] = {
stage('Preparation') {
echo 'Preparation'
println(i);
}
stage('Build') {
echo 'Build'
println(i);
}
}]
}
parallel executions
}
}
}
}
}
Other useful examples for the parallel function can be found here
Per my header, I'm receiving the following error for my jenkins setup:
Unknown stage section "stage". Starting with version 0.5, steps in a stage must be in a 'steps' block. #line xxx, column xx.
stage('First Parallel Stage') {
^
My configuration:
pipeline {
stages {
stage('Header_1'){
steps{}
}
stage('Header_2'){
steps{}
}
parallel{
stage('First Parallel Stage'){
environment{}
}
stages {
stage('Another_One'){
steps{}
}
}
}
}
}
I've tried putting an empty steps{} in stage('First Parallel Stage') and tried putting it inside steps. I'm unsure what could be wrong.
You'll need to put stages that are grouped together into a stage and parallel must be within a stage too. Full working example:
pipeline {
agent any
stages {
stage('Header_1') {
steps {
echo '1'
}
}
stage('Header_2') {
steps {
echo '2'
}
}
stage('Parallel') { // add this
parallel {
stage('First Parallel Stage') {
environment {
TEST = 3
}
steps {
echo "$TEST"
}
}
stage('Execute this together') { // add this
stages {
stage('Another_One') {
steps {
echo "4"
}
}
stage('Yet Another_One') {
steps {
echo "5"
}
}
}
}
}
}
}
}
Please note that you can't have parallel{} inside of parallel{}, but you can chain them.
On BlueOcean it then looks like the following:
I have the following (part of a) pipeline
stages {
stage('bootstrap') {
parallel {
stage("Linux") {
agent { label 'linux' }
steps {
sh 'bootstrap.sh'
}
}
stage("Windows") {
agent { label 'Windows' }
steps {
bat 'bootstrap.bat'
}
}
}
}
stage('devenv') {
parallel {
stage('Linux') {
agent { label 'linux' }
steps {
sh 'devenv.sh'
}
}
stage('Windows') {
agent { label 'Windows' }
steps {
bat 'devenv.bat'
}
}
}
}
}
post {
always {
echo "Done"
}
}
The problem is that when bootstrap.bat fails on windows, the devenv step is now considered failed, and the linux devenv won't continue. I would like to have the results of the linux pipeline even if the windows one fails early.
An option would be to separate the stages so that linux full pipeline is on one branch of the parallel execution, and windows is on the other, but maybe there's a trick I am not aware of, because I tried it and it does not seem to be acceptable syntax.
Edit
Suggested fix does not work. This is the pipeline
pipeline {
agent none
parallel {
stage('Linux') {
agent { label 'linux' }
stages {
stage('bootstrap') {
sh "ls"
}
stage('devenv') {
sh "ls"
}
}
}
stage('windows') {
agent { label 'Windows' }
stages {
stage('bootstrap') {
bat 'dir'
}
stage('devenv') {
bat 'dir'
}
}
}
}
}
This is the error message
WorkflowScript: 8: Undefined section "parallel" # line 8, column 5.
parallel {
^
WorkflowScript: 1: Missing required section "stages" # line 1, column 1.
pipeline {
^
I have this Jenkinsfile below. I am trying to get the key of a map but I am getting "java.lang.NoSuchMethodError: No such DSL method 'get' found among steps". Can someone help me to resolve this?
def country_capital = {
[Australia : [best: 'xx1', good: 'xx2', bad: 'xx3'],
America : [best: 'yy1', good: 'yy2', bad: 'yy3']]
}
pipeline {
agent any
stages {
stage('Test Map') {
steps {
script {
echo country_capital.get('Australia')['best']
}
}
}
}
}
You can get the value using this way
def country_capital = [
Australia: [
best: 'xx1',
good: 'xx2',
bad: 'xx3'
],
America: [
best: 'yy1',
good: 'yy2',
bad: 'yy3'
]
]
pipeline {
agent any
stages {
stage('Test Map') {
steps {
script {
echo country_capital['Australia'].best
}
}
}
}
}
// Output
xx1
For the above example one can also do
country_capital.each { capital_key, capital_value ->
try {
echo "Testing ${capital_value.best}..."
}
catch(ex){
echo "Test failed: ${capital_value.bad}" }
}
I need to launch a dynamic set of tests in a declarative pipeline.
For better visualization purposes, I'd like to create a stage for each test.
Is there a way to do so?
The only way to create a stage I know is:
stage('foo') {
...
}
I've seen this example, but I it does not use declarative syntax.
Use the scripted syntax that allows more flexibility than the declarative syntax, even though the declarative is more documented and recommended.
For example stages can be created in a loop:
def tests = params.Tests.split(',')
for (int i = 0; i < tests.length; i++) {
stage("Test ${tests[i]}") {
sh '....'
}
}
As JamesD suggested, you may create stages dynamically (but they will be sequential) like that:
def list
pipeline {
agent none
options {buildDiscarder(logRotator(daysToKeepStr: '7', numToKeepStr: '1'))}
stages {
stage('Create List') {
agent {node 'nodename'}
steps {
script {
// you may create your list here, lets say reading from a file after checkout
list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
}
}
post {
cleanup {
cleanWs()
}
}
}
stage('Dynamic Stages') {
agent {node 'nodename'}
steps {
script {
for(int i=0; i < list.size(); i++) {
stage(list[i]){
echo "Element: $i"
}
}
}
}
post {
cleanup {
cleanWs()
}
}
}
}
}
That will result in:
dynamic-sequential-stages
If you don't want to use for loop, and generated pipeline to be executed in parallel then, here is an answer.
def jobs = ["JobA", "JobB", "JobC"]
def parallelStagesMap = jobs.collectEntries {
["${it}" : generateStage(it)]
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Note that all generated stages will be executed into 1 node.
If you are willing to executed the generated stages to be executed into different nodes.
def agents = ['master', 'agent1', 'agent2']
// enter valid agent name in array.
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
You can of course add more than 1 parameters and can use collectEntries for 2 parameters.
Please remember return in function generateStage is must.
#Jorge Machado: Because I cannot comment I had to post it as an answer. I've solved it recently. I hope it'll help you.
Declarative pipeline:
A simple static example:
stage('Dynamic') {
steps {
script {
stage('NewOne') {
echo('new one echo')
}
}
}
}
Dynamic real-life example:
// in a declarative pipeline
stage('Trigger Building') {
when {
environment(name: 'DO_BUILD_PACKAGES', value: 'true')
}
steps {
executeModuleScripts('build') // local method, see at the end of this script
}
}
// at the end of the file or in a shared library
void executeModuleScripts(String operation) {
def allModules = ['module1', 'module2', 'module3', 'module4', 'module11']
allModules.each { module ->
String action = "${operation}:${module}"
echo("---- ${action.toUpperCase()} ----")
String command = "npm run ${action} -ddd"
// here is the trick
script {
stage(module) {
bat(command)
}
}
}
}
You might want to take a look at this example - you can have a function return a closure which should be able to have a stage in it.
This code shows the concept, but doesn't have a stage in it.
def transformDeployBuildStep(OS) {
return {
node ('master') {
wrap([$class: 'TimestamperBuildWrapper']) {
...
} } // ts / node
} // closure
} // transformDeployBuildStep
stage("Yum Deploy") {
stepsForParallel = [:]
for (int i = 0; i < TargetOSs.size(); i++) {
def s = TargetOSs.get(i)
def stepName = "CentOS ${s} Deployment"
stepsForParallel[stepName] = transformDeployBuildStep(s)
}
stepsForParallel['failFast'] = false
parallel stepsForParallel
} // stage
Just an addition to what #np2807 and #Anton Yurchenko have already presented: you can create stages dynamically and run the in parallel by simply delaying list of stages creation (but keeping its declaration), e.g. like that:
def parallelStagesMap
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent { label 'master' }
stages {
stage('Create List of Stages to run in Parallel') {
steps {
script {
def list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
// you may create your list here, lets say reading from a file after checkout
// personally, I like to use scriptler scripts and load the as simple as:
// list = load '/var/lib/jenkins/scriptler/scripts/load-list-script.groovy'
parallelStagesMap = list.collectEntries {
["${it}" : generateStage(it)]
}
}
}
}
stage('Run Stages in Parallel') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
That will result in Dynamic Parallel Stages:
I use this to generate my stages which contain a Jenkins job in them.
build_list is a list of Jenkins jobs that i want to trigger from my main Jenkins job, but have a stage for each job that is trigger.
build_list = ['job1', 'job2', 'job3']
for(int i=0; i < build_list.size(); i++) {
stage(build_list[i]){
build job: build_list[i], propagate: false
}
}
if you are using Jenkinsfile then, I achieved it via dynamically creating the stages, running them in parallel and also getting Jenkinsfile UI to show separate columns. This assumes parallel steps are independent of each other (otherwise don't use parallel) and you can nest them as deep as you want (depending upon the # of for loops you'll nest for creating stages).
Jenkinsfile Pipeline DSL: How to Show Multi-Columns in Jobs dashboard GUI - For all Dynamically created stages - When within PIPELINE section see here for more.