I figure I’m doing something unorthodox here, but I’d like to stick to declarative for convenience while dynamically generating parallel steps.
I found a way to do something like that, but mixing both paradigms, which doesn’t seem to work well with the BlueOcean UI (multiple stages inside each parallel branch do not show up properly).
The closest I got was with something like this:
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = []
for (account in accounts()) {
jobs[] = stage(account) {
steps {
echo "Step for $account"
}
}
}
return jobs
}
# this is inside a shared library, called by my Jenkinsfile, like what is described
# under "Defining Declarative Pipelines in Shared Libraries" in
# https://www.jenkins.io/blog/2017/09/25/declarative-1/
def call() {
pipeline {
stages {
stage('Build all variations') {
parallel parallelJobs()
}
}
}
}
The problem is Jenkins errors like this:
Expected a block for parallel # line X, column Y.
parallel parallelJobs()
^
So, I was wondering if there is a way I could transform that list of stages, returned by parallelJobs(), into the block expected by Jenkins...
Yes, you can. You need to return a map of stages. Following is a working pipeline example.
pipeline {
agent any
stages {
stage('Parallel') {
steps {
script {
parallel parallelJobs()
}
}
}
}
}
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = [:]
for (account in accounts()) {
jobs[account] = { stage(account) {
echo "Step for $account"
}
}
}
return jobs
}
I have this pipeline that generates dynamic stages based on file content. I want the stages to run on different containers, so I thought to move the agent{docker{image ''}}} label to inside the stage in the generateStage() function but it's not possible since it's a scripted pipeline. How can I run these stages on separate containers, and still run them parallelly and generate them dynamically?
Would really appreciate your help.
Thanks!
def generateStage(job) {
return {
stage("stage: job") {
//do something
}
}
}
pipeline{
agent none
stages{
.
.
.
stage('parallel stages') {
agent {
docker{
image 'some-image:tag'
}
}
steps {
script {
def list = ["STAGE-A", "STAGE-B"....] // DYNAMIC LIST CREATED FROM A FILE
parallelStages = list.collectEntries{
["$it": generateStage(it)]
}
parallel parallelStages
}
}
}
Instead of using the agent option, you can do something like the below.
def generateStage(job) {
return {
stage("stage: job") {
docker.image('your-image').inside {
sh 'DO SOMETHING'
}
}
}
}
I have a Jenkinsfile that looks like this:
static def randomUser() {
final def POOL = ["a".."z"].flatten()
final Random rand = new Random(System.currentTimeMillis())
return (0..5).collect { POOL[rand.nextInt(POOL.size())] }.join("")
}
pipeline {
agent any
environment {
//CREATOR = sh(script: "randomUser()", returnStdout: true)
CREATOR = "fixed-for-now"
...
}
stages {
...
stage("Terraform Plan") {
when { not { branch "master" } }
steps {
sh "terraform plan -out=plan.out -var creator=${CREATOR} -var-file=env.tfvars "
}
}
...
stage("Terraform Destroy") {
when { not { branch "master" } }
steps {
sh "terraform destroy -auto-approve -var creator=${CREATOR} -var-file=env.tfvars "
}
}
...
}
My problem is I cannot call randomUser while being inside the environment block. I would need to have the CREATOR variable as a random string every time. I would prefer to have CREATOR as a global environment variable since it's going to be used in many stages.
Is there a way to achieve (or workaround) this?
Given your specific use case, it might be better to use the CREATOR variable as a parameter instead of an environment variable, and to assign its defaultValue as the return of your randomUser method.
pipeline {
agent any
parameters {
string(name: 'CREATOR', defaultValue: sh(script: "randomUser()", returnStdout: true))
}
...
}
You can then use it in your pipeline like so:
stage("Terraform Plan") {
when { not { branch "master" } }
steps {
sh "terraform plan -out=plan.out -var creator=${params.CREATOR} -var-file=env.tfvars "
}
}
This way you have a correctly assigned and useful defaultValue for CREATOR, but with the ability to override it per-pipeline when necessary.
You can achieve this by removing environment block and defining global variable CREATOR before pipeline block
def CREATOR
pipeline {
agent any
stages {
stage('Initialize the variables') {
steps{
script{
CREATOR = randomUser()
}
}
}
...
I need to launch a dynamic set of tests in a declarative pipeline.
For better visualization purposes, I'd like to create a stage for each test.
Is there a way to do so?
The only way to create a stage I know is:
stage('foo') {
...
}
I've seen this example, but I it does not use declarative syntax.
Use the scripted syntax that allows more flexibility than the declarative syntax, even though the declarative is more documented and recommended.
For example stages can be created in a loop:
def tests = params.Tests.split(',')
for (int i = 0; i < tests.length; i++) {
stage("Test ${tests[i]}") {
sh '....'
}
}
As JamesD suggested, you may create stages dynamically (but they will be sequential) like that:
def list
pipeline {
agent none
options {buildDiscarder(logRotator(daysToKeepStr: '7', numToKeepStr: '1'))}
stages {
stage('Create List') {
agent {node 'nodename'}
steps {
script {
// you may create your list here, lets say reading from a file after checkout
list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
}
}
post {
cleanup {
cleanWs()
}
}
}
stage('Dynamic Stages') {
agent {node 'nodename'}
steps {
script {
for(int i=0; i < list.size(); i++) {
stage(list[i]){
echo "Element: $i"
}
}
}
}
post {
cleanup {
cleanWs()
}
}
}
}
}
That will result in:
dynamic-sequential-stages
If you don't want to use for loop, and generated pipeline to be executed in parallel then, here is an answer.
def jobs = ["JobA", "JobB", "JobC"]
def parallelStagesMap = jobs.collectEntries {
["${it}" : generateStage(it)]
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Note that all generated stages will be executed into 1 node.
If you are willing to executed the generated stages to be executed into different nodes.
def agents = ['master', 'agent1', 'agent2']
// enter valid agent name in array.
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
You can of course add more than 1 parameters and can use collectEntries for 2 parameters.
Please remember return in function generateStage is must.
#Jorge Machado: Because I cannot comment I had to post it as an answer. I've solved it recently. I hope it'll help you.
Declarative pipeline:
A simple static example:
stage('Dynamic') {
steps {
script {
stage('NewOne') {
echo('new one echo')
}
}
}
}
Dynamic real-life example:
// in a declarative pipeline
stage('Trigger Building') {
when {
environment(name: 'DO_BUILD_PACKAGES', value: 'true')
}
steps {
executeModuleScripts('build') // local method, see at the end of this script
}
}
// at the end of the file or in a shared library
void executeModuleScripts(String operation) {
def allModules = ['module1', 'module2', 'module3', 'module4', 'module11']
allModules.each { module ->
String action = "${operation}:${module}"
echo("---- ${action.toUpperCase()} ----")
String command = "npm run ${action} -ddd"
// here is the trick
script {
stage(module) {
bat(command)
}
}
}
}
You might want to take a look at this example - you can have a function return a closure which should be able to have a stage in it.
This code shows the concept, but doesn't have a stage in it.
def transformDeployBuildStep(OS) {
return {
node ('master') {
wrap([$class: 'TimestamperBuildWrapper']) {
...
} } // ts / node
} // closure
} // transformDeployBuildStep
stage("Yum Deploy") {
stepsForParallel = [:]
for (int i = 0; i < TargetOSs.size(); i++) {
def s = TargetOSs.get(i)
def stepName = "CentOS ${s} Deployment"
stepsForParallel[stepName] = transformDeployBuildStep(s)
}
stepsForParallel['failFast'] = false
parallel stepsForParallel
} // stage
Just an addition to what #np2807 and #Anton Yurchenko have already presented: you can create stages dynamically and run the in parallel by simply delaying list of stages creation (but keeping its declaration), e.g. like that:
def parallelStagesMap
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent { label 'master' }
stages {
stage('Create List of Stages to run in Parallel') {
steps {
script {
def list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
// you may create your list here, lets say reading from a file after checkout
// personally, I like to use scriptler scripts and load the as simple as:
// list = load '/var/lib/jenkins/scriptler/scripts/load-list-script.groovy'
parallelStagesMap = list.collectEntries {
["${it}" : generateStage(it)]
}
}
}
}
stage('Run Stages in Parallel') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
That will result in Dynamic Parallel Stages:
I use this to generate my stages which contain a Jenkins job in them.
build_list is a list of Jenkins jobs that i want to trigger from my main Jenkins job, but have a stage for each job that is trigger.
build_list = ['job1', 'job2', 'job3']
for(int i=0; i < build_list.size(); i++) {
stage(build_list[i]){
build job: build_list[i], propagate: false
}
}
if you are using Jenkinsfile then, I achieved it via dynamically creating the stages, running them in parallel and also getting Jenkinsfile UI to show separate columns. This assumes parallel steps are independent of each other (otherwise don't use parallel) and you can nest them as deep as you want (depending upon the # of for loops you'll nest for creating stages).
Jenkinsfile Pipeline DSL: How to Show Multi-Columns in Jobs dashboard GUI - For all Dynamically created stages - When within PIPELINE section see here for more.
I have a pipeline job that uses two separate nodes (one for build, one for test), and I'd like to share a variable between two of these blocks of code in my Jenkinsfile. I assume it's possible, but I'm very new to groovy and the Jenkinsfile concept. Here is the relevant code so far:
node('build') {
stage('Checkout') {
checkout scm
}
stage('Build') {
bat(script: 'build')
def rev = readFile('result')
}
}
node('test') {
stage('Test') {
def SDK_VERSION = "5.0.0001.${rev}"
bat "test.cmd ${env.BUILD_URL} ${SDK_VERSION}"
archiveArtifacts artifacts: 'artifacts/**/*.xml'
junit 'artifacts/**/*.xml'
}
}
I want to assign the "rev" variable in the build stage, but then concatenate it to the SDK_VERSION variable in the Test stage. My error is:
groovy.lang.MissingPropertyException: No such property: rev for class: groovy.lang.Binding
Just define the variable before your node block:
def rev = ''
node('build') {
stage('Checkout') {
checkout scm
}
stage('Build') {
bat(script: 'build')
rev = readFile('result')
}
}
In a declarative pipeline, #mkobit's answer won't work. You can, however, switchinto script mode explicitly and use its scoping, e.g. like so:
...
steps {
script {
def foo = sh script: "computeFoo", returnStdout: true
node('name') {
script {
someStep()
}
}
}
}
...