I have a declarative Jenkins pipeline which contains several identical stages running in parallel. I would like to reuse these sections, however so far the best I've been able to achieve is as follows:
stage('Deploy & E2E') {
parallel {
stage('E2E 1') {
agent { ... }
environment { ... }
steps {
script { runE2ESteps() }
}
post { ... }
}
stage('E2E 2') {
agent { ... }
environment { ... }
steps {
script { runE2ESteps() }
}
post { ... }
}
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
}
}
I am able to partially reuse bits, by extracting the steps out into a method, but there is still quite a bit of duplication in the agent, environment and post blocks. Ideally I want to write something like:
stage('Deploy & E2E') {
parallel {
script {
for (int i = 0; i < 5; i++) {
stage('E2E ${i}') {
agent { ... }
environment { ... }
steps { ... }
post { ... }
}
}
}
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
}
}
But apparently I can't do this in declarative land - Jenkins seems very finicky about what precisely you're allowed to have at various points in the file.
I've tried a few other approaches, too, but none of them have worked:
Using matrix for the E2E block seemed promising, but you're not allowed to nest matrix within parallel so this didn't pan out (because there are other, unrelated stages we want running in parallel to the E2E steps)
Using the in-line parallel command. This doesn't achieve what I want because I need each stage running in its own agent (not parallelised within the same agent).
I've attached a diagram of what the pipeline looks like - it's behaving exactly as I want right now, just with more code duplication than I'd like. If we want to change the parallelism of our E2E, it currently involves copy+pasting or removing a block of code, rather than (ideally) changing one number somewhere.
Maybe this just isn't possible, but I figured I might as well chuck out a post to see if there are any ideas that I missed.
stage('Deploy & E2E') {
parallel {
script {
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
for (int i = 0; i < 5; i++) {
stage('E2E ${i}') {
agent { ... }
environment { ... }
steps { ... }
post { ... }
}
}
}
}
}
have a try
We never found a solution to this in Jenkins. In the end, we've moved to Github Actions where this is possible with a matrix job.
My pipeline looks like that:
pipeline{
...
post {
always {
archiveArtifacts artifacts: 'artifacts/**/*'
script {
...
}
rtp stableText: '${FILE:artifacts/summary.html}', parserName: 'HTML'
}
success {
script {
...
}
}
}
}
I'd like that the script which is executed on success, was executed also on unstable, how can I achieve that?
Is there a way to specify success or unstable {?
Or is there a way to declare the action to take somewhere else and "invoke" it in a success and in an unstable tags?
you can also do like below
def commonPostSteps() {
echo "Hello World"
script {
def x =10
print x + 20
}
}
pipeline {
agent any;
stages {
stage('one') {
steps {
echo "${env.STAGE_NAME}"
}
}
}
post {
always {
echo "post always"
}
success {
commonPostSteps()
}
unstable {
commonPostSteps()
}
}
}
I have multiple jenkinsifles, doing basically the same thing:
pipeline{
parameters { ... }
environment { ... }
stages {
stage ('setup') { ... }
stage ('run') {
agent { AGENT }
steps { STEPS }
}
}
The STEPS & AGENT parameters are values I get in the setup stage. Is it possible to define somewhere a function that returns a stage?
e.g.
def stage_factory(name, agent, steps, post ...){
return
stage (name) {
agent { agnet }
steps { steps }
post { post }
}
}
}
which later will be called inside the pipeline, right after the setup stage
?
The following works in scripted pipeline, you need to try the declarative syntax yourself. Note the use of surrounding {}
def stage_factory(name, agent, steps, post ...){
return {
node(agent){
stage (name) {
steps()
}
}
}
}
With this approach you need to put the post action in try-catch blocks, but this is the gist of it.
If you change it like so, you can even pass the steps to it as you would expect from a Jenkins stage.
def stage_factory(name, agent){
return { steps ->
node(agent){
stage (name) {
steps()
}
}
}
}
usage:
def myDtage = stage_factory("foo", "bar")
myStage{
//...
}
I have something like this:
stages {
stage('createTemplate') {
parallel {
stage('template_a') {
creating template a
}
stage('template_b') {
creating template b
}
}
}
stage('deployVm') {
parallel {
stage('deploy_a') {
deploy vm a
}
stage('deploy_b') {
deploy vm b
}
}
}
}
How can I make sure that deployVm stages run when and only when respective createTemplate stages were successful?
You may want to run one parallel like this:
parallel {
stage('a') {
stages {
stage ('template_a') { ... }
stage ('deploy_a') { ... }
}
stage('b') {
stages {
stage ('template_b') { ... }
stage ('deploy_b') { ... }
}
}
}
This will make sure only stages that deploy are the ones following successful template stages.
I need to launch a dynamic set of tests in a declarative pipeline.
For better visualization purposes, I'd like to create a stage for each test.
Is there a way to do so?
The only way to create a stage I know is:
stage('foo') {
...
}
I've seen this example, but I it does not use declarative syntax.
Use the scripted syntax that allows more flexibility than the declarative syntax, even though the declarative is more documented and recommended.
For example stages can be created in a loop:
def tests = params.Tests.split(',')
for (int i = 0; i < tests.length; i++) {
stage("Test ${tests[i]}") {
sh '....'
}
}
As JamesD suggested, you may create stages dynamically (but they will be sequential) like that:
def list
pipeline {
agent none
options {buildDiscarder(logRotator(daysToKeepStr: '7', numToKeepStr: '1'))}
stages {
stage('Create List') {
agent {node 'nodename'}
steps {
script {
// you may create your list here, lets say reading from a file after checkout
list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
}
}
post {
cleanup {
cleanWs()
}
}
}
stage('Dynamic Stages') {
agent {node 'nodename'}
steps {
script {
for(int i=0; i < list.size(); i++) {
stage(list[i]){
echo "Element: $i"
}
}
}
}
post {
cleanup {
cleanWs()
}
}
}
}
}
That will result in:
dynamic-sequential-stages
If you don't want to use for loop, and generated pipeline to be executed in parallel then, here is an answer.
def jobs = ["JobA", "JobB", "JobC"]
def parallelStagesMap = jobs.collectEntries {
["${it}" : generateStage(it)]
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Note that all generated stages will be executed into 1 node.
If you are willing to executed the generated stages to be executed into different nodes.
def agents = ['master', 'agent1', 'agent2']
// enter valid agent name in array.
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
You can of course add more than 1 parameters and can use collectEntries for 2 parameters.
Please remember return in function generateStage is must.
#Jorge Machado: Because I cannot comment I had to post it as an answer. I've solved it recently. I hope it'll help you.
Declarative pipeline:
A simple static example:
stage('Dynamic') {
steps {
script {
stage('NewOne') {
echo('new one echo')
}
}
}
}
Dynamic real-life example:
// in a declarative pipeline
stage('Trigger Building') {
when {
environment(name: 'DO_BUILD_PACKAGES', value: 'true')
}
steps {
executeModuleScripts('build') // local method, see at the end of this script
}
}
// at the end of the file or in a shared library
void executeModuleScripts(String operation) {
def allModules = ['module1', 'module2', 'module3', 'module4', 'module11']
allModules.each { module ->
String action = "${operation}:${module}"
echo("---- ${action.toUpperCase()} ----")
String command = "npm run ${action} -ddd"
// here is the trick
script {
stage(module) {
bat(command)
}
}
}
}
You might want to take a look at this example - you can have a function return a closure which should be able to have a stage in it.
This code shows the concept, but doesn't have a stage in it.
def transformDeployBuildStep(OS) {
return {
node ('master') {
wrap([$class: 'TimestamperBuildWrapper']) {
...
} } // ts / node
} // closure
} // transformDeployBuildStep
stage("Yum Deploy") {
stepsForParallel = [:]
for (int i = 0; i < TargetOSs.size(); i++) {
def s = TargetOSs.get(i)
def stepName = "CentOS ${s} Deployment"
stepsForParallel[stepName] = transformDeployBuildStep(s)
}
stepsForParallel['failFast'] = false
parallel stepsForParallel
} // stage
Just an addition to what #np2807 and #Anton Yurchenko have already presented: you can create stages dynamically and run the in parallel by simply delaying list of stages creation (but keeping its declaration), e.g. like that:
def parallelStagesMap
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent { label 'master' }
stages {
stage('Create List of Stages to run in Parallel') {
steps {
script {
def list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
// you may create your list here, lets say reading from a file after checkout
// personally, I like to use scriptler scripts and load the as simple as:
// list = load '/var/lib/jenkins/scriptler/scripts/load-list-script.groovy'
parallelStagesMap = list.collectEntries {
["${it}" : generateStage(it)]
}
}
}
}
stage('Run Stages in Parallel') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
That will result in Dynamic Parallel Stages:
I use this to generate my stages which contain a Jenkins job in them.
build_list is a list of Jenkins jobs that i want to trigger from my main Jenkins job, but have a stage for each job that is trigger.
build_list = ['job1', 'job2', 'job3']
for(int i=0; i < build_list.size(); i++) {
stage(build_list[i]){
build job: build_list[i], propagate: false
}
}
if you are using Jenkinsfile then, I achieved it via dynamically creating the stages, running them in parallel and also getting Jenkinsfile UI to show separate columns. This assumes parallel steps are independent of each other (otherwise don't use parallel) and you can nest them as deep as you want (depending upon the # of for loops you'll nest for creating stages).
Jenkinsfile Pipeline DSL: How to Show Multi-Columns in Jobs dashboard GUI - For all Dynamically created stages - When within PIPELINE section see here for more.