I have declarative pipeline as below and it runs 2* and 3* stages in parallel, pasted the blue ocean diagram below.
pipeline {
agent { label 'my_node' }
options {
timestamps()
parallelsAlwaysFailFast()
}
stages {
stage('1') {
steps {
script {
step([$class: 'WsCleanup'])
}
}
}
stage('2') {
parallel {
stage("2.1") {
steps {
script {
sh 'echo hi 2.1'
}
}
}
stage("2p") {
steps {
script {
sh 'echo hi 2p'
}
}
}
}
}
stage('3') {
parallel {
stage('3.1') {
steps {
script {
sh """
echo hi 3.1
"""
}
}
}
stage('3.2') {
steps {
script {
sh """
echo "hi 3.2"
"""
}
}
}
}
}
stage('4') {
steps {
script {
sh "echo end"
}
}
}
}
}
But I am looking to run 2p in parallel to 2* and 3*, like shown below, is there a way?
I tried to use paralle under parallel, to start 2p in parallel to 2 and 3, and nested parallel to run 3.1. and 3.2 underneath, but declarative pipeline is not allowing nested parallel.
You can't do this only with Declarative syntax. But you can achieve this with a combination of Scripted and Declarative syntax. One thing to note is, AFAIK there is no visualization support for nested parallel stages as of now. There is a feature request for this here.
Following is a sample pipeline you can use as a reference for your use case.
pipeline {
agent any
options {
timestamps()
parallelsAlwaysFailFast()
}
stages {
stage('1') {
steps {
script {
step([$class: 'WsCleanup'])
}
}
}
stage('2 AND 3') {
steps {
script {
parallel getWrappedStages()
}
}
}
stage('4') {
steps {
script {
sh "echo end"
}
}
}
}
}
def getWrappedStages() {
stages = [:]
stages["Step2.1"] = { stage('2.1') {
sh """
echo hi 2.1
"""
}
parallel parallel3xstages()
}
stages["Step2.p"] = { stage('2.p') {
sh """
echo hi 2.p
"""
}
}
return stages
}
def parallel3xstages() {
stages = [:]
stages["Step3.1"] = { stage('3.1') {
sh """
echo hi 3.1
"""
}
}
stages["Step3.2"] = { stage('3.2') {
sh """
echo hi 3.2
"""
}
}
return stages
}
Related
I have a Jenkins pipeline with some jobs running in parallel. I want to manage the timing out of one of them and keep the pipeline running. I have followed this very well-done guide, but I can't avoid that when my job times out; all the other parallel jobs and the whole process fail too.
Following, how I have implemented the pipeline:
pipeline {
agent any;
stages {
stage('Stage-1') {
steps {
sh 'job-1.sh'
}
}
stage('Stage-2') {
failFast false
parallel {
stage('Stage-2-1') {
steps {
sh 'job-2-1.sh'
}
}
stage('Stage2-2') {
stages {
stage('job-2-2-1') {
options {
timeout(time: 10, unit: "SECONDS")
}
steps {
script {
Exception caughtException = null
catchError(buildResult: 'SUCCESS', stageResult: 'ABORTED') {
try {
sh 'job-2-2-1.sh'
}
catch (org.jenkinsci.plugins.workflow.steps.FlowInterruptedException e) {
echo "Time out"
TIMEOUT="1"
} catch (Throwable e) {
caughtException = e
}
}
if (caughtException) {
error caughtException.message
}
}
}
}
stage('job-2-2-2') {
when {
expression { TIMEOUT=="0" }
}
steps {
sh 'job-2-2-2.sh'
}
}
}
}
stage('Stage-2-3') {
steps {
sh 'job-2-3.sh'
}
}
}
}
stage('Stage-3') {
steps {
sh 'job-3.sh'
}
}
}
}
I tried to set failFast to false because I don't know if it has a default value.
How can I keep the other jobs run and get the successful final build?
Thanks
I have a Jenkinsfile like this
pipeline {
agent { label 'master' }
stages {
stage('1') {
steps {
script {
sh '''#!/bin/bash
source $EXPORT_PATH_SCRIPT
cd $SCRIPT_PATH
python -m scripts.test.test
'''
}
}
}
stage('2') {
steps {
script {
sh '''#!/bin/bash
source $EXPORT_PATH_SCRIPT
cd $SCRIPT_PATH
python -m scripts.test.test
'''
}
}
}
}
}
As you can see, In both stages I am using the same script and calling the same file.
Can I move this step to a function in Jenkinsfile and call that function in script? like this
def execute script() {
return {
sh '''#!/bin/bash
source $EXPORT_PATH_SCRIPT
cd $SCRIPT_PATH
python -m scripts.test.test
'''
}
}
Yes, It's possible like below example:
Jenkinsfile
def doIt(name) {
return "The name is : ${name}"
}
def executeScript() {
sh "echo HelloWorld"
}
pipeline {
agent any;
stages {
stage('01') {
steps {
println doIt("stage 01")
executeScript()
}
}
stage('02') {
steps {
println doIt("stage 02")
executeScript()
}
}
}
}
I've a pipeline where multiple stages run in parallel but if any of the stages fails, I want to get its name to show failed stage. With following code, even if it fails in first stage ie Checking Redmine access, it always show last stage as failed i.e. Checking Reviewers list. This is because it runs in parallel and latest assigned value is picked up.
pipeline {
agent {
label {
label "<machine-ip>"
customWorkspace "workspace/RedmineAndReviewboardProject/SVNCheckoutTest"
}
}
stages {
stage('Verify inputs') {
parallel {
stage('Checking Redmine access') {
steps {
script {
FAILED_STAGE = env.STAGE_NAME
echo "Redmine"
sh'''hello'''
}
}
}
stage('Checking SVN access') {
steps {
script {
FAILED_STAGE = env.STAGE_NAME
echo "SVN"
}
}
}
stage('Checking Reviewers list') {
steps {
script {
FAILED_STAGE = env.STAGE_NAME
echo "Reviewer"
}
}
}
}
}
}
post {
failure {
script {
echo "Failed stage is " + FAILED_STAGE
}
}
}
}
Is there any way I can get exactly failed stage of parallel running stages? or it will be also ok with me if parent stage name is returned as failed stage.
I believe you can use a post { failure { block for each stage : see https://www.jenkins.io/doc/book/pipeline/syntax/#post
pipeline {
agent {
label {
label "<machine-ip>"
customWorkspace "workspace/RedmineAndReviewboardProject/SVNCheckoutTest"
}
}
stages {
stage('Verify inputs') {
parallel {
stage('Checking Redmine access') {
steps {
script {
echo "Redmine"
sh'''hello'''
}
}
post {
failure {
script {
echo "Failed stage is ${STAGE_NAME}"
}
}
}
}
stage('Checking SVN access') {
steps {
script {
FAILED_STAGE = env.STAGE_NAME
echo "SVN"
}
}
post {
failure {
script {
echo "Failed stage is ${STAGE_NAME}"
}
}
}
}
stage('Checking Reviewers list') {
steps {
script {
FAILED_STAGE = env.STAGE_NAME
echo "Reviewer"
}
}
post {
failure {
script {
echo "Failed stage is ${STAGE_NAME}"
}
}
}
}
}
}
}
}
There is a way to execute steps/post-actions on different nodes in parallel described in this article: https://jenkins.io/blog/2017/09/25/declarative-1/
stage('Run Tests') {
parallel {
stage('Test On Windows') {
agent {
label "windows"
}
steps {
bat "run-tests.bat"
}
post {
always {
junit "**/TEST-*.xml" // DUPLICATE CODE
}
}
}
stage('Test On Linux') {
agent {
label "linux"
}
steps {
sh "run-tests.sh"
}
post {
always {
junit "**/TEST-*.xml" // DUPLICATE CODE
}
}
}
}
}
Is there any possibility to execute same stages on multiple nodes without duplicating code?
Something like this:
stage('Run Tests') {
parallel {
stage("Test On ${NODE_NAME}") {
agents {
label "windows"
label "linux"
}
steps {
// do test steps
}
post {
always {
junit "**/TEST-*.xml"
}
}
}
}
}
You can create dynamic stages, but for your case not needed
pipeline {
agent any
stages {
stage ("Test") {
steps {
script {
testStages = ["Windows", "Linux"].collectEntries {
["${it}" : runTests(it)]
}
parallel testStages
}
}
}
}
}
def runTests(def name){
return {
node(name) {
stage("Run on ${name}") {
script {
command = "run-tests"
try {
switch(name.toLowerCase()) {
case "windows":
command += ".bat"
break;
case "linux":
command += ".sh"
break;
}
echo command
} catch (Exception ex) {
echo ex
} finally {
echo "post ${name}"
}
}
}
}
}
}
Declarative Matrix worked best for me:
pipeline {
agent none
stages {
stage('BuildAndTest') {
matrix {
agent {
label "${PLATFORM}-agent"
}
axes {
axis {
name 'PLATFORM'
values 'linux', 'windows'
}
}
stages {
stage('Test') {
steps {
echo "Do Test for ${PLATFORM}"
}
}
}
post {
always {
junit "**/TEST-*.xml"
}
}
}
}
}
}
This pipeline will execute the defined stages incl. post build actions on both platforms without any code duplication.
Quote from a Jenkins blog post about declartive matrix:
An equivalent pipeline created without matrix would easily be several
times larger, and much harder to understand and maintain.
This minimal pipeline works:
pipeline {
agent any
stages {
stage('test') {
parallel {
stage('test-1') {
steps {
sh "echo test1"
}
}
stage('test-2') {
steps {
sh "echo test2"
}
}
}
}
}
}
We have several test stages inside the parallel block and so we run into Method Code too large error that the Jenkins guys apparently aren't planning to fix.
I'd like to have my parallel stages defined in an outside function, like this:
pipeline {
agent any
stages {
stage('test') {
parallel test_func()
}
}
}
def test_func() {
return {
stage('test-1') {
steps {
sh "echo test1"
}
}
stage('test-2') {
steps {
sh "echo test2"
}
}
}
}
This however doesn't work with lots of variations of syntax that we've tried.
Thanks in advance!
Check my answer to this question.
You need to put your parallel stages in a variable, not a function. So, it could be somethink like this:
def test_func = [
"test1": {
echo "test1"
},
"test2": {
echo "test2"
}
]
pipeline {
agent any
stages {
stage('test') {
steps {
script {
parallel test_func
}
}
}
}
}
As seen in the examples here, it expects a map of an arbitrary identifier as key and a closure containing the code as value. So something (untested) like:
def test_func() {
return [
test1: {
stage('test-1') {
steps {
sh "echo test1"
}
}
},
test2: {
stage('test-2') {
steps {
sh "echo test2"
}
}
}
]
}
You could do it like this:
def createStages() {
stage_map = [:]
stage_map.put('test-1', {echo 'test1'})
stage_map.put('test-2', {echo 'test2'})
return stage_map
}
pipeline {
agent any
stages {
stage('test') {
steps{
script { parallel(createStages()) }
}
}
}
}