I have a very long and complex pipeline that I'm rewriting after a major jenkins upgrade.
What I'd like to to is declaring my stages as variables,then execute them in the main node body: I can do this easily for the parallel stages, but I want to have the same style also for the sequential ones.
After a lot of tests, the only way I found to make this work was using "fake" parallel calls around all single sequential stages (ugly), I'm sure there is a better solution but seems like I can't find the proper step...shame on me.
Here's my example:
stage1 = { stage("one") {
println "stage one"
} }
stage2 = { stage("two") {
println "stage two"
} }
stage3 = { stage("three") {
println "stage three"
} }
node {
parallel (
"one" : stage1 ,
"two" : stage2
)
HERE I WANT TO CALL stage3 Closure, possibly giving a map like in the parallel above
}
You should be able to do this with the run method.
stage3.run()
I do not know if this is safe to use.
Related
following code is throwing below error.
if(!SkipLanguageComponentTests){
^
WorkflowScript: : Groovy compilation error(s) in script. Error(s): "Ambiguous expression could be either a parameterless closure expression or an isolated open code block;
solution: Add an explicit closure parameter list,
script {
2 errors
`
def SkipLanguageComponentTests = false;
pipeline {
parameters {
booleanParam(name: 'SkipLanguageComponentTests', defaultValue: false, description: 'XYZ')
}
stages {
stage('Checkout Source') {
steps {
checkout scm
}
}
stage("Component & Language Tests"){
steps{
parallel (
"componentTestsTask":{
//component test start
dir("docker"){
sh script: "docker-compose -f blah blah\""
}
// some xyz step here
//component test ends here
},
"integrationTestTasks":{
// language test script starts
if(!SkipLanguageComponentTests){
//run lang test and publish report
} else {
echo "Skip Language Component Tests"
}
// language test script ends
}
)
}
}
}
`
I have tried as per the documentation https://www.jenkins.io/blog/2017/09/25/declarative-1/
I have tried this based on the answer mentioned in : Running stages in parallel with Jenkins workflow / pipeline
stage("Parallel") { steps { parallel ( "firstTask" : { //do some stuff }, "secondTask" : { // Do some other stuff in parallel } ) } }
Can someone help me to resolve this ?
Ok, here is the working version of your pipeline - with proper IF inside:
parameters {
booleanParam(name: 'SkipLanguageComponentTests', defaultValue: false, description: '')
}
agent { label 'master' }
stages {
stage("Component & Language Tests") {
parallel {
stage("componentTestsTask") {
steps {
//component test start
echo "docker"
// some xyz step here
//component test ends here
}
}
stage("integrationTestTasks") {
steps {
script {
// language test script starts
if (!params.SkipLanguageComponentTests) {
echo "not skipped"
//run lang test and publish report
} else {
echo "Skip Language Component Tests"
}
}
}
// language test script ends
}
}
}
}
}
This pipeline is not optimal, use below information to improve it.
Notes:
you are using declarative pipeline so I think it is better to stay with parallel section expressed in declarative way
help is here: Jenkins doc about parallel
there is scripted pipeline as well Jenkins doc about scripted pipeline
as I stated in original answer, you have to use params to refer to input parameter properly
if you are using code you have to enclose it in script section - this is like putting scripted pipeline inside declarative one
IF statement can also be done declaratively: Jenkins doc about WHEN
I recommend not to mix these 2 styles: so if you are using both for some good reason they should be separated from one another as much as possible for example by using function or library inside declarative pipeline - the goal is to have pipeline as clear and readable as possible
You are using input variable, so try to refer as it should be done for input in Jenkins:
if(!params.SkipLanguageComponentTests)
I have 2 questions regarding jenkins
1. What is the difference between the 2 methods?
Method 1
pipeline {
agent any
stages {
stage('Example') {
steps {
echo 'Hello World'
}
}
}
Method 2
node ("jenkins-nodes") {
stage("git clone"){
echo 'Hello World' }
}
As I understand in the first method I can add Post section that will be running regardless of the result of the job. I wish to add the same post section for the second method, but it is not working. Any ideas?
What is the difference between the 2 methods?
As Noam Helmer wrote, pipeline{} is declarative syntax and and node{} is scripted syntax.
In general I recommend to always use pipeline{} as it makes common tasks easier to write and visualization using Blue Ocean plugin works best with declarative pipeline.
When declarative pipeline becomes too unflexible, you can insert scripted blocks using the script{} step in a declarative pipeline:
pipeline {
agent any
stages {
stage('Example') {
steps {
script {
echo 'Hello World'
}
}
}
}
}
A cleaner approach is to define a function, which is scripted by definition and use that as a custom step in declarative pipeline. Note this works even without script{}!
pipeline {
agent any
stages {
stage('Example') {
steps {
myStep 42
}
}
}
}
void myStep( def x ) {
echo "Hello World $x" // prints "Hello World 42"
}
In complex pipelines that use lots of custom code, I usually have one function per stage. This keeps the pipeline{} clean and makes it easy to see the overall structure of the pipeline, without script{} clutter all over the place.
As I understand in the first method I can add Post section that will
be running regardless of the result of the job. I wish to add the same
post section for the second method, but it is not working. Any ideas?
post{} is only available in declarative pipeline, but not within a scripted pipeline or a scripted section of a declarative pipeline. You can use try{} catch{} finally{} instead. catch{} runs only when an error occurs, finally{} always runs. You can use both or either of catch{} and finally{}.
node ("jenkins-nodes") {
stage("git clone"){
echo 'Hello World'
try {
// some steps that may fail
}
catch( Exception e ) {
echo "An error happened: $e"
// Rethrow exception, to let the build fail.
// If you remove "throw", the error would be ignored by Jenkins.
throw
}
finally {
echo "Cleaning up some stuff"
}
}
}
I want to add a matrix section to the following pipeline. I want the pipeline to run on 4 nodes with each node running a different stage that is specified in the for loop (e.g. one node runs AD, the other runs CD, the other runs DC, and the last runs DISP_A. It then repeats this behavior for the rest of the list until it is done iterating to the end of the list).
I have looked at the documentation and have not come up with any concrete answers to my question.
pipeline
{
agent none
stages
{
stage ('Test')
{
steps
{
script
{
def test_proj_choices = ['AD', 'CD', 'DC', 'DISP_A', 'DISP_PROC', 'EGI', 'FD', 'FLT', 'FMS_C', 'IFF', 'liblO', 'libNGC', 'libSC', 'MISCMP_MP', 'MISCMP_GP', 'NAV_MGR', 'RADALT', 'SYS', 'SYSIO15', 'SYSIO42', 'SYSRED', 'TACAN', 'VOR_ILS', 'VPA', 'WAAS', 'WCA']
for (choice in test_proj_choices)
{
stage ("${choice}")
{
echo "Running ${choice}"
build job: "UH60Job", parameters: [string(name: "TEST_PROJECT", value: choice), string(name: "SCADE_SUITE_TEST_ACTION", value: "all"), string(name: "VIEW_ROOT", value: "myview")]
}
}
}
}
}
}
}
I don't think your expectation can be decorated with Jenkins matrix DSL, as matrix DSL works like a single or multi-dimensional array.
But you can do something similar by writing a small groovy logic.
Below is one small example similar to your expectation:
The expectation of this example will run one task in one Jenkins agent in a distributed fashion.
Meaning it will run this way (Task - agent) A - will run on- agent1, B - will run on- agent2, C -will run on- agent3, D - will run on- agent4, E - will run on- agent1, ....
node {
agent=['agent1','agent2','agent3', 'agent4']
tasks=['A','B','C','D','E','F','G','H','I','J','K','L']
int nodeCount=0
tasks.each {
node(agent[nodeCount]) {
stage("build") {
println "Task - ${it} running on ${agent[nodeCount]}"
}
}
nodeCount=nodeCount+1
if(nodeCount == agent.size()){
nodeCount=0
}
}
}
Jenkins agent no need to be hardcode, by using Jenkins groovy api, you can easily find all the available and active agent from Jenkins, like below.
agent=[]
for (a in hudson.model.Hudson.instance.slaves) {
if(!a.getComputer().isOffline()) {
agent << a.name
}
}
I would like to setup parallel stages as described in the image
In this instance, the setup is pretty heavy so has to be done once before the parallel group starts of G1, G2 and G3. At the same time, the stage : Initial Checks has 2 items that I would like to run in parallel.
Is this possible in the Declarative Pipeline or do I have to resort to a script?
I couldnt see in the documentation the ability for this to work
Stages {
stage ('P1') {
}
stage ('P2 Setup') {}
stage ('P2') {
//Here it can contain either Steps or Parallel. I can only do
parallel {
stage ('g1') {} //Parallel tests
stage ('g2') {}
stage ('g3') {}
}
}
stage ('P2 Cleanup') {}
}
Have you encountered similar situations and what have your solutions been like?
Ofcourse 1 solution is to make Setup and Cleanup as part of every group, but like I said, its pretty intensive and I would only take it on if what the diagram indicates isn't possible.
Solution 2
stage ('p2') {
script {
//Some scripting here to get the result?
}
}
Pipeline
This is not supported by a DSL or declarative pipeline yet. You are essentially looking for nested parallel stages as mentioned here
Issue is still open with Jenkins community which you can watch here
In your given case, you can launch stage P1 and stage setup in parallel. However, it is important to start P1 as a background process because from your graph it appears that P1 is a time-intensive operation. Once group stage completes, you can collect the status of P1 and proceed to S2.
stages{
stage('Build'){
steps{
echo "Build"
}
}
stage('Init'){
parallel{
stage('P1'){steps{ echo "launch p1 in background"}}
stage('setup'){steps{echo "setup"}}
}
}
stage('Group'){
parallel{
stage('P1'){steps{echo "p1"}}
stage('P2'){steps{echo "p2"}}
stage('P3'){steps{echo "p3"}}
}
}
stage('Cleanup'){
steps{
echo "cleanup"
}
}
stage('Check P1 status'){
steps{
echo "Check"
}
}
stage('S2'){
steps{
echo "S2"
}
}
}
I think you are looking for this
node {
stage("P1"){}
stage("p2") {}
stage("p3") {
parallel (
"firstTask" : {
},
"secondTask" : {
}
)
}
}
Please note: the question is based on the old, now called "scripted" pipeline format. When using "declarative pipelines", parallel blocks can be nested inside of stage blocks (see Parallel stages with Declarative Pipeline 1.2).
I'm wondering how parallel steps are supposed to work with Jenkins workflow/pipeline plugin, esp. how to mix them with build stages. I know about the general pattern:
parallel(firstTask: {
// Do some stuff
}, secondTask: {
// Do some other stuff in parallel
})
However, I'd like to run couple of stages in parallel (on the same node, which has multiple executors), so I tried to add stages like this:
stage 'A'
// Do some preparation stuff
parallel(firstTask: {
stage 'B1'
// Do some stuff
}, secondTask: {
stage 'B2'
// Do some other stuff in parallel
})
stage 'C'
// Finalizing stuff
This does not work as expected. The "do stuff" tasks are executed in parallel, but the parallel stages end immediately and do not incorporate the stuff they should contain. As a consequence, the Stage View does not show the correct result and also does not link the logs.
Can I build different stages in parallel, or is the "parallel" step only meant to be used within a single stage?
You may not place the deprecated non-block-scoped stage (as in the original question) inside parallel.
As of JENKINS-26107, stage takes a block argument. You may put parallel inside stage or stage inside parallel or stage inside stage etc. However visualizations of the build are not guaranteed to support all nestings; in particular
The built-in Pipeline Steps (a “tree table” listing every step run by the build) shows arbitrary stage nesting.
The Pipeline Stage View plugin will currently only display a linear list of stages, in the order they started, regardless of nesting structure.
Blue Ocean will display top-level stages, plus parallel branches inside a top-level stage, but currently no more.
JENKINS-27394, if implemented, would display arbitrarily nested stages.
that syntax is now deprecated, you will get this error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 14: Expected a stage # line 14, column 9.
parallel firstTask: {
^
WorkflowScript: 14: Stage does not have a name # line 14, column 9.
parallel secondTask: {
^
2 errors
You should do something like:
stage("Parallel") {
steps {
parallel (
"firstTask" : {
//do some stuff
},
"secondTask" : {
// Do some other stuff in parallel
}
)
}
}
Just to add the use of node here, to distribute jobs across multiple build servers/ VMs:
pipeline {
stages {
stage("Work 1"){
steps{
parallel ( "Build common Library":
{
node('<Label>'){
/// your stuff
}
},
"Build Utilities" : {
node('<Label>'){
/// your stuff
}
}
)
}
}
All VMs should be labelled as to use as a pool.
I have just tested the following pipeline and it works
parallel firstBranch: {
stage ('Starting Test')
{
build job: 'test1', parameters: [string(name: 'Environment', value: "$env.Environment")]
}
}, secondBranch: {
stage ('Starting Test2')
{
build job: 'test2', parameters: [string(name: 'Environment', value: "$env.Environment")]
}
}
This Job named 'trigger-test' accepts one parameter named 'Environment'
Job 'test1' and 'test2' are simple jobs:
Example for 'test1'
One parameter named 'Environment'
Pipeline : echo "$env.Environment-TEST1"
On execution, I am able to see both stages running in the same time
I think this has been officially implemented now:
https://jenkins.io/blog/2017/09/25/declarative-1/
As #Quartz mentioned, you can do something like
stage('Tests') {
parallel(
'Unit Tests': {
container('node') {
sh("npm test --cat=unit")
}
},
'API Tests': {
container('node') {
sh("npm test --cat=acceptance")
}
}
)
}