I'm running some tests with Jenkins Pipeline using parallelism and I noticed that Blue Ocean behaves weirdly with job results. I declared a global class with a variable to handle the current build status based on the parallel jobs results.
It works, however the parallel steps status does not seem to be getting set correctly within the Blue Ocean UI.
I prepared an example below to show what is going on. The pipeline definition is:
pipeline {
stages {
stage('Tests') {
steps {
script {
parallel(
a: {
echo "This is branch a"
manager.build.#result = hudson.model.Result.SUCCESS
},
b: {
echo "This is branch b"
manager.build.#result = hudson.model.Result.ABORTED
},
c: {
echo "This is branch c"
manager.build.#result = hudson.model.Result.UNSTABLE
},
d: {
echo "This is branch d"
manager.build.#result = hudson.model.Result.FAILURE
try {sh "exit 1"}catch(e){echo 'failure'}
}
)
manager.build.#result = hudson.model.Result.UNSTABLE
}
}
}
}
}
The last row changes the build result correctly (whatever value I give it - SUCCESS, FAILURE, UNSTABLE or ABORTED).
However, the result for each parallel step are always SUCCESS. The only exception is when I give the general build the status of UNSTABLE, then all parallel steps become UNSTABLE as well. Below follows an illustration:
I expected each parallel step to have it's own result, it would be very useful for debugging test results, since I need to run tests in different machines / devices in parallel.
Is it a Blue Ocean problem or am I doing something wrong here?
Related
I want to add a matrix section to the following pipeline. I want the pipeline to run on 4 nodes with each node running a different stage that is specified in the for loop (e.g. one node runs AD, the other runs CD, the other runs DC, and the last runs DISP_A. It then repeats this behavior for the rest of the list until it is done iterating to the end of the list).
I have looked at the documentation and have not come up with any concrete answers to my question.
pipeline
{
agent none
stages
{
stage ('Test')
{
steps
{
script
{
def test_proj_choices = ['AD', 'CD', 'DC', 'DISP_A', 'DISP_PROC', 'EGI', 'FD', 'FLT', 'FMS_C', 'IFF', 'liblO', 'libNGC', 'libSC', 'MISCMP_MP', 'MISCMP_GP', 'NAV_MGR', 'RADALT', 'SYS', 'SYSIO15', 'SYSIO42', 'SYSRED', 'TACAN', 'VOR_ILS', 'VPA', 'WAAS', 'WCA']
for (choice in test_proj_choices)
{
stage ("${choice}")
{
echo "Running ${choice}"
build job: "UH60Job", parameters: [string(name: "TEST_PROJECT", value: choice), string(name: "SCADE_SUITE_TEST_ACTION", value: "all"), string(name: "VIEW_ROOT", value: "myview")]
}
}
}
}
}
}
}
I don't think your expectation can be decorated with Jenkins matrix DSL, as matrix DSL works like a single or multi-dimensional array.
But you can do something similar by writing a small groovy logic.
Below is one small example similar to your expectation:
The expectation of this example will run one task in one Jenkins agent in a distributed fashion.
Meaning it will run this way (Task - agent) A - will run on- agent1, B - will run on- agent2, C -will run on- agent3, D - will run on- agent4, E - will run on- agent1, ....
node {
agent=['agent1','agent2','agent3', 'agent4']
tasks=['A','B','C','D','E','F','G','H','I','J','K','L']
int nodeCount=0
tasks.each {
node(agent[nodeCount]) {
stage("build") {
println "Task - ${it} running on ${agent[nodeCount]}"
}
}
nodeCount=nodeCount+1
if(nodeCount == agent.size()){
nodeCount=0
}
}
}
Jenkins agent no need to be hardcode, by using Jenkins groovy api, you can easily find all the available and active agent from Jenkins, like below.
agent=[]
for (a in hudson.model.Hudson.instance.slaves) {
if(!a.getComputer().isOffline()) {
agent << a.name
}
}
I know that in a scripted jenkins pipeline I can use Utils.markStageSkippedForConditional to mark an entire stage as skipped, which gets the 'execution line' to dip around the stage in the Blue Ocean UI.
But if I've got a stage that has parallel tasks under it, something like this:
stage('Stage 1') {
parallel 'Task 1' : {
node('requirements') {
doTask1()
}
}, 'Task 2' : {
when (mustDoTask2) { // this 'when' function calls Utils.markStageSkippedForConditional
node('requirements') {
doTask2()
}
}
}
}
Then the entire stage is marked as skipped (unsurprisingly I suppose given the Utils function name) and Blue Ocean renders the pipeline as never executing any of the parallel tasks (even though they do execute).
So is there an alternative to Utils.markStageSkippedForConditional that I can use to just mark an individual parallel task as skipped?
The alternative I've got at the moment is for the parallel tasks to check within the 'node' statement as to whether they should execute anything, which works but when you look at the pipeline in Blue Ocean it looks like those tasks are always executed, which isn't ideal.
#PatriceM. suggested putting each task into its own stage, which is the best solution I've got at the moment:
stage('Stage 1') {
parallel 'Task 1' : {
node('requirements') {
doTask1()
}
}, 'Task 2' : {
stage('Task 2') { // new nested stage that can be marked as skipped by the 'when'
when (mustDoTask2) {
node('requirements') {
doTask2()
}
}
}
}
}
Note that the 'Task 2' parallel task has its own stage. This means the 'when' can mark that stage as skipped, rather than the parent stage 'Stage 1'. It's not necessary to add the extra nested stage to 'Task 1' as it doesn't have a 'when' to potentially skip it.
This results in Blue Ocean showing the skipped stages in grey (not with the line dipping around them) and has the log 'queued waiting for run to start' until the entire run has finished when it will change to 'this stage has no steps' and shows the final logs of the entire run. It's not 100% perfect but shows what's actually happened fairly well so good enough for now!
I have a very long and complex pipeline that I'm rewriting after a major jenkins upgrade.
What I'd like to to is declaring my stages as variables,then execute them in the main node body: I can do this easily for the parallel stages, but I want to have the same style also for the sequential ones.
After a lot of tests, the only way I found to make this work was using "fake" parallel calls around all single sequential stages (ugly), I'm sure there is a better solution but seems like I can't find the proper step...shame on me.
Here's my example:
stage1 = { stage("one") {
println "stage one"
} }
stage2 = { stage("two") {
println "stage two"
} }
stage3 = { stage("three") {
println "stage three"
} }
node {
parallel (
"one" : stage1 ,
"two" : stage2
)
HERE I WANT TO CALL stage3 Closure, possibly giving a map like in the parallel above
}
You should be able to do this with the run method.
stage3.run()
I do not know if this is safe to use.
I'm looking into moving our scripted pipelines to declarative pipelines.
I'm using the when key word to skip stages
stage('test') {
// Only do anything if we are on the master branch
when { branch 'master' }
//...
}
This works, however the skipped stage is shown as green. I would prefer if it was shown as gray in the pipeline overview. Is there a way to achieve this?
As you mentioned in your comment I suggest you to use Jenkins Blue Ocean when working with pipelines.
It provides a more modern and user friendly view for your pipeline projects. Even the pipeline itself is display in a much more convenient way.
If the stage is appearing green for you then it's likely still actually running. A skipped stage should look like this in the Jenkins classic stage view. Consider the following code sample, which has three stages, the middle stage being skipped conditionally with the when directive.
pipeline {
agent any
stages {
stage('Always run 1') {
steps { echo "hello world" }
}
stage('Conditionally run') {
when {
expression { return false }
}
steps { echo "doesn't get printed" }
}
stage("Always run 2") {
steps { echo "hello world again" }
}
}
}
This should produce the following line in your build log
Stage "Conditionally run" skipped due to when conditional
Another answerer of this question mentioned Blue Ocean, which definitely presents a beautiful presentation of the stage view. Here is an image of how a skipped stage looks in the Blue Ocean stage view. Note that Blue Ocean is a UI and your job's underlying pipeline code will be the same regardless of which UI you choose to use.
Please note: the question is based on the old, now called "scripted" pipeline format. When using "declarative pipelines", parallel blocks can be nested inside of stage blocks (see Parallel stages with Declarative Pipeline 1.2).
I'm wondering how parallel steps are supposed to work with Jenkins workflow/pipeline plugin, esp. how to mix them with build stages. I know about the general pattern:
parallel(firstTask: {
// Do some stuff
}, secondTask: {
// Do some other stuff in parallel
})
However, I'd like to run couple of stages in parallel (on the same node, which has multiple executors), so I tried to add stages like this:
stage 'A'
// Do some preparation stuff
parallel(firstTask: {
stage 'B1'
// Do some stuff
}, secondTask: {
stage 'B2'
// Do some other stuff in parallel
})
stage 'C'
// Finalizing stuff
This does not work as expected. The "do stuff" tasks are executed in parallel, but the parallel stages end immediately and do not incorporate the stuff they should contain. As a consequence, the Stage View does not show the correct result and also does not link the logs.
Can I build different stages in parallel, or is the "parallel" step only meant to be used within a single stage?
You may not place the deprecated non-block-scoped stage (as in the original question) inside parallel.
As of JENKINS-26107, stage takes a block argument. You may put parallel inside stage or stage inside parallel or stage inside stage etc. However visualizations of the build are not guaranteed to support all nestings; in particular
The built-in Pipeline Steps (a “tree table” listing every step run by the build) shows arbitrary stage nesting.
The Pipeline Stage View plugin will currently only display a linear list of stages, in the order they started, regardless of nesting structure.
Blue Ocean will display top-level stages, plus parallel branches inside a top-level stage, but currently no more.
JENKINS-27394, if implemented, would display arbitrarily nested stages.
that syntax is now deprecated, you will get this error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 14: Expected a stage # line 14, column 9.
parallel firstTask: {
^
WorkflowScript: 14: Stage does not have a name # line 14, column 9.
parallel secondTask: {
^
2 errors
You should do something like:
stage("Parallel") {
steps {
parallel (
"firstTask" : {
//do some stuff
},
"secondTask" : {
// Do some other stuff in parallel
}
)
}
}
Just to add the use of node here, to distribute jobs across multiple build servers/ VMs:
pipeline {
stages {
stage("Work 1"){
steps{
parallel ( "Build common Library":
{
node('<Label>'){
/// your stuff
}
},
"Build Utilities" : {
node('<Label>'){
/// your stuff
}
}
)
}
}
All VMs should be labelled as to use as a pool.
I have just tested the following pipeline and it works
parallel firstBranch: {
stage ('Starting Test')
{
build job: 'test1', parameters: [string(name: 'Environment', value: "$env.Environment")]
}
}, secondBranch: {
stage ('Starting Test2')
{
build job: 'test2', parameters: [string(name: 'Environment', value: "$env.Environment")]
}
}
This Job named 'trigger-test' accepts one parameter named 'Environment'
Job 'test1' and 'test2' are simple jobs:
Example for 'test1'
One parameter named 'Environment'
Pipeline : echo "$env.Environment-TEST1"
On execution, I am able to see both stages running in the same time
I think this has been officially implemented now:
https://jenkins.io/blog/2017/09/25/declarative-1/
As #Quartz mentioned, you can do something like
stage('Tests') {
parallel(
'Unit Tests': {
container('node') {
sh("npm test --cat=unit")
}
},
'API Tests': {
container('node') {
sh("npm test --cat=acceptance")
}
}
)
}