I am trying to run a job with parallel and sequential stages. My code looks like this, each stage will be a groovy script therefore I am running with parallel name: and not parallel {}
stage('Start profiling') {
steps {
script{
parallel stageParallel: {
stage("parallel"){
echo"parallel"
}
}, runTest: {
stage("sequential 1"){
echo "sequential 1 "
}
stage("sequential 2"){
echo "sequential 2 "
}
}
}
}
}
Image of Blue Ocean
As you can see it gets executed, but can´t see it on Open Blue Ocean as a step.
Image of stage view
Displaying sequential stages within parallel stages tends to be flaky in Blue Ocean. There is an open bug relating to your problem in the Jenkins bug backlog, but it hasn't been looked into since 2019. Unfortunately there is little you can do from your end since you pipeline code is obviously correct.
Related
Is it possible in a Matrix-cell of a Jenkins declarative pipeline to execute a later stage on the pipeline's 'initial/main' agent, after a previous stage of the same cell has been executed on another agent (identified by label)?
To put this into context, I want to build native-binaries for different platforms in a Jenkins declarative pipeline using a matrix stage where each cell is responsible to
collect the native's sources for that platform
build the native-binaries from the sources for that platform
collect the just build native-binaries and distribute them to the platform specific artefacts
Step two has to be performed on special agents, which are prepared to build the binaries for a particular platform and are identified by their label. Step one and three has to be performed on the initial agent, the pipeline's 'main' agent where the sources are checkout from SCM. In the end the native-binaries are bundled together and distributed from the pipeline's inital/main agent. To transfer of sources and binaries stash/unstash is used.
A exemplary, simplified pseudo pipeline would look like:
pipeline {
agent { label 'basic' }
// Declarative SCM checkout configured in the mutli-branch pipeline job-config
stages {
stage('Build binaries') {
matrix {
axes {
axis {
name 'PLATFORM'
values 'linux', 'windows'
}
}
stages {
stage("Collect sources") {
steps {
<Collect native's sources for ${PLATFORM}> in "${WORKSPACE}/native.sources.${PLATFORM}"
dir("native.sources.${PLATFORM}") {
stash "sources.${PLATFORM}"
}
}
}
stage('Build binaries') {
options { skipDefaultCheckout() }
agent { label "natives-${PLATFORM}" }
steps {
unstash "sources.${PLATFORM}"
<Build native binaries from unstashed sources into 'libs' folder >
dir('libs') {
stash "binaries.${PLATFORM}"
}
}
}
stage('Collect and distribute binaries') {
agent {
<initial/pipeline-agent>
}
steps {
dir("libs.${PLATFORM}") {
unstash "binaries.${PLATFORM}"
}
}
}
}
}
}
stage('Bundle and distribute') {
...
}
}
}
But the question is, how do I tell Jenkins to execute the third stage of the matrix on the initial/pipeline agent again?
If I simply don't specify an agent for the third-stage the execution is:
Stage on Pipeline-Agent
Stage on Native-Build-Agent
Stage on Native-Build-Agent
but I want:
Stage on Pipeline-Agent
Stage on Native-Build-Agent
Stage on Pipeline-Agent
In the syntax-reference I didn't find a agent parameter like agent { <initial/pipeline-agent> }:
https://www.jenkins.io/doc/book/pipeline/syntax/#agent
https://www.jenkins.io/doc/book/pipeline/syntax/#matrix-cell-directives
The agent section describes a boolean option reuseNode, but it is only "valid for docker and dockerfile".
The only workaround I found so far, was to define a second matrix and move the execution of the third stage to that. This works as expect and the stage is executed on the pipeline-agent, but has the drawback that the matrix-stage has to be specified twice as well as its when-conditions.
Appendix
The problem probably also exists, when using per stage agents in an ordinary linear pipeline.
So I spent the whole day trying to figure out how to configure a simple Jenkins Pipeline with multiple Docker images and I am not happy at all.
I need a few stages (prepare, build, test, docs) executed on a couple of different docker containers (currently I just picked three standard Python containers). And it would be nice if those would run in parallel, but I only found this solution, which combines all stages into a single one (and thus creates a not so informative overview in the Blue Ocean UI): Jenkins Pipeline Across Multiple Docker Images
So I ended up with the configuration below, which is ugly as hell (code repetition everywhere), but more or less creates an good looking overview in the classic UI:
A not so informative overview in the Blue Ocean UI
And an acceptable test overview from junit, which combines all the tests from each stage but if any test is failing, the corresponding "version" is shown:
The most annoying thing however is, you cannot see which step has failed. If Python 2.7 fails, everything else is also marked as failed and you don't even see which stage failed.
I tried so many different approaches and I am wondering how this should be done. This should be such a common thing to do with Jenkins, so I guess I have some general misunderstandings in this (for me absolutely new) pipeline/nodes/labels/stages/steps/declarative/scripted/groovy/blueocean stuff...
It should be possible to define a list of docker containers some (maybe customisable stages/steps) for each of them and run them in parallel and having it displayed nicely in Blue Ocean and in Classic UI, shouldn't it?
node {
stage("Python 2.7.14") {
checkout scm
docker.image('python:2.7.14').inside { // just a dummy for now
stage("Prepare") { sh 'python --version' }
stage("Build") { sh 'ls -al' }
}
}
stage("Python 3.5.4") {
checkout scm
docker.image('python:3.5.4').inside {
stage("Prepare") { sh 'python -m venv venv' }
stage("Build") {
sh """
. venv/bin/activate
make install-dev
"""
}
stage('Test') {
sh """
. venv/bin/activate
make test
"""
}
stage('Docs') {
sh """
. venv/bin/activate
make doc-dependencies
cd docs
make html
"""
}
}
}
stage("Python 3.6.4") {
checkout scm
docker.image('python:3.5.4').inside {
stage("Prepare") { sh 'python -m venv venv' }
stage("Build") {
sh """
. venv/bin/activate
make install-dev
"""
}
stage('Test') {
sh """
. venv/bin/activate
make test
"""
}
stage('Docs') {
sh """
. venv/bin/activate
make doc-dependencies
cd docs
make html
"""
}
}
}
}
Update: this is how it looks like in the Blue Ocean UI when a step fails, int this case "Test" in both Python 3.5.4 and 3.6.4 failed but it looks like everything has failed.
Also the Python 2.7.14 and 3.5.4 stages are collapsed and cannot be viewed separately. If I click on one of them, all the steps are shown in green although in this case . venv/bin/activate make test failed:
So this is what I ended up with. There are surely better solutions, but I have to move on. I hope to gather some (better) answers in time, I'll not mark this as "the solution" yet ;)
First, some credits to Stephen Kings slides (the title says "Declarative" but there are some nice examples regarding the scripted Pipeline): (Declarative) Jenkins Pipelines
Here is my gist on GitHub with the following snippet:
def docker_images = ["python:2.7.14", "python:3.5.4", "python:3.6.2"]
def get_stages(docker_image) {
stages = {
docker.image(docker_image).inside {
stage("${docker_image}") {
echo 'Running in ${docker_image}'
}
stage("Stage A") {
switch (docker_image) {
case "python:2.7.14":
sh 'exit 123' // for python 2.7.14 we force an error for fun
break
default:
sh 'sleep 10' // for any other docker image, we sleep 10s
}
sh 'echo this is stage A' // this is executed for all
}
stage("Stage B") {
sh 'sleep 5'
sh 'echo this is stage B'
}
stage("Stage C") {
sh 'sleep 8'
sh 'echo this is stage C'
}
}
}
return stages
}
node('master') {
def stages = [:]
for (int i = 0; i < docker_images.size(); i++) {
def docker_image = docker_images[i]
stages[docker_image] = get_stages(docker_image)
}
parallel stages
}
I tried to make it easy to use:
you add your Docker images in a list at the top and then you define the stages in the get_stages() function
add the common stages and steps
if any Docker image needs special treatment (like python:2.7.14 in my example), you can use a simple switch. This could also be realised with a double map for the special cases ('images'->'stage'='steps') and a fallback double map for defaults, but I'll leave it as an exercise for the reader. (to be honest, I could not figure out the correct, supported Groovy-lang syntax)
This is how it looks like when everything is fine in both the Classic and the Blue Ocean UIs (it's known that the Blue Ocean UI fails to display multiple stages in parallel runs, see JENKINS-38442):
Classic UI
Blue Ocean UI
And this is the output if Stage A in python:2.7.14 fails:
Classic UI
Blue Ocean UI
I'm going to use Jenkins pipeline plugin to test several binaries A B C on several nodes 1 2 3.
In the end of my test I would like to have every single result of all possible combinations. So my Pipe may not abort when a single stage fails. It should proceed.
eg: A1 green, A2 green, A3 red, B1 green, B2 red, ..., C3 green
But when the first binary returns with an value unequal zero ("Binary not working on the system") it's stage is marked as FAILURE and any other stages are skipped.
Is there a possibility in Jenkins Pipeline to mark a stage as "UNSTABLE" but proceed with running the other tests?
According to Continue Jenkins job after failed stage while marking stage as failed can't mark this step as failed. The solution of this in running tasks in parallel is not working for my setup. So is it possible to safely mark it as something else? Is it possible to manipulate the result of a stage?
This question How to continue past a failing stage in Jenkins declarative pipeline syntax intents to use a scripted pipeline. I would like to avoid that if it is possible to do it in an other way.
pipeline {
agent {label 'master'}
stages {
stage('A1') {
agent {label 'Node1'}
steps {
sh 'binA'
}
}
stage('A2') {
agent {label 'Node1'}
steps {
sh 'binB' // If this bin fails, all following stages are skipped
}
}
// ...
stage('C3'){
agent {label 'Node3'}
steps {
sh 'binC'
}
}
}
}
Declarative Pipeline: Though using currentBuild.result = 'UNSTABLE' works in declarative pipelines too, Blue Ocean displays all stages as unstable irrespective of which stage fails.
To mark only specific stages as unstable, use the step unstable(message: String) as described here within your stage and install/update the following plugins:
Pipeline: Basic Steps to 2.16 or newer
Pipeline: API Plugin to 2.34 or newer
Pipeline: Groovy to 2.70 or newer
Pipeline Graph Analysis to 1.10 or newer
Sample pipeline stage:
stage('Sign Code') {
steps {
script {
try {
pwd()
sh "<YOUR SCRIPT HERE>"
}
catch (err) {
unstable(message: "${STAGE_NAME} is unstable")
}
}
}
}
Note: This also marks the overall build status as unstable.
There is now a more elegant solution, that not only allows you to set a stage and the job result to unstable. Using catchError, you can set any combination of stage and build result:
pipeline {
agent any
stages {
stage('1') {
steps {
sh 'exit 0'
}
}
stage('2') {
steps {
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
sh "exit 1"
}
}
}
stage('3') {
steps {
sh 'exit 0'
}
}
}
}
In the example above, all stages will execute, the pipeline will be successful, but stage 2 will show as failed:
As mentioned above, you can freely choose the buildResult and stageResult. You can even fail the build and continue the execution of the pipeline.
Just make sure your Jenkins is up to date, since this is a fairly new feature. (Pipeline: Basic Steps needs to be 2.18 or newer)
For scripted pipeline, you can use try .. catch blocks inside the stages and then set currentBuild.result = 'UNSTABLE'
in the exception handler.
I have some code that needs running (build, test, and packages in actuality but for example just running tox) on different OSes. Currently my Jenkinsfile looks like thus:
pipeline {
// Where to run stuff.
agent {
node {
label 'CentOS7'
customWorkspace '/home/build/jenkins/workspace/pipelines/ook'
}
}
// What to run goes here.
stages {
stage('Tox') {
steps {
sh 'tox -v --recreate'
}
}
}
// Clean up after ourselves.
post {
failure {
mail subject: "\u2639 ${env.JOB_NAME} (${env.BUILD_NUMBER}) has failed",
body: """Build ${env.BUILD_URL} is failing!
Somebody should do something about that\u2026""",
to: "devs#example.com",
replyTo: "devs#example.com",
from: 'jenkins#example.com'
}
}
}
}
The middle bit, I want to run on two different nodes: one for OS 1 and one for OS 2.
How do I do that?
Sure, you would want to label your slave nodes somehow. I didn't look up what tox is, but maybe like 'os_linux' and 'os_mac', and then you can use the node step in your Jenkinsfile to run some commands in the context of each slave. So your Tox stage might look like:
stage('Tox') {
steps {
node('os_linux') {
sh 'tox -v --recreate'
}
node('os_mac') {
sh 'tox -v --recreate'
}
}
}
This will run the tasks in serial, and Jenkinsfile syntax also supports doing those two tox commands in parallel on different nodes. Use the "Pipeline Syntax" link in the left nav of your Jenkins UI (only on pipeline jobs) to play around with node and parallel. Rock on.
I'm building jobs that will be using Jenkins piplines extensively. Our nodes are designated per project by their tags, but unlike regular jobs the pipeline build does not seem to have the "Restrict where this project can be run" checkbox. How can I specify on which node the pipeline will run the way I do for regular jobs?
You specify the desired node or tag when you do the node step:
node('specialSlave') {
// Will run on the slave with name or tag specialSlave
}
See https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#node-allocate-node for an extended explanation of the arguments to node.
Edit 2019: This answer (and question) was made back in 2017, back then there only was one flavor of Jenkins pipeline, scripted pipeline, since then declarative pipeline has been added. So above answer is true for scripted pipeline, for answers regarding declarative pipeline please see other answers below.
Choosing a node which has the label X in a declarative pipeline with json format:
pipeline {
agent { label 'X' }
...
...
}
You also can apply multiple labels with or (||) or with and (&&) operator.
Running the job on any of the nodes which has label X or label Y:
agent { label 'X || Y' }
Running the job only on nodes which have both label:
agent { label 'X && Y' }
More in the Jenkins Pipeline reference guide.
ps: if you are reading this you probably have just started using Jenkins pipeline and you are not sure if you should use declarative or scripted pipeline. Short answer: it's better to start with declarative. From jenkins.io:
Declarative and Scripted Pipelines are constructed fundamentally
differently. Declarative Pipeline is a more recent feature of Jenkins
Pipeline which:
provides richer syntactical features over Scripted Pipeline syntax, and
is designed to make writing and reading Pipeline code easier.
To be clear, because Pipeline has two Syntax, there are two ways to achieve that.
Declarative
pipeline {
agent none
stages {
stage('Build') {
agent { label 'slave-node' }
steps {
echo 'Building..'
sh '''
'''
}
}
}
post {
success {
echo 'This will run only if successful'
}
}
}
Scripted
node('your-node') {
try {
stage 'Build'
node('build-run-on-this-node') {
sh ""
}
} catch(Exception e) {
throw e
}
}
Agent or Node where we should not execute the jenkins job :
This is the negation of the problem statment i.e. node where not to run
It was most weird solution to me but issue has been already raised to jenkins community
agent { label '!build-agent-name' }
If you need to run entire jenkins pipeline to run on single node, use the following format
pipeline {
agent {
label 'test1'
}
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
echo 'Testing..'
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}
if you need to execute each stage in different nodes, use the below format,
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
node("test1"){
echo 'Testing..'
}
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}