I'm trying to figure out what options do I have,
when trying to build a good pipeline for CICD for a monorepo,
I'm trying to have something like this (this is only a pseudo pipeline)
and not really what I'm using ATM in my monorepo (or what I will have).
Explanation:
Pre: understand what I should build, test, etc..
Build dynamically a parallel step which will give me the later explained capabilities.
Foo: run the parallel and comfortably wait:)
This is the only way I thought of getting this features:
* Build process among the P’s can be shared and I can generate some waitUntil statements
to make this works, I guess...
* Every P’s is independent from the other, if one Ut of P2 fails f.e, it doesn't affect the other progress
of the pipeline, or if I want, it's only a failFast configuration
* Every step within the way is again not related to the progress of other P’s,
so when Ut finishes in any of the P's it starts immediately it's St.
(thought this might changed according to some configuration I'll probably need)
The main problems with that is:
1. I'm losing the control the Restart single steps (since I can only restart Top level steps)
2. It requires me to do a lot more with Scripted Pipeline, which looks like the support of BlueOcean
(which is kind of critical to me), is questionable...
seems that BlueOcean is more supported within the scope of the Declarative Pipeline.
NOTE: It probably looks like I can split every P’s to a another jenkins job
but, this will require me to wait a lot of time in checkout workspace+preparation of the monorepo,
and like I said the "build" step may have shared between the P’s and it's more efficient to do this like that
I will appreciate every feedback or any suggestion:)
There's no problem whatsoever with doing what you want with a Declarative pipeline, since stage can have a stages child. So:
pipeline {
stages {
stage("Pre") { }
stage("Foo") {
parallel {
stage ("P1") {
stages {
stage("P1-Build") {}
stage("P1-Ut") {}
stage("P1-St") {}
}
}
stage ("P2") {
stages {
stage("P2-Build") {}
stage("P2-Ut") {}
}
}
// etc..
Stages P1..P4 will run in parallel but within each their Build-unittest-test stages will run sequentially.
You won't be able to restart separate stages but it's not a good feature anyway.
Related
We have an existing Jenkins install that is testing firmware running on an embedded tart. The multi-stage pipeline looks something like: Checkout -> Build -> Download -> Smoke tests -> Unit tests. This is working great, except it takes 9 hours to run the pipeline. To speed things up and also to test different target variants we have added 3 more targets to the system (UUT#1, #2, and so on).
My question is, what is the most straightforward way to allow the parallelization happen while also restricting the suites to UUTs with specific properties. For example, our Unit tests contain about 10 different suites (suite1 suite2 and so on), and what I’d like to do is spread those out amongst the 4 UUTs (thus having 4 suites running at a time) but restrict the execution this way:
Suite1 can only run on a UUT that has ‘USB’
Suite2 can only run on a UUT that has ‘LCD-display’
Suite3 can run anywhere
.. and so on, then my UUTs might have properties like:
UUT#1 ‘USB LCD-display’
UUT#2 ‘Ethernet’
UUT#3 ‘RS-232 USB’
Etc.
Reading about agents, it seems that a label on an agent may allow this, but agents seem to carry a lot of overhead and I’m not sure if they’re appropriate.
Long-time Jenkins user, but this is the first time I’ve ever attempted anything this complicated and pipelines are a new concept for me.
A straightforward way is to use the Lockable Resources plugin.
This can be used as a step as well as a stage option (undocumented). The latter comes in handy if you have nested stages which all depend on the resource to be locked.
Stage option in declarative pipeline
pipeline {
agent any
stages {
stage('Test') {
options {
// Lock a single resource from all resources labeled 'mylabel'
lock( label: 'mylabel',
quantity: 1,
variable: 'MyResourceName' )
}
steps { // or 'parallel' or 'stages'
echo "Locked resource $MyResourceName"
sleep 10
echo "Resource will be unlocked after this stage"
}
}
}
}
Step in scripted pipeline
node {
stage('Test') {
lock( label: 'mylabel',
quantity: 1,
variable: 'MyResourceName' ) {
echo "Locked resource $MyResourceName"
sleep 10
echo "Resource will be unlocked after this stage"
}
}
}
Caveats
If lock is used as a step in declarative pipeline, you may get an error:
Missing required parameter: "resource"
This seems to be a little bug in argument checking. According to the documentation, you only need to specify either resource or label parameter. Simply pass null as the value for this parameter.
If parameter quantity is not specified, all resources that match the given label will be locked.
You can use propagate on a build job as described here:
https://jenkins.io/doc/pipeline/steps/pipeline-build-step/
So you can use something like this to prevent a failing step from failing the complete build:
build(job: 'example-job', propagate: false)
Is there a way to use this for a stage or a step? I know i can surround it with a try/catch and that does works almost as i want. It does ignore the failing the stage and resumes the rest of the build, but it does not display the stage as failed. For now i write all failing stages to a variable and output that on a later stage, but this is not ideal.
If i cant suppress propagation in a stage/step, is there maybe a way to use the build() call to do the same? Maybe if i move it to another pipeline and call that via build()?
Any help appreciated.
With catchError you can prevent a failing step from failing the complete build:
pipeline {
agent any
stages {
stage('1') {
steps {
sh 'exit 0'
}
}
stage('2') {
steps {
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
sh "exit 1"
}
}
}
stage('3') {
steps {
sh 'exit 0'
}
}
}
}
In the example above, all stages will execute, the pipeline will be successful, but stage 2 will show as failed:
As you might have guessed, you can freely choose the buildResult and stageResult, in case you want it to be unstable or anything else. You can even fail the build and continue the execution of the pipeline.
Just make sure your Jenkins is up to date, since this is a fairly new feature.
There are currently lots of suggestions for the scripted syntax, but for the declarative syntax work is in progress to support this.
Track the progress of https://issues.jenkins-ci.org/browse/JENKINS-26522 which groups all of the pieces together to achieve this. It has some interesting bits already marked as 'Resolved' (meaning a code change was made), such as https://issues.jenkins-ci.org/browse/JENKINS-49764 ( "Allow define a custom status for pipeline stage"). Unfortunately, I cannot find references to any of the tickets involved in the Jenkins changelog which would make sense since the parent ticket is not yet finished.
Of interest might also be the following : https://issues.jenkins-ci.org/browse/JENKINS-45579 which was reopened due to an issue. The environment for this is :
Admittedly, there are a confusing number of tickets tracking this work, but that is probably due to the fact that the functionality being implemented has a number of use-cases.
Another interesting ticket is "Individual Pipeline steps and stages/blocks should have Result statuses" , for which I was able to find a related PR: https://github.com/jenkinsci/workflow-api-plugin/pull/63
It is worth noting that the declarative pipeline was always designed as being opinionated and as such was not meant to support everything possible with the scripted syntax. For more complicated workflows and use-cases where it does not serve your needs, scripted syntax may be the only (and recommended?) option.
For needs such as the one you've stated, if enough noise is made, the declarative pipeline will probably be modified in due course to support it.
tell me please, can I integrate one jenkinsfile with declarative pipeline into another?
The idea is to run in parallel several processes that are divided into different jenkinsfiles.
stage('run-parallel-branches') {
steps {
parallel(
a: {
echo "call Jenkinsfile 1"
},
b: {
echo "call Jenkinsfile 2"
}
)
}
}
Thanks for the help.
You have two solutions:
If your goal is to avoid repeating code, you can use Shared Libraries and extract the work done in Jenkinsfile1 and Jenkinsfile2 into a library, and then call this library instead.
Your two Jenkinsfile can have their dedicated jobs, and you would call them in the parallel stages (it will wait and propagate errors): build 'myJob1' & build 'myJob2'
The problem with this approach, however, is that the invoked job will not necessarily have the same SCM commit as the parent one (if commits entered meanwhile).
As both node step and stage step provide scoped {} syntax, what is the best practice for defining their topology inside groovy code?
Exhibit A
node ("NodeName") {
stage ("a stage inside node"){
// do stuff here
}
}
Exhibit B
stage ("a stage holding a node") {
node ("NodeName"){
// do stuff here
}
}
This depends on your actual needs.
As long as you can run your complete pipeline on a single node, I would wrap the stages in a node so that the pipeline is not blocked by busy executors.
As soon as you use the parallel step, then you don't really have a choice besides having stage around node allocations.
There are (at least for me) no issues around mixing that, i.e., have the first 2-3 stages executed on the same node and then one stage that executes on multiple nodes within parallel.
With node { stage { ... } } each stage will share the same working folder and all the files from the previous stage will be there for the next stage.
With stage { node { ... } } you need to stash/unstash files between each stage. If you have a large repository, and especially if you have a large folder of dependencies like node_modules, this repeated stash/unstash could end up being a significant, or even majority, or your build time.
IMO I would generally start with the first syntax, node { stage { ... } } as preferred. If you have individual build stages that take time and can benefit from parallelism, then switching to stage { node { ... } } might be better, as long as the time gained in parallelization is not lost in stashing.
Update:
I tested the exact effect of swapping nesting on one of our builds. with a bunch of stages inside a node, the total build time is just over one minute. With a node inside each stage, total build time is almost five minutes. Big difference.
The Jenkins Pipeline plugin (aka Workflow) can be extended with other Multibranch plugins to build branches and pull requests automatically.
What would be the preferred way to run multiple configurations? For example, building with Java 7 and Java 8. This is often called matrix configuration (because of the multiple combinations such as language version, framework version, ...) or build variants.
I tried:
executing them serially as separate stage steps. Good, but takes more time than necessary.
executing them inside a parallel step, with or without nodes allocated inside them. Works but I cannot use the stage step inside parallel for known limitations on how it would be visualized.
Is there a recommended way to do this?
TLDR: Jenkins.io wants you to use nodes for each build.
Jenkins.io: In pipeline coding contexts, a "node" is a step that does two things, typically by enlisting help from available executors on agents:
Schedules the steps contained within it to run by adding them to the Jenkins build queue (so that as soon as an executor slot is free on a node, the appropriate steps run)
It is a best practice to do all material work, such as building or running shell scripts, within nodes, because node blocks in a stage tell Jenkins that the steps within them are resource-intensive enough to be scheduled, request help from the agent pool, and lock a workspace only as long as they need it.
Vanilla Jenkins Node blocks within a stage would look like:
stage 'build' {
node('java7-build'){ ... }
node('java8-build'){ ... }
}
Further extending this notion Cloudbees writes about parallelism and distributed builds with Jenkins. Cloudbees workflow for you might look like:
stage 'build' {
parallel 'java7-build':{
node('mvn-java7'){ ... }
}, 'java8-build':{
node('mvn-java8'){ ... }
}
}
Your requirements of visualizing the different builds in the pipeline would could be satisfied with either workflow, but I trust the Jenkins documentation for best practice.
EDIT
To address the visualization #Stephen would like to see, He's right - it doesn't work! The issue has been raised with Jenkins and is documented here, the resolution of involving the use of 'labelled blocks' is still in progress :-(
Q: Is there documentation letting pipeline users not to put stages inside of parallel steps?
A: No, and this is considered to be an incorrect usage if it is done; stages are only valid as top-level constructs in the pipeline, which is why the notion of labelled blocks as a separate construct has come to be ... And by that, I mean remove stages from parallel steps within my pipeline.
If you try to use a stage in a parallel job, you're going to have a bad time.
ERROR: The ‘stage’ step must not be used inside a ‘parallel’ block.
I would suggest Declarative Matrix as a preferred way to run multiple configurations in Jenkins. It allows you to execute the defined stages for every configuration without code duplication.
Example:
pipeline {
agent none
stages {
stage('Test') {
matrix {
agent {
label "${NODENAME}"
}
axes {
axis {
name 'NODENAME'
values 'java7node', 'java8node'
}
}
stages {
stage('Test') {
steps {
echo "Do Test for ${NODENAME}"
}
}
}
}
}
}
}
Note that declarative Matrix is a native declarative Pipeline feature, so no additional Plugin installation needed.
Jenkins blog post about the matrix directive.
As noted by #StephenKing, Blue Ocean will show parallel branches better than the current stage view. A planned upcoming version of the stage view will be able to show all the branches, though it will not visually indicate any nesting structure (would look the same as if you ran the configurations serially).
In any event, the deeper issue is that you will essentially only get a pass/fail status for the build overall, pending a resolution to JENKINS-27395 and related requests.
In order to test each commit on several platforms, I've used this base Jenkinsfile skeleton:
def test_platform(label, with_stages = false)
{
node(label)
{
// Checkout
if (with_stages) stage label + ' Checkout'
...
// Build
if (with_stages) stage label + ' Build'
...
// Tests
if (with_stages) stage label + ' Tests'
...
}
}
/*
parallel ( failFast: false,
Windows: { test_platform("Windows") },
Linux: { test_platform("Linux") },
Mac: { test_platform("Mac") },
)
*/
test_platform("Windows", true)
test_platform("Mac", true)
test_platform("Linux", true)
With this it's relatively easy to switch from a sequential to a parallel execution, each of them having their pros and cons:
Parallel execution runs much faster, but it doesn't contain the stages labelling
Sequential execution is much slower, but you get a detailed report thanks to stages, labelled as "Windows Checkout", "Windows Build", "Windows Tests", "Mac Checkout", etc.)
I'm using the sequential execution for the time being, until I find a better solution.
It seems like there is relief coming at least with the BlueOcean UI. Here is what I got (the tk-* nodes are the parallel steps):