I have a supposedly simple task: I want to run a job on multiple agents in parallel.
Even though I'm a bit of a noob with Jenkins, I've googled a bit, and got to the conclusion that the preferred solution is to use Matrix directive.
I've read Matrix official docs and this blog and still can't solve it completely.
But I'm close, so I assume I just need a bit of help.
The agents I need job to run on - have label: 'vms'.
The below pipeline will run job on some of required agents that have 'vms' label - equivalent to the amount of values for DUMMY_AXIS axis.
For example, 'vms' label has 3 agents, pipeline below will run stages on 2 out of 3.
How to fix the issue, so that stages would run once on each agent from given label , regardless of how many agents there are?
pipeline {
agent none
stages {
stage('Update TestHostAgent') {
matrix {
agent {
label 'vms'
}
axes {
axis {
name 'DUMMY_AXIS'
values 'dummy_val_1', 'dummy_val_2'
}
}
stages {
stage('Build') {
steps {
echo "Build stage"
}
}
stage('Test') {
steps {
script {
echo "Test Step"
}
}
}
}
}
}
}
}
Related
What is modern best practice for multi-configuration builds (with Jenkins)?
I want to support multiple branches and multiple configurations.
For example for each version V1, V2 of the software I want builds targeting
platforms P1 and P2.
We have managed to set up multi-branch declarative pipelines. Each build has its own docker so its easy to support multiple platforms.
pipeline {
agent none
stages {
stage('Build, test and deploy for P1) {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P1.Dockerfile'
}
}
steps {
sh buildit...
}
}
stage('Build, test and deploy for P2) {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P2.Dockerfile'
}
}
steps {
sh buildit...
}
}
}
}
This gives one job covering multiple platforms but there is no separate red/blue status for each platform.
There is good argument that this does not matter as you should not release unless the build works for all platforms.
However, I would like a separate status indicator for each configuration. This suggests I should use a multi-configuration build which triggers a parameterised build for each configuration as below (and the linked question):
pipeline {
parameters {
choice(name: 'Platform',choices: ['P1', 'P2'], description: 'Target OS platform', )
}
agent {
filename someMagicToGetDockerfilePathFromPlatform()
}
stages {
stage('Build, test and deploy for P1) {
steps {
sh buildit...
}
}
}
}
There are several problems with this:
A declarative pipeline has more constraints over how it is scripted
Multi-configuration builds cannot trigger declarative pipelines (even with the parameterized triggers plugin I get "project is not buildable").
This also begs the question what use are parameters in declarative pipelines?
Is there a strategy that gives the best of both worlds i.e:
pipeline as code
separate status indicators
limited repetition?
This is a partial answer. I think others with better experience will be able to improve on it.
This is currently untested. I may be barking up the wrong tree.
Please comment or add a better answer.
Do not use pipeline parameters except where you need user input
Use a hybrid of a scripted and declarative pipeline
(see also https://stackoverflow.com/a/46675227/1569204)
Have a function which declares a pipeline based on parameters:
(see also https://jenkins.io/doc/book/pipeline/shared-libraries/)
Use nodes to create visible indicators in the pipeline (at least in blue ocean)
So something like the following:
def build(string platform) {
switch(platform) {
case P1:
dockerFile = 'foo'
indicator = 'build for foo'
break
case P2:
dockerFile = 'bar'
indicator = 'build for bar'
break
}
pipeline {
agent {
dockerfile {
filename "$dockerFile"
}
node {
label "$indicator"
}
}
stages {
steps {
echo "build it"
}
}
}
}
The relevant code could be moved to a shared library (even if you don't actually need to share it).
I think the cleanest approach is to have this all in a pipeline similar to the first one you presented, the only modification I would see here is making those parallel, so you would actually try and build/test for both platforms.
To reuse the previous stage's workspace you could do: reuseNode true
Something similar to this flow, that would have parallel build for platforms
pipeline {
agent 'docker'
stages {
stage('Common pre') { ... }
stage('Build all platforms') {
parallel {
stage('Build, test and deploy for P1') {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P1.Dockerfile'
reuseNode true
}
}
steps {
sh buildit...
}
}
stage('Build, test and deploy for P2') {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P2.Dockerfile'
reuseNode true
}
}
steps {
sh buildit...
}
}
}
}
stage('Common post parallel') { ... }
}
}
I would like to setup parallel stages as described in the image
In this instance, the setup is pretty heavy so has to be done once before the parallel group starts of G1, G2 and G3. At the same time, the stage : Initial Checks has 2 items that I would like to run in parallel.
Is this possible in the Declarative Pipeline or do I have to resort to a script?
I couldnt see in the documentation the ability for this to work
Stages {
stage ('P1') {
}
stage ('P2 Setup') {}
stage ('P2') {
//Here it can contain either Steps or Parallel. I can only do
parallel {
stage ('g1') {} //Parallel tests
stage ('g2') {}
stage ('g3') {}
}
}
stage ('P2 Cleanup') {}
}
Have you encountered similar situations and what have your solutions been like?
Ofcourse 1 solution is to make Setup and Cleanup as part of every group, but like I said, its pretty intensive and I would only take it on if what the diagram indicates isn't possible.
Solution 2
stage ('p2') {
script {
//Some scripting here to get the result?
}
}
Pipeline
This is not supported by a DSL or declarative pipeline yet. You are essentially looking for nested parallel stages as mentioned here
Issue is still open with Jenkins community which you can watch here
In your given case, you can launch stage P1 and stage setup in parallel. However, it is important to start P1 as a background process because from your graph it appears that P1 is a time-intensive operation. Once group stage completes, you can collect the status of P1 and proceed to S2.
stages{
stage('Build'){
steps{
echo "Build"
}
}
stage('Init'){
parallel{
stage('P1'){steps{ echo "launch p1 in background"}}
stage('setup'){steps{echo "setup"}}
}
}
stage('Group'){
parallel{
stage('P1'){steps{echo "p1"}}
stage('P2'){steps{echo "p2"}}
stage('P3'){steps{echo "p3"}}
}
}
stage('Cleanup'){
steps{
echo "cleanup"
}
}
stage('Check P1 status'){
steps{
echo "Check"
}
}
stage('S2'){
steps{
echo "S2"
}
}
}
I think you are looking for this
node {
stage("P1"){}
stage("p2") {}
stage("p3") {
parallel (
"firstTask" : {
},
"secondTask" : {
}
)
}
}
I'm using declarative Jenkins pipelines to run some of my build pipelines and was wondering if it is possible to define multiple agent labels.
I have a number of build agents hooked up to my Jenkins and would like for this specific pipeline to be able to be built by various agents that have different labels (but not by ALL agents).
To be more concrete, let's say I have 2 agents with a label 'small', 4 with label 'medium' and 6 with label 'large'. Now I have a pipeline that is very resource-low and I want it to be executed on only a 'small'- or 'medium'-sized agent, but not on a large one as it may cause larger builds to wait in the queue for an unnecessarily long time.
All the examples I've seen so far only use one single label.
I tried something like this:
agent { label 'small, medium' }
But it failed.
I'm using version 2.5 of the Jenkins Pipeline Plugin.
You can see the 'Pipeline-syntax' help within your Jenkins installation and see the sample step "node" reference.
You can use exprA||exprB:
node('small||medium') {
// some block
}
EDIT: I misunderstood the question. This answer is only if you know
which specific agent you want to run for each stage.
If you need multiple agents you can declare agent none and then declare the agent at each stage.
https://jenkins.io/doc/book/pipeline/jenkinsfile/#using-multiple-agents
From the docs:
pipeline {
agent none
stages {
stage('Build') {
agent any
steps {
checkout scm
sh 'make'
stash includes: '**/target/*.jar', name: 'app'
}
}
stage('Test on Linux') {
agent {
label 'linux'
}
steps {
unstash 'app'
sh 'make check'
}
post {
always {
junit '**/target/*.xml'
}
}
}
stage('Test on Windows') {
agent {
label 'windows'
}
steps {
unstash 'app'
bat 'make check'
}
post {
always {
junit '**/target/*.xml'
}
}
}
}
}
This syntax appears to work for me:
agent { label 'linux && java' }
As described in Jenkins pipeline documentation and by Vadim Kotov one can use operators in label definition.
So in your case if you want to run your jobs on nodes with specific labels, the declarative way goes like this:
agent { label('small || medium') }
And here are some examples from Jenkins page using different operators
// with AND operator
agent { label('windows && jdk9 )') }
// a more complex one
agent { label('postgres && !vm && (linux || freebsd)') }
Notes
When constructing those definitions one just needs to consider following rules/restrictions:
All operators are left-associative
Labels or agent names can be surrounded with quotation marks if they contain characters that would conflict with the operator syntax
Expressions can be written without whitespace
Jenkins will ignore whitespace when evaluating expressions
Matching labels or agent names with wildcards or regular expressions is not supported
An empty expression will always evaluate to true, matching all agents
Create a another label call 'small-or-medium' that has 6 all agents. Then in Jenkinsfile:
agent { label 'small-or-medium' }
I'm looking into moving our scripted pipelines to declarative pipelines.
I'm using the when key word to skip stages
stage('test') {
// Only do anything if we are on the master branch
when { branch 'master' }
//...
}
This works, however the skipped stage is shown as green. I would prefer if it was shown as gray in the pipeline overview. Is there a way to achieve this?
As you mentioned in your comment I suggest you to use Jenkins Blue Ocean when working with pipelines.
It provides a more modern and user friendly view for your pipeline projects. Even the pipeline itself is display in a much more convenient way.
If the stage is appearing green for you then it's likely still actually running. A skipped stage should look like this in the Jenkins classic stage view. Consider the following code sample, which has three stages, the middle stage being skipped conditionally with the when directive.
pipeline {
agent any
stages {
stage('Always run 1') {
steps { echo "hello world" }
}
stage('Conditionally run') {
when {
expression { return false }
}
steps { echo "doesn't get printed" }
}
stage("Always run 2") {
steps { echo "hello world again" }
}
}
}
This should produce the following line in your build log
Stage "Conditionally run" skipped due to when conditional
Another answerer of this question mentioned Blue Ocean, which definitely presents a beautiful presentation of the stage view. Here is an image of how a skipped stage looks in the Blue Ocean stage view. Note that Blue Ocean is a UI and your job's underlying pipeline code will be the same regardless of which UI you choose to use.
I have one Pipeline job for each component in my Jenkins 2.0. All of them
consist of many stages (build, UT, IT etc.), so they're working as a
pipeline for a component.
The components are depending on each other in a specified order, so I used "Build after other projects are built" (I also tried JobFanIn Plugin) to trigger these "mini-pipelines" after each other. This works like a pipeline of "mini pipelines"
I'd like to visualize the relationship between the jobs. For this purpose I've found 2 plugins:
Delivery Pipeline Plugin
Build Pipeline Plugin
Both introduce a new View type, but none of them supports the "Pipeline" or "Multibranch pipeline" job types (introduced in Jenkins 2.0), these jobs are not visible in the related dropdown list on the view config page.
How can I visualize the relation of these job types? Is there any other plugin which supports these types?
Thinking about this.
I don't think a visualisation of multi branch pipelines makes sense in the same way it would for a single branch build.
The reason is that each bench of a mb pipeline can have a different build configuration. Eg with master triggering a promotion job but branch doing something else or nothing.
Do the best one could do I think is trace an individual build number and it's links. Can't do it at the job level.
Jenkins blue ocean plugins give the rich view to visualize all types (parallel, sequential stages) view out of the box.
Let say if you have a pipeline like this
pipeline {
agent any;
stages {
stage('build') {
stages {
stage('compile') {
steps {
echo "steps for unitest"
}
}
stage('security scan') {
parallel {
stage('sonarqube') {
steps {
echo "steps for parallel sonarqube"
}
}
stage('blackduck') {
steps {
echo "steps for parallel blackduck"
}
}
}
}
stage('package') {
steps {
echo "steps for package"
}
}
}
}
stage('deployment') {
stages {
stage('dev') {
steps {
echo "Development"
}
}
stage('pp') {
when { branch 'master' }
steps {
echo "PreProduction"
}
}
stage('prod') {
when {
branch 'master'
beforeInput true
}
input {
message "Deploy to production?"
id "simple-input"
}
steps {
echo "Production"
}
}
}
}
}
}
It will visualize like this :
is this what you are looking for?
Note- it can customize. but this view is per build ..you can't create a dashboard from it and combine it all in one