I have something like this:
stages {
stage('createTemplate') {
parallel {
stage('template_a') {
creating template a
}
stage('template_b') {
creating template b
}
}
}
stage('deployVm') {
parallel {
stage('deploy_a') {
deploy vm a
}
stage('deploy_b') {
deploy vm b
}
}
}
}
How can I make sure that deployVm stages run when and only when respective createTemplate stages were successful?
You may want to run one parallel like this:
parallel {
stage('a') {
stages {
stage ('template_a') { ... }
stage ('deploy_a') { ... }
}
stage('b') {
stages {
stage ('template_b') { ... }
stage ('deploy_b') { ... }
}
}
}
This will make sure only stages that deploy are the ones following successful template stages.
Related
Is it possible to specify agent 'any' only when a condition is met?
Context: I have a parallel pipeline with some common steps. The pipeline runs tests on Linux and Windows on two different agents. There are parameter defining if both, or only Linux, or only Windows should run. I want to use as little agents as possible.
With the following pipeline, the Linux tests are executed on the main agent. The Windows tests on a different agent. So if only Linux needs to run, I only use 1 agent, which is good. But if only Windows needs to run, the main agent will be blocked until the Windows agent finishes.
pipeline {
agent any // main agent
parameters {
booleanParam( name: 'RUN_WINDOWS' )
booleanParam( name: 'RUN_LINUX' )
}
stages {
stage('Common step') {
...
}
stage("Test") {
parallel {
stage('Windows') {
agent any
when {
expression { return params.RUN_WINDOWS }
}
stages {
stage('Test Windows') {
...
}
}
}
stage('Linux') { // no agent specified, so will use the main agent
when {
expression { return params.RUN_LINUX }
}
stages {
stage('Test Linux') {
...
}
}
}
}
post {
...
}
}
}
}
I'd need something like this for the Windows stage, where it only specifies an agent based on a condition:
if (params.RUN_WINDOWS && params.LINUX) { agent any }
EDIT: I'm looking for a solution without using labels.
As a workaround, instead of setting any you can set agent labels conditionally. Something like below.
def labels = "Windows"
if (params.RUN_WINDOWS && params.LINUX) {
labels = "windows || linux"
}
pipeline {
agent any // main agent
parameters {
booleanParam( name: 'RUN_WINDOWS' )
booleanParam( name: 'RUN_LINUX' )
}
stages {
stage('Common step') {
...
}
stage("Test") {
parallel {
stage('Windows') {
agent {
label "${labelSelected}"
}
when {
expression { return params.RUN_WINDOWS }
}
stages {
stage('Test Windows') {
...
}
}
}
stage('Linux') { // no agent specified, so will use the main agent
when {
expression { return params.RUN_LINUX }
}
stages {
stage('Test Linux') {
...
}
}
}
}
post {
...
}
}
}
}
I am building to 2 different environments in the same pipeline and I want to make the cleanup for both environments in parallel.
As I understood, parallel does not work inside the post step: post step parallel.
Any suggestions? Example of my code:
post {
always {
script{
cleanup(env1)
cleanup(env2)
}
}
}
def cleanup(env) {
withEnv(env) {
sh "./cleanup.py"
}
}
The parallel keyword can work inside a post condition as long as it is encapsulated inside a script block, as the script blocks is just a fallback to the scripted pipeline which will allow you to run parallel execution wherever you want.
The following should work fine:
post {
always{
script {
def environments = ['env1', 'env2', 'env3']
parallel environments.collectEntries {
["Cleanup ${it}" : {
cleanup(it)
}]
}
}
}
}
def cleanup(env) {
withEnv(env) {
sh "./cleanup.py"
}
}
Just don't forget to allocate an agent using the node keyword if the steps in the post section are required to run on a specific agent.
A better idea in my opinion is to clean up after the fact, before you possibly lost the node to another job:
parallel {
stage('env1') {
agent { node { label "env1" }}
steps {
script {
println "Inside env1"
}
}
post {
cleanup { script { my_cleanup_func("env1") } }
}
}
stage('env2') {
agent { node { label "env2" }}
steps {
script {
println "Inside env2"
}
}
post {
cleanup { script { my_cleanup_func("env2") } }
}
}
...
def my_cleanup_func(String env) {
// ...
}
I have a declarative Jenkins pipeline which contains several identical stages running in parallel. I would like to reuse these sections, however so far the best I've been able to achieve is as follows:
stage('Deploy & E2E') {
parallel {
stage('E2E 1') {
agent { ... }
environment { ... }
steps {
script { runE2ESteps() }
}
post { ... }
}
stage('E2E 2') {
agent { ... }
environment { ... }
steps {
script { runE2ESteps() }
}
post { ... }
}
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
}
}
I am able to partially reuse bits, by extracting the steps out into a method, but there is still quite a bit of duplication in the agent, environment and post blocks. Ideally I want to write something like:
stage('Deploy & E2E') {
parallel {
script {
for (int i = 0; i < 5; i++) {
stage('E2E ${i}') {
agent { ... }
environment { ... }
steps { ... }
post { ... }
}
}
}
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
}
}
But apparently I can't do this in declarative land - Jenkins seems very finicky about what precisely you're allowed to have at various points in the file.
I've tried a few other approaches, too, but none of them have worked:
Using matrix for the E2E block seemed promising, but you're not allowed to nest matrix within parallel so this didn't pan out (because there are other, unrelated stages we want running in parallel to the E2E steps)
Using the in-line parallel command. This doesn't achieve what I want because I need each stage running in its own agent (not parallelised within the same agent).
I've attached a diagram of what the pipeline looks like - it's behaving exactly as I want right now, just with more code duplication than I'd like. If we want to change the parallelism of our E2E, it currently involves copy+pasting or removing a block of code, rather than (ideally) changing one number somewhere.
Maybe this just isn't possible, but I figured I might as well chuck out a post to see if there are any ideas that I missed.
stage('Deploy & E2E') {
parallel {
script {
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
for (int i = 0; i < 5; i++) {
stage('E2E ${i}') {
agent { ... }
environment { ... }
steps { ... }
post { ... }
}
}
}
}
}
have a try
We never found a solution to this in Jenkins. In the end, we've moved to Github Actions where this is possible with a matrix job.
I have the following two pipelines in my repository
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
gradlew (":build")
}
}
}
}
void gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent none
stages {
stage("PR checks") {
parallel {
stage("build 1") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":one:build")
}
}
stage("build 2") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":two:build")
}
}
}
}
}
}
void gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
As you can see, there is some code duplication between the two - ANDROID_EMULATOR and void gradlew(..).
I would like to move them into their own shared.groovy file:
#Field String ANDROID_EMULATOR = "android-emulator"
void gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
And be able to import it into my other pipelines with a single line of code. Gradle allows this to be done with apply('shared.groovy').
Jenkins seems to allow only shared libraries (which are global), and load statements (which need to be loaded as a part of a node, which does not scale well). Does Jenkins lack support for this basic style of code sharing here?
You can use the pipeline load which is more simple than using shared library, especially when you hope the shared.groovy in the same repo as your Jenkinsfiles.
// shared.groovy
def gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
return this // the return this must be have
// pipeline 1
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
scripts {
shared = load 'shared.groovy'
shared.gradlew (":build")
}
}
}
}
}
// pipeline 2
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
scripts {
shared = load 'shared.groovy'
shared.gradlew ("one:build")
}
}
}
}
}
Jenkins shared libraries have a well defined folder structure https://www.jenkins.io/doc/book/pipeline/shared-libraries/#directory-structure
You can try:
to implement this folder structure in a subfolder in your repo
to use a Dynamic Retrieval with a scm config that will check out the specific folder
I afraid it is too complicated and even not possible
I think the best approach is to create global shared libraries repo and to implement a gradleBuild custom step . In that case your code will be like
Pipeline 1:
#Library('somelib')
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
gradleBuild ":build"
}
}
}
}
Pipeline 2:
#Library('somelib')
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent none
stages {
stage("PR checks") {
parallel {
stage("build 1") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":one:build")
}
}
stage("build 2") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":two:build")
}
}
}
}
}
}
Shared libraries vars/gradeBuild.groovy file:
def call(String tasks) {
sh "./gradlew $tasks --profile"
}
I have 3 different jobs (Build, Undeploy and Deploy), which want to execute Build and Undeploy in parallel and after that Deploy.
From search got to know that Build Flow Plugin got deprecated.
Please suggest a plugin.
You can write Jenkins file with the below format:-
pipeline {
stages {
agent { node { label 'master' } }
stage('Build/Undeploy') {
parallel {
stage('Build') {
agent { node { label 'Build' } }
steps {
script {
//Call your build script
}
}
}
stage('Undeploy') {
agent { node { label 'Undeploy' } }
steps {
script {
//Call your undeploy script
}
}
}
}
}
stage('Deploy'){
agent { node { label 'Deploy' } }
steps {
script {
//Call your deploy script
}
}
}
}
}