I have a declarative Jenkins pipeline which contains several identical stages running in parallel. I would like to reuse these sections, however so far the best I've been able to achieve is as follows:
stage('Deploy & E2E') {
parallel {
stage('E2E 1') {
agent { ... }
environment { ... }
steps {
script { runE2ESteps() }
}
post { ... }
}
stage('E2E 2') {
agent { ... }
environment { ... }
steps {
script { runE2ESteps() }
}
post { ... }
}
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
}
}
I am able to partially reuse bits, by extracting the steps out into a method, but there is still quite a bit of duplication in the agent, environment and post blocks. Ideally I want to write something like:
stage('Deploy & E2E') {
parallel {
script {
for (int i = 0; i < 5; i++) {
stage('E2E ${i}') {
agent { ... }
environment { ... }
steps { ... }
post { ... }
}
}
}
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
}
}
But apparently I can't do this in declarative land - Jenkins seems very finicky about what precisely you're allowed to have at various points in the file.
I've tried a few other approaches, too, but none of them have worked:
Using matrix for the E2E block seemed promising, but you're not allowed to nest matrix within parallel so this didn't pan out (because there are other, unrelated stages we want running in parallel to the E2E steps)
Using the in-line parallel command. This doesn't achieve what I want because I need each stage running in its own agent (not parallelised within the same agent).
I've attached a diagram of what the pipeline looks like - it's behaving exactly as I want right now, just with more code duplication than I'd like. If we want to change the parallelism of our E2E, it currently involves copy+pasting or removing a block of code, rather than (ideally) changing one number somewhere.
Maybe this just isn't possible, but I figured I might as well chuck out a post to see if there are any ideas that I missed.
stage('Deploy & E2E') {
parallel {
script {
stage('Other unrelated stage, not wanting reuse') {
agent { ... }
environment { ... }
steps {
// something else
}
post { ... }
}
for (int i = 0; i < 5; i++) {
stage('E2E ${i}') {
agent { ... }
environment { ... }
steps { ... }
post { ... }
}
}
}
}
}
have a try
We never found a solution to this in Jenkins. In the end, we've moved to Github Actions where this is possible with a matrix job.
Related
I figure I’m doing something unorthodox here, but I’d like to stick to declarative for convenience while dynamically generating parallel steps.
I found a way to do something like that, but mixing both paradigms, which doesn’t seem to work well with the BlueOcean UI (multiple stages inside each parallel branch do not show up properly).
The closest I got was with something like this:
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = []
for (account in accounts()) {
jobs[] = stage(account) {
steps {
echo "Step for $account"
}
}
}
return jobs
}
# this is inside a shared library, called by my Jenkinsfile, like what is described
# under "Defining Declarative Pipelines in Shared Libraries" in
# https://www.jenkins.io/blog/2017/09/25/declarative-1/
def call() {
pipeline {
stages {
stage('Build all variations') {
parallel parallelJobs()
}
}
}
}
The problem is Jenkins errors like this:
Expected a block for parallel # line X, column Y.
parallel parallelJobs()
^
So, I was wondering if there is a way I could transform that list of stages, returned by parallelJobs(), into the block expected by Jenkins...
Yes, you can. You need to return a map of stages. Following is a working pipeline example.
pipeline {
agent any
stages {
stage('Parallel') {
steps {
script {
parallel parallelJobs()
}
}
}
}
}
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = [:]
for (account in accounts()) {
jobs[account] = { stage(account) {
echo "Step for $account"
}
}
}
return jobs
}
this should be fairly basic, but when I research I come to things like gerrit triggrs and whatnot, which seem way too complicated for doing something simple like this.
I would like to do something like either this in the JobDSL script:
pipelineJob('deploy-game') {
definition {
environmentVariables {
env('ENVIRONMENT', "${ENVIRONMENT}")
keepBuildVariables(true)
}
cpsScm {
scm {
git{
remote {
url('https://blabla.git')
credentials('gitlab-credentials')
}
branches('${gitlabsourcebranch}')
}
}
scriptPath('path/to/this.jenkinsfile')
}
triggers {
gitlabPush {
buildOnMergeRequestEvents(true)
if ($gitlabMergeRequestState == 'merged') // this part
}
}
}
}
Or, trigger on all MR events, and then filter out in the pipeline script:
pipeline {
agent none
environment {
ENVIRONMENT = "${ENVIRONMENT}"
}
triggers {
$gitlabMergeRequestState == 'merged' // this one
}
stages {
stage ('do-stuff') {
agent {
label 'agent'
}
steps {
sh 'some commands ...'
}
}
}
}
How do I do this ?
So this is how it should be, I hope this is what you are looking for it.
pipelineJob('Job_Name') {
definition {
cpsScm {
lightweight(true)
triggers {
gitlabPush {
buildOnMergeRequestEvents(true) // it will trigger build when MR is opened.
buildOnPushEvents(true)
commentTrigger('retry a build') // When you write the comment on MR on gitlab. it will also trigger build
enableCiSkip(true)
rebuildOpenMergeRequest('source')
skipWorkInProgressMergeRequest(false)
targetBranchRegex('.*master.*|.*release.*') //This mean only push happened to master or release then only trigger jenkins build. Do not trigger build on normal feature branch push until the MR is opened.
}
}
configure {
it / triggers / 'com.dabsquared.gitlabjenkins.GitLabPushTrigger' << secretToken('ADD_TOKEN_FROM_JENKINS_JOB')
}
scm {
git {
remote {
credentials('ID')
url("git#URL.git")
branch("refs/heads/master")
}
}
}
scriptPath("jenkinsfile")
}
}
}
I am building to 2 different environments in the same pipeline and I want to make the cleanup for both environments in parallel.
As I understood, parallel does not work inside the post step: post step parallel.
Any suggestions? Example of my code:
post {
always {
script{
cleanup(env1)
cleanup(env2)
}
}
}
def cleanup(env) {
withEnv(env) {
sh "./cleanup.py"
}
}
The parallel keyword can work inside a post condition as long as it is encapsulated inside a script block, as the script blocks is just a fallback to the scripted pipeline which will allow you to run parallel execution wherever you want.
The following should work fine:
post {
always{
script {
def environments = ['env1', 'env2', 'env3']
parallel environments.collectEntries {
["Cleanup ${it}" : {
cleanup(it)
}]
}
}
}
}
def cleanup(env) {
withEnv(env) {
sh "./cleanup.py"
}
}
Just don't forget to allocate an agent using the node keyword if the steps in the post section are required to run on a specific agent.
A better idea in my opinion is to clean up after the fact, before you possibly lost the node to another job:
parallel {
stage('env1') {
agent { node { label "env1" }}
steps {
script {
println "Inside env1"
}
}
post {
cleanup { script { my_cleanup_func("env1") } }
}
}
stage('env2') {
agent { node { label "env2" }}
steps {
script {
println "Inside env2"
}
}
post {
cleanup { script { my_cleanup_func("env2") } }
}
}
...
def my_cleanup_func(String env) {
// ...
}
I have multiple jenkinsifles, doing basically the same thing:
pipeline{
parameters { ... }
environment { ... }
stages {
stage ('setup') { ... }
stage ('run') {
agent { AGENT }
steps { STEPS }
}
}
The STEPS & AGENT parameters are values I get in the setup stage. Is it possible to define somewhere a function that returns a stage?
e.g.
def stage_factory(name, agent, steps, post ...){
return
stage (name) {
agent { agnet }
steps { steps }
post { post }
}
}
}
which later will be called inside the pipeline, right after the setup stage
?
The following works in scripted pipeline, you need to try the declarative syntax yourself. Note the use of surrounding {}
def stage_factory(name, agent, steps, post ...){
return {
node(agent){
stage (name) {
steps()
}
}
}
}
With this approach you need to put the post action in try-catch blocks, but this is the gist of it.
If you change it like so, you can even pass the steps to it as you would expect from a Jenkins stage.
def stage_factory(name, agent){
return { steps ->
node(agent){
stage (name) {
steps()
}
}
}
}
usage:
def myDtage = stage_factory("foo", "bar")
myStage{
//...
}
I have something like this:
stages {
stage('createTemplate') {
parallel {
stage('template_a') {
creating template a
}
stage('template_b') {
creating template b
}
}
}
stage('deployVm') {
parallel {
stage('deploy_a') {
deploy vm a
}
stage('deploy_b') {
deploy vm b
}
}
}
}
How can I make sure that deployVm stages run when and only when respective createTemplate stages were successful?
You may want to run one parallel like this:
parallel {
stage('a') {
stages {
stage ('template_a') { ... }
stage ('deploy_a') { ... }
}
stage('b') {
stages {
stage ('template_b') { ... }
stage ('deploy_b') { ... }
}
}
}
This will make sure only stages that deploy are the ones following successful template stages.