I have an external function to call multiple times (with different variable values) from my jenkins pipeline and was hoping to make this call parallel.
Here's my pipeline code.
stepsForParallel = [:]
parallel_list = ['apple','banana','orange']
def groovyFile = load "${path}/runFruit.groovy"
for(value in parallel_list){
stepsForParallel.put("Run for fruit-${value}", groovyFile.runForFruit(value))
}
parallel stepsForParallel
And my groovy file called 'runFruit' is as follows.
#!/usr/bin/env groovy
def runForFruit(fruitValue){
..do something here..
}
return this
For this code the pipeline still runs sequentially.
Perhaps because in this line inside the for loop
stepsForParallel.put("Run for fruit-${value}", groovyFile.runForFruit(value)) an actual call to the external function is being made?
Any suggestions on how to achieve this parallelization?
Take a look at the Documentation for the parallel step, it takes a map from branch names (the name of the parallel executions) to closures which are the code that will be executed per parallel stage. In your case you are not passing a closure, but rather executing the function itself.
parallel firstBranch: {
// do something
}, secondBranch: {
// do something else
},
failFast: true|false
So you need to construct your code to fit the parallel format by passing a closure that contains your code to execute as the map value.
In your case it will look something like (using collectEntries):
parallel_list = ['apple','banana','orange']
def groovyFile = load "${path}/runFruit.groovy"
stepsForParallel = parallel_list.collectEntries{ fruit ->
["Run for fruit-${fruit}" : {
groovyFile.runForFruit(fruit)
}]
}
parallel stepsForParallel
Or if you prefer your format using the for loop:
stepsForParallel = [:]
parallel_list = ['apple','banana','orange']
def groovyFile = load "${path}/runFruit.groovy"
for(value in parallel_list){
stepsForParallel.put("Run for fruit-${value}", { groovyFile.runForFruit(value) })
}
parallel stepsForParallel
You can also use it in below way :
def fruit_list = ['apple','banana','orange']
def groovyFile
def stepsForParallel = [:]
pipeline {
agent any
stages {
stage('Hello') {
steps {
script{
groovyFile = load "runFruit.groovy"
stepsForParallel = fruit_list.collectEntries {
["${it}" : groovyFile.runForFruit(it)]
}
}
}
}
}
}
parallel stepsForParallel
Output:
Related
I figure I’m doing something unorthodox here, but I’d like to stick to declarative for convenience while dynamically generating parallel steps.
I found a way to do something like that, but mixing both paradigms, which doesn’t seem to work well with the BlueOcean UI (multiple stages inside each parallel branch do not show up properly).
The closest I got was with something like this:
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = []
for (account in accounts()) {
jobs[] = stage(account) {
steps {
echo "Step for $account"
}
}
}
return jobs
}
# this is inside a shared library, called by my Jenkinsfile, like what is described
# under "Defining Declarative Pipelines in Shared Libraries" in
# https://www.jenkins.io/blog/2017/09/25/declarative-1/
def call() {
pipeline {
stages {
stage('Build all variations') {
parallel parallelJobs()
}
}
}
}
The problem is Jenkins errors like this:
Expected a block for parallel # line X, column Y.
parallel parallelJobs()
^
So, I was wondering if there is a way I could transform that list of stages, returned by parallelJobs(), into the block expected by Jenkins...
Yes, you can. You need to return a map of stages. Following is a working pipeline example.
pipeline {
agent any
stages {
stage('Parallel') {
steps {
script {
parallel parallelJobs()
}
}
}
}
}
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = [:]
for (account in accounts()) {
jobs[account] = { stage(account) {
echo "Step for $account"
}
}
}
return jobs
}
I'm in need to run two defined functions parallel in Jenkins pipeline.
As defined in jenkins, the keyword parallel used with jobs, seems don't work with function calling.
What I've tried is -
def first_func(){
echo "first function"
}
def second_func(){
echo "second function"
}
node {
task = [:]
function_lists = ['first_func()', 'second_func()']
stage ('build') {
for (job in function_lists) {
task[job] = { '${job}' }
}
parallel task
}
}
don't actually call the functions. Is there any way to do so in jenkins?
Yes this can be achieved in below way:
def first_func(){
echo "first function"
}
def second_func(){
echo "second function"
}
node {
def task = [:]
stage ('build') {
// Loop through list
['first_func', 'second_func'].each {
def a = it;
task[a] = { "${a}"()}
}
parallel task
}
}
Output :
Actually my Jenkinsfile looks like this:
#Library('my-libs') _
myPipeline{
my_build_stage(project: 'projectvalue', tag: '1.0' )
my_deploy_stage()
}
I am trying to pass these two variables (project and tag) to my build_stage.groovy, but it is not working.
What is the correct syntax to be able to use $params.project or $params.tag in my_build_stage.groovy?
Please see the below code which will pass parameters.
In your Jenkinsfile write below code:
// Global variable is used to get data from groovy file(shared library file)
def mylibrary
def PROJECT_VALUE= "projectvalue"
def TAG = 1
pipeline{
agent{}
stages{
stage('Build') {
steps {
script {
// Load Shared library Groovy file mylibs.Give your path of mylibs file which will contain all your function definitions
mylibrary= load 'C:\\Jenkins\\mylibs'
// Call function my_build stage and pass parameters
mylibrary.my_build_stage(PROJECT_VALUE, TAG )
}
}
}
stage('Deploy') {
steps {
script {
// Call function my_deploy_stage
mylibrary.my_deploy_stage()
}
}
}
}
}
Create a file named : mylibs(groovy file)
#!groovy
// Write or add Functions(definations of stages) which will be called from your jenkins file
def my_build_stage(PROJECT_VALUE,TAG_VALUE)
{
echo "${PROJECT_VALUE} : ${TAG_VALUE}"
}
def my_deploy_stage()
{
echo "In deploy stage"
}
return this
This really helpful answer, got me 95% of the way there. Using this solution, I'm able to start n build stages in parallel. However, the map of parallel stages is essentially hardcoded. I want to be able to create it dynamically. The first step in this process is changing parallelStagesMap from a map, to a function that returns a map.
Unfortunatey, this small change causes my build to fail without any apparent error logs related to syntax.
How can I accomplish this? Am I using malformed Groovy syntax? I'd be grateful for any help.
def jobs = ["JobA", "JobB", "JobC"]
def parallelStagesMap() { // This is now a function that returns a map.
return jobs.collectEntries {
["${it}" : generateStage(it)]
}
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
sh script: "sleep 15"
}
}
}
pipeline {
agent any
stages {
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap() // I call the function here.
}
}
}
}
}
I got a working solution! It's not perfect, because I would like to extract the jobs.collectEntries part to my own function, but now I can define the contents of my parallel stages inline, instead of at the top of the file!
I tried writing a function matching the same signature as Map.collectEntries: ({ Closure -> Map }), but the Jenkins build fails without any logs once it hits my function. If someone's able to work that out, I'd be grateful.
def jobs = ["JobA", "JobB", "JobC"]
pipeline {
agent any
stages {
stage('parallel stage') {
steps {
script {
parallel jobs.collectEntries { j ->
["${j}" : { job -> return {
stage("stage: ${job}") {
echo "This is ${job}."
sh script: "sleep 15"
}
}}(j)]
}
}
}
}
}
}
I'd like to ask for help with a a Jenkins groovy pipeline, copied from here:
Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?
I'd like for a several sets of vars to be passed in a under a map, for several stages under a parallel run. However, only the last set (square brackets at the bottom of the map) gets registered for my map.
When the parallel stage runs, the map iterates successfully, but only with the last set (currently install_Stage(it)), ignoring other sets. Meaning that I get a pipeline showing four "stage: install ${product}" stages in parallel, and that's it. I'd like to get three parallels with four stages (network setup, revert, and install), as per my code below:
#!groovy
#Library('ci_builds')
def products = ["A", "B", "C", "D"]
def parallelStagesMap = products.collectEntries {
switch (it) {
case "A":
static_ip_address = "10.100.100.6"; static_vm_name = "install-vm1"; version = "14.1.60"
break
case "B":
static_ip_address = "10.100.100.7"; static_vm_name = "install-vm2"; version = "15.1"
break
case "C":
static_ip_address = "10.100.100.8"; static_vm_name = "install-vm3"; version = "15.1"
break
case "D":
static_ip_address = "10.100.100.9"; static_vm_name = "install-vm4"; version = "15.2"
break
default:
static_ip_address = "The product name is not on the switch list - please enter an ip address"
version = "The product name is not on the switch list - please enter a version"
break
}
["${it}" : network_reg(it)]
["${it}" : revert_to_snapshot_Stage(it)]
["${it}" : install_Stage(it)]
}
def network_reg(product) {
return {
stage("stage: setup network for ${product}") {
echo "setting network on ${static_vm_name} with ${static_ip_address}."
sh script: "sleep 15"
}
}
}
def revert_to_snapshot_Stage(product) {
return {
stage("stage: revert ${product}") {
echo "reverting ${static_vm_name} for ${product} on ${static_ip_address}."
sh script: "sleep 15"
}
}
}
def install_Stage(product) {
return {
stage("stage: install ${product}") {
echo "installing ${product} on ${static_ip_address}."
sh script: "sleep 15"
}
}
}
pipeline {
agent any
stages {
stage('non-parallel env check') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
The network_reg and revert_to_snapshot_Stage won't run (unless I place them as the last set instead of ["${it}" : install_Stage(it)] , in which case, again, only the one of the parallel stages is run)
I don't mind a different approach to run several map definitions, but others such as: How to define and iterate over map in Jenkinsfile don't allow for a full multi variable map (more than a key+value pair)
Any help would be appreciated, Thanks!
I assume you have a similar issue like I had trying to dynamically build the parallel branches for parallel execution.
Two things were very important:
Make a copy of the loop variable (in you case: it) and use that copy only inside the parallel branch; if you don't all branches (closures) will reference the very same variable which of course will have the same value. That is particular to closures. See also: http://groovy-lang.org/closures.html.
Don't use collectEntries{}. Stick to the java-style loops as groovy loops most of the time do not work properly. Some .each{} constructs may work already but if in doubt switch to the java loops. See also: Impossibility to iterate over a Map using Groovy within Jenkins Pipeline
Following stripped down example works for me. I believe you'll be able to adjust it to your needs.
def products = ["A", "B", "C", "D"]
def parallelStagesMap = [:]
// use java-style loop
for (def product: products) {
// make a copy to ensure that each closure will get it's own variable
def copyOfProduct = product
parallelStagesMap[product] = {echo "install_Stage($copyOfProduct)"}
}
echo parallelStagesMap.toString()
pipeline {
agent any
stages {
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
If it still doesn't work: Check whether there's and upgrade your Pipeline: Groovy plugin as they usually fix lot of issues which usually work in groovy but won`t in pipeline.
You may want to check following related question which contains a minimal example as well:
Currying groovy CPS closure for parallel execution