I am trying to run a list of builds in parallel using a for loop because the code is getting bigger and bigger.
I have a global list with the names of projects
#Field def final String[] projectsList = ['project1','project2', 'project3'....]
stages {
stage('Parallel Build') {
steps{
script{
def branches = [:]
for(int i = 0;i<15;i++) {
branches["Build "+projectsList[i]] = {buildProject(i)}
}
parallel branches
}
}
}
The Build projects method takes the name of the project from the global list and builds it using maven.
The thing is, the project at index 15 (Which shouldn't be built) is building 15 times in parallel. As if it is waiting until the for loop ends and then assigns the same available i value (15) in this case to all the methods.
Do you have any idea how can I solve this ?
Your issue lies with how you make (mis)use of the Groovy closure concept, i.e. the part where you define a closure in the loop's body that makes use of the iteration variable i, that is { buildProject(i) } :)
What happens exactly is nicely described here. This is, indeed, a common "gotcha" with other languages offering functional programming features, as well (e.g. JavaScript).
The easiest (hardly the most elegant, though) solution is to define a variable within the loop that receives the current i value and make use of that within the closure:
def branches = [:]
for(i = 0; i < 15; i++) {
def curr = i
branches["Build ${projectsList[i]}"] = { buildProject(curr) }
}
parallel branches
(I've also used a bit more idiomatic Groovy like String interpolation).
A more elegant, less verbose, Groovy-like solution that iterates through the range of projects would be:
(0..<projectsList.size()).each { i ->
branches["Build ${projectsList[i]}"] = { buildProject(i) }
}
Related
I came across this message with the code below
in JENKINS-44085.
If I already have a map of branches that contains 50 items, but I want to parallel them 5 at a time, how do I need to modify this code?
My code already has a map of 50 items in a var named branches.
// put a number of items into the queue to allow that number of branches to run
for (int i=0;i<MAX_CONCURRENT;i++) {
latch.offer("$i")
}
for (int i=0; i < 500; i++) {
def name = "$i"
branches[name] = {
def thing = null
// this will not allow proceeding until there is something in the queue.
waitUntil {
thing = latch.pollFirst();
return thing != null;
}
try {
echo "Hello from $name"
sleep time: 5, unit: 'SECONDS'
echo "Goodbye from $name"
}
finally {
// put something back into the queue to allow others to proceed
latch.offer(thing)
}
}
}
timestamps {
parallel branches
}
This question is a bit old, but for me the problem was also relevant yesterday. In some cases your Jenkins jobs may be light on Jenkins but high on some other system, so you want to limit it for that system. In my opinion using max executors per build agent is not the right way to do that because if your Jenkins cluster scales you will have to adjust stuff.
To answer your question, you probably want to do something like this:
Create a branches map with numeric indexes "0", "1", etc.
In the try block of that code you pasted have something like: build(my_branches[name])
At least that's how I was using that same workaround before. But then someone at work pointed out a better solution. I also commented this simpler solution in the Jira ticket you refered to. It requires the Lockable Resources Plugin: https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin
Go to: http://<your Jenkins URL>/configure and add X lockable resources with label "XYZ".
Use in your code as such:
def tests = [:]
for (...) {
def test_num="$i"
tests["$test_num"] = {
lock(label: "XYZ", quantity: 1, variable: "LOCKED") {
println "Locked resource: ${env.LOCKED}"
build(job: jobName, wait: true, parameters: parameters)
}
}
}
parallel tests
The nice thing about this is that you can use this across different jobs. In our case different jobs have a load on XYZ so having these global locks are very handy.
Original situation:
I have a job in Jenkins that is running an ant script. I easily managed to test this ant script on more then one software version using a "Multi-configuration project".
This type of project is really cool because it allows me to specify all the versions of the two software that I need (in my case Java and Matlab) an it will run my ant script with all the combinations of my parameters.
Those parameters are then used as string to be concatenated in the definition of the location of the executable to be used by my ant.
example: env.MATLAB_EXE=/usr/local/MATLAB/${MATLAB_VERSION}/bin/matlab
This is working perfectly but now I am migrating this scripts to a pipline version of it.
Pipeline migration:
I managed to implement the same script in a pipeline fashion using the Parametrized pipelines pluin. With this I achieve the point in which I can manually select which version of my software is going to be used if I trigger the build manually and I also found a way to execute this periodically selecting the parameter I want at each run.
This solution seems fairly working however is not really satisfying.
My multi-config project had some feature that this does not:
With more then one parameter I can set to interpolate them and execute each combination
The executions are clearly separated and in build history/build details is easy to recognize which settings hads been used
Just adding a new "possible" value to the parameter is going to spawn the desired executions
Request
So I wonder if there is a better solution to my problem that can satisfy also the point above.
Long story short: is there a way to implement a multi-configuration project in jenkins but using the pipeline technology?
I've seen this and similar questions asked a lot lately, so it seemed that it would be a fun exercise to work this out...
A matrix/multi-config job, visualized in code, would really just be a few nested for loops, one for each axis of parameters.
You could build something fairly simple with some hard coded for loops to loop over a few lists. Or you can get more complicated and do some recursive looping so you don't have to hard code the specific loops.
DISCLAIMER: I do ops much more than I write code. I am also very new to groovy, so this can probably be done more cleanly, and there are probably a lot of groovier things that could be done, but this gets the job done, anyway.
With a little work, this matrixBuilder could be wrapped up in a class so you could pass in a task closure and the axis list and get the task map back. Stick it in a shared library and use it anywhere. It should be pretty easy to add some of the other features from the multiconfiguration jobs, such as filters.
This attempt uses a recursive matrixBuilder function to work through any number of parameter axes and build all the combinations. Then it executes them in parallel (obviously depending on node availability).
/*
All the config axes are defined here
Add as many lists of axes in the axisList as you need.
All combinations will be built
*/
def axisList = [
["ubuntu","rhel","windows","osx"], //agents
["jdk6","jdk7","jdk8"], //tools
["banana","apple","orange","pineapple"] //fruit
]
def tasks = [:]
def comboBuilder
def comboEntry = []
def task = {
// builds and returns the task for each combination
/* Map the entries back to a more readable format
the index will correspond to the position of this axis in axisList[] */
def myAgent = it[0]
def myJdk = it[1]
def myFruit = it[2]
return {
// This is where the important work happens for each combination
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//We won't declare a specific agent this part
node {
println "fruit=${myFruit}"
}
}
}
/*
This is where the magic happens
recursively work through the axisList and build all combinations
*/
comboBuilder = { def axes, int level ->
for ( entry in axes[0] ) {
comboEntry[level] = entry
if (axes.size() > 1 ) {
comboBuilder(axes[1..-1], level + 1)
}
else {
tasks[comboEntry.join("-")] = task(comboEntry.collect())
}
}
}
stage ("Setup") {
node {
println "Initial Setup"
}
}
stage ("Setup Combinations") {
node {
comboBuilder(axisList, 0)
}
}
stage ("Multiconfiguration Parallel Tasks") {
//Run the tasks in parallel
parallel tasks
}
stage("The End") {
node {
echo "That's all folks"
}
}
You can see a more detailed flow of the job at http://localhost:8080/job/multi-configPipeline/[build]/flowGraphTable/ (available under the Pipeline Steps link on the build page.
EDIT:
You can move the stage down into the "task" creation and then see the details of each stage more clearly, but not in a neat matrix like the multi-config job.
...
return {
// This is where the important work happens for each combination
stage ("${it.join('-')}--build") {
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//Node irrelevant for this part
node {
println "fruit=${myFruit}"
}
}
}
...
Or you could wrap each node with their own stage for even more detail.
As I did this, I noticed a bug in my previous code (fixed above now). I was passing the comboEntry reference to the task. I should have sent a copy, because, while the names of the stages were correct, when it actually executed them, the values were, of course, all the last entry encountered. So I changed it to tasks[comboEntry.join("-")] = task(comboEntry.collect()).
I noticed that you can leave the original stage ("Multiconfiguration Parallel Tasks") {} around the execution of the parallel tasks. Technically now you have nested stages. I'm not sure how Jenkins is supposed to handle that, but it doesn't complain. However, the 'parent' stage timing is not inclusive of the parallel stages timing.
I also noticed is that when a new build starts to run, on the "Stage View" of the job, all the previous builds disappear, presumably because the stage names don't all match up. But after the build finishes running, they all match again and the old builds show up again.
And finally, Blue Ocean doesn't seem to vizualize this the same way. It doesn't recognize the "stages" in the parallel processes, only the enclosing stage (if it is present), or "Parallel" if it isn't. And then only shows the individual parallel processes, not the stages within.
Points 1 and 3 are not completely clear to me, but I suspect you just want to use “scripted” rather than “Declarative” Pipeline syntax, in which case you can make your job do whatever you like—anything permitted by matrix project axes and axis filters and much more, including parallel execution. Declarative syntax trades off syntactic simplicity (and friendliness to “round-trip” editing tools and “linters”) for flexibility.
Point 2 is about visualization of the result, rather than execution per se. While this is a complex topic, the usual concrete request which is not already supported by existing visualizations like Blue Ocean is to be able to see test results distinguished by axis combination. This is tracked by JENKINS-27395 and some related issues, with design in progress.
The conditionals of Jenkins Declarative Pipeline is built around branches namely.
I would like to evaluate whether a specific folder is changed within any branch and then run the stages.
Something like:
stage {
when {
folderX changed
}
}
To give you a better idea of why I need this feature, let me elaborate.
The project I am working on exists out of a few modules (let's say micro services). Although every module can have its own branch or even repository, we have chosen to put them together in their own folders, so we can always keep everything clean in the master.
Our Jenkins pipleine has a stage for every module. However, we do not want to rebuild every module if nothing is changed in that folder.
I finally solved the problem by a script block which I found here:
when{ expression {
def changeLogSets = currentBuild.changeSets
for (int i = 0; i < changeLogSets.size(); i++) {
def entries = changeLogSets[i].items
for (int j = 0; j < entries.length; j++) {
def entry = entries[j]
def files = new ArrayList(entry.affectedFiles)
for (int k = 0; k < files.size(); k++) {
def file = files[k]
if(file.path.contains("FolderOfInterest")){
return true
}
}
}
}
return false
}
}
There are couple problems here:
The feature you're proposing does not exist. You could submit a ticket in the jenkins issue tracker for that.
Also, I think you're suggesting that you want to look at code changes across branches. This is not a standard use case for pipelines. Usually, you want to build a specific branch of a project (when it changes, for example), so you are going to feel moderate to intense pain trying to do what you're suggesting.
If you want to see what has changed on the current branch from within a Jenkisfile, you can use currentBuild.changeSets (docs). You could combine that with if statements and whatnot, within a script block if you want to use a declarative pipeline like you seem to be suggesting.
Keep at it and you'll figure it out. Good luck!
I have a requirement to run a set of tasks for a build in parallel, The tasks for the build are dynamic it may change. I need some help in the implementation of that below are the details of it.
I tasks details for a build will be generated dynamically in an xml which will have information of which tasks has to be executed in parallel/serial
example:
say there is a build A.
Which had below task and the order of execution , first task 1 has to be executed next task2 and task3 will be executed in parallel and next is task 4
task1
task2,task3
task4
These details will be in an xml dynamically generated , how can i parse that xml and schedule task accordingly using pipeline plugin. I need some idea to start of with.
You can use Groovy to read the file from the workspace (readFile) and then generate the map containing the different closures, similar to the following:
parallel(
task2: {
node {
unstash('my-workspace')
sh('...')
}
},
task3: {
node {
unstash('my-workspace')
sh('...')
}
}
}
In order to generate such data structure, you simply iterate over the task data read using XML parsing in Groovy over the file contents you read previously.
By occasion, I gave a talk about pipelines yesterday and included very similar example (presentation, slide 34ff.). In contrast, I read the list of "tasks" from another command output. The complete code can be found here (I avoid pasting all of this here and instead refer to this off-site resource).
The kind of magic bit is the following:
def parallelConverge(ArrayList<String> instanceNames) {
def parallelNodes = [:]
for (int i = 0; i < instanceNames.size(); i++) {
def instanceName = instanceNames.get(i)
parallelNodes[instanceName] = this.getNodeForInstance(instanceName)
}
parallel parallelNodes
}
def Closure getNodeForInstance(String instanceName) {
return {
// this node (one per instance) is later executed in parallel
node {
// restore workspace
unstash('my-workspace')
sh('kitchen test --destroy always ' + instanceName)
}
}
}
I'm trying to create a job that will run a certain flow multiple times, each time with different parameters on multiple nodes in parallel.
I have a csv file, on which each line contains the requested parameters for a run.
I tried using multi configuration job, and I read about the dynamic axis, but I don't quite understand how to use it with the data from my csv file.
I also saw build flow and workflow plugins, but again, I couldn't understand how to use it with my csv file.
I'd appreciate if anyone can give me ideas how to solve this.
Thanks in advance,
Sivan
Beneath a solution without eachLine closure (works in Jenkins ver. 2.89.3).
Some closures like eachLine still seem to be broken in Jenkins.
def nodes = [:]
readFile("input.csv").split('\n').eachWithIndex { line, index ->
def params = line.split(',')
nodes[name] = {
// ...
}
If you don't need the counter, you can use 'each' instead
readFile("input.csv").split('\n').each { line -> ... }
Using the workflow plugin you could read the file, parse the contents with standard groovy then set up your nodes - something like
def nodes = [:]
readFile("myfile.csv").eachLine { line, count ->
def params = line.split(',')
nodes["line${count}"] = {
node {
// do stuff
}
}
}
parallel nodes
if you dont need the count variable you could use splitEachLine instead
def nodes = [:]
readFile("abc.csv").splitEachLine(/,/) { runName, param2, p3 ->
nodes[runName] = {
// dostuff with param2, p3
}
}