I'm trying to create a job that will run a certain flow multiple times, each time with different parameters on multiple nodes in parallel.
I have a csv file, on which each line contains the requested parameters for a run.
I tried using multi configuration job, and I read about the dynamic axis, but I don't quite understand how to use it with the data from my csv file.
I also saw build flow and workflow plugins, but again, I couldn't understand how to use it with my csv file.
I'd appreciate if anyone can give me ideas how to solve this.
Thanks in advance,
Sivan
Beneath a solution without eachLine closure (works in Jenkins ver. 2.89.3).
Some closures like eachLine still seem to be broken in Jenkins.
def nodes = [:]
readFile("input.csv").split('\n').eachWithIndex { line, index ->
def params = line.split(',')
nodes[name] = {
// ...
}
If you don't need the counter, you can use 'each' instead
readFile("input.csv").split('\n').each { line -> ... }
Using the workflow plugin you could read the file, parse the contents with standard groovy then set up your nodes - something like
def nodes = [:]
readFile("myfile.csv").eachLine { line, count ->
def params = line.split(',')
nodes["line${count}"] = {
node {
// do stuff
}
}
}
parallel nodes
if you dont need the count variable you could use splitEachLine instead
def nodes = [:]
readFile("abc.csv").splitEachLine(/,/) { runName, param2, p3 ->
nodes[runName] = {
// dostuff with param2, p3
}
}
Related
I have a file called apps.txt which has three app names in it
frontendapp
authorizationservice
connectorservice*
In jenkins pipeline i want to perform some operation on them one by one so i am trying to get them in groovy list using this code -
list = readFile.readFileLineByLine("${workspace}/apps.txt").collect {it}
for (item in list) {
println "I need to perform some operations on files"
}
But getting groovy.lang.MissingPropertyException.
If i use file class like this - list = new File("${workspace}/apps.txt").collect {it} then it search for a file on Jenkins master node only and i get fileNotFoundException.
If i use list = readFile("${workspace}/apps.txt").collect {it} then list gets values character by character. How i can get app names from apps.txt inorder to perform operation on each app.
Your attempts are close, but mix things up.
Those are working ways:
def list = new File("${workspace}/apps.txt").text.readLines()
Note the .text call inbetween.
def list = readFile("${workspace}/apps.txt").readLines()
Or with the helper Jenkins provides.
Side note: .collect{it} is just .collect() - and usually is only
needed to copy a list. Since the read lines from the file are already
eagerly read, those are copies.
This work for me in a windows worker
script {
def list = readFile("filesChanged").readLines()
for(item in list){
print item
bat('type '+item)
}
}
I am retrieving JSON object from a URL using httpRequest in a groovy script.
pipeline {
agent any
stages {
stage ('Extract Data') {
steps {
script {
def response = httpRequest \
authentication: 'user', \
httpMode: 'GET', \
url: "https://example.com/data"
writeFile file: 'output.json', text: response.content
def data = readFile(file: 'output.json')
def details = new groovy.json.JsonSlurperClassic().parseText(data)
echo "Data: ${details.fields.customfield}"
}
}
}
}
}
I am interested in the customfieldstring. The format of the string is:
Application!01.01.01 TestSuite1,TestSuite2,TestSuite3,TestSuite4 Product!01.01.01,Product2!01.01.02
I would like to parse the string into 3 data sets:
Map of Applications [Application: version] (there will always be one Appliction)
List of TestSuites [TestSuite1,...,TestSuite]
Map of Prodcts [Product1: version,..., ProductN: version].
However, I am not sure how to do this.
Are there any Jenkins Groovy libraries that I can use to do this in a declarative pipeline?
EDIT
Based on the answer below I can see that I can make a map in the following way:
def applications = groups[0].split(',').collect { it.split('!') }.collectEntries { [(it):it] }
In the example I have:
application = [Application: Application]
How do I get:
application = [Application: 01.01.01]
EDIT2
Note the following output:
def applications = groups[0].split(',').collect { it.split('!') }
[[Application, 01.01.01]]
There're no libraries I'm aware of that will have functionality to parse the data but, since you know the format of the data it's easy to parse them manually.
There are 3 groups in the input (applications, suites, products) separated by a character. To get the groups you need:
def input = "Application!01.01.01 TestSuite1,TestSuite2,TestSuite3,TestSuite4 Product!01.01.01,Product2!01.01.02"
def groups = input.split(' ')
To process the applications you need to split group 0 with , character (just in case there are many applications). You got a list of pairs in format: name!version. Every pair must be splitted with !, so you get a list of lists in format: [[name, version]]. From the last structure it's easy to create a map. All steps together:
def applications = groups[0].split(',').collect { it.split('!') }.collectEntries { [(it[0]):it[1]] }
Getting the list of suites is easy, just split group 1 with , character:
def suites = groups[1].split(',')
Finally, products are analogical to the list of applications but this time group 2 should be used:
def products = groups[2].split(',').collect { it.split('!') }.collectEntries { [(it[0]):it[1]] }
You can simplifier your issue by using pipeline utility step: readJSON
def data = readJSON(file: 'output.json')
echo data.fields.customfield
I found a method. Groovy can convert the values of an Object array and convert them into a map with the toSpreadMap(). However, the array must have an even number of elements.
def appList = ['DevOpsApplication', '01.01.01']
def appMap = appList.toSpreadMap()
For some better answers please refer to this
I am trying to run a list of builds in parallel using a for loop because the code is getting bigger and bigger.
I have a global list with the names of projects
#Field def final String[] projectsList = ['project1','project2', 'project3'....]
stages {
stage('Parallel Build') {
steps{
script{
def branches = [:]
for(int i = 0;i<15;i++) {
branches["Build "+projectsList[i]] = {buildProject(i)}
}
parallel branches
}
}
}
The Build projects method takes the name of the project from the global list and builds it using maven.
The thing is, the project at index 15 (Which shouldn't be built) is building 15 times in parallel. As if it is waiting until the for loop ends and then assigns the same available i value (15) in this case to all the methods.
Do you have any idea how can I solve this ?
Your issue lies with how you make (mis)use of the Groovy closure concept, i.e. the part where you define a closure in the loop's body that makes use of the iteration variable i, that is { buildProject(i) } :)
What happens exactly is nicely described here. This is, indeed, a common "gotcha" with other languages offering functional programming features, as well (e.g. JavaScript).
The easiest (hardly the most elegant, though) solution is to define a variable within the loop that receives the current i value and make use of that within the closure:
def branches = [:]
for(i = 0; i < 15; i++) {
def curr = i
branches["Build ${projectsList[i]}"] = { buildProject(curr) }
}
parallel branches
(I've also used a bit more idiomatic Groovy like String interpolation).
A more elegant, less verbose, Groovy-like solution that iterates through the range of projects would be:
(0..<projectsList.size()).each { i ->
branches["Build ${projectsList[i]}"] = { buildProject(i) }
}
I want to trigger Jenkins job on "hyp-z" and "hyp-x" build nodes. I tried to write it this way but getting "There are no nodes with the label ‘hyp-x&&hyp-z’"
node ('hyp-z&&hyp-x') {
// write something here
}
What is the mistake I am doing and what is the exact working format?
This isn't possible in this form.
The && expression is for narrowing down your pool of nodes for certain features.
e.g. I want to run on a node which has the label UBUNTU and DOCKER.
As opposed to running on two different nodes with those labels.
You can use parallel block to do what you are wanting.
If you are using the Declarative syntax then see this article
https://jenkins.io/blog/2017/09/25/declarative-1/
or here for scripted
https://jenkins.io/doc/book/pipeline/jenkinsfile/#parallel-execution
I have tried this in two ways
def labels = ["hyp-x", "hyp-z"]
def builders = [:]
for (x in labels) {
def label = x
builders[label] = {
node(label) {
// build script
}
}
}
parallel builders
Above code is working as expected from Jenkinsfile but I see both builds triggered in one common job and log looks clumsy. So I have tried 2nd approach like below
Created Jenkinsfile.x and Jenkinsfile.z and each file represents x and z platform build.
I have a requirement to run a set of tasks for a build in parallel, The tasks for the build are dynamic it may change. I need some help in the implementation of that below are the details of it.
I tasks details for a build will be generated dynamically in an xml which will have information of which tasks has to be executed in parallel/serial
example:
say there is a build A.
Which had below task and the order of execution , first task 1 has to be executed next task2 and task3 will be executed in parallel and next is task 4
task1
task2,task3
task4
These details will be in an xml dynamically generated , how can i parse that xml and schedule task accordingly using pipeline plugin. I need some idea to start of with.
You can use Groovy to read the file from the workspace (readFile) and then generate the map containing the different closures, similar to the following:
parallel(
task2: {
node {
unstash('my-workspace')
sh('...')
}
},
task3: {
node {
unstash('my-workspace')
sh('...')
}
}
}
In order to generate such data structure, you simply iterate over the task data read using XML parsing in Groovy over the file contents you read previously.
By occasion, I gave a talk about pipelines yesterday and included very similar example (presentation, slide 34ff.). In contrast, I read the list of "tasks" from another command output. The complete code can be found here (I avoid pasting all of this here and instead refer to this off-site resource).
The kind of magic bit is the following:
def parallelConverge(ArrayList<String> instanceNames) {
def parallelNodes = [:]
for (int i = 0; i < instanceNames.size(); i++) {
def instanceName = instanceNames.get(i)
parallelNodes[instanceName] = this.getNodeForInstance(instanceName)
}
parallel parallelNodes
}
def Closure getNodeForInstance(String instanceName) {
return {
// this node (one per instance) is later executed in parallel
node {
// restore workspace
unstash('my-workspace')
sh('kitchen test --destroy always ' + instanceName)
}
}
}