Parse Data Using Jenkins Groovy Pipeline Script - jenkins

I am retrieving JSON object from a URL using httpRequest in a groovy script.
pipeline {
agent any
stages {
stage ('Extract Data') {
steps {
script {
def response = httpRequest \
authentication: 'user', \
httpMode: 'GET', \
url: "https://example.com/data"
writeFile file: 'output.json', text: response.content
def data = readFile(file: 'output.json')
def details = new groovy.json.JsonSlurperClassic().parseText(data)
echo "Data: ${details.fields.customfield}"
}
}
}
}
}
I am interested in the customfieldstring. The format of the string is:
Application!01.01.01 TestSuite1,TestSuite2,TestSuite3,TestSuite4 Product!01.01.01,Product2!01.01.02
I would like to parse the string into 3 data sets:
Map of Applications [Application: version] (there will always be one Appliction)
List of TestSuites [TestSuite1,...,TestSuite]
Map of Prodcts [Product1: version,..., ProductN: version].
However, I am not sure how to do this.
Are there any Jenkins Groovy libraries that I can use to do this in a declarative pipeline?
EDIT
Based on the answer below I can see that I can make a map in the following way:
def applications = groups[0].split(',').collect { it.split('!') }.collectEntries { [(it):it] }
In the example I have:
application = [Application: Application]
How do I get:
application = [Application: 01.01.01]
EDIT2
Note the following output:
def applications = groups[0].split(',').collect { it.split('!') }
[[Application, 01.01.01]]

There're no libraries I'm aware of that will have functionality to parse the data but, since you know the format of the data it's easy to parse them manually.
There are 3 groups in the input (applications, suites, products) separated by a character. To get the groups you need:
def input = "Application!01.01.01 TestSuite1,TestSuite2,TestSuite3,TestSuite4 Product!01.01.01,Product2!01.01.02"
def groups = input.split(' ')
To process the applications you need to split group 0 with , character (just in case there are many applications). You got a list of pairs in format: name!version. Every pair must be splitted with !, so you get a list of lists in format: [[name, version]]. From the last structure it's easy to create a map. All steps together:
def applications = groups[0].split(',').collect { it.split('!') }.collectEntries { [(it[0]):it[1]] }
Getting the list of suites is easy, just split group 1 with , character:
def suites = groups[1].split(',')
Finally, products are analogical to the list of applications but this time group 2 should be used:
def products = groups[2].split(',').collect { it.split('!') }.collectEntries { [(it[0]):it[1]] }

You can simplifier your issue by using pipeline utility step: readJSON
def data = readJSON(file: 'output.json')
echo data.fields.customfield

I found a method. Groovy can convert the values of an Object array and convert them into a map with the toSpreadMap(). However, the array must have an even number of elements.
def appList = ['DevOpsApplication', '01.01.01']
def appMap = appList.toSpreadMap()
For some better answers please refer to this

Related

How does this jenkins groovy script work?

This is my groovy Script on Jenkins
def gitURL = "http://bitbucket.webapp.intern.de/scm/myproject.git"
def command = "git ls-remote -h $gitURL"
def proc = command.execute() // line 3
proc.waitFor()
if ( proc.exitValue() != 0 ) {
println "Error, ${proc.err.text}"
System.exit(-1)
}
def branches = proc.in.text.readLines().collect { // line 9
it.replaceAll(/[a-z0-9]*\trefs\/heads\//, '') // line 10
}
return branches
First I do not understand anything. How on line 3 can you call execute on command which is supposed to be a string ??
What type is variable branches in line 9 ???
Is branches a String ??
What is it.replaceAll in line 10 ??? I know replaceAll as method from String. But it has 2 parameters.
I do not see 2 parameters here.
Now I somehow understand branches contains all branches. What I want to do. I only want to have
branches which contain "-REST" it their names. How can I do this ??
My intention was using java. But it does not work.
List<String> branchesNew = new ArrayList<String>();
for(String branch : branchesNew)
{
if(branch.contains("-REST"))
branchesNew.add(branch);
}
line 3: this is a Groovy feature, see http://groovy-lang.org/groovy-dev-kit.html#process-management
line 9: it's a collection (a list, for instance) given as output of the collect method. see also http://groovy-lang.org/groovy-dev-kit.html#_working_with_collections
line 10: the second argument is ''. it replaces the regexp matches (branch prefixes) with an empty string.
You can filter all your wanted branches using the findAll method on collections. Following the given example:
def branchesNew = proc.in.text.readLines().findAll{ it.contains('-REST') }
Look at Filtering and searching for all related manipulation methods.

How to assign an array to a Env variable in Jenkinsfile

I am trying to run a pipeline that has several servers. I want to do some actions in several servers at a time when selecting a choice parameter. My idea is to select a choice parameter 'APPLICATION' and execute some actions on 4 different servers sequentially (one server at a time). I am trying to put the environment variables assigning the value os the servers in an array and then ask for the environment variable to execute the actions.
pipeline {
agent {
node {
label 'master'
}
}
environment {
APPLICATION = ['veappprdl001','veappprdl002','veappprdl003','veappprdl004']
ROUTER = ['verouprdl001','verouprdl002']
}
parameters {
choice(name: 'SERVER_NAME', choices: ['APPLICATION','ROUTER'], description: 'Select Server to Test' )
}
stages {
stage ('Application Sync') {
steps {
script {
if (env.SERVER_NAME == 'APPLICATION') {
sh """
curl --location --request GET 'http://${SERVER_NAME}//configuration-api/localMemory/update/ACTION'
"""
}
}
}
}
} }
I want to execute the action on all the servers of the 'APPLICATION' variable if is selected the 'APPLICATION' parameter in 'Build with parameters'.
Any Help would be appreciate it.
Thanks
You can't store a value of an array type in the environment variable. Whatever you are trying to assign to the env variable gets automatically cast to the string type. (I explained it in more detail in the following blog post or this video.) So when you try to assign an array, what you assign is its toString() representation.
However, you can solve this problem differently. Instead of trying to assign an array, you can store a string of values with a common delimiter (like , for instance.) Then in the part that expects to work with a list of elements, you simply call tokenize(",") method to produce a list of string elements. Having that you can iterate and do things in sequence.
Consider the following example that illustrates this alternative solution.
pipeline {
agent any
environment {
APPLICATION = "veappprdl001,veappprdl002,veappprdl003,veappprdl004"
}
stages {
stage("Application Sync") {
steps {
script {
env.APPLICATION.tokenize(",").each { server ->
echo "Server is $server"
}
}
}
}
}
}
When you run such a pipeline you will get something like this:

Not that kind of Map exception with Jenkins and Groovy

I have a string in groovy that I want to convert into a map. When I run the code on my local computer through a groovy script for testing, I have no issues and a lazy map is returned. I can then convert that to a regular map and life goes on. When I try the same code through my Jenkins DSL pipeline, I run into the exception
groovy.json.internal.Exceptions$JsonInternalException: Not that kind of map
Here is the code chunk in question:
import groovy.json.*
String string1 = "{value1={blue green=true, red pink=true, gold silver=true}, value2={red gold=false}, value3={silver brown=false}}"
def stringToMapConverter(String stringToBeConverted){
formattedString = stringToBeConverted.replace("=", ":")
def jsonSlurper = new JsonSlurper().setType(JsonParserType.LAX)
def mapOfString = jsonSlurper.parseText(formattedString)
return mapOfString
}
def returnedValue = stringToMapConverter(string1)
println(returnedValue)
returned value:
[value2:[red gold:false], value1:[red pink:true, gold silver:true, blue green:true], value3:[silver brown:false]]
I know that Jenkins and Groovy differ in various ways, but from searches online others suggest that I should be able to use the LAX JsonSlurper library within my groovy pipeline. I am trying to avoid hand rolling my own string to map converter and would prefer to use a library if it's out there. What could be the difference here that would cause this behavior?
Try to use
import groovy.json.*
//#NonCPS
def parseJson(jsonString) {
// Would like to use readJSON step, but it requires a context, even for parsing just text.
def lazyMap = new JsonSlurper().setType(JsonParserType.LAX).parseText(jsonString.replace("=", ":").normalize())
// JsonSlurper returns a non-serializable LazyMap, so copy it into a regular map before returning
def m = [:]
m.putAll(lazyMap)
return m
}
String string1 = "{value1={blue green=true, red pink=true, gold silver=true}, value2={red gold=false}, value3={silver brown=false}}"
def returnedValue = parseJson(string1)
println(returnedValue)
println(JsonOutput.toJson(returnedValue))
You can find information about normalize here.

Get Build Number from another job in Jenkins as a variable

I am running a Build Flow job that executes multiple builds in parallel and then use the Post-build Action to Publish HTML reports.
How can I get the build number of each of the individual jobs as a variable so I can use when fetching the HTML report?
EDIT
This is what my parallel code looks like:
parallel (
{ uarr = build("Baseline - Secure - UARR", param1: build.properties.get("number")) },
{ login = build("Baseline - Secure - Login", param2: build.properties.get("number")) }
)
And this is what i tried using when using the Publish HTML reports for the Index page[s], but it's not seeing ${param1} as a variable and trying to find it literally:
*Secure Baseline*Secure_UARR-${param1}.html
This is what I'm using in the Maven build job and it is working great at finding the report with the correct filename that contains the build number:
*Secure Baseline*Secure_UARR-${BUILD_NUMBER}.html
The problem is, if I use that same logic in the Build Flow parallel job, it uses the build number of that job, not the Maven job that creates the report. (I hope that makes sense)
You can store the job references in a variable
parallel(
job1: { def n = build("JOB_NAME", PARAM_1: "value-1", PARAM_2: true, ...) }
...
)
or even store them in an array if you like
def jobs = [:]
parallel(
job1: {
def n1 = build("job1", param1: "value1", ...)
jobs["job1"] = n1.number
},
job1: {
def n2 = build("job2", param1: "value1", ...)
jobs["job2"] = n2.number
},
job1: {
def n3 = build("job3", param1: "value1", ...)
jobs["job3"] = n3.number
},
jobm: {
def nm = build("jobm", param1: "value1", ...)
jobs["jobm"] = n1.number
}
)
And then you can read the map when you want
The following answer shows how to export this as environment variables.
def buildEnv = build.getEnvVars();
buildEnv.putAll(jobs)
import org.jenkinsci.plugins.envinject.EnvInjectPluginAction
def envInjectAction = build.getAction(EnvInjectPluginAction.class);
envInjectAction.overrideAll(buildEnv)
You can the use it in your postbuild step(s) as $job1, $job2, ..., $jobm.
This resolved bug suggests that you then can use it in the HTML-publisher plugin (I am not verry familiar with the plugin)

How to run Jenkins build steps in a loop from csv file

I'm trying to create a job that will run a certain flow multiple times, each time with different parameters on multiple nodes in parallel.
I have a csv file, on which each line contains the requested parameters for a run.
I tried using multi configuration job, and I read about the dynamic axis, but I don't quite understand how to use it with the data from my csv file.
I also saw build flow and workflow plugins, but again, I couldn't understand how to use it with my csv file.
I'd appreciate if anyone can give me ideas how to solve this.
Thanks in advance,
Sivan
Beneath a solution without eachLine closure (works in Jenkins ver. 2.89.3).
Some closures like eachLine still seem to be broken in Jenkins.
def nodes = [:]
readFile("input.csv").split('\n').eachWithIndex { line, index ->
def params = line.split(',')
nodes[name] = {
// ...
}
If you don't need the counter, you can use 'each' instead
readFile("input.csv").split('\n').each { line -> ... }
Using the workflow plugin you could read the file, parse the contents with standard groovy then set up your nodes - something like
def nodes = [:]
readFile("myfile.csv").eachLine { line, count ->
def params = line.split(',')
nodes["line${count}"] = {
node {
// do stuff
}
}
}
parallel nodes
if you dont need the count variable you could use splitEachLine instead
def nodes = [:]
readFile("abc.csv").splitEachLine(/,/) { runName, param2, p3 ->
nodes[runName] = {
// dostuff with param2, p3
}
}

Resources