Bulk update parameter in jobs - jenkins

We have a lot of jobs that all perform SCM checkout based on a Build Parameter value: say REPO_URL=ssh://url. Over time there accumulated small differences in names and values of these parameters: REPOURL, repo_url, =ssh://url/, =ssh://url:port, etc.
We need to reduce them to a common denominator with a single parameter name and a single value. How do we bulk update parameters in 50+ jobs?

Using Jenkins Script Console.
NOTE: these are essentially destructive operations, so make sure you tested your code on some spare jobs before running it in production!!!
Change default value of a parameter
Jenkins.instance.getAllItems(Job)
// filter jobs by name if needed
.findAll { it.fullName.startsWith('sandbox/tmp-magic') }
.each {
it
.getProperty(ParametersDefinitionProperty)
.getParameterDefinition('MAGIC_PARAMETER')
// `each` ensures nothing happens if `get` returns null; also see paragraph below
.each {
it.defaultValue = 'shmagic'
}
// the job has changed, but next config reload (f.x. at restart) will overwrite our changes
// so we need to save job config to its config.xml file
it.save()
}
Instead of .getParameterDefinition('MAGIC_PARAMETER') you can use
.parameterDefinitions
.findAll { it.name == 'MAGIC_PARAMETER' }
, changing predicate in findAll if you need f.x. to change value of multiple parameters with different names - then you iterate over found definitions via each{}.
Change parameter name (and value)
This is slightly more tricky, since apparently you cannot edit name of ParameterDefinition, only replace one in a list.
Jenkins.instance.getAllItems(Job)
.findAll { it.fullName.startsWith('sandbox/tmp-magic') }
.each {
def parameters = it.getProperty(ParametersDefinitionProperty).parameterDefinitions
def oldParameter = parameters.find { it.name == 'FOO' }
// avoid changing jobs without this parameter
if (!oldParameter)
return
def idx = parameters.indexOf(oldParameter)
// preserve original value if necessary
def oldValue = oldParameter.defaultValue
parameters[idx] = new StringParameterDefinition('GOOD_FOO', oldValue)
it.save()
}
Bonus points: replace value for SCM step in Freestyle and Pipeline From SCM job
Some of our jobs use MercurialSCM plugin, and some use MultiSCM plugin to checkout multiple repos, so this is what I tested it with.
import hudson.plugins.mercurial.MercurialSCM
import org.jenkinsci.plugins.multiplescms.MultiSCM
import org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition
import org.jenkinsci.plugins.workflow.job.WorkflowJob
Jenkins.instance.getAllItems(Job)
.findAll { it.fullName.startsWith('sandbox/tmp-magic') }
.each {
print "Checking $it ... "
if (it.class == FreeStyleProject && it.scm) {
println "Freestyle"
it.scm = replaceWhateverScm(it.scm)
it.save()
} else if (it.class == WorkflowJob) {
print "Pipeline ... "
def flow = it.definition
if (flow.class == CpsScmFlowDefinition) {
println "ScmFlow"
def scm = replaceWhateverScm(flow.scm)
def newFlow = new CpsScmFlowDefinition(scm, flow.scriptPath)
newFlow.lightweight = flow.lightweight
it.definition = newFlow
it.save()
} else
println "unsupported definition"
} else
println "unsupported job"
}
def replaceWhateverScm(scm) {
if (scm.class == MercurialSCM) {
println "replacing MercurialSCM"
return replaceMercurialSource(scm)
}
if (scm.class == MultiSCM) {
println "replacing MultiSCM"
// cannot replace part of MultiSCM, replace whole scm instead
return new MultiSCM(
scm.configuredSCMs
.collect { (it.class == MercurialSCM) ? replaceMercurialSource(it) : it }
)
}
throw new Exception("unknown class ${scm.class}")
}
def replaceMercurialSource(MercurialSCM original) {
if (!original.source.toLowerCase().contains('repo_url'))
return original
def s = new MercurialSCM('<new_url>')
for (v in ["browser","clean","credentialsId","disableChangeLog","installation","modules","revision","revisionType","subdir",]) {
s."$v" = original."$v"
}
return s
}```

Related

Groovy script to implement "How to get a parameter depend of other parameter in Hudson or Jenkins"?

Jenkins v2.289.3
I'm trying to implement the Active Choices plugin discussed in one of the answers in in How to get a parameter depend of other parameter in Hudson or Jenkins, but unsure how to implement my second script to get the value from the first parameter.
I have a MyFolder job folder with multi-pipeline job called Builders, with branches like master, release/*, feature/*. So the full name of the jobs in the folder will be MyFolder/Builders/release%2F1.0.0 for the release/1.0.0 job for example (%2F is the escape character for /).
I then created a second job in the same folder called DeployBranchVersion, whose goal is to execute deployment code that deploys a chosen branch and one its corresponding successful build numbers. I therefore need to pass 2 parameters to the deployment code, GIT_BRANCH and VERSION.
My first Active Choices parameter gets these branches using the following script, and assigns the choice to the GIT_BRANCH parameter.
Job name: MyFolder/DeployBranchVersion
Parameter name: GIT_BRANCH
Groovy script:
def gettags = "git ls-remote -h -t https://username:password#bitbucket.org/organization/myrepo.git".execute()
def branches = []
def t1 = []
gettags.text.eachLine {branches.add(it)}
for(i in branches)
t1.add(i.split()[1].replaceAll('\\^\\{\\}', '').replaceAll('refs/heads/', '').replaceAll('refs/tags/', ''))
t1 = t1.unique()
return t1
This returns a drop-down list of the branches in my repo, and the chosen one is assigned to the GIT_BRANCH parameter.
Now how do I setup the second Active Choices Reactive parameter to reference the above choice? I have the following Groovy code that works in a non-Active-Choice parameter setup. How can I modify it to work in this case? The BUILD_JOB_NAME needs to reference the GIT_BRANCH value from the first parameter?
import hudson.model.*
BUILD_JOB_NAME = "some_reference_to_GIT_BRANCH" // ??????????
def getJobs() {
def hi = Hudson.instance
return hi.getItems(Job)
}
def getBuildJob() {
def buildJob = null
def jobs = getJobs()
(jobs).each { job ->
if (job.fullName == BUILD_JOB_NAME) {
buildJob = job
}
}
return buildJob
}
def getAllBuildNumbers(Job job) {
def buildNumbers = []
(job.getBuilds()).each { build ->
def status = build.getBuildStatusSummary().message
if (status.contains("stable") || status.contains("normal")) {
buildNumbers.add("${build.displayName}")
}
}
return buildNumbers
}
def buildJob = getBuildJob()
return getAllBuildNumbers(buildJob)
I tried setting it this way to ho avail.
BUILD_JOB_NAME = "MyFolder/Builders/$GIT_BRANCH"
Turns out I was doing it correctly, I just had a buggy 2nd script. Here's the good one. I realized that GIT_BRANCH values had / in them so I had to replace them with the equivalent escape character %2F.
import hudson.model.*
BRANCH = GIT_BRANCH.replaceAll("/", "%2F")
BUILD_JOB_NAME = "MyFolder/Builders/$BRANCH"
def getJobs() {
def hi = Hudson.instance
return hi.getAllItems(Job.class)
}
def getBuildJob() {
def buildJob = null
def jobs = getJobs()
(jobs).each { job ->
if (job.fullName == BUILD_JOB_NAME) {
buildJob = job
}
}
return buildJob
}
def getAllBuildNumbers(Job job) {
def buildNumbers = []
(job.getBuilds()).each { build ->
def status = build.getBuildStatusSummary().message
if ((status.contains("stable") || status.contains("normal")) &&
build.displayName.contains("-")) {
buildNumbers.add(build.displayName)
}
}
return buildNumbers
}
def buildJob = getBuildJob()
return getAllBuildNumbers(buildJob)

fetch source values from jenkins extended choice parameter

I have added an extended choice paramter. Now the source values are lin1, lin2, lin3 as listed in screenshot
now when I run,
If I select lin1 then I get param3 = lin1,
If I select lin1 and lin2 then I get param2 - lin1,lin2 ( delimiter is comma )
The question here is, inside jenkins pipeline how can get what all source values were set when the param was created. In short, without selecting any of the checkboxes, want to get the list of the possible values probably in a list
Eg:
list1 = some_method(param3)
// expected output >> list1 = [lin,lin2,lin3]
Let me know if this description is not clear.
The user who runs this does not have configure access ( we dont want to give configure access to anonynmous user ) Hence the job/config.xml idea will not work here
As requested you can also get the values dynamically:
import hudson.model.*
import org.jenkinsci.plugins.workflow.job.*
import com.cwctravel.hudson.plugins.extended_choice_parameter.ExtendedChoiceParameterDefinition
def getJob(name) {
def hi = Hudson.instance
return hi.getItemByFullName(name, Job)
}
def getParam(WorkflowJob job, String paramName) {
def prop = job.getProperty(ParametersDefinitionProperty.class)
for (param in prop.getParameterDefinitions()) {
if (param.name == paramName) {
return param
}
}
return null
}
pipeline {
agent any
parameters {
choice(name: 'FOO', choices: ['1','2','3','4'])
}
stages {
stage('test') {
steps {
script {
def job = getJob(JOB_NAME)
def param = getParam(job, "FOO")
if (param instanceof ChoiceParameterDefinition) {
// for the standard choice parameter
print param.getChoices()
} else if (param instanceof ExtendedChoiceParameterDefinition) {
// for the extended choice parameter plugin
print param.getValue()
}
}
}
}
}
}
As you can see it requires a lot of scripting, so just must either disable the Groovy sandbox or approve most of the calls on the script approval page.
I couldn't find any variable or method to get the parameter list. I guess it's somehow possible through a undocumented method on the param or currentBuild maps.
A possible solution to your problem could be defining the map outside of the pipeline and then just use that variables like this:
def param3Choices = ['lin1', 'lin2', 'lin3']
pipeline {
parameters {
choice(name: 'PARAM3', choices: param3Choices, description: '')
}
stage('Debug') {
steps {
echo param.PARAM3
print param3Choices
}
}
}

Why can't Jenkins find job object using extensible choice/system groovy choice parameter?

I followed Groovy to list all jobs to get a list of my job names, and got the exact "fullName" of my job, which is...
Microservice/build/feature%2Fdev-26387-split-micro-deploy
Now I have this to try to get all the successful build numbers for this job. The return values are just for debugging to see which value I'm returning, and I'm returning a list with the number "43", meaning, the getJobs() function returned a null, meaning it wasn't able to find the job object. Why?
import hudson.model.*
BUILD_JOB_NAME = "Microservice/build/feature%2Fdev-26387-split-micro-deploy"
def getJobs() {
def hi = Hudson.instance
return hi.getItems(Job)
}
def getBuildJob() {
def buildJob = null
def jobs = getJobs()
(jobs).each { job ->
if (job.fullName == BUILD_JOB_NAME) {
// WHY IS THIS NOT WORKING???
buildJob = job
}
}
return buildJob
}
def getAllBuildNumbers(Job job) {
try {
def buildNumbers = []
(job.getBuilds()).each { build ->
def status = build.getBuildStatusSummary().message
if (status.contains("stable") || status.contains("normal")) {
buildNumbers.add(build.number)
}
}
/// return buildNumbers
return ["44"]
}
catch (Throwable t) {
return ["45"]
}
}
def buildJob = getBuildJob()
if (buildJob == null) {
return ['43']
}
return getAllBuildNumbers(buildJob)
Use return hi.getAllItems(Job) instead and it should work. Also, ensure your BUILD_JOB_NAME is set to correct job name. To check the job name, run Hudson.instance.getAllItems(Job).each { println(it.fullName) } from Jenkins -> Manage Jenkins -> Script Console

How to trigger multiple down stream jobs in jenkins dynamically based on some input parameter

Scenario: I want to trigger few down stream jobs(Job A and Job B ....) dynamically based on the input parameter received by the current job.
import hudson.model.*
def values = ${configname}.split(',')
def currentBuild = Thread.currentThread().executable
println ${configname}
println ${sourceBranch}
values.eachWithIndex { item, index ->
println item
println index
def job = hudson.model.Hudson.instance.getJob(item)
def params = new StringParameterValue('upstream_job', ${sourceBranch})
def paramsAction = new ParametersAction(params)
def cause = new hudson.model.Cause.UpstreamCause(currentBuild)
def causeAction = new hudson.model.CauseAction(cause)
hudson.model.Hudson.instance.queue.schedule(job, 0, causeAction, paramsAction)
}
How about something like this? I was getting a comma separated list from the upstream system and I splitted them as individaul string which is internally jobs. Making a call by passing each individual strings.
this Jenkinsfile would do that:
#!/usr/bin/env groovy
pipeline {
agent { label 'docker' }
parameters {
string(name: 'myHotParam', defaultValue: '', description: 'What is your param, sir?')
}
stages {
stage('build') {
steps {
script {
if (params.myHotParam == 'buildEverything') {
build 'mydir/jobA'
build 'mydir/jobB'
}
}
}
}
}
}

How to tell Jenkins "Build every project in folder X"?

I have set up some folders (Using Cloudbees Folder Plugin).
It sounds like the simplest possible command to be able to tell Jenkins: Build every job in Folder X.
I do not want to have to manually create a comma-separated list of every job in the folder. I do not want to add to this list whenever I want to add a job to this folder. I simply want it to find all the jobs in the folder at run time, and try to build them.
I'm not finding a plugin that lets me do that.
I've tried using the Build Pipeline Plugin, the Bulk Builder Plugin, the MultiJob plugin, and a few others. None seem to support the use case I'm after. I simply want any Job in the folder to be built. In other words, adding a job to this build is as simple as creating a job in this folder.
How can I achieve this?
I've been using Jenkins for some years and I've not found a way of doing what you're after.
The best I've managed is:
I have a "run every job" job (which contains a comma-separated list of all the jobs you want).
Then I have a separate job that runs periodically and updates the "run every job" job as new projects come and go.
One way to do this is to create a Pipeline job that runs Groovy script to enumerate all jobs in the current folder and then launch them.
The version below requires the sandbox to be disabled (so it can access Jenkins.instance).
def names = jobNames()
for (i = 0; i < names.size(); i++) {
build job: names[i], wait: false
}
#NonCPS
def jobNames() {
def project = Jenkins.instance.getItemByFullName(currentBuild.fullProjectName)
def childItems = project.parent.items
def targets = []
for (i = 0; i < childItems.size(); i++) {
def childItem = childItems[i]
if (!childItem instanceof AbstractProject) continue;
if (childItem.fullName == project.fullName) continue;
targets.add(childItem.fullName)
}
return targets
}
If you use Pipeline libraries, then the following is much nicer (and does not require you to allow a Groovy sandbox escape:
Add the following to your library:
package myorg;
public String runAllSiblings(jobName) {
def names = siblingProjects(jobName)
for (def i = 0; i < names.size(); i++) {
build job: names[i], wait: false
}
}
#NonCPS
private List siblingProjects(jobName) {
def project = Jenkins.instance.getItemByFullName(jobName)
def childItems = project.parent.items
def targets = []
for (def i = 0; i < childItems.size(); i++) {
def childItem = childItems[i]
if (!childItem instanceof AbstractProject) continue;
if (childItem.fullName == jobName) continue;
targets.add(childItem.fullName)
}
return targets
}
And then create a pipeline with the following code:
(new myorg.JobUtil()).runAllSiblings(currentBuild.fullProjectName)
Yes, there are ways to simplify this further, but it should give you some ideas.
I developed a Groovy script that does this. It works very nicely. There are two Jobs, initBuildAll, which runs the groovy script and then launches the 'buildAllJobs' jobs. In my setup, I launch the InitBuildAll script daily. You could trigger it another way that works for you. We aren't full up CI, so daily is good enough for us.
One caveat: these jobs are all independent of one another. If that's not your situation, this may need some tweaking.
These jobs are in a separate Folder called MultiBuild. The jobs to be built are in a folder called Projects.
import com.cloudbees.hudson.plugins.folder.Folder
import javax.xml.transform.stream.StreamSource
import hudson.model.AbstractItem
import hudson.XmlFile
import jenkins.model.Jenkins
Folder findFolder(String folderName) {
for (folder in Jenkins.instance.items) {
if (folder.name == folderName) {
return folder
}
}
return null
}
AbstractItem findItem(Folder folder, String itemName) {
for (item in folder.items) {
if (item.name == itemName) {
return item
}
}
null
}
AbstractItem findItem(String folderName, String itemName) {
Folder folder = findFolder(folderName)
folder ? findItem(folder, itemName) : null
}
String listProjectItems() {
Folder projectFolder = findFolder('Projects')
StringBuilder b = new StringBuilder()
if (projectFolder) {
for (job in projectFolder.items.sort{it.name.toUpperCase()}) {
b.append(',').append(job.fullName)
}
return b.substring(1) // dump the initial comma
}
return b.toString()
}
File backupConfig(XmlFile config) {
File backup = new File("${config.file.absolutePath}.bak")
FileWriter fw = new FileWriter(backup)
config.writeRawTo(fw)
fw.close()
backup
}
boolean updateMultiBuildXmlConfigFile() {
AbstractItem buildItemsJob = findItem('MultiBuild', 'buildAllProjects')
XmlFile oldConfig = buildItemsJob.getConfigFile()
String latestProjectItems = listProjectItems()
String oldXml = oldConfig.asString()
String newXml = oldXml;
println latestProjectItems
println oldXml
def mat = newXml =~ '\\<projects\\>(.*)\\<\\/projects\\>'
if (mat){
println mat.group(1)
if (mat.group(1) == latestProjectItems) {
println 'no Change'
return false;
} else {
// there's a change
File backup = backupConfig(oldConfig)
def newProjects = "<projects>${latestProjectItems}</projects>"
newXml = mat.replaceFirst(newProjects)
XmlFile newConfig = new XmlFile(oldConfig.file)
FileWriter nw = new FileWriter(newConfig.file)
nw.write(newXml)
nw.close()
println newXml
println 'file updated'
return true
}
}
false
}
void reloadMultiBuildConfig() {
AbstractItem job = findItem('MultiBuild', 'buildAllProjects')
def configXMLFile = job.getConfigFile();
def file = configXMLFile.getFile();
InputStream is = new FileInputStream(file);
job.updateByXml(new StreamSource(is));
job.save();
println "MultiBuild Job updated"
}
if (updateMultiBuildXmlConfigFile()) {
reloadMultiBuildConfig()
}
A slight variant on Wayne Booth's "run every job" approach. After a little head scratching I was able to define a "run every job" in Job DSL format.
The advantage being I can maintain my job configuration in version control. e.g.
job('myfolder/build-all'){
publishers {
downstream('myfolder/job1')
downstream('myfolder/job2')
downstream('myfolder/job2')
}
}
Pipeline Job
When running as a Pipeline job you may use something like:
echo jobNames.join('\n')
jobNames.each {
build job: it, wait: false
}
#NonCPS
def getJobNames() {
def project = Jenkins.instance.getItemByFullName(currentBuild.fullProjectName)
project.parent.items.findAll {
it.fullName != project.fullName && it instanceof hudson.model.Job
}.collect { it.fullName }
}
Script Console
Following code snippet can be used from the script console to schedule all jobs in some folder:
import hudson.model.AbstractProject
Jenkins.instance.getAllItems(AbstractProject.class).each {
if(it.fullName =~ 'path/to/folder') {
(it as AbstractProject).scheduleBuild2(0)
}
}
With some modification you'd be able to create a jenkins shared library method (requires to run outside the sandbox and needs #NonCPS), like:
import hudson.model.AbstractProject
#NonCPS
def triggerItemsInFolder(String folderPath) {
Jenkins.instance.getAllItems(AbstractProject.class).each {
if(it.fullName =~ folderPath) {
(it as AbstractProject).scheduleBuild2(0)
}
}
}
Reference pipeline script to run a parent job that would trigger other jobs as suggested by #WayneBooth
pipeline {
agent any
stages {
stage('Parallel Stage') {
parallel {
stage('Parallel 1') {
steps {
build(job: "jenkins_job_1")
}
}
stage('Parallel 2') {
steps {
build(job: "jenkins_job_2")
}
}
}
}
}
The best way to run an ad-hoc command like that would be using the Script Console (can be found under Manage Jenkins).
The console allows running Groovy Script - the script controls Jenkins functionality. The documentation can be found under Jenkins JavaDoc.
A simple script triggering immediately all Multi-Branch Pipeline projects under the given folder structure (in this example folder/subfolder/projectName):
import org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject
import hudson.model.Cause.UserIdCause
Jenkins.instance.getAllItems(WorkflowMultiBranchProject.class).findAll {
return it.fullName =~ '^folder/subfolder/'
}.each {
it.scheduleBuild(0, new UserIdCause())
}
The script was tested against Jenkins 2.324.

Resources