I am creating jobs for use with Terraform. There are several environments and the number is growing all the time. Rather than update both the pipeline file and the jobdsl file as the parameters change, I started working from the standpoint of scanning the repo for environment files and updating the pipeline and jobdsl file as needed.
My jobdsl script:
#Library('mylib') _
params = [
"serviceName": "infrastructure-${repo}",
"repoUrl": "${repoUrl}",
"sshCredentials": 'git-readonly',
"environment": "${env.Environment}",
"configParams": getTFConfigs(
repoUrl,
"env/${env.AccountName}/${env.AWSRegion}/${env.Environment}")
]
template = libraryResource('dslTemplates/infra.groovy')
jobDsl scriptText: helpers.renderTemplate(template, params)
shared library method: getTFConfigs
#!/usr/bin/env groovy
#NonCPS
import java.util.zip.ZipEntry
import java.util.zip.ZipInputStream
def call(String repoUrl, String filter=""){
def gitProc = new ProcessBuilder(
"git",
"archive",
"--format=zip",
"--remote=${repoUrl}",
"main").start()
def zipIn = new ZipInputStream(gitProc.inputStream)
def zipMembers = []
while (true) {
def ZipEntry entry = zipIn.getNextEntry()
if (entry == null) break
if ( (entry.getName()).contains(filter) ) {
entryName = entry.getName()
zipMembers.push("${entryName}")
}
}
println zipMembers
return zipMembers
}
dslTemplates/infra.groovy template
pipelineJob("${serviceName}") {
description("Apply TF for ${serviceName} to all environment configurations")
definition {
parameters {
<% configParams.each { %>
booleanParam(name: "<%= "${it}" %>", defaultValue: true, description: "<%= "${it}" %>" )
<% } %>
}
logRotator {
numToKeep(20)
}
cpsScm {
scm {
git{
remote{
url("${repoUrl}")
credentials("${sshCredentials}")
branch('*/main')
}
}
}
scriptPath('infra.groovy')
}
}
}
Template result
...
definition {
parameters {
booleanParam(name: env1.tfvars, defaultValue: true, description: env1.tfvars )
booleanParam(name: env2.tfvars, defaultValue: true, description: env2.tfvars )
}
...
When the seed job runs and executes the code, the parameters should be updated with a checkbox for each environment. However, the jobdsl fails with this:
ERROR: (script, line 6) No signature of method: javaposse.jobdsl.dsl.helpers.BuildParametersContext.booleanParam() is applicable for argument types: (java.util.LinkedHashMap) values: [[name:env1.tfvars, defaultValue:true, ...]]
Possible solutions: booleanParam(java.lang.String), booleanParam(java.lang.String, boolean), booleanParam(java.lang.String, boolean, java.lang.String)
Finished: FAILURE
I have tried to applying "toString()" at various steps and cannot seem to find any solution to this.
I have tried to write the entire jobdsl script to a file and read it back in using "jobDsl targets: filename" and got the same result!
Banging my head! as it were!
Thanks
It looks like you used Pipeline Syntax for the parameters in DSL script. If you want to define a parameter in a DSL script do not use name, defaultValue and description. (See Job DSL Plugin)
booleanParam('BOOL_PARAM', true, 'This is a boolean param')
Related
Using the declarative pipeline syntax, I want to be able to define parameters based on an array of repos, so that when starting the build, the user can check/uncheck the repos that should not be included when the job runs.
final String[] repos = [
'one',
'two',
'three',
]
pipeline {
parameters {
booleanParam(name: ...) // static param
// now include a booleanParam for each item in the `repos` array
// like this but it's not allowed
script {
repos.each {
booleanParam(name: it, defaultValue: true, description: "Include the ${it} repo in the release?")
}
}
}
// later on, I'll loop through each repo and do stuff only if its value in `params` is `true`
}
Of course, you can't have a script within the parameters block, so this won't work. How can I achieve this?
Using the Active Choices Parameter plugin is probably the best choice, but if for some reason you can't (or don't want to) use a plugin, you can still achieve dynamic parameters in a Declarative Pipeline.
Here is a sample Jenkinsfile:
def list_wrap() {
sh(script: 'echo choice1 choice2 choice3 choice4 | sed -e "s/ /\\n/g"', , returnStdout: true)
}
pipeline {
agent any
stages {
stage ('Gather Parameters') {
steps {
timeout(time: 30, unit: 'SECONDS') {
script {
properties([
parameters([
choice(
description: 'List of arguments',
name: 'service_name',
choices: 'DEFAULT\n' + list_wrap()
),
booleanParam(
defaultValue: false,
description: 'Whether we should apply changes',
name: 'apply'
)
])
])
}
}
}
}
stage ('Run command') {
when { expression { params.apply == true } }
steps {
sh """
echo choice: ${params.service_name} ;
"""
}
}
}
}
This embeds a script {} in a stage, which calls a function, which runs a shell script on the agent/node of the Declarative Pipeline, and uses the script's output to set the choices for the parameters. The parameters are then available in the next stages.
The gotcha is that you must first run the job with no build parameters in order for Jenkins to populate the parameters, so they're always going to be one run out of date. That's why the Active Choices Parameter plugin is probably a better idea.
You could also combine this with an input command to cause the pipeline to prompt the user for a parameter:
script {
def INPUT_PARAMS = input message: 'Please Provide Parameters', ok: 'Next',
parameters: [
choice(name: 'ENVIRONMENT', choices: ['dev','qa'].join('\n'), description: 'Please select the Environment'),
choice(name: 'IMAGE_TAG', choices: getDockerImages(), description: 'Available Docker Images')]
env.ENVIRONMENT = INPUT_PARAMS.ENVIRONMENT
env.IMAGE_TAG = INPUT_PARAMS.IMAGE_TAG
}
Credit goes to Alex Lashford (https://medium.com/disney-streaming/jenkins-pipeline-with-dynamic-user-input-9f340fb8d9e2) for this method.
You can use CHOICE parameter of Jenkins in which user can select a repository.
pipeline {
agent any
parameters
{
choice(name: "REPOS", choices: ['REPO1', 'REPO2', 'REPO3'])
}
stages {
stage ('stage 1') {
steps {
// the repository selected by user will be printed
println("$params.REPOS")
}
}
}
}
You can also use the plugin Active Choices Parameter if you want to do multiple select : https://plugins.jenkins.io/uno-choice/#documentation
You can visit pipeline syntax and configure in below way to generate code snippet and you can put it in the jenkins file:
Copy the snippet code and paste it in jenkinsfile at the start.
I'm hoping to find a way to remove duplicated stages in a Declarative Jenkinsfile by loading environment variables based on the branch.
Currently I have something like:
#Library("MySharedLibrary#v1") _
String tagBaseDev = "repo.org/myspace/image:dev"
String tagBaseTest = "repo.org/myspace/image:test"
String tagBaseProd = "repo.org/myspace/image:prod"
pipeline {
agent none
stages {
// Always Run This
stage ('Maven Build and Unit Tests') {
agent {label 'docker-slave'}
steps {
sharedLibraryBuild mavenGoals:'clean package', additionalProps:['ci.env':'']
stash 'artifacts'
}
}
// Dev Only
stage ('Build Dev Docker Image and Push') {
when {
branch 'dev'
}
agent {label 'docker-slave'}
steps {
unstash 'artifacts'
sharedLibraryDockerImageBuildPush tag:"$tagBaseDev"
}
}
// Test Only
stage ('Build Test Docker Image and Push') {
when {
branch 'test'
}
agent {label 'docker-slave'}
steps {
unstash 'artifacts'
sharedLibraryDockerImageBuildPush tag:"$tagBaseTest"
}
}
// Prod Only
stage ('Build Prod Docker Image and Push') {
when {
branch 'prod'
}
agent {label 'docker-slave'}
steps {
unstash 'artifacts'
sharedLibraryDockerImageBuildPush tag:"$tagBaseProd"
}
}
}
}
I want to be able to reduce that into one stage block and dynamically load in the needed $tagBaseXXX based on branch. This is just an example but I'm planning to have four or five variables that will have different values for each environment.
My thought was to create EnvDev, EnvTest, and EnvProd maps with corresponding values and then create a EnvMap which is a Map that correlates the branch name to Environment Map. For instance:
def EnvDev = [
url: "dev.com",
tag: "dev",
var: "Dev Var"
]
def EnvProd = [
url: "prod.com",
tag: "prod",
var: "prod Var"
]
def EnvMap = [
dev: EnvDev,
prod: EnvProd
]
I then try to create a Shared Library call that looks something like this:
def call(String branch, Map envMapping) {
Map use_me = envMapping.get("${branch}")
String url = use_me.get("URL")
echo ("${url}")
}
With the idea being to pass the Map and pull the corresponding environment map based on the branch and then use the variables as needed.
So I have something like this:
#Library("MySharedLibrary#v1") _
def EnvDev = [
url: "dev.com",
tag: "dev",
var: "Dev Var"
]
def EnvProd = [
url: "prod.com",
tag: "prod",
var: "prod Var"
]
def EnvMap = [
dev: EnvDev,
prod: EnvProd
]
pipeline {
agent {label 'docker-slave'}
stages {
stage ('Test Env Vars') {
steps {
echo "$env.GIT_BRANCH"
sharedLibrarySetupEnv branch: "$env.GIT_BRANCH", evnMapping: EnvMap
}
}
}
}
But I get the following error:
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: setupEnv.call() is applicable for argument types: (java.util.LinkedHashMap) values: [[branch:dev, env_mapping:[dev:[url:dev.com, tag:dev, var:Dev Var], ...]]]
Possible solutions: call(java.lang.String, java.util.Map), wait(), any(), wait(long), main([Ljava.lang.String;), each(groovy.lang.Closure)
Is there an easier way to accomplish what I'm trying to do?
This is my first time trying to write a Shared Library function so I'm guessing it may just be some Groovy syntax/concept I'm not familiar with.
Thanks!
your function signature is def call(String branch, Map envMapping), but your calling is branch: xxx, env_mapping:xxx.
Change to sharedLibrarySetupEnv branch: "$env.GIT_BRANCH", envMapping: EnvMap
The problem was with how I was trying to invoke the Shared Library function. I thought I was able to reference the variable names which led to the Jenkinsfile/pipeline passing a LinkedHashMap to the shared library and not two separate variables.
There are two solutions to this:
Have the Shared Library call method take in a Map<String, Object> parms and within the call reference the variables with parms.varname.
Shared Library:
def call(Map<String, Object> parms) {
echo "${parms.branch}"
Map use_this_map = parms.envMapping.get(branch)
}
Jenkinsfile:
setupEnv branch: "$env.GIT_BRANCH", envMapping: EnvMap
Don't pass the variable names in the Jenkinsfile and have the Shared Library call method take in corresponding variables.
Shared Library:
def call(String branch, Map<String, Map> envMapping) {
echo "${branch}"
Map use_this_map = envMapping.get(branch)
}
Jenkinsfile:
setupEnv $env.GIT_BRANCH, EnvMap
You can use the variable BRANCH_NAME and put it in a condition like below:-
if (env.BRANCH_NAME == master)
{
//set all the environment variable you need
} else {
//variable required if the condition doesn't match
}
You can use REGEX in the condition.
I have a multibranch pipeline in Jenkins.
I defined multiple check boxes (over 20) for each parameter to be passed to a script, which then starts my application and runs corresponding test case (this might not be an optimal solution but this framework was created before I started at current company and I am not going to refactor it):
booleanParam(name: 'cluster_number', defaultValue: false, description: '')
booleanParam(name: 'post_cluster_wu', defaultValue: false, description: '')
etc.
I need to collect user selection for each checkbox (true-false). I would prefer to do it in a loop, like this:
sh """
for (element in params) {
// testing:
echo "${element.key} ${element.value}"
}
"""
but keep getting an error:
[Pipeline] End of Pipeline
groovy.lang.MissingPropertyException: No such property: element for class: groovy.lang.Binding
at groovy.lang.Binding.getVariable(Binding.java:63)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:264)
at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:288)
Also tried to put loop outside of shell script. No luck so far.
steps {
echo "username: ${params.OWNER_USERNAME}"
for (element in params) {
echo "${element.key} ${element.value}"
}
...
Wonder if anyone was able to loop through params?
Thanks in advance!
This works:
pipeline {
agent any
parameters {
booleanParam(name: 'alpha', defaultValue: true)
booleanParam(name: 'beta', defaultValue: true)
booleanParam(name: 'gamma', defaultValue: false)
}
stages {
stage('only') {
steps {
script {
params.keySet().each {
echo "The value of the ${it} parameter is: ${params[it]}"
}
}
}
}
}
}
I have a Jenkinsfile that takes a bunch of params ( 50 aprox.), and other 50 for input processing:
pipeline {
agent { label 'ansible24' }
parameters {
string(name: 'NAME', defaultValue: 'Nightly Valid', description: ' instance name')
// ... x50
}
script {
def filename = "configuration.yml"
def yaml = readYaml file: filename
yaml.global.name = "${params.NAME}".toString()
// ... x50
}
Tomorrow, I will also have a validation for each field.
How could I extract this logic in separated files?
I already saw this: How do you load a groovy file and execute it
but it doesn't help a lot for the case of params and my case is not scripted.
Any idea ?
We've got a set of groovy scripts that our users invoke in their jenkinsfile that sets some common job properties. However, we haven't been able to figure out how to preserve their existing parameters when we do this update.
snippet of our groovy code:
def newParamsList = []
def newbool = booleanParam(defaultValue: false, description: "deploy", name: "deploy_flag")
newParamsList.add(newbool)
def newParams = parameters(newParamsList)
properties([ //job property declaration
jobProperties,
disableConcurrentBuilds(),
newParams,
addSchedule,
])
However, this overwrites the parameter definitions, so if the user had specified a different parameter definition in their jenkins file before invoking our groovy, it's been wiped out.
I can get access to the existing parameters using currentBuild.rawBuild.getAction(ParametersAction), but if I understand correctly, I need the ParameterDefinition not the ParameterValue in order to set the property. I tried currentBuild.rawBuild.getAction(ParametersDefinitionProperty.class) thinking I could use that like ParametersAction, but it returns null.
Is it possible to get the parameter definitions inside the groovy being called from a Jenkinsfile? Or is there a different way that would let us add an additional parameter to the job without wiping out the existing ones currently defined in the jenkinsfile?
So the way we do this, is treat it all like a simple list, then join them together. So jenkinsfile's first get a list from the shared library, before adding their own to the list and then they set the params (not the shared library)
Repos jenkinsfiles do this:
#!groovy
#Library('shared') _
// Call shared libaray for common params
def paramList = jobParams.listParams ([
"var1": "value",
"var2": "value2"
])
// Define repo specific params
def addtionalParams = [
booleanParam(defaultValue: false, name: 'SOMETHING', description: 'description?'),
booleanParam(defaultValue: false, name: 'SOMETHING_ELSE', description: 'description?'),
]
// Set Jenkins job properties, combining both
properties([
buildDiscarder(logRotator(numToKeepStr: '20')),
parameters(paramList + addtionalParams)
])
// Do repo stuff
Our shared library looks like this:
List listParams(def body = [:]) {
//return list of parameters
config = BuildConfig.resolve(body)
// Always common params
def paramsList = [
choice(name: 'ENV', choices: ['dev', 'tst'].join('\n'), description: 'Environment'),
string(name: 'ENV_NO', defaultValue: "1", description: 'Environment number'),
]
// Sometimes common params, switch based on jenkinsfile input
def addtionalParams = []
switch (config.var1) {
case 'something':
case 'something2':
addtionalParams = [
choice(name: 'AWS_REGION', choices: ['us-west-2'].join('\n'), description: 'AWS Region to build/deploy'),
]
break
case 'something3':
addtionalParams = [
string(name: 'DEBUG', defaultValue: '*', description: 'Namespaces for debug logging'),
]
break
}
return paramsList + addtionalParams
}
We did the following groovy code to retrieve the parameters definitions and add new parameters to existing ones (we don't have any knowledge about what the user will put as parameters). If you have something more simple, I take it:
boolean isSupported = true
// nParamsis the List of new parameters to add //
Map initParamsMap = this.initializeParamsMap(nParams)
currentBuild.rawBuild.getParent().getProperties().each { k, v ->
if (v instanceof hudson.model.ParametersDefinitionProperty) {
// get each parameter definition
v.parameterDefinitions.each { ParameterDefinition paramDef ->
String param_symbol_name = null
// get the symbol name from the nested DescriptorImpl class
paramDef.class.getDeclaredClasses().each {
if(it.name.contains('DescriptorImpl')){
param_symbol_name = it.getAnnotation(Symbol).value().first()
}
}
// ... processing... //
if( !initParamsMap.containsKey(paramDef.name) ) {
//Valid parameter types are booleanParam, choice, file, text, password, run, or string.
if (param_symbol_name == 'choice') {
String defaultParamVal = paramDef.defaultParameterValue == null ? null : paramDef.defaultParameterValue.value
tempParams.add(
"$param_symbol_name"(name: paramDef.name,
defaultValue: defaultParamVal,
description: paramDef.description,
choices: paramDef.choices)
)
} else if (param_symbol_name == 'run') {
logError {"buildParametersArray does not support yet already existing RunParameterDefinition " +
"in current job parameters list, so the job parameters will not be modified"}
isSupported = false
} else {
tempParams.add(
"$param_symbol_name"(name: paramDef.name,
defaultValue: paramDef.defaultParameterValue.value,
description: paramDef.description)
)
}
}
}
}
}
if( isSupported) {
properties([parameters(tempParams)])
}
I think you can also do something like this:
// Get existing ParameterDefinitions
existing = currentBuild.rawBuild.parent.properties
.findAll { it.value instanceof hudson.model.ParametersDefinitionProperty }
.collectMany { it.value.parameterDefinitions }
// Create new params and merge them with existing ones
jobParams = [
booleanParam(name: 'boolean_param', defaultValue: false)
/* other params */
] + existing
// Create properties
properties([
parameters(jobParams)
])
Note: But you should either run it in a non-sandboxed environment or use with #NonCPS
There is an example how to add additional string parameter NEW_PARAM into job with name test:
job = Jenkins.instance.getJob("test")
ParametersDefinitionProperty params = job.getProperty(ParametersDefinitionProperty.class);
List<ParameterDefinition> newParams = new ArrayList<>();
newParams.addAll(params.getParameterDefinitions());
newParams.add(new StringParameterDefinition("NEW_PARAM", "default_value"));
job.removeProperty(params);
job.addProperty(new ParametersDefinitionProperty(newParams));