In my Jenkins pipeline, I've got a yaml file that I need to apply to multiple environments, and separate environment specific yaml files that I'd like to inject or merge into the default file and write as a new file.
I've looked at readYaml and writeYaml here: https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/ But I'm not finding a good way of merging multiple files.
A simple example of what I'd like to achieve is here:
# config.yaml
config:
num_instances: 3
instance_size: large
# dev-overrides.yaml
config:
instance_size: small
# dev-config.yaml (desired output after merging dev-overrides.yaml in config.yaml)
config
num_instances: 3
instance_size: small
The Jenkins implementation of readYaml uses SnakeYAML as processor and supports YAML 1.1. You could possibly use the merge operator to accomplish your goal. But the merge operator has been removed in YAML 1.2. Thus I would not advise using this feature even it's currently available.
I would instead merge the objects with some Groovy code like this:
Map merge(Map... maps) {
Map result = [:]
maps.each { map ->
map.each { k, v ->
result[k] = result[k] instanceof Map ? merge(result[k], v) : v
}
}
result
}
def config = readYaml text: """
config:
num_instances: 3
instance_size: large
"""
def configOverrides = readYaml text: """
config:
instance_size: small
"""
// Showcasing what the above code does:
println "merge(config, configOverrides): " + merge(config, configOverrides)
// => [config:[num_instances:3, instance_size:small]]
println "merge(configOverrides, config): " + merge(configOverrides, config)
// => [config:[instance_size:large, num_instances:3]]
// Write to file
writeYaml file: 'dev-config.yaml', data: merge(config, configOverrides)
Inspired by https://stackoverflow.com/a/27476077/1549149
Related
I was wondering what is the best approach to reading in the yaml config and setting environment variables.
For example, my yaml config looks like this:
amps-ml:
models:
- name: app-sample
type: sagemaker
inference:
image: project_id.dkr.ecr.us-west-2.amazonaws.com/template-model-bert:test_1
data: s3://my_project/sagemaker/huggingface-pytorch-inference-recommender/sentiment-analysis/model/model.tar.gz
endpoint: amp-app-endpoint-test
model_name: sample-model
endpoint_config_name: amp-app-config
model_package_group_name: sample-package-group
endpoint_instance_count: 1,
endpoint_instance_type: ml.m5.large
I essentially want to set environment variables in my Jenkins pipeline for all the variables under inference.
Try
def yaml = readYAML file: "your-file.yaml"
yaml["amps-ml"]["models"][0]["inference"].each {name, value ->
env["$name"] = value
}
You can also iterate the models instead of using explicit index (0)
I'm trying to generate Jenkins pipelines using the pipelineJob function in the jobDSL pluging, but cannot pass parameters from the DSL to the pipeline script. I have several projects that use what is essentially the same Jenkinsfile, with differences only in a few steps. I'm trying to use the JobDSL plugin to generate these pipelines on the fly, with the values I want changed in them interpreted to match the parameters to the DSL.
I've tried just about every combination of string interpretation that I can in the pipeline script, as well as in the DSL, but cannot get Jenkins/groovy to interpret variables in the pipeline script.
I'm calling the job DSL in a pipeline step:
def projectName = "myProject"
def envs = ['DEV','QA','UAT']
def repositoryURL = 'myrepo.com'
jobDsl targets: ['jobs/*.groovy'].join('\n'),
additionalParameters: [
project: projectName,
environments: envs,
repository: repositoryURL
],
removedJobAction: 'DELETE',
removedViewAction: 'DELETE'
The DSL is as follows:
pipelineJob("${project} pipeline") {
displayName('Pipeline')
definition {
cps {
script(readFileFromWorkspace(pipeline.groovy))
}
}
}
pipeline.groovy:
pipeline {
agent any
environment {
REPO = repository
}
parameters {
choice name: "ENVIRONMENT", choices: environments
}
stages {
stage('Deploy') {
steps {
echo "Deploying ${env.REPO} to ${params.ENVIRONMENT}..."
}
}
}
}
The variables that I pass in additionalParameters are interpreted in the jobDSL script; a pipeline with the correct name does get generated. The problem is that the variables are not passed to the pipeline script read from the workspace - the Jenkins configuration for the generated pipeline looks exactly the same as the file, without any interpretation on the variables.
I've made a number of attempts at getting the string to interpret, including a lot of variations of "${environments}", ${environments}, $environments, \$environments...I can't find any that work. I've also tried reading the file as a gstringImpl:
script("${readFileFromWorkspace(pipeline.groovy)}")
Does anyone have any ideas as to how I can make variables propagate down to the pipeline script? I know that I could just use a for loop to do string.replaceAll() on the script text, but that seems cumbersome; there's got to be a better way.
I've come up with a way to make this work. It's not what I'd prefer, which is having the string contents of the file implicitly interpreted during job creation, but it does work; it just adds an extra step.
import groovy.text.SimpleTemplateEngine
def fileContents = readFileFromWorkspace "pipeline.groovy"
def engine = new SimpleTemplateEngine()
template = engine.createTemplate(fileContents).make(binding.getVariables()).toString()
pipelineJob("${project} pipeline") {
displayName('Pipeline')
definition {
cps {
script(template)
}
}
}
This reads a file from your workspace, then uses it as a template with the binding variables. The other changes needed to make this work are escaping any variables used in your Jenkinsfile script, like \${VARIABLE} so that they are expanded at runtime, not at the time you build the job. Any variables you want to be expanded at job creation should be referenced as ${VARIABLE}.
You could achieve what you're trying to do by defining environment variables in the pipelineJob and then using those variables in your pipeline.
They are a bit limited because environment variables are strings, but it should work for basic stuff
Ex.:
//job-dsl
pipelineJob('example') {
environmentVariables {
// these vars could be specified by parameters of this job
env('repository', 'blah')
env('environments', "a,b,c"]) //comma separated string
}
displayName('Pipeline')
definition {
cps {
script(readFileFromWorkspace(pipeline.groovy))
}
}
}
}
And then in the pipeline:
//pipeline.groovy
pipeline {
agent any
environment {
REPO = env.repository
}
parameters {
choice name: "ENVIRONMENT", choices: env.environments.split(',')
//note the need to split the comma separated string above
}
}
You need to use the complete job name as a variable without the quotes. E.g., if JOBNAME is a parameter containing the entire job name:
pipelineJob(JOBNAME) {
displayName('Pipeline')
definition {
cps {
script(readFileFromWorkspace(pipeline.groovy))
}
}
}
I am retrieving JSON object from a URL using httpRequest in a groovy script.
pipeline {
agent any
stages {
stage ('Extract Data') {
steps {
script {
def response = httpRequest \
authentication: 'user', \
httpMode: 'GET', \
url: "https://example.com/data"
writeFile file: 'output.json', text: response.content
def data = readFile(file: 'output.json')
def details = new groovy.json.JsonSlurperClassic().parseText(data)
echo "Data: ${details.fields.customfield}"
}
}
}
}
}
I am interested in the customfieldstring. The format of the string is:
Application!01.01.01 TestSuite1,TestSuite2,TestSuite3,TestSuite4 Product!01.01.01,Product2!01.01.02
I would like to parse the string into 3 data sets:
Map of Applications [Application: version] (there will always be one Appliction)
List of TestSuites [TestSuite1,...,TestSuite]
Map of Prodcts [Product1: version,..., ProductN: version].
However, I am not sure how to do this.
Are there any Jenkins Groovy libraries that I can use to do this in a declarative pipeline?
EDIT
Based on the answer below I can see that I can make a map in the following way:
def applications = groups[0].split(',').collect { it.split('!') }.collectEntries { [(it):it] }
In the example I have:
application = [Application: Application]
How do I get:
application = [Application: 01.01.01]
EDIT2
Note the following output:
def applications = groups[0].split(',').collect { it.split('!') }
[[Application, 01.01.01]]
There're no libraries I'm aware of that will have functionality to parse the data but, since you know the format of the data it's easy to parse them manually.
There are 3 groups in the input (applications, suites, products) separated by a character. To get the groups you need:
def input = "Application!01.01.01 TestSuite1,TestSuite2,TestSuite3,TestSuite4 Product!01.01.01,Product2!01.01.02"
def groups = input.split(' ')
To process the applications you need to split group 0 with , character (just in case there are many applications). You got a list of pairs in format: name!version. Every pair must be splitted with !, so you get a list of lists in format: [[name, version]]. From the last structure it's easy to create a map. All steps together:
def applications = groups[0].split(',').collect { it.split('!') }.collectEntries { [(it[0]):it[1]] }
Getting the list of suites is easy, just split group 1 with , character:
def suites = groups[1].split(',')
Finally, products are analogical to the list of applications but this time group 2 should be used:
def products = groups[2].split(',').collect { it.split('!') }.collectEntries { [(it[0]):it[1]] }
You can simplifier your issue by using pipeline utility step: readJSON
def data = readJSON(file: 'output.json')
echo data.fields.customfield
I found a method. Groovy can convert the values of an Object array and convert them into a map with the toSpreadMap(). However, the array must have an even number of elements.
def appList = ['DevOpsApplication', '01.01.01']
def appMap = appList.toSpreadMap()
For some better answers please refer to this
The bazel build flag --workspace_status_command supports calling a script to retrieve e.g. repository metadata, this is also known as build stamping and available in rules like java_binary.
I'd like to create a custom rule using this metadata.
I want to use this for a common support function. It should receive the git version and some other attributes and create a version.go output file usable as a dependency.
So I started a journey looking at rules in various bazel repositories.
Rules like rules_docker support stamping with stamp in container_image and let you reference the status output in attributes.
rules_go supports it in the x_defs attribute of go_binary.
This would be ideal for my purpose and I dug in...
It looks like I can get what I want with ctx.actions.expand_template using the entries in ctx.info_file or ctx.version_file as a dictionary for substitutions. But I didn't figure out how to get a dictionary of those files. And those two files seem to be "unofficial", they are not part of the ctx documentation.
Building on what I found out already: How do I get a dict based on the status command output?
If that's not possible, what is the shortest/simplest way to access workspace_status_command output from custom rules?
I've been exactly where you are and I ended up following the path you've started exploring. I generate a JSON description that also includes information collected from git to package with the result and I ended up doing something like this:
def _build_mft_impl(ctx):
args = ctx.actions.args()
args.add('-f')
args.add(ctx.info_file)
args.add('-i')
args.add(ctx.files.src)
args.add('-o')
args.add(ctx.outputs.out)
ctx.actions.run(
outputs = [ctx.outputs.out],
inputs = ctx.files.src + [ctx.info_file],
arguments = [args],
progress_message = "Generating manifest: " + ctx.label.name,
executable = ctx.executable._expand_template,
)
def _get_mft_outputs(src):
return {"out": src.name[:-len(".tmpl")]}
build_manifest = rule(
implementation = _build_mft_impl,
attrs = {
"src": attr.label(mandatory=True,
allow_single_file=[".json.tmpl", ".json_tmpl"]),
"_expand_template": attr.label(default=Label("//:expand_template"),
executable=True,
cfg="host"),
},
outputs = _get_mft_outputs,
)
//:expand_template is a label in my case pointing to a py_binary performing the transformation itself. I'd be happy to learn about a better (more native, fewer hops) way of doing this, but (for now) I went with: it works. Few comments on the approach and your concerns:
AFAIK you cannot read in (the file and perform operations in Skylark) itself...
...speaking of which, it's probably not a bad thing to keep the transformation (tool) and build description (bazel) separate anyways.
It could be debated what constitutes the official documentation, but ctx.info_file may not appear in the reference manual, it is documented in the source tree. :) Which is case for other areas as well (and I hope that is not because those interfaces are considered not committed too yet).
For sake of comleteness in src/main/java/com/google/devtools/build/lib/skylarkbuildapi/SkylarkRuleContextApi.java there is:
#SkylarkCallable(
name = "info_file",
structField = true,
documented = false,
doc =
"Returns the file that is used to hold the non-volatile workspace status for the "
+ "current build request."
)
public FileApi getStableWorkspaceStatus() throws InterruptedException, EvalException;
EDIT: few extra details as asked in the comment.
In my workspace_status.sh I would have for instance the following line:
echo STABLE_GIT_REF $(git log -1 --pretty=format:%H)
In my .json.tmpl file I would then have:
"ref": "${STABLE_GIT_REF}",
I've opted for shell like notation of text to be replaced, since it's intuitive for many users as well as easy to match.
As for the replacement, relevant (CLI kept out of this) portion of the actual code would be:
def get_map(val_file):
"""
Return dictionary of key/value pairs from ``val_file`.
"""
value_map = {}
for line in val_file:
(key, value) = line.split(' ', 1)
value_map.update(((key, value.rstrip('\n')),))
return value_map
def expand_template(val_file, in_file, out_file):
"""
Read each line from ``in_file`` and write it to ``out_file`` replacing all
${KEY} references with values from ``val_file``.
"""
def _substitue_variable(mobj):
return value_map[mobj.group('var')]
re_pat = re.compile(r'\${(?P<var>[^} ]+)}')
value_map = get_map(val_file)
for line in in_file:
out_file.write(re_pat.subn(_substitue_variable, line)[0])
EDIT2: This is how the Python script is how I expose the python script to rest of bazel.
py_binary(
name = "expand_template",
main = "expand_template.py",
srcs = ["expand_template.py"],
visibility = ["//visibility:public"],
)
Building on Ondrej's answer, I now use somthing like this (adapted in SO editor, might contain small errors):
tools/bazel.rc:
build --workspace_status_command=tools/workspace_status.sh
tools/workspace_status.sh:
echo STABLE_GIT_REV $(git rev-parse HEAD)
version.bzl:
_VERSION_TEMPLATE_SH = """
set -e -u -o pipefail
while read line; do
export "${line% *}"="${line#* }"
done <"$INFILE" \
&& cat <<EOF >"$OUTFILE"
{ "ref": "${STABLE_GIT_REF}"
, "service": "${SERVICE_NAME}"
}
EOF
"""
def _commit_info_impl(ctx):
ctx.actions.run_shell(
outputs = [ctx.outputs.outfile],
inputs = [ctx.info_file],
progress_message = "Generating version file: " + ctx.label.name,
command = _VERSION_TEMPLATE_SH,
env = {
'INFILE': ctx.info_file.path,
'OUTFILE': ctx.outputs.version_go.path,
'SERVICE_NAME': ctx.attr.service,
},
)
commit_info = rule(
implementation = _commit_info_impl,
attrs = {
'service': attr.string(
mandatory = True,
doc = 'name of versioned service',
),
},
outputs = {
'outfile': 'manifest.json',
},
)
In my jenkins pipeline I am working with properties stored in file.
I can read properties from file and add new items to the map using this code, but I do not understand how to persist my changes.
node('hozuki-best-girl') {
def propertiesPath = "${env.hozuki_properties}"
def props = readProperties file: propertiesPath
props['versionCode'] = 100500
}
What should I do in order to persist my changes? There is no writeProperties method here https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#code-readproperties-code-read-properties-from-files-in-the-workspace-or-text
you can use yaml format instead of properties.
it also simple and human readable and in jenkins-pipeline there are read and write operations for yaml
or you can use this kind of code:
#NonCPS
def propsToString(Map map){
return new StringWriter().with{w-> (map as Properties).store(w, null); w; }.toString()
}
writeFile file: propertiesPath, text: propsToString(props)
The Phoenix AutoTest Plugin has a writeProperties step.