Modify a YAML playbook in a Jenkins pipeline - jenkins

I have a delivery.yaml used by Ansible to deploy data :
---
- name: Deploy to Filer on host
hosts: host
tasks:
- name: Copy files
copy: src={{ item.src }} dest={{ item.dest }} mode=0775
with_items:
- { src: 'data1.xml', dest: '' }
- { src: 'data2.so', dest: '' }
- { src: 'data3.exe', dest: '' }
- { src: 'data4.lib', dest: '' }
I need to complete the "dest" value according to the data extension :
xml files => target1
so files => target2
exe files => target3
lib files => target4
How to write it ? I'm not used with groovy language. Currently, I write this but it does not work :
stage('YAML test'){
steps{
script{
yamlData = readYaml file: 'delivery.yaml'
yamlData.tasks.with_items.find{it.src.endsWith("xml")}.dest="target1"
writeYaml file: 'delivery_temp.yaml', data: yamlData
sh "cat delivery_temp.yaml"
}
}
}
I got this error :
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.util.ArrayList.endsWith() is applicable for argument types: (java.lang.String) values: [xml]

The problem here is, that your outermost data type is already a list.
So Groovy will use the implicit spread operator. What is implied here
is:
yamlData*.tasks*.with_items.find{ it*.src.endWith ... }
So you actually want to do the transformation for each element on the
outer most level:
yamlData.each{ it.tasks.each { task -> task.with_items.find{it.src.endsWith("xml")}.dest="target1" } }
Whether the find here is correct depends on the data (this will only
find the first one and it will fail if there is none). Most likely you
want findAll instead (which results in a list of all results and the
.dest assignment would still work due to the implicit spread operator).
And to get rid of the copy and paste code for
doing this for all four types (or how many there will be), you are
better off extracting the transformations out as data (e.g. a map from
suffix to target)

Related

How do I use sh in Jenkins Global library

I am creating my own global library for Jenkins, which I have hosted on github, and to simplify some run-of-the-mill tasks, I wanted to add a function that returns the GIT tag.
Therefore I created something like this:
class Myclass{
static String getGitTag() {
return "${sh(returnStdout: true, script: 'git tag --sort version:refname | tail -1').trim()}"
}
}
... which results in this error:
No signature of method: static com.stevnsvig.jenkins.release.ReleaseUtil.sh()
So I'm left with two questions:
Is the solution to import the sh() library that Jenkins' groovy flavor obviously already has imported? (and if so how)
What is the best practice here? I am wondering why there isn't a GIT_TAG global variable when you use declarative pipelines, and something like this should (in my opinion) be easy as pie.
EDIT #1:
static String getGitTag() {
stdout = script.sh(script: "git tag --sort version:refname | tail -1", returnStdout: true)
return stdout.trim()
}
produces a similar error:
No signature of method: static com.stevnsvig.jenkins.release.ReleaseUtil.sh() is applicable for argument types: (java.util.LinkedHashMap) values: [[returnStdout:true, script:git tag --sort version:refname | tail -1]]
EDIT #2:
static String getGitTag() {
def stdout = "git tag --sort version:refname | tail -1".execute()
return stdout.in.text
}
completes, but the output is blank. Running the same command with pwd returns / which indicaes that the environment is not set, which makes sense, since all the commands running under Jenkins are designed to rununder pipelines
EDIT #3:
I went hunting for the import. Stumbled across the Jenkins CI project on github and started searching the many repositories. Found a promising one... and put a file called pwd.groovy in /vars with this content:
import org.jenkinsci.plugins.workflow.steps.durable_task.ShellStep
static String getPWD() {
def ret = ShellStep.sh(returnStdout: true, script: "git tag --sort version:refname | tail -1").trim()
echo "currently in ${ret}"
}
The error I got is a variation of the same. I guess since itsa plugin, the definition is different...
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: static org.jenkinsci.plugins.workflow.steps.durable_task.ShellStep.sh() is applicable ...
Option 1) Use Groovy execute to run cmd and get its output as below
tag = "git tag --sort version:refname | tail -1".execute().text
Option 2) Use Jenkins pipeline step sh.
One concept need to get clear: the context of sh is global function is when sh used directly inside Jenkinsfile.
In your case, sh is used outside the Jenkinsfile. To make better understand I give an example Jenkinsfile.
pipeline {
stages('foo') {
steps {
sh 'pwd'
// In above sh step, there is an implicit `this` which represents the
// global object for Jenkinsfile, you can image sh 'pwd' to this.sh 'pwd'
//
// Thus if you want to use `sh` outside Jenkinsfile, you must pass down the
// implicit `this` into the file where you used `sh`
}
}
}
To address your issue
// ReleaseUtil.groovy
static String getGitTag(steps) {
// here `steps` is the global object for Jenkinsfile
// you can use other pipeline step here by `steps`
steps.echo 'test use pipeline echo outside Jenkinsfile'
steps.withCredentials([steps.string(credentialsId: 'git_hub_auth', variable: 'GIT_AUTH_TOKEN')]) {
steps.echo '....'
steps.sh '....'
}
return steps.sh(returnStdout: true, script:"git tag --sort version:refname | tail -1").trim()
}
// Jenkinsfile
import com.stevnsvig.jenkins.release.ReleaseUtil
pipeline {
stages('foo') {
steps {
ReleaseUtil.getGitTag(this)
}
}
}

Each loop failing in Groovy DSL Pipeline for Jenkins Jobs Builder

I'm trying to create dynamic Jenkins job pipeline stages based on an array of values but I can't seem to get the loop functioning as expected, it complains about the syntax I'm using but I can't figure it out, is this a Groovy issue?
Approach
uat_nodes:
- 'node1'
- 'node2'
dsl: |
stage('Update UAT dist') {{
build job: '{key}-{module}-DP-BuildNamedDist-UAT'
}}
def UAT_NODES = {uat_nodes}
UAT_NODES.each { UAT_NODE ->
stage('Deploy code to UAT node: ' . ${{UAT_NODE}}) {{
build job: '{key}-{module}-DP-UAT-Nodes', parameters: [
string(name: 'LIMIT', value: '${{UAT_NODE}}'),
string(name: 'PLAYBOOK', value: '{playbook}')
]
}}
}
Error
WorkflowScript: 8: Ambiguous expression could be either a parameterless closure expression or an isolated open code block;
solution: Add an explicit closure parameter list, e.g. {it -> ...}, or force it to be treated as an open block by giving it a label, e.g. L:{...} # line 8, column 56.
e to UAT node: ' . ${{UAT_NODE}}) {{
As the error states, there is a problem with this piece of code: . ${{UAT_NODE}}
If strings would have a $ method, that would call it with a closure inside a closure, that returns UAT_NODE.
I can only assume, that you want to concat strings here similar to perl or php. This is not how it works in groovy.
Use: "Deploy code to UAT node: ${UAT_NODE}". Note the double quotes "! Single quotes ' won't give you replacement (this is every other string you are using in your code).

Rollup JS build using rollup-plugin-modify plugin not replacing all string instances in the build files (using 'replace-in-file' plugin instead)

I wanted to tackle file cache updating in a Svelte app and wanted this part of the rollup build. I decide to add a querystring parameter to the file references (such as 'index.html?v=0.1') in the distributed build scripts. I created a constant '__cVersion__' in my rollup.config.js script and tried to use the 'rollup-plugin-modify', but that only updated my main.js and App.svelte code (the files being compiled). I also tried the '#rollup/plugin-replace' plugin with the same results. I needed the files I was also copying (not building) from src to public to also to have instances of '__cVersion__' replaced in the scripts.
The following was my initial rollup.config.js export function (the string replacement that did not work):
export default {
input: 'src/main.js',
output: {
sourcemap: true,
format: 'iife',
name: 'app',
file: 'public/build/bundle.js'
},
plugins: [
// this only seems to work on the main.js and .svelte files
modify({
'__cVersion__': 'c0.1.19'
}),
svelte({
dev: !production,
css: css => {
css.write('public/build/bundle.css');
}
}),
copy({
targets: [{
src: 'src/bs4.4.1.css',
dest: 'public/'
},
{
src: 'src/sw.js',
dest: 'public/'
},
{
src: 'src/index.html',
dest: 'public/'
},
{
src: 'src/manifest.json',
dest: 'public/'
},
{
src: 'src/images/*',
dest: 'public/images/'
}
]
}),
resolve({
browser: true,
dedupe: ['svelte']
}),
commonjs(),
// cache files
workbox({
mode: 'injectManifest',
options: {
swSrc: 'src/sw.js',
swDest: 'public/sw.js',
globDirectory: 'public',
globPatterns: [
'**/*.{html,json,js,css,png,map}',
'./manifest.json',
'./images/**',
'./bs4.4.1.css',
'./index.html'
]
}
}),
!production && serve(),
!production && livereload('public'),
production && terser()
],
watch: {
clearScreen: false
}
};
By default, the rollup-plugin-copy plugin will trigger on rollup's buildEnd hook.
Setting the hook to writeBundle fixed this issue for me, like so:
copy({
targets: [
{
src: ...,
dest: ...
},
...
],
hook: "writeBundle",
}),
I decided to take a different approach since I realized the build process needed to complete before I tried to replace '__cVersion__' in the files. After some trial and error I settled on this code:
https://github.com/kuhlaid/svelte2/releases/tag/v0.1.7
If you search the source code for '__cVersion__' you will see where I am adding the file revision string to try and force a file cache update...however, that didn't fully fix the issue.
I then looked at the service worker (sw.js) and realized the Workbox 'injectManifest' was actually handling the file revisions. The only problem with my current setup was that I had added '__cVersion__' constants to my scripts, but Workbox never saw the replacements since Workbox processed the service worker before I replaced the constants.
What I probably need to do is copy the src files to a 'staging directory' where I can replace the 'cache' constants in the scripts and then run the build rollup off of the staging files. This 'should' cause Workbox to treat the files as updated and thus assign them different revision numbers in the service worker file. I will try and update this thread when I have that issue worked out.

Jenkins Pipeline - inserting variable in shell creates a new line

I am using Choice param in my jenkins file to select environment as follows:
pipeline {
agent any
parameters {
choice(
name: 'ENVIRONMENT_URL',
choices: "https://beta1.xyz.com\nhttps://beta2.xyz.com\nhttps://beta3.xyz.com",
description: 'interesting stuff' )
}
in the Stage section, i have the following piece
stage('execute tests') {
steps {
script {
sh """URL=${ENVIRONMENT_URL} npm run e2e:tests"""
sh 'echo -e "\nTest Run Completed.\n"'
}
}
}
However, when i run the pipeline job by selecting the choice parameter i added, the following gets executed (the inserted choice param creates a line break):
+ URL=https://beta1.xyz.com
+ npm run e2e:tests
Using the variable is causing a line break and that's what is causing issue.
I have tried different ways to avoid line break. Tried using a variable but that didn't help. tried with different quotations, that didn't either.
What can i do to avoid line break ?
You can use the trim method on a String class type to remove trailing whitespace and newline:
sh "URL=${params.ENVIRONMENT_URL.trim()} npm run e2e:tests"
Note I also specified your parameter is in the params map and removed the triple quotes as those are for multiline string formatting.
Alternatively, you can specify the choices as an array instead of as a multiline string. The choices argument would then appear like:
choice(
name: 'ENVIRONMENT_URL',
choices: ['https://beta1.xyz.com', 'https://beta2.xyz.com', 'https://beta3.xyz.com'],
description: 'interesting stuff'
)
Either solution would fix your issue.

Jenkins Shared Library resource not found

I have just started looking into a shared libarary with jenkins in order to combine a load of scripts and pipelines across multiple repos that are pretty much identical.
I have the shared lib loaded and working but when tryign to execute the scripts i the resources folder i keep geting not found errors:
../releaseTagging-EO2DMYOPJ6JGB6JT5Q2RSFJWJWWPALA7F25H7CQNYBEV4ITTEB6Q#tmp/build.sh: not found
I am creating a copy of the file using the following:
createTempLocation(String path) {
String tmpDir = pwd tmp: true
return tmpDir + File.separator + new File(path).getName()
}
and
copyGlobalLibraryScriptcall(String srcPath, String destPath = null) {
destPath = destPath ?: createTempLocation(srcPath)
writeFile file: destPath, text: libraryResource(srcPath)
echo "copyGlobalLibraryScript: copied ${srcPath} to ${destPath}"
sh "chmod +x ${destPath}"
echo "added executable permissions to ${destPath}"
return destPath
}
I am then calling the last function thusly:
runBuild(Map config) {
def script = copyGlobalLibraryScript('build.sh')
sh script
}
(i realise i can collapse the above function in to one line)
This in turn then gets called via (trimed the whole file to relevent part):
pipeline {
agent any
stages {
stage('Build') {
steps {
timestamps {
checkout scm
bbNotify( key: buildKey, name: BuildName) {
runBuild()
}
stash includes: '**', name: 'RelToSTAN'
}
}
}
}
This all fails with the error at the top of the question, however when sshing on to the build server i can find that file int he location specified.
I dont understand why Jenkins cannot find it and execute it.
The issue will be the following:
When using a java File object it‘ll always refer to some location on the Jenkins master. And of course it usually cannot run inside the sandbox.
On the other hand the readFile and writeFile methods always refer to some path on the build agent reserved by the node block where the call is encapsulated.
Long story short: Do not use the File class. Unfortunately you’ll need to create the temp path manually. But that shouldn’t be too hard.

Resources