I have a yaml file that contains reference to some variable that define in pipeline.
I need some way to evaluate this $ to real value after I read the yaml.
yaml file
chart_folder: test_chart_${my_suffix}
lint:
enable: false
pipeline look like below
pipeline{
agent{
label "my_node"
}
stages{
stage("test"){
steps{
script {
def my_suffix = "test"
def my_yaml = readYaml file: "my_file.yaml"
echo my_yaml.chart_folder
}
}
}
}
}
The output of execution is
....
[Pipeline] readYaml
[Pipeline] echo
test_chart_${my_suffix}
[Pipeline] }
....
and I want to get the chart_folder as evaluated string
....
[Pipeline] readYaml
[Pipeline] echo
test_chart_test
[Pipeline] }
....
How can I do it?
You can't interpolate, but you can override.
def my_yaml = readYaml file: 'my_file.yaml', text: "chart_folder: 'test_chart_test'"
Related
(edited/updated from original post to attempt to address confusion about what the problem is)
The problem is: Values that are set in a Jenkinsfile environment section are not added to the object returned by env.getEnvironment()
The question is: How do I get a map of the complete environment, including values that were assigned in the environment section? Because env.getEnvironment() doesn't do that.
Example Jenkinsfile:
pipeline {
agent any
environment {
// this is not included in env.getEnvironment()
ONE = '1'
}
stages {
stage('Init') {
steps {
script {
// this is included in env.getEnvironment()
env['TWO'] = '2'
}
}
}
stage('Test') {
steps {
script {
// get env values as a map (for passing to groovy methods)
def envObject = env.getEnvironment()
// see what env.getEnvironment() looks like
// notice ONE is not present in the output, but TWO is
// ONE is set using ONE = '1' in the environment section above
// TWO is set using env['TWO'] = '2' in the Init stage above
println envObject.toString()
// for good measure loop through the env.getEnvironment() map
// and print any value(s) named ONE or TWO
// only TWO: 2 is output
envObject.each { k,v ->
if (k == 'ONE' || k == 'TWO') {
println "${k}: ${v}"
}
}
// now show that both ONE and TWO are indeed in the environment
// by shelling out and using the env linux command
// this outputs ONE=1 and TWO=2
sh 'env | grep -E "ONE|TWO"'
}
}
}
}
}
Output (output of envObject.toString() shortened to ... except relevant part):
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Init)
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] script
[Pipeline] {
[Pipeline] echo
[..., TWO:2]
[Pipeline] echo
TWO: 2
[Pipeline] sh
+ env
+ grep -E ONE|TWO
ONE=1
TWO=2
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Notice ONE is missing from the env.getEnvironment() object, but TWO is present.
Also notice that both ONE and TWO are set in the actual environment and I am not asking how to access the environment or how to iterate through the values returned by env.getEnvironment(). The issue is that env.getEnvironment() does not return all the values in the environment, it excludes any values that were set inside the environment section of the Jenkinsfile.
I don't have a "why" answer for you, but you can cheat and get a map by parsing the output from env via the readProperties step.
def envMap = readProperties(text: sh(script: 'env', returnStdout: true))
println(envMap.getClass())
println("${envMap}")
I would get the env and convert it to map with the help of properties
pipeline {
agent any
environment {
// this is not included in env.getEnvironment()
ONE = '1'
}
stages {
stage('Init') {
steps {
script {
// this is included in env.getEnvironment()
env['TWO'] = '2'
}
}
}
stage('Test') {
steps {
script {
def envProp = readProperties text: sh (script: "env", returnStdout: true).trim()
Map envMapFromProp = envProp as Map
echo "ONE=${envMapFromProp.ONE}\nTWO=${envMapFromProp.TWO}"
// now show that both ONE and TWO are indeed in the environment
// by shelling out and using the env linux command
// this outputs ONE=1 and TWO=2
sh 'env | grep -E "ONE|TWO"'
}
}
}
}
}
Output of env.getEnvironment() method will not return a list or Map, Hence it's difficult to iterate with each but there are some workaround you can do to make this work.
import groovy.json.JsonSlurper
pipeline {
agent any;
environment {
ONE = 1
TWO = 2
}
stages {
stage('debug') {
steps {
script {
def jsonSlurper = new JsonSlurper()
def object = jsonSlurper.parseText(env.getEnvironment().toString())
assert object instanceof Map
object.each { k,v ->
echo "Key: ${k}, Value: ${v}"
}
}
}
}
}
}
Note - env.getEnvironment().toString() will give you a JSON String . While parsing the JOSN string if groovy jsonSlurper.parseText found any special character it will through an error
You can also explore a little bit around env Jenkins API and find an appropriate method that will either return a Map or List so that you can use each
I've inherited some Jenkins pipeline and try to improve it. Jenkins and groovy is quite fresh topic for me, so most probably I'm doing something wrong.
I'm using Jenkins ver. 2.121.3
Main aim was to add build parameter to do some extra cleaning during build. So I've added parameter CLEAN_FIRST with Boolean type and default value false to a job configuration and did something like this in pipeline:
// CLEAN_FIRST = false
// def prefix = CLEAN_FIRST ? "" : "REM"
pipeline {
agent none
stages {
stage('Some step') {
steps {
script {
node('master') {
cleanWs()
try {
def prefix = CLEAN_FIRST ? "" : "REM"
echo "CLEAN_FIRST=$CLEAN_FIRST prefix=$prefix"
bat (label: 'build third party',
script: """
$prefix call cleanSomthing.bat
call doOtherStuff.bat
"""
} finally {
echo "some stuff"
}
} // node
} // script
} // steps
} // stage
} // stages
} // pipeline
Now this doesn't work as expected. "REM" prefix is not added.
Echo prints:
CLEAN_FIRST=false prefix=
And bat invokes cleanSomthing.bat which I wish to avoid (to save on build times).
I've tried to make prefix global, but with same result.
Most probably this is caused by some evaluation order or scoping issue, but I can't put finger on it.
Can someone give me a clue why it doesn't work? How to fix it?
Answered own question. Is this problem fixed on some version of Jenkins?
replace
def prefix = CLEAN_FIRST ? "" : "REM"
with
def prefix = params.CLEAN_FIRST ? "" : "REM"
Ok I've found source of problems. It is a bit funny.
When running this pipeline (tested on Mac machine since it had empty job queue):
pipeline {
agent none
stages {
stage('Some step') {
steps {
script {
node('Mac') {
cleanWs()
try {
def logic = true
def prefix = CLEAN_FIRST ? "Ole" : "REM"
def typeLogic = logic.getClass()
def typeParam = CLEAN_FIRST.getClass()
echo "typeLogic=$typeLogic typeParam=$typeParam"
echo "CLEAN_FIRST=$CLEAN_FIRST prefix=$prefix"
sh (script: """
echo prefix=$prefix
""")
} finally {
echo "some stuff"
}
} // node
} // script
} // steps
} // stage
} // stages
} // pipeline
I've got this outcome:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] stage
[Pipeline] { (Some step)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on master in /Users/builder/jenkins/workspace/EIbuild_MacOS
[Pipeline] {
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] echo
typeLogic=class java.lang.Boolean typeParam=class java.lang.String
[Pipeline] echo
CLEAN_FIRST=false prefix=Ole
[Pipeline] sh
[EIbuild_MacOS] Running shell script
+ echo prefix=Ole
prefix=Ole
[Pipeline] echo
some stuff
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
So now source the problem is obvious.
Jenkins in configuration promises variable of type Boolean, but in fact provides type String with values are "true" or "false" which are always evaluated as true when used as condition since both values are not empty strings :).
I have a shared global function on PublishGitHub.groovy looks like this:
#!/usr/bin/env groovy
def call(body)
{
def config = [:]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = config
echo "\u001B[32mINFO: Publishing...\u001B[m"
body()
echo "\u001B[32mINFO: End Publish...\u001B[m"
}
And a code on my JenkinsFile:
environment {
VERSION = "v1.3.${env.BUILD_NUMBER}"
}
stages {
stage ('Publish WebAPI'){
steps{
echo "\u001B[32mINFO: Start Publish...\u001B[m"
PublishGitHub{
echo "This is a body with version: ${env.VERSION}"
}
}
}
}
And this is my output:
[Pipeline] echo
INFO: Start Publish...
[Pipeline] echo
INFO: Publishing...
[Pipeline] }
And follow next error:
java.lang.NullPointerException: Cannot get property 'VERSION' on null
object
Because inside the body I do not have access to the environment variables?
Your shared library code runs outside of the workflow CPS context, that is why closure you pass to the vars script does not recognize env property. You can fix this problem by passing a reference to the workflow script. If you call your function like this
PublishGitHub(this) {
echo "This is a body with version: ${env.VERSION}"
}
and you apply a small modification to vars/PublishGitHub.groovy script like:
#!/usr/bin/env groovy
def call(config, body) {
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = config
echo "\u001B[32mINFO: Publishing...\u001B[m"
body()
echo "\u001B[32mINFO: End Publish...\u001B[m"
}
then you will run your pipeline successfully:
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Publish WebAPI)
[Pipeline] echo
[32mINFO: Start Publish...[m
[Pipeline] echo
[32mINFO: Publishing...[m
[Pipeline] echo
This is a body with version: v1.3.537
[Pipeline] echo
[32mINFO: End Publish...[m
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
If you want to limit the scope for the shared library, you can always simply pass env instead of this and change vars/PublishGitHub.groovy to something like this:
#!/usr/bin/env groovy
def call(env, body) {
def config = [
env: env
]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = config
echo "\u001B[32mINFO: Publishing...\u001B[m"
body()
echo "\u001B[32mINFO: End Publish...\u001B[m"
}
In this scenario you give your shared library an access to environment variables only.
In order to make the environment variables that you have defined in your Jenkinsfile available in your shared library code you have to pass a this parameter on the call to your shared library method.
For example (below is a partial extract only of a full pipeline file):
// JENKINS-42730
#Library('pipeline-shared-library')_
import org.blah.MySharedLibraryClass
// END JENKINS_42730
pipeline {
agent { any }
environment {
FOO = (new MySharedLibraryClass(config, this)).myMethod("StringVar1", "StringVar2")
}
}
My Shared Library:
package org.blah
import groovy.text.SimpleTemplateEngine
public class MySharedLibraryClass implements Serializable {
def engine = new SimpleTemplateEngine()
def config
def steps
def ArtifactoryHelper(config, steps) {
this.config = config
this.steps = steps
}
def log(msg){
//Allows me to print to Jenkins console
steps.println(msg)
}
def myMethod(var1, var2) {
....
}
The this parameter I referred to above maps to steps in the shared library code. You should then be able to resolve "VERSION=${steps.env.VERSION}" in your shared library code.
Also see this post.
Notes:
pipeline-shared-library is the ID I gave the library in Manage Jenkins > Configure System
Szymon's answer works. I added "p.env = env"
def toParam(final Closure body) {
final def p = [:]
p.env = env
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = p
body()
return p
}
How can I write out environment variable(s) with writeFile in Jenkins pipelines?
It seems such an easy task but I can't find any documentation on how to get it to work.
I tried $VAR, ${VAR} and ${env.VAR}, nothing works...?
In a declarative pipeline (using a scripted block for writeFile) it will look like this:
pipeline {
agent any
environment {
SENTENCE = 'Hello World\n'
}
stages {
stage('Write') {
steps {
script {
writeFile file: 'script.txt', text: env.SENTENCE
}
}
}
stage('Verify') {
steps {
sh 'cat script.txt'
}
}
}
}
Output:
...
[Pipeline] { (Verify)
[Pipeline] sh
[test] Running shell script
+ cat script.txt
Hello World
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
If you want to avoid groovy, this will work too:
writeFile file: 'script.txt', text: "${SENTENCE}"
To combine your env var with text you can do:
...
environment {
SENTENCE = 'Hello World'
}
...
writeFile file: 'script.txt', text: env.SENTENCE + ' is my newest sentence!\n'
Despite following this answer and others, I am unable to successfully use a local groovy file in my Jenkinsfile (both are in the same repository).
def deployer = null
...
...
...
pipeline {
agent {
label 'cf_slave'
}
options {
skipDefaultCheckout()
disableConcurrentBuilds()
}
stages {
stage ("Checkout SCM") {
steps {
checkout scm
}
}
...
...
...
stage ("Publish CF app") {
steps {
script {
STAGE_NAME = "Publish CF app"
deployer = fileLoader.load ('deployer')
withCredentials(...) {
if (BRANCH_NAME == "develop") {
...
...
...
} else {
deployer.generateManifest()
}
}
}
}
}
...
...
}
deployer.groovy:
#!/usr/bin/env groovy
def generateManifest() {
sh "..."
echo "..."
}
In the console log (stack):
[Pipeline] stage
[Pipeline] { (Publish CF app)
[Pipeline] script
[Pipeline] {
[Pipeline] echo
before loading groovy file
[Pipeline] echo
Loading from deployer.groovy
[Pipeline] load
[Pipeline] // load
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
Update:
It seems the problem was not with loading the file but rather with the contents of the file, where I execute the following which apparently does not play well:
sh "node $(pwd)/config/mustacher manifest.template.yml config/environments/common.json config/environments/someFile.json"
echo "..."
When only the echo is there, this is the stack.
So not the sh "node ..." nor the echo work. Even changing it just to sh "pwd" fails as well. What could it be? the syntax in the file? the way it is called in the pipeline?
If I will make the same node call in the pipeline (for example in the withCredentials if statement, it works.
Add a return this to the bottom of the deployer.groovy file, and then change you load step to use relative path and extension to groovy file like load('deployer.groovy').
The return this is documented on jenkins.io:
Takes a filename in the workspace and runs it as Groovy source text.
The loaded file can contain statements at top level or just load and run a closure. For example:
def pipeline
node('slave') {
pipeline = load 'pipeline.groovy'
pipeline.functionA()
}
pipeline.functionB()
pipeline.groovy
def pipelineMethod() {
...code
}
return this
Where pipeline.groovy defines functionA and functionB functions (among others) before ending with return this