I am using Jenkins to trigger an Ansible playbook. I would like to pass in an array of computer names in order for the Ansible playbook to trigger the playbook and image the machines one by one.
In order to do this I believe I need a foreach loop.
I have little skill in groovy/Jenkins and have ran into an issue.
error = "Expected a symbol # line..." The line that this is referring to is HOSTS.each {item ->
Can someone please assist me? My script is below (I have edited out some private data)
pipeline {
agent any
// every day
triggers {
cron('H 7 * * *')
}
environment {
HOSTS = ['node1','node2']
}
stages {
stage('MachineDB Scheduler') {
steps {
HOSTS.each { item -> // review
HOSTNAME = ${item} // review
ansibuildPlaybookperf(
sshUser: env.USER,
vaultUser: env.USER,
server: "$SERVER",
dir: "$HOMEDIR/$BUILD_TAG",
playbook: "$PLAYBOOK",
extras: "--vault-password-file passmgr.sh",
extraVars: "$VARS_JENKINS"
)
}
}
}
}
}
I don't really know a lot about ansible but maybe this helps.
This pipeline shows a list of PC's. In the declarative pipeline I call a groovy function which is defined after the pipeline. In this function I go through the list and past every PC-name.
def list = [
'PCNAME1',
'PCNAME2',
'PCNAME3'
]
pipeline {
agent any
stages {
stage('Loop through PCs') {
steps {
loopPC(list)
}
}
}
}
def loopPC(list){
list.each {
println "Computer ${it}"
}
}
OUTPUT:
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Loop through PCs)
[Pipeline] echo
Computer PCNAME1
[Pipeline] echo
Computer PCNAME2
[Pipeline] echo
Computer PCNAME3
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
You can also use a script block in your declarative pipeline to execute the script immediately. It's less clean but maybe easier in the beginning and to make it work (and closer to your attempt):
def list = [
'PCNAME1',
'PCNAME2',
'PCNAME3'
]
pipeline {
agent any
stages {
stage('Loop through PCs') {
steps {
script {
list.each {
println "Computer ${it}"
}
}
}
}
}
}
Related
(edited/updated from original post to attempt to address confusion about what the problem is)
The problem is: Values that are set in a Jenkinsfile environment section are not added to the object returned by env.getEnvironment()
The question is: How do I get a map of the complete environment, including values that were assigned in the environment section? Because env.getEnvironment() doesn't do that.
Example Jenkinsfile:
pipeline {
agent any
environment {
// this is not included in env.getEnvironment()
ONE = '1'
}
stages {
stage('Init') {
steps {
script {
// this is included in env.getEnvironment()
env['TWO'] = '2'
}
}
}
stage('Test') {
steps {
script {
// get env values as a map (for passing to groovy methods)
def envObject = env.getEnvironment()
// see what env.getEnvironment() looks like
// notice ONE is not present in the output, but TWO is
// ONE is set using ONE = '1' in the environment section above
// TWO is set using env['TWO'] = '2' in the Init stage above
println envObject.toString()
// for good measure loop through the env.getEnvironment() map
// and print any value(s) named ONE or TWO
// only TWO: 2 is output
envObject.each { k,v ->
if (k == 'ONE' || k == 'TWO') {
println "${k}: ${v}"
}
}
// now show that both ONE and TWO are indeed in the environment
// by shelling out and using the env linux command
// this outputs ONE=1 and TWO=2
sh 'env | grep -E "ONE|TWO"'
}
}
}
}
}
Output (output of envObject.toString() shortened to ... except relevant part):
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Init)
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] script
[Pipeline] {
[Pipeline] echo
[..., TWO:2]
[Pipeline] echo
TWO: 2
[Pipeline] sh
+ env
+ grep -E ONE|TWO
ONE=1
TWO=2
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Notice ONE is missing from the env.getEnvironment() object, but TWO is present.
Also notice that both ONE and TWO are set in the actual environment and I am not asking how to access the environment or how to iterate through the values returned by env.getEnvironment(). The issue is that env.getEnvironment() does not return all the values in the environment, it excludes any values that were set inside the environment section of the Jenkinsfile.
I don't have a "why" answer for you, but you can cheat and get a map by parsing the output from env via the readProperties step.
def envMap = readProperties(text: sh(script: 'env', returnStdout: true))
println(envMap.getClass())
println("${envMap}")
I would get the env and convert it to map with the help of properties
pipeline {
agent any
environment {
// this is not included in env.getEnvironment()
ONE = '1'
}
stages {
stage('Init') {
steps {
script {
// this is included in env.getEnvironment()
env['TWO'] = '2'
}
}
}
stage('Test') {
steps {
script {
def envProp = readProperties text: sh (script: "env", returnStdout: true).trim()
Map envMapFromProp = envProp as Map
echo "ONE=${envMapFromProp.ONE}\nTWO=${envMapFromProp.TWO}"
// now show that both ONE and TWO are indeed in the environment
// by shelling out and using the env linux command
// this outputs ONE=1 and TWO=2
sh 'env | grep -E "ONE|TWO"'
}
}
}
}
}
Output of env.getEnvironment() method will not return a list or Map, Hence it's difficult to iterate with each but there are some workaround you can do to make this work.
import groovy.json.JsonSlurper
pipeline {
agent any;
environment {
ONE = 1
TWO = 2
}
stages {
stage('debug') {
steps {
script {
def jsonSlurper = new JsonSlurper()
def object = jsonSlurper.parseText(env.getEnvironment().toString())
assert object instanceof Map
object.each { k,v ->
echo "Key: ${k}, Value: ${v}"
}
}
}
}
}
}
Note - env.getEnvironment().toString() will give you a JSON String . While parsing the JOSN string if groovy jsonSlurper.parseText found any special character it will through an error
You can also explore a little bit around env Jenkins API and find an appropriate method that will either return a Map or List so that you can use each
How to access parameters values inside different stages in Jenkins pipeline.
So far I have done:
env_vars = 'Initial value'
pipeline {
agent {label 'master'}
stages {
stage('Inject-Env-Vars') {
steps {
script {
env_vars = params.collect{string(name: it.key, value: it.value)} //Collecting all the params here
}
}
}
stage('add_env_variable') {
steps {
script {
env_vars.add(string(name: 'Change_Reason', value: "sam")) ////Adding an extra param
}
}
}
stage('Parent') {
parallel {
stage('RCP') {
steps {
echo "$env_vars"
echo "${env_vars.Change_Reason}" //// want to print value of Change_Reason
}
}
}
}
}
}
Output:
[Pipeline] { (RCP)
[Pipeline] echo
[#string(name=Change_Reason,value=sam)]
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch RCP
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such field found: field org.jenkinsci.plugins.structs.describable.UninstantiatedDescribable Change_Reason
Please let me know if there is a way to get the param value in different stages.
env_vars seems to be an Array and you expect to access it as a Map.
My project has many common variables for many other projects, so I use Jenkins Shared Library and created a vars/my_vars.groovy file where I defined my variables and return Map of them:
class my_vars {
static Map varMap = [:]
static def loadVars (Map config) {
varMap.var1 = "val1"
varMap.var2 = "val2"
// Many more variables ...
return varMap
}
}
I load the Shared Library in my Jenkinsfile, and call the function in the environment bullet, as I want those variables to be as environment variables .
Jenkinsfile:
pipeline {
environment {
// initialize common vars
common_vars = my_vars.loadVars()
} // environment
stages {
stage('Some Stage') {
// ...
}
}
post {
always {
script {
// Print environment variables
sh "env"
} // script
} // always
} // post
} // pipeline
The thing is that the environment bullet gets KEY=VALUE pairs, thus my common_vars map is loaded like a String value (I can see that on sh "env").
...
vars=[var1:val1, var2:val2]
...
What is the correct way to declare those values as an environment variables?
My target to get this:
...
var1=val1
var2=val2
...
Pipeline's environment variables store only String values. That is why when you assign a map to env.common_vars variables it stores map.toString() equivalent.
If you want to rewrite key-values from a map to the environment variables, you can iterate the variables map and assign each k-v pair to something like env."$k" = v. You can do that by calling a class method inside the environment block - that way you can be sure that the environment variables are assigned no matter which stage your pipeline gets restarted from. Consider the following example:
class MyVars {
private Map config = [
var1: "val1",
var2: "val2"
]
String initializeEnvironmentVariables(final Script script) {
config.each { k,v ->
script.env."$k" = v
}
return "Initialization of env variables completed!"
}
}
pipeline {
agent any
environment {
INITIALIZE_ENV_VARIABLES_FROM_MAP = "${new MyVars().initializeEnvironmentVariables(this)}"
}
stages {
stage("Some stage") {
steps {
echo "env.var1 = ${env.var1}"
}
}
}
post {
always {
script {
sh 'printenv | grep "var[0-9]\\+"'
}
}
}
}
In this example, we use MyVars class to store some global config map (it can be a part of a shared library, here, for simplicity, it is a part of the Jenkinsfile). We use INITIALIZE_ENV_VARIABLES_FROM_MAP environment variable assignment to call MyVars.initializeEnvironmentVariables(this) method that can access env from the script parameter. Calling this method from inside environment block has one significant benefit - it guarantees that environment variables will be initialized even if you restart the pipeline from any stage.
And here is the output of this exemplary pipeline:
Running on Jenkins in /home/wololock/.jenkins/workspace/pipeline-env-map
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Some stage)
[Pipeline] echo
env.var1 = val1
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ grep 'var[0-9]\+'
+ printenv
var1=val1
var2=val2
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
As you can see we it sets env.var1 and env.var2 from the map encapsulated in MyVars class. Both variables can be accessed inside the pipeline step, script block or even inside the shell environment variables.
As far as I know there is no easy way to do this in declarative pipeline (e.g. in the environment directive. Instead, what you can do is to setup the environment outside of the declarative definition, like this:
my_vars.loadVars().each { key, value ->
env[key] = value
}
// Followed by your pipelines definition:
pipeline {
stages {
stage('Some Stage') {
// ...
}
}
// ...
} // pipeline
As an full example:
class my_vars {
static Map varMap = [:]
static def loadVars (Map config) {
varMap.var1 = "val1"
varMap.var2 = "val2"
// Many more variables ...
return varMap
}
}
my_vars.loadVars().each { key, value ->
env[key] = value
}
pipeline {
agent any
stages {
stage("Some stage") {
steps {
echo "env.var1 = ${env.var1}"
}
}
}
}
Which outputs the following when built:
Started by user xxx
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on yyy in /zzz
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Some stage)
[Pipeline] echo
env.var1 = val1
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Edit; If your class (my_vars) is located in a shared library (MySharedLibrary):
library 'MySharedLibrary' // Will load vars/my_vars.groovy
my_vars.loadVars().each { key, value ->
env[key] = value
}
pipeline {
agent any
stages {
stage("Some stage") {
steps {
echo "env.var1 = ${env.var1}"
}
}
}
}
You don't have to return a map of your environment variables from your shared library. You can simply set them in a shared library method, the method will run in the same container as your pipeline.
In you shared library vars/ directory:
def setVars() {
env.var1 = "var1"
env.var2 = "var2"
env.var3 = "var3"
}
In your pipeline:
pipeline {
agent any
stages {
stage("Setup") {
steps {
script {
imported_shared_lib.setVars()
}
}
}
}
}
Others mentioned the need to preserve the environment variables even if you restart the pipeline from a certain stage. In my experiments, the variables are preserved using this method, even if the setVars() method is not called in the environment{} block.
I've inherited some Jenkins pipeline and try to improve it. Jenkins and groovy is quite fresh topic for me, so most probably I'm doing something wrong.
I'm using Jenkins ver. 2.121.3
Main aim was to add build parameter to do some extra cleaning during build. So I've added parameter CLEAN_FIRST with Boolean type and default value false to a job configuration and did something like this in pipeline:
// CLEAN_FIRST = false
// def prefix = CLEAN_FIRST ? "" : "REM"
pipeline {
agent none
stages {
stage('Some step') {
steps {
script {
node('master') {
cleanWs()
try {
def prefix = CLEAN_FIRST ? "" : "REM"
echo "CLEAN_FIRST=$CLEAN_FIRST prefix=$prefix"
bat (label: 'build third party',
script: """
$prefix call cleanSomthing.bat
call doOtherStuff.bat
"""
} finally {
echo "some stuff"
}
} // node
} // script
} // steps
} // stage
} // stages
} // pipeline
Now this doesn't work as expected. "REM" prefix is not added.
Echo prints:
CLEAN_FIRST=false prefix=
And bat invokes cleanSomthing.bat which I wish to avoid (to save on build times).
I've tried to make prefix global, but with same result.
Most probably this is caused by some evaluation order or scoping issue, but I can't put finger on it.
Can someone give me a clue why it doesn't work? How to fix it?
Answered own question. Is this problem fixed on some version of Jenkins?
replace
def prefix = CLEAN_FIRST ? "" : "REM"
with
def prefix = params.CLEAN_FIRST ? "" : "REM"
Ok I've found source of problems. It is a bit funny.
When running this pipeline (tested on Mac machine since it had empty job queue):
pipeline {
agent none
stages {
stage('Some step') {
steps {
script {
node('Mac') {
cleanWs()
try {
def logic = true
def prefix = CLEAN_FIRST ? "Ole" : "REM"
def typeLogic = logic.getClass()
def typeParam = CLEAN_FIRST.getClass()
echo "typeLogic=$typeLogic typeParam=$typeParam"
echo "CLEAN_FIRST=$CLEAN_FIRST prefix=$prefix"
sh (script: """
echo prefix=$prefix
""")
} finally {
echo "some stuff"
}
} // node
} // script
} // steps
} // stage
} // stages
} // pipeline
I've got this outcome:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] stage
[Pipeline] { (Some step)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on master in /Users/builder/jenkins/workspace/EIbuild_MacOS
[Pipeline] {
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] echo
typeLogic=class java.lang.Boolean typeParam=class java.lang.String
[Pipeline] echo
CLEAN_FIRST=false prefix=Ole
[Pipeline] sh
[EIbuild_MacOS] Running shell script
+ echo prefix=Ole
prefix=Ole
[Pipeline] echo
some stuff
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
So now source the problem is obvious.
Jenkins in configuration promises variable of type Boolean, but in fact provides type String with values are "true" or "false" which are always evaluated as true when used as condition since both values are not empty strings :).
Despite following this answer and others, I am unable to successfully use a local groovy file in my Jenkinsfile (both are in the same repository).
def deployer = null
...
...
...
pipeline {
agent {
label 'cf_slave'
}
options {
skipDefaultCheckout()
disableConcurrentBuilds()
}
stages {
stage ("Checkout SCM") {
steps {
checkout scm
}
}
...
...
...
stage ("Publish CF app") {
steps {
script {
STAGE_NAME = "Publish CF app"
deployer = fileLoader.load ('deployer')
withCredentials(...) {
if (BRANCH_NAME == "develop") {
...
...
...
} else {
deployer.generateManifest()
}
}
}
}
}
...
...
}
deployer.groovy:
#!/usr/bin/env groovy
def generateManifest() {
sh "..."
echo "..."
}
In the console log (stack):
[Pipeline] stage
[Pipeline] { (Publish CF app)
[Pipeline] script
[Pipeline] {
[Pipeline] echo
before loading groovy file
[Pipeline] echo
Loading from deployer.groovy
[Pipeline] load
[Pipeline] // load
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
Update:
It seems the problem was not with loading the file but rather with the contents of the file, where I execute the following which apparently does not play well:
sh "node $(pwd)/config/mustacher manifest.template.yml config/environments/common.json config/environments/someFile.json"
echo "..."
When only the echo is there, this is the stack.
So not the sh "node ..." nor the echo work. Even changing it just to sh "pwd" fails as well. What could it be? the syntax in the file? the way it is called in the pipeline?
If I will make the same node call in the pipeline (for example in the withCredentials if statement, it works.
Add a return this to the bottom of the deployer.groovy file, and then change you load step to use relative path and extension to groovy file like load('deployer.groovy').
The return this is documented on jenkins.io:
Takes a filename in the workspace and runs it as Groovy source text.
The loaded file can contain statements at top level or just load and run a closure. For example:
def pipeline
node('slave') {
pipeline = load 'pipeline.groovy'
pipeline.functionA()
}
pipeline.functionB()
pipeline.groovy
def pipelineMethod() {
...code
}
return this
Where pipeline.groovy defines functionA and functionB functions (among others) before ending with return this