How do you handle global variables in a declarative pipeline? - jenkins

Previously asked a question about how to overwrite variables defined in an environment directive and it seems that's not possible.
I want to set a variable in one stage and have it accessible to other stages.
In a declarative pipeline it seems the only way to do this is in a script{} block.
For example I need to set some vars after checkout. So at the end of the checkout stage I have a script{} block that sets those vars and they are accessible in other stages.
This works, but it feels wrong. And for the sake of readability I'd much prefer to declare these variables at the top of the pipeline and have them overwritten. So that would mean having a "set variables" stage at the beginning with a script{} block that just defines vars- thats ugly.
I'm pretty sure I'm missing an obvious feature here. Do declarative pipelines have a global variable feature or must I use script{}

This is working without an error,
def my_var
pipeline {
agent any
environment {
REVISION = ""
}
stages {
stage('Example') {
steps {
script{
my_var = 'value1'
}
}
}
stage('Example2') {
steps {
script{
echo "$my_var"
}
}
}
}
}

Like #mkobit says, you can define the variable to global level out of pipeline block. Have you tried that?
def my_var
pipeline {
agent any
stages {
stage('Example') {
steps {
my_var = 'value1'
}
}
stage('Example2') {
steps {
printl(my_var)
}
}
}
}

For strings, add it to the 'environment' block:
pipeline {
environment {
myGlobalValue = 'foo'
}
}
But for non-string variables, the easiest solution I've found for declarative pipelines is to wrap the values in a method.
Example:
pipeline {
// Now I can reference myGlobalValue() in my pipeline.
...
}
def myGlobalValue() {
return ['A', 'list', 'of', 'values']
// I can also reference myGlobalValue() in other methods below
def myGlobalSet() {
return myGlobalValue().toSet()
}

#Sameera's answer is good for most use cases. I had a problem with appending operator += though. So this did NOT work (MissingPropertyException):
def globalvar = ""
pipeline {
stages {
stage("whatever) {
steps {
script {
globalvar += "x"
}
}
}
}
}
But this did work:
globalvar = ""
pipeline {
stages {
stage("whatever) {
steps {
script {
globalvar += "x"
}
}
}
}
}

The correct syntax is:
For global static variable
somewhere at the top of the file, before pipeline {, declare:
def MY_VAR = 'something'
For global variable that you can edit and reuse accross stages:
At the top of your file, add an import to Field:
import groovy.transform.Field
somewhere before pipeline {, declare:
#Field def myVar
then inside your step, inside a script, set the variable
stage('some stage') {
steps {
script {
myVar = 'I mutate myVar with success'
}
}
}
to go even further, you can declare functions:
before the pipeline {
def initSteps() {
cleanWs()
checkout scm
}
and then
stages {
stage('Init') {
steps {
initSteps()
}
}
}

This worked for me
pipeline {
agent any
stages {
stage('Example') {
steps {
script{
env.my_var = 'value1'
}
}
}
stage('Example2') {
steps {
printl(my_var)
}
}
}
}

Related

Declarative dynamic parallel stages

I figure I’m doing something unorthodox here, but I’d like to stick to declarative for convenience while dynamically generating parallel steps.
I found a way to do something like that, but mixing both paradigms, which doesn’t seem to work well with the BlueOcean UI (multiple stages inside each parallel branch do not show up properly).
The closest I got was with something like this:
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = []
for (account in accounts()) {
jobs[] = stage(account) {
steps {
echo "Step for $account"
}
}
}
return jobs
}
# this is inside a shared library, called by my Jenkinsfile, like what is described
# under "Defining Declarative Pipelines in Shared Libraries" in
# https://www.jenkins.io/blog/2017/09/25/declarative-1/
def call() {
pipeline {
stages {
stage('Build all variations') {
parallel parallelJobs()
}
}
}
}
The problem is Jenkins errors like this:
Expected a block for parallel # line X, column Y.
parallel parallelJobs()
^
So, I was wondering if there is a way I could transform that list of stages, returned by parallelJobs(), into the block expected by Jenkins...
Yes, you can. You need to return a map of stages. Following is a working pipeline example.
pipeline {
agent any
stages {
stage('Parallel') {
steps {
script {
parallel parallelJobs()
}
}
}
}
}
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = [:]
for (account in accounts()) {
jobs[account] = { stage(account) {
echo "Step for $account"
}
}
}
return jobs
}

How to create a generic Jenkins stage, where the agent, and the steps are parameters?

I have multiple jenkinsifles, doing basically the same thing:
pipeline{
parameters { ... }
environment { ... }
stages {
stage ('setup') { ... }
stage ('run') {
agent { AGENT }
steps { STEPS }
}
}
The STEPS & AGENT parameters are values I get in the setup stage. Is it possible to define somewhere a function that returns a stage?
e.g.
def stage_factory(name, agent, steps, post ...){
return
stage (name) {
agent { agnet }
steps { steps }
post { post }
}
}
}
which later will be called inside the pipeline, right after the setup stage
?
The following works in scripted pipeline, you need to try the declarative syntax yourself. Note the use of surrounding {}
def stage_factory(name, agent, steps, post ...){
return {
node(agent){
stage (name) {
steps()
}
}
}
}
With this approach you need to put the post action in try-catch blocks, but this is the gist of it.
If you change it like so, you can even pass the steps to it as you would expect from a Jenkins stage.
def stage_factory(name, agent){
return { steps ->
node(agent){
stage (name) {
steps()
}
}
}
}
usage:
def myDtage = stage_factory("foo", "bar")
myStage{
//...
}

Call Groovy method inside Jenkinsfile's environment block

I have a Jenkinsfile that looks like this:
static def randomUser() {
final def POOL = ["a".."z"].flatten()
final Random rand = new Random(System.currentTimeMillis())
return (0..5).collect { POOL[rand.nextInt(POOL.size())] }.join("")
}
pipeline {
agent any
environment {
//CREATOR = sh(script: "randomUser()", returnStdout: true)
CREATOR = "fixed-for-now"
...
}
stages {
...
stage("Terraform Plan") {
when { not { branch "master" } }
steps {
sh "terraform plan -out=plan.out -var creator=${CREATOR} -var-file=env.tfvars "
}
}
...
stage("Terraform Destroy") {
when { not { branch "master" } }
steps {
sh "terraform destroy -auto-approve -var creator=${CREATOR} -var-file=env.tfvars "
}
}
...
}
My problem is I cannot call randomUser while being inside the environment block. I would need to have the CREATOR variable as a random string every time. I would prefer to have CREATOR as a global environment variable since it's going to be used in many stages.
Is there a way to achieve (or workaround) this?
Given your specific use case, it might be better to use the CREATOR variable as a parameter instead of an environment variable, and to assign its defaultValue as the return of your randomUser method.
pipeline {
agent any
parameters {
string(name: 'CREATOR', defaultValue: sh(script: "randomUser()", returnStdout: true))
}
...
}
You can then use it in your pipeline like so:
stage("Terraform Plan") {
when { not { branch "master" } }
steps {
sh "terraform plan -out=plan.out -var creator=${params.CREATOR} -var-file=env.tfvars "
}
}
This way you have a correctly assigned and useful defaultValue for CREATOR, but with the ability to override it per-pipeline when necessary.
You can achieve this by removing environment block and defining global variable CREATOR before pipeline block
def CREATOR
pipeline {
agent any
stages {
stage('Initialize the variables') {
steps{
script{
CREATOR = randomUser()
}
}
}
...

How to set Jenkins Declarative Pipeline environments with Global Variables?

I am trying to do this
pipeline {
agent any
environment {
LOCAL_BUILD_PATH=env.WORKSPACE+'/build/'
}
stages {
stage('Stuff'){
steps{
echo LOCAL_BUILD_PATH
}
}
}
}
Result:
null/build/
How can I use Global Environments to create my environments?
So this is method that I ended up using
pipeline {
agent {
label 'master'
}
stages {
stage ("Setting Variables"){
steps {
script{
LOCAL_BUILD_PATH = "$env.WORKSPACE/build"
}
}
}
stage('Print Varliabe'){
steps{
echo LOCAL_BUILD_PATH
}
}
}
}
You can use something like this...
LOCAL_BUILD_PATH="${env.WORKSPACE}/build/"
Remember: use "(double quote) for variable in string
I think you should use:
steps {
echo "${env.LOCAL_BUILD_PATH}"
}
as in "environment" step you're defining environmental variables which are later accessible by env.your-variable-name
This a scope issue. Declare the variable, at the top and set it to null. Something like
def var = null
You should be able to set the value in a block/closure/stage and access it in another

Can I create dynamically stages in a Jenkins pipeline?

I need to launch a dynamic set of tests in a declarative pipeline.
For better visualization purposes, I'd like to create a stage for each test.
Is there a way to do so?
The only way to create a stage I know is:
stage('foo') {
...
}
I've seen this example, but I it does not use declarative syntax.
Use the scripted syntax that allows more flexibility than the declarative syntax, even though the declarative is more documented and recommended.
For example stages can be created in a loop:
def tests = params.Tests.split(',')
for (int i = 0; i < tests.length; i++) {
stage("Test ${tests[i]}") {
sh '....'
}
}
As JamesD suggested, you may create stages dynamically (but they will be sequential) like that:
def list
pipeline {
agent none
options {buildDiscarder(logRotator(daysToKeepStr: '7', numToKeepStr: '1'))}
stages {
stage('Create List') {
agent {node 'nodename'}
steps {
script {
// you may create your list here, lets say reading from a file after checkout
list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
}
}
post {
cleanup {
cleanWs()
}
}
}
stage('Dynamic Stages') {
agent {node 'nodename'}
steps {
script {
for(int i=0; i < list.size(); i++) {
stage(list[i]){
echo "Element: $i"
}
}
}
}
post {
cleanup {
cleanWs()
}
}
}
}
}
That will result in:
dynamic-sequential-stages
If you don't want to use for loop, and generated pipeline to be executed in parallel then, here is an answer.
def jobs = ["JobA", "JobB", "JobC"]
def parallelStagesMap = jobs.collectEntries {
["${it}" : generateStage(it)]
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Note that all generated stages will be executed into 1 node.
If you are willing to executed the generated stages to be executed into different nodes.
def agents = ['master', 'agent1', 'agent2']
// enter valid agent name in array.
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
You can of course add more than 1 parameters and can use collectEntries for 2 parameters.
Please remember return in function generateStage is must.
#Jorge Machado: Because I cannot comment I had to post it as an answer. I've solved it recently. I hope it'll help you.
Declarative pipeline:
A simple static example:
stage('Dynamic') {
steps {
script {
stage('NewOne') {
echo('new one echo')
}
}
}
}
Dynamic real-life example:
// in a declarative pipeline
stage('Trigger Building') {
when {
environment(name: 'DO_BUILD_PACKAGES', value: 'true')
}
steps {
executeModuleScripts('build') // local method, see at the end of this script
}
}
// at the end of the file or in a shared library
void executeModuleScripts(String operation) {
def allModules = ['module1', 'module2', 'module3', 'module4', 'module11']
allModules.each { module ->
String action = "${operation}:${module}"
echo("---- ${action.toUpperCase()} ----")
String command = "npm run ${action} -ddd"
// here is the trick
script {
stage(module) {
bat(command)
}
}
}
}
You might want to take a look at this example - you can have a function return a closure which should be able to have a stage in it.
This code shows the concept, but doesn't have a stage in it.
def transformDeployBuildStep(OS) {
return {
node ('master') {
wrap([$class: 'TimestamperBuildWrapper']) {
...
} } // ts / node
} // closure
} // transformDeployBuildStep
stage("Yum Deploy") {
stepsForParallel = [:]
for (int i = 0; i < TargetOSs.size(); i++) {
def s = TargetOSs.get(i)
def stepName = "CentOS ${s} Deployment"
stepsForParallel[stepName] = transformDeployBuildStep(s)
}
stepsForParallel['failFast'] = false
parallel stepsForParallel
} // stage
Just an addition to what #np2807 and #Anton Yurchenko have already presented: you can create stages dynamically and run the in parallel by simply delaying list of stages creation (but keeping its declaration), e.g. like that:
def parallelStagesMap
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent { label 'master' }
stages {
stage('Create List of Stages to run in Parallel') {
steps {
script {
def list = ["Test-1", "Test-2", "Test-3", "Test-4", "Test-5"]
// you may create your list here, lets say reading from a file after checkout
// personally, I like to use scriptler scripts and load the as simple as:
// list = load '/var/lib/jenkins/scriptler/scripts/load-list-script.groovy'
parallelStagesMap = list.collectEntries {
["${it}" : generateStage(it)]
}
}
}
}
stage('Run Stages in Parallel') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
That will result in Dynamic Parallel Stages:
I use this to generate my stages which contain a Jenkins job in them.
build_list is a list of Jenkins jobs that i want to trigger from my main Jenkins job, but have a stage for each job that is trigger.
build_list = ['job1', 'job2', 'job3']
for(int i=0; i < build_list.size(); i++) {
stage(build_list[i]){
build job: build_list[i], propagate: false
}
}
if you are using Jenkinsfile then, I achieved it via dynamically creating the stages, running them in parallel and also getting Jenkinsfile UI to show separate columns. This assumes parallel steps are independent of each other (otherwise don't use parallel) and you can nest them as deep as you want (depending upon the # of for loops you'll nest for creating stages).
Jenkinsfile Pipeline DSL: How to Show Multi-Columns in Jobs dashboard GUI - For all Dynamically created stages - When within PIPELINE section see here for more.

Resources