Here is the content of my Jenkinsfile :
node {
// prints only the first element 'a'
[ 'a', 'b', 'c' ].each {
echo it
}
}
When executing the job in Jenkins (with the Pipeline plugin), only the first item in the list is printed.
Can someone explain me this strange behavior? Is it a bug? or is it just me not understanding the Groovy syntax?
Edit : the for (i in items) works as expected :
node {
// prints 'a', 'b' and 'c'
for (i in [ 'a', 'b', 'c' ]) {
echo i
}
}
The accepted answer here states that it's a known bug, and uses a workaround that didn't work for me, so I'll offer an update with what I've found lately.
Despite the resolution of JENKINS-26481 (fairly recent, as of this writing) many people may be stuck with an older version of Jenkins where the fix is not available. For-loop iteration over a literal list might work sometimes but related issues like JENKINS-46749 and JENKINS-46747 seem to continue to bedevil many users. Also, depending on the exact context in your Jenkinsfile, possibly echo will work whereas sh fails, and things might fail silently or they might crash the build with serialization failures.
If you dislike surprises (skipped loops and silent failures) and if you want your Jenkinsfiles to be the most portable across multiple versions of Jenkins, the main idea seems to be that you should always use classic counters in your for-loops and ignore other groovy features.
This gist is the best reference I've seen and spells out many cases that you'd think should work the same but have surprisingly different behaviour. It's a good starting place to establish sanity checks and debug your setup, regardless of what kind of iteration you're looking at and regardless of whether you're trying to use #NonCPS, do your iteration directly inside node{}, or call a separate function.
Again, I take no credit for the work itself but I'm embedding the gist of iteration test cases below for posterity:
abcs = ['a', 'b', 'c']
node('master') {
stage('Test 1: loop of echo statements') {
echo_all(abcs)
}
stage('Test 2: loop of sh commands') {
loop_of_sh(abcs)
}
stage('Test 3: loop with preceding SH') {
loop_with_preceding_sh(abcs)
}
stage('Test 4: traditional for loop') {
traditional_int_for_loop(abcs)
}
}
#NonCPS // has to be NonCPS or the build breaks on the call to .each
def echo_all(list) {
list.each { item ->
echo "Hello ${item}"
}
}
// outputs all items as expected
#NonCPS
def loop_of_sh(list) {
list.each { item ->
sh "echo Hello ${item}"
}
}
// outputs only the first item
#NonCPS
def loop_with_preceding_sh(list) {
sh "echo Going to echo a list"
list.each { item ->
sh "echo Hello ${item}"
}
}
// outputs only the "Going to echo a list" bit
//No NonCPS required
def traditional_int_for_loop(list) {
sh "echo Going to echo a list"
for (int i = 0; i < list.size(); i++) {
sh "echo Hello ${list[i]}"
}
}
// echoes everything as expected
Thanks to #batmat on #jenkins IRC channel for answering this question!
It's actually a known bug : JENKINS-26481.
A workaround for this issue is to expand all the commands to a flat text file as a groovy script. Then use load step to load the file and execute.
For example:
#NonCPS
def createScript(){
def cmd=""
for (i in [ 'a', 'b', 'c' ]) {
cmd = cmd+ "echo $i"
}
writeFile file: 'steps.groovy', text: cmd
}
Then call the function like
createScript()
load 'steps.groovy'
Here is example loop example with curl without NonCPS :
#!/usr/bin/env groovy
node('master') {
stagesWithTry([
'https://google.com/',
'https://github.com',
'https://releases.hashicorp.com/',
'https://kubernetes-charts.storage.googleapis.com',
'https://gcsweb.istio.io/gcs/istio-release/releases'
])
stage ('ALlinOneStage'){
stepsWithTry([
'https://google.com/',
'https://github.com',
'https://releases.hashicorp.com/',
'https://kubernetes-charts.storage.googleapis.com',
'https://gcsweb.istio.io/gcs/istio-release/releases'
])
}
}
//loop in one stage
def stepsWithTry(list){
for (int i = 0; i < list.size(); i++) {
try {
sh "curl --connect-timeout 15 -v -L ${list[i]}"
} catch (Exception e) {
echo "Stage failed, but we continue"
}
}
}
//loop in multiple stage
def stagesWithTry(list){
for (int i = 0; i < list.size(); i++) {
try {
stage(list[i]){
sh "curl --connect-timeout 15 -v -L ${list[i]}"
}
} catch (Exception e) {
echo "Stage failed, but we continue"
}
}
}
Related
How do I get the following post section to run?
The idea is to generate parallel running stages as per the contents of myList.
stage("Dynamic stages"){
steps{
stageMap = [:]
script {
for (element in myList) {
stageMap[element] = {
stage("Stage 1 of ${element}"){
echo "Stage 1"
}
stage("Stage 2 ${element}"){
echo "Stage 2"
post{
failure{
echo "Stage 2 failed!"
}
}
}
}
}
}
parallel stageMap
}
}
The build fails with the following error
Also: java.lang.NoSuchMethodError: No such DSL method 'post' found among steps...
The Jenkins doc however says it is allowed
Sure enough there is a big list of allowed steps and post is not among those.
Am I missing something or is the doc not clear about this case?
post{} is only available in declarative pipeline, but not within a scripted section of a declarative pipeline. You can use try{} catch{} instead.
There is another error: You are using loop variable element within a closure, which doesn't work as expected. The closure captures the element variable, but when the closure runs, its value will always be the last value of the loop. To fix this, I have assigned the loop variable to new local variable curElement, which will be a new instance for every iteration, so it gets captured as expected.
stage("Dynamic stages"){
steps{
stageMap = [:]
script {
for (element in myList) {
def curElement = element
stageMap[element] = {
stage("Stage 1 of ${curElement}"){
echo "Stage 1"
}
stage("Stage 2 ${curElement}"){
echo "Stage 2"
try {
// some steps that may fail
}
catch( Exception e ) {
echo "Stage 2 failed! Error: $e"
throw // Rethrow exception, to let the build fail.
}
}
}
}
}
parallel stageMap
}
}
I came here from this post Defining a variable in shell script portion of Jenkins Pipeline
My situation is the following I have a pipeline that is updating some files and generating a PR in my repo if there are changes in the generated files (they change every couple of weeks or less).
At the end of my pipeline I have a post action to send the result by email to our teams connector.
I wanted to know if I could somehow generate a variable and include that variable in my email.
It looks something like this but off course it does not work.
#!groovy
String WasThereAnUpdate = '';
pipeline {
agent any
environment {
GRADLE_OPTS = '-Dorg.gradle.java.home=$JAVA11_HOME'
}
stages {
stage('File Update') {
steps {
sh './gradlew updateFiles -P updateEnabled'
}
}
stage('Create PR') {
steps {
withCredentials(...) {
sh '''
if [ -n \"$(git status --porcelain)\" ]; then
WasThereAnUpdate=\"With Updates\"
...
else
WasThereAnUpdate=\"Without updates\"
fi
'''
}
}
}
}
post {
success {
office365ConnectorSend(
message: "Scheduler finished: " + WasThereAnUpdate,
status: 'Success',
color: '#1A5D1C',
webhookUrl: 'https://outlook.office.com/webhook/1234'
)
}
}
}
I've tried referencing my variable in different ways ${}, etc... but I'm pretty sure that assignment is not working.
I know I probably could do it with a script block but I'm not sure how I would put the script block inside the SH itself, not sure this would be possible.
Thanks to the response from MaratC https://stackoverflow.com/a/64572833/5685482 and this documentation
I'll do it something like this:
#!groovy
def date = new Date()
String newBranchName = 'protoUpdate_'+date.getTime()
pipeline {
agent any
stages {
stage('ensure a diff') {
steps {
sh 'touch oneFile.txt'
}
}
stage('AFTER') {
steps {
script {
env.STATUS2 = sh(script:'git status --porcelain', returnStdout: true).trim()
}
}
}
}
post {
success {
office365ConnectorSend(
message: "test ${env.STATUS2}",
status: 'Success',
color: '#1A5D1C',
webhookUrl: 'https://outlook.office.com/webhook/1234'
)
}
}
In your code
sh '''
if [ -n \"$(git status --porcelain)\" ]; then
WasThereAnUpdate=\"With Updates\"
...
else
WasThereAnUpdate=\"Without updates\"
fi
'''
Your code creates a sh session (most likely bash). That session inherits the environment variables from the process that started it (Jenkins). Once it runs git status, it then sets a bash variable WasThereAnUpdate (which is a different variable from likely named Groovy variable.)
This bash variable is what gets updated in your code.
Once your sh session ends, bash process gets destroyed, and all of its variables get destroyed too.
This whole process has no influence whatsoever on Groovy variable named WasThereAnUpdate that just stays what it was before.
I'm trying to create parallel stages in jenkins pipeline for say with this example
node {
stage('CI') {
script {
doDynamicParallelSteps()
}
}
}
def doDynamicParallelSteps(){
tests = [:]
for (f in ["Branch_1", "Branch_2", "Branch_3"]) {
tests["${f}"] = {
node {
stage("${f}") {
echo "${f}"
}
}
}
}
parallel tests
}
I'm expecting to see "Branch_1", "Branch_2", "Branch_3" and instead I'm getting "Branch_3", "Branch_3", "Branch_3"
I don't understand why. Can you please help ?
Short answer: On the classic view, the stage names are displaying the last value of the variable ${f}. Also, all the echo are echoing the same value. You need to change the loop.
Long Answer: Jenkins does not allow to have multiple stages with the same name so this could never happen successfully :)
On your example, you can see it fine on Blue Ocean:
Also, on console output, the names are right too.
On Jenkins classic view, the stage names have the last value of the variable ${f}. The last value is being printed on the classic view for the stage name, and all the echo are the same.
Solution: Change your loop. This worked fine for me.
node {
stage('CI') {
script {
doDynamicParallelSteps()
}
}
}
def void doDynamicParallelSteps(){
def branches = [:]
for (int i = 0; i < 3 ; i++) {
int index=i, branch = i+1
branches["branch_${branch}"] = {
stage ("Branch_${branch}"){
node {
sh "echo branch_${branch}"
}
}
}
}
parallel branches
}
This has to do with closures and iteration, but in the end this might fix it:
for (f in ["Branch_1", "Branch_2", "Branch_3"]) {
def definitive_name = f
tests[definitive_name] = {
I'd like to ask for help with a a Jenkins groovy pipeline, copied from here:
Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?
I'd like for a several sets of vars to be passed in a under a map, for several stages under a parallel run. However, only the last set (square brackets at the bottom of the map) gets registered for my map.
When the parallel stage runs, the map iterates successfully, but only with the last set (currently install_Stage(it)), ignoring other sets. Meaning that I get a pipeline showing four "stage: install ${product}" stages in parallel, and that's it. I'd like to get three parallels with four stages (network setup, revert, and install), as per my code below:
#!groovy
#Library('ci_builds')
def products = ["A", "B", "C", "D"]
def parallelStagesMap = products.collectEntries {
switch (it) {
case "A":
static_ip_address = "10.100.100.6"; static_vm_name = "install-vm1"; version = "14.1.60"
break
case "B":
static_ip_address = "10.100.100.7"; static_vm_name = "install-vm2"; version = "15.1"
break
case "C":
static_ip_address = "10.100.100.8"; static_vm_name = "install-vm3"; version = "15.1"
break
case "D":
static_ip_address = "10.100.100.9"; static_vm_name = "install-vm4"; version = "15.2"
break
default:
static_ip_address = "The product name is not on the switch list - please enter an ip address"
version = "The product name is not on the switch list - please enter a version"
break
}
["${it}" : network_reg(it)]
["${it}" : revert_to_snapshot_Stage(it)]
["${it}" : install_Stage(it)]
}
def network_reg(product) {
return {
stage("stage: setup network for ${product}") {
echo "setting network on ${static_vm_name} with ${static_ip_address}."
sh script: "sleep 15"
}
}
}
def revert_to_snapshot_Stage(product) {
return {
stage("stage: revert ${product}") {
echo "reverting ${static_vm_name} for ${product} on ${static_ip_address}."
sh script: "sleep 15"
}
}
}
def install_Stage(product) {
return {
stage("stage: install ${product}") {
echo "installing ${product} on ${static_ip_address}."
sh script: "sleep 15"
}
}
}
pipeline {
agent any
stages {
stage('non-parallel env check') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
The network_reg and revert_to_snapshot_Stage won't run (unless I place them as the last set instead of ["${it}" : install_Stage(it)] , in which case, again, only the one of the parallel stages is run)
I don't mind a different approach to run several map definitions, but others such as: How to define and iterate over map in Jenkinsfile don't allow for a full multi variable map (more than a key+value pair)
Any help would be appreciated, Thanks!
I assume you have a similar issue like I had trying to dynamically build the parallel branches for parallel execution.
Two things were very important:
Make a copy of the loop variable (in you case: it) and use that copy only inside the parallel branch; if you don't all branches (closures) will reference the very same variable which of course will have the same value. That is particular to closures. See also: http://groovy-lang.org/closures.html.
Don't use collectEntries{}. Stick to the java-style loops as groovy loops most of the time do not work properly. Some .each{} constructs may work already but if in doubt switch to the java loops. See also: Impossibility to iterate over a Map using Groovy within Jenkins Pipeline
Following stripped down example works for me. I believe you'll be able to adjust it to your needs.
def products = ["A", "B", "C", "D"]
def parallelStagesMap = [:]
// use java-style loop
for (def product: products) {
// make a copy to ensure that each closure will get it's own variable
def copyOfProduct = product
parallelStagesMap[product] = {echo "install_Stage($copyOfProduct)"}
}
echo parallelStagesMap.toString()
pipeline {
agent any
stages {
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
If it still doesn't work: Check whether there's and upgrade your Pipeline: Groovy plugin as they usually fix lot of issues which usually work in groovy but won`t in pipeline.
You may want to check following related question which contains a minimal example as well:
Currying groovy CPS closure for parallel execution
Question
I have simple parallel pipeline (see code) which I use together with Jenkins 2.89.2. Additionally I use parameters and now want to be able to in-/decrease the number of deployVM A..Z stages automatically by providing the parameter before job execution.
How can I dynamically build my pipeline by providing a parameter?
Researched so far:
Jenkins pipeline script created dynamically - Not getting this to work with my Jenkins version
Can I create dynamically stages in a Jenkins pipeline? - Not working either
Code
The pseudo code of what I want - dynamic generation:
pipeline {
agent any
parameters {
string(name: 'countTotal', defaultValue: '3')
}
stages {
stage('deployVM') {
def list = [:]
for(int i = 0; i < countTotal.toInteger; i++) {
list += stage("deployVM ${i}") {
steps {
script {
sh "echo p1; sleep 12s; echo phase${i}"
}
}
}
}
failFast true
parallel list
}
}
}
The code I have so far - executes parallel but is static:
pipeline {
agent any
stages {
stage('deployVM') {
failFast true
parallel {
stage('deployVM A') {
steps {
script {
sh "echo p1; sleep 12s; echo phase1"
}
}
}
stage('deployVM B') {
steps {
script {
sh "echo p1; sleep 20s; echo phase2"
}
}
}
}
}
}
}
Although the question assumes using declarative pipeline I would suggest to use scripted pipeline because it's way more flexible.
Your task can be accomplished this way
properties([
parameters([
string(name: 'countTotal', defaultValue: '3')
])
])
def stages = [failFast: true]
for (int i = 0; i < params.countTotal.toInteger(); i++) {
def vmNumber = i //alias the loop variable to refer it in the closure
stages["deployVM ${vmNumber}"] = {
stage("deployVM ${vmNumber}") {
sh "echo p1; sleep 12s; echo phase${vmNumber}"
}
}
}
node() {
parallel stages
}
Also take a look at snippet generator which allows you to generate some scripted pipeline code.
Using Declarative pipeline also you can achieve this.
Follow my answer HERE
In above link answer I have used Var.collectEntries but map also can be used.
#Vitalii
I wrote similiar code piece, but unfoutunelty, all three element been loopped all shows the last one, not sure if it had something to do with groovy / jenkinsfile itself, that some clouse / reference went break with wrong usage
my purpose is to distribute tasks to specific work nodes
node_candicates = ["worker-1", "worder-2", "worker-3"]
def jobs = [:]
for (node_name in node_candidates){
jobs["run on $node_name"] = { // good
stage("run on $node_name"){ // all show the third
node(node_name){ // all show the third
print "on $node_name"
sh "hostname"
}
}
}
}
parallel jobs
it went totally Ok if I expand / explain the loop, instead of loop over it, like
parallel worker_1: {
stage("worker_1"){
node("worker_1"){
sh """hostname ; pwd """
print "on worker_1"
}
}
}, worker_2: {
stage("worker_2"){
node("worker_2"){
sh """hostname ; pwd """
print "on worker_2"
}
}
}, worker_3: {
stage("worker_3"){
node("worker_3"){
sh """hostname ; pwd """
print "on worker_3"
}
}
}