How to do custom sort of Map for example by value in Jekins pipeline script?
This code doesn't quite work in Jenkins pipeline script:
Map m =[ james :"silly boy",
janny :"Crazy girl",
jimmy :"funny man",
georges:"massive fella" ]
Map sorted = m.sort { a, b -> a.value <=> b.value }
The map is still not sorted.
I decided to crate a separate question with better name and tags, because many people were struggling to find an answer here:
Groovy custom sort a map by value
You will have to create a separate method with #NonCPS annotation for that:
#NonCPS
def getSorted(def toBeSorted){
toBeSorted.sort(){ a, b -> b.value <=> a.value }
}
And then call it from the pipeline script.
Map unsortedMap =[ james :"silly boy",
janny :"Crazy girl",
jimmy :"funny man",
georges:"massive fella" ]
def sortedMap = getSorted(unsortedMap)
params name
1.xx
2.xx
...
pipeline {
agent {
kubernetes {
inheritFrom 'seunggabi-batch'
defaultContainer 'seunggabi-batch'
}
}
environment {
COUNTRY = "kr"
ENV = "prod"
CLASS = "seunggabi.batch.job.SparkSubmitJob"
}
stages {
stage('Run Job') {
steps {
script {
ARGS = sorted(params).collect { /$it.value/ } join ","
}
sh "/app/static/sh/emr.sh 1 20 ${COUNTRY} ${ENV} ${CLASS} \"${ARGS}\""
}
}
}
}
#NonCPS
def sorted(def m){
m.sort { /$it.key/ }
}
Related
Among the possible steps one can use in a Jenkins pipeline, there is one with the name step, subtitled General Build Step. https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#step-general-build-step . I need to iterate on calling this step based on the contents of a file. I have created a groovy script to read the file and perform the iteration, but I am not sure how to create the equivalent of my step() in the groovy script. Here is the general format of the step I am trying to perform:
stage ('title') {
steps {
step([
$class: 'UCDeployPublisher',
siteName: 'literal string',
deploy: [
$class: 'com.urbancode.jenkins.plugins.ucdeploy.DeployHelper$DeployBlock',
param1: 'another literal string',
param2: 'yet another string'
]
])
}
}
The script step I have developed looks like this:
steps {
script {
def content = readFile(file:'data.csv', encoding:'UTF-8');
def lines = content.split('\n');
for (line in lines) {
// want to insert equivalent groovy code for the basic build step here
}
}
}
I'm expecting there is probably a trivial answer here. I'm just out of my element in the groovy/java world and I am not sure how to proceed. I have done extensive research, looked at source code for Jenkins, looked at plugins, etc. I am stuck!
Check the following, simply move your UCDeployPublisher to a new function and call that from your loop.
steps {
script {
def content = readFile(file:'data.csv', encoding:'UTF-8');
def lines = content.split('\n');
for (line in lines) {
runUCD(line)
}
}
}
// Groovy function
def runUCD(def n) {
stage ("title $n") {
steps {
step([
$class: 'UCDeployPublisher',
siteName: 'literal string',
deploy: [
$class: 'com.urbancode.jenkins.plugins.ucdeploy.DeployHelper$DeployBlock',
param1: 'another literal string',
param2: 'yet another string'
]
])
}
}
}
This is showing the code related to my comment on the accepted answer
pipeline {
stages {
stage ('loop') {
steps {
script {
... groovy to read/parse file and call runUCD
}
}
}
}
}
def runUCD(def param1, def param2) {
stage ("title $param1") {
step([
....
])
}
}
I figure I’m doing something unorthodox here, but I’d like to stick to declarative for convenience while dynamically generating parallel steps.
I found a way to do something like that, but mixing both paradigms, which doesn’t seem to work well with the BlueOcean UI (multiple stages inside each parallel branch do not show up properly).
The closest I got was with something like this:
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = []
for (account in accounts()) {
jobs[] = stage(account) {
steps {
echo "Step for $account"
}
}
}
return jobs
}
# this is inside a shared library, called by my Jenkinsfile, like what is described
# under "Defining Declarative Pipelines in Shared Libraries" in
# https://www.jenkins.io/blog/2017/09/25/declarative-1/
def call() {
pipeline {
stages {
stage('Build all variations') {
parallel parallelJobs()
}
}
}
}
The problem is Jenkins errors like this:
Expected a block for parallel # line X, column Y.
parallel parallelJobs()
^
So, I was wondering if there is a way I could transform that list of stages, returned by parallelJobs(), into the block expected by Jenkins...
Yes, you can. You need to return a map of stages. Following is a working pipeline example.
pipeline {
agent any
stages {
stage('Parallel') {
steps {
script {
parallel parallelJobs()
}
}
}
}
}
def accounts() {
return ["dynamic", "list"]
}
def parallelJobs() {
jobs = [:]
for (account in accounts()) {
jobs[account] = { stage(account) {
echo "Step for $account"
}
}
}
return jobs
}
I'm trying to create parallel stages in jenkins pipeline for say with this example
node {
stage('CI') {
script {
doDynamicParallelSteps()
}
}
}
def doDynamicParallelSteps(){
tests = [:]
for (f in ["Branch_1", "Branch_2", "Branch_3"]) {
tests["${f}"] = {
node {
stage("${f}") {
echo "${f}"
}
}
}
}
parallel tests
}
I'm expecting to see "Branch_1", "Branch_2", "Branch_3" and instead I'm getting "Branch_3", "Branch_3", "Branch_3"
I don't understand why. Can you please help ?
Short answer: On the classic view, the stage names are displaying the last value of the variable ${f}. Also, all the echo are echoing the same value. You need to change the loop.
Long Answer: Jenkins does not allow to have multiple stages with the same name so this could never happen successfully :)
On your example, you can see it fine on Blue Ocean:
Also, on console output, the names are right too.
On Jenkins classic view, the stage names have the last value of the variable ${f}. The last value is being printed on the classic view for the stage name, and all the echo are the same.
Solution: Change your loop. This worked fine for me.
node {
stage('CI') {
script {
doDynamicParallelSteps()
}
}
}
def void doDynamicParallelSteps(){
def branches = [:]
for (int i = 0; i < 3 ; i++) {
int index=i, branch = i+1
branches["branch_${branch}"] = {
stage ("Branch_${branch}"){
node {
sh "echo branch_${branch}"
}
}
}
}
parallel branches
}
This has to do with closures and iteration, but in the end this might fix it:
for (f in ["Branch_1", "Branch_2", "Branch_3"]) {
def definitive_name = f
tests[definitive_name] = {
I have a set of static environmental variables in the environmental directive section of a declarative pipeline. These values are available to every stage in the pipeline.
I want the values to change based on an arbitrary condition.
Is there a way to do this?
pipeline {
agent any
environment {
if ${params.condition} {
var1 = '123'
var2 = abc
} else {
var1 = '456'
var2 = def
}
}
stages {
stage('One') {
steps {
script {
...
echo env.var1
echo env.var2
...
}
}
}
}
stag('Two'){
steps {
script {
...
echo env.var1
echo env.var2
...
}
}
}
Looking for the same thing I found a nice answer in other question:
Basically is to use the ternary conditional operator
pipeline {
agent any
environment {
var1 = "${params.condition == true ? "123" : "456"}"
var2 = "${params.condition == true ? abc : def}"
}
}
Note: keep in mind that in the way you wrote your question (and I did my answer) the numbers are Strings and the letters are variables.
I would suggest you to create a stage "Environment" and declare your variable according to the condition you want, something like below:-
pipeline {
agent any
environment {
// Declare variables which will remain same throughout the build
}
stages {
stage('Environment') {
agent { node { label 'master' } }
steps {
script {
//Write condition for the variables which need to change
if ${params.condition} {
env.var1 = '123'
env.var2 = abc
} else {
env.var1 = '456'
env.var2 = def
}
sh "printenv"
}
}
}
stage('One') {
steps {
script {
...
echo env.var1
echo env.var2
...
}
}
}
stage('Two'){
steps {
script {
...
echo env.var1
echo env.var2
...
}
}
}
}
}
Suppose we want to use optional params for downstream job if it is called from upsteam job, and default params if downsteam job is called by itself.
But we don't want to have "holder" params with default value in downstream for some reason.
This could be done via groovy function:
upstream Jenkinsfile - param CREDENTIALS_ID is passed downsteam
pipeline {
stage {
steps {
build job: "my_downsteam_job_name",
parameters [string(name: 'CREDENTIALS_ID', value: 'other_credentials_id')]
}
}
}
downstream Jenkinsfile - if param CREDENTIALS_ID not passed from upsteam, function returns default value
def getCredentialsId() {
if(params.CREDENTIALS_ID) {
return params.CREDENTIALS_ID;
} else {
return "default_credentials_id";
}
}
pipeline {
environment{
TEST_PASSWORD = credentials("${getCredentialsId()}")
}
}
you can get another level of flexibility, using maps:
stage("set_env_vars") {
steps {
script {
def MY_MAP1 = [A: "123", B: "456", C: "789"]
def MY_MAP2 = [A: "abc", B: "def", C: "ghi"]
env.var1 = MY_MAP1."${env.switching_var}"
env.var2 = MY_MAP2."${env.switching_var}"
}
}
}
This way, more choices are possible.
Using Jenkins pipeline we have our own build script. Also all of our projects have a rakefile which is what we use to do a lot of the building steps. Our typical jenkins build executes 3 rake tasks but we do have some exceptions and that has to do when we have a angular website we try to build with it.
I've configured my pipeline like this:
buildGitProject {
repository='https://anonymous.visualstudio.com/Project/_git/my-csharp-project-with-angular'
branchName= 'master'
solutionName='MyCSharpSolution.sln'
emailTo='someone#aol.com'
preRakeCommands=['install_npm_dependencies', 'ng_build']
}
that relies on our build script which is this:
def call(body) {
def args= [:]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = args
body()
def agentName = "windows && ${args.branchName}"
def remoteConfig = org.pg.RemoteConfigFactory.create(args.repository);
pipeline {
agent none
options {
buildDiscarder(logRotator(numToKeepStr: org.pg.Settings.BUILDS_TO_KEEP))
skipStagesAfterUnstable()
timestamps()
}
stages {
stage("checkout") {
agent any
steps {
checkoutFromGit(remoteConfig, args.branchName)
}
}
stage('build') {
agent{node{ label agentName as String}}
steps {
buildSolution(args.solutionName, args.get('preRakeCommands', []), args.get('postRakeCommands', []))
}
}
stage('test') {
agent{node{ label agentName as String}}
steps {
testSolution(args.solutionName)
}
}
}
}
}
which fails in the build stage.
buildSolution.groovy
def call(String solutionName, ArrayList preRakeCommands, ArrayList postRakeCommands) {
unstash 'ws'
String[] rakeCommands = [
"build_solution[${solutionName}, Release, Any CPU]",
"copy_to_deployment_folder",
"execute_dev_dropkick"
]
String[] combinedRakeCommand = (preRakeCommands.plus(rakeCommands).plus(postRakeCommands)) as String[]
executeRake( combinedRakeCommand )
stash name: 'deployment', includes: 'deployment/**/*'
}
executeRake.groovy
def call(String... rakeTasks) {
def safeRakeTasks = rakeTasks.collect{ "\"$it\"" }.join(' ');
bat script: "rake ${safeRakeTasks}"
}
in the jenkins build log it says:
08:43:09 C:\jenkins_repos\Project\my-csharp-project-with-angular>rake "install_npm_dependencies" "ng_build" "[Ljava.lang.String;#11bd466"
I have no idea how or why it is using a string pointer because I thought that plus concated arrays and ArrayList... Plus it is in Jenkins so it is a pain to test.
List a = ['a1','a2','a3']
String [] s = ['s1','s2','s3']
List b = ['b1','b2','b3']
println a.plus(s as List).plus(b)
output:
[a1, a2, a3, s1, s2, s3, b1, b2, b3]
Another approach:
List a = ['a1','a2','a3']
String[] s = ['s1','s2','s3']
List b = ['b1','b2','b3']
println ([*a,*s,*b])
alternatively
println a + [*s] + b
which should perform better